more frequent updates?

[kylebakerio]

I see the output here, but I'd love to get more frequent updates, and not be dependent on you doing your updates by hand. How could we update this code so that it scrapes and updates this more frequently, and updates the output here automatically? I'll use it for now, but this is a big hesitation for me.

Also, I'll be converting this to JSON; should be pretty easy to just include that as a file here (or to just output the JSON here directly, either way), if you're open to that.

[bumbeishvili]

@kylebakerio you will get JSON output if you use d3.csv for loading, it's very easy to use and very convenient.

I am against additional JSON files because I only will be maintaining csv files, I don't have time to maintain JSON files as well if it requires manual intervention. Now, I can't see how can we do it automatically

On the other hand, frequent updates will be very cool, say in every 20-30 minutes, we can easily scrap the Worldometers page and get relevant information and update data here.

I think it will require diving into Github API, I don't have time to do it now, but I will think about it. Seems like a good personal project. JSON output will be easy to generate as well (Though CSV will be a better choice because size is much smaller, hence loading time will be smaller)

If you want and you have time to make reliable and regular changes, I'd happily add you as a contributor

[bumbeishvili]

Just a note, I created a small node.js app, which can be used to update files on Github automatically.
bumbeishvili/git-sync-repo
Here is result
git-sync-test-repo commits

[kylebakerio]

I ended up using csvToJson (https://github.com/Keyang/node-csvtojson).

Still took a lot of massaging to get the data into the form I wanted to work with, though. Still, glad to have it!

Also, neat project there--are you going to be using that app to update your CSV's here? Or what are your thoughts?

Yeah, I'd be happy to be a contributor. My site is almost ready to 'beta launch', and its current iteration relies on this data--would be nice if the data was updated more often, already was thinking about what API to jump to next, but if I could just improve this data source that'd be nice too.

[bumbeishvili]

Also, neat project there--are you going to be using that app to update your CSV's here? Or what are your thoughts?

That was my intention, but I spent more time to it than I thought and still without major luck

I don't expect it to be automated in a short period of time, so maybe it's better for you if you jump to other API

[kylebakerio]

Can you explain to me what the whole update sequence is? I'm not familiar with this 'notebook' concept (?). My impression is you have some code on a website (for some reason?) that you run, it goes and scrapes and formats stuff, gives you a csv output, and you upload that here.

Is that right? What am I missing, and what are the limitations you're working with?

[bumbeishvili]

Update sequence is the following:

• Between 11:00-11:59 a.m (GMT+4) - I am visiting this notebook

This notebook itself just runs js code automatically, which scrapes yesterdays worldometers data and merges it with data currently exists on Github

• Then I am just copying fields from textareas which can be found in a notebook and pasting to the relevant text files.

For your dashboard, you could use same approach to load live data, this is what I am doing on my dashboard - https://coronavirus.davidb.dev/ . Fetching current worldometers data, merging it with current data existing on Github and then displaying stats