21 hours ago, but no new data?

[kylebakerio]

I assume it's a temporary blip from maybe worldometer reporting late or something, but I noticed that even though the csv's were updated 21 hours ago (on the 21st), the latest data is from the 20th. (Noticed because my site wasn't able to get new data all day.)

Any idea what caused this? Will it be back to normal tomorrow?

[kylebakerio]

btw--you aren't still doing this by hand, right?

[bumbeishvili]

@kylebakerio it looks like you are right, let me check it

[bumbeishvili]

@kylebakerio I removed 20 Aug data and added it again from Worldometer. I added 21 Aug data as well.

Regarding your comment that data was updated 21 hours ago and it was still showing 20 August data. That was correct behavior. I was going to add 21 Aug data when I saw your comment, so please be aware that new data is added every day, between 08:00-13:00 (GST), so there may be some gaps when data is not updated.

Any idea what caused this? Will it be back to normal tomorrow?

Not sure, I did everything as before. Today I created a new notebook, which will make past 2 days data recovering easier, so It's good that I saw your comment.

btw--you aren't still doing this by hand, right?

Yes, 2 minutes every day. Could not find time to finalize automation of this

[kylebakerio]

Cool, glad I made the effort to comment.

But I check my site every day, it was off by a day. Not sure what you mean by "correct behavior"--it was showing data for the 20th late in the evening of the 22nd--usually it's updated early in the morning to get the previous day's data. Which is to say, on the evening of the 22nd, it should show the data for the 21st, etc. (I am pretty familiar with your timing and API behavior because I've written code that starts checking for updated datasets based on what time of day it is, and when one is detected, runs a re-deploy of a static site.)

Regardless, thanks for the fix.

I applaud your consistency, but this doing it by hand can't go on forever. Given that this may be going on well into 2021...

What roadblock did you run into in automating?

[bumbeishvili]

First roadblock is hosting of node.js app, which needs git access on the server

[kylebakerio]

you mean, be able to run 'git' commands from a server?

I could make a little node app you could deploy to heroku that has a template for running shell commands. You'd then code in the github login sequence, and then the github commit commands.

If you had that done, is that all you'd need?

[bumbeishvili]

you mean, be able to run 'git' commands from a server?

@kylebakerio yes, exactly

I could make a little node app you could deploy to Heroku that has a template for running shell commands. You'd then code in the GitHub login sequence, and then the GitHub commit commands.

I already made that app

Here it is - https://github.com/bumbeishvili/git-sync-repo
It pushes files to this repo - https://github.com/bumbeishvili/git-sync-test-repo

I tried it hosting on Heroku already, but no luck. Probably, because of my limited knowledge.

I gave you access to both of those repositories and if you can host it on Heroku and provide sample instructions it will be a big help.

Git-sync-repo - https://github.com/bumbeishvili/git-sync-repo/invitations
Git-sync-test-repo - https://github.com/bumbeishvili/git-sync-test-repo/invitations

If you had that done, is that all you'd need?

This will be only 10 % of the whole work, but I have a very clear idea of what to do in the remaining 90%, so you can say that this is all I need.

Note: if done, I won't be able to complete all work right away, since I am working on several paid projects currently, but I will get it done eventually

[bumbeishvili]

BTW - after that, it will provide near real-time data, (not previous day data), since I will be able to run it say in every 30 m.

[kylebakerio]

I'm also in a sprint for some paid work right now, but should get some availability in a week and a half.