What features are essential before twitter goes away?

[lenaschimmel]

Hypothesis: the data that we try to download and archive with this tool might become unavailable for good, even though I don't know if it'll be days, weeks or months until it happens. It's not a huge difference if Twitter goes down entirely, if the APIs we are using fail or are switched off, if twitter loses or deletes the data, if they mass-block users, etc.

So I'm approaching development like this: focus on downloading data first and foremost, and leave everything else for later. If / when the Twitter API is offline, we still have time to crunch the data locally, add better formatting or usability to the outputted html and markdown, add more output formats, etc.

This reasoning leads me to believe that even advanced downloading / archiving features, which are only relevant for a small share of our users, might be more urgent than the processing and formatting that everybody will need and use. That's not saying that they are more important.

How do you (as users and/or developers of this script) feel about this approach? What data should be downloaded by default? What data might be important to some, but not all users? When can the feature "downloading" be considerd as "finished" and (my) work can focus on the offline processing of the data?

(Side note: I'm still working on the PR #97 "Base feature: Download tweets which are missing or incomplete", and I'm quite unhappy that I still couldn't finish it. Also, I know it's hard to grasp the progress on that PR an the related #122 "More work on downloadtweets". I plan to write a not-very-technical update soon.)

[lenaschimmel]

FYI: For impatient minds like ourselves, @flauschzelle and I are currently working on a quick-and-dirty version to get data downloaded now, quickly and possibly without user input, so that you can automate it.

In this version, we merged several features from branches and PRs that already work, but where the code does not necessarily meet the standards of the main repo. It's in the archivemode branch of my fork.

It accepts command line parameters like this: ./parser.py --archive-folder <PATH_TO_ARCHIVE> and asks no questions because it has defaults configured for each potential question. You can change those defaults in parser.py - get_config.