Weekly_Log

Dev log for miso

Week 1:

This week is mostly going to be looking into web scraping, and how to make plugins for node.js

• puppeteer

• how to make plugins

Goals:

• reorganize code base
• Add new sites to the list
• Different Download options
• Add CLI interface

Week 2:

This week, starts the restructuring and organization of the code-base. Didnt make any real progress on the general scraping code, so I decided to focus on reorganization.

Goals:

• reorganize code base
  • Starting to restucture
• Add new sites to the list
• Different Download options
• Add CLI interface

Next week: Prepare for presentation

Week 3:

Gave a presentation at the end of week 2, someone recomended looking into cypress

At the moment, still looking at puppeteer. Looking into 2 things:

1. adding new sites a. xkcd for testing purposes b. pixiv for first test scrape
2. Adding CLI interface

Reorganization is more or less done. Need to write some general purpose scraping code that I can use for multiple sites. Still looking into puppeteer at this point.

Goals:

• reorganize code base
  • Starting to restucture
  • addition: currently all js files, could try to have config files for different sites instead
• Add new sites to the list
  • Pixiv
  • xkcd
• Different Download options
• Add CLI interface
  • in progress

Week 4:

Was able to get the basic general scraping code written, and added 2 sites to the list of scrap-able sites (nh, pixiv). Going to add more, and that should be a much less painful process. Ended up not going with cypress, because I was already deep with puppeteer.

A few things to look at for next week:

1. different download options
2. adding sites like archive.org that dont necessarily scrape, but have a lot of things to download.

Goals:

• reorganize code base
  • Starting to restucture
  • addition: currently all js files, could try to have config files for different sites instead
• Add new sites to the list
  • Pixiv
  • xkcd
  • nh
  • Archive.org
• Different Download options
  • in progress
• Add CLI interface
  • in progress

Next week looking into adding CLI and more sites as always.

Checkpoint 0

Part of discussion for Open Source software: Richard Stallman

Open Source tends to have a problem with branding and marketing. With it being so decentralized, and many of the people who are involved being more technical rather than outspoken, there often tends to be a void of a figure head that people outside of the sphere can go to as a figure head. If we look at Linux, the closest thing to a figurehead is Linus Torvalds , however, Linus has said on many occasions how he isn't a people person, and does very little to encourage the adoption of the OS. We can contrast this to Mac OS, which in its early days had Steve Jobs, who was a very charismatic and helped with pushing Mac OS's image and adoption. In this space devoid of someone to push and represent it, Stallman fulfills a very crucial role as the next best thing.

Richard Stallman is a very divisive figure in the open source community. As someone who used to be one of the main contributors to emacs, GCC, and many other GNU project software, he is a very prolific programmer as well as a head of the Free software movement. His efforts of pushing Free software, and his relative presence in the public have made him made him a very key figurehead in open source, and arguably the face of Free software. In this, he has many things going for him: for one, his zeal. The way he is uncompromising about free software helps to draw attention to the movement, and related ones like Open Source. However, there are a few problems with him as a figure head, namely how derisive he can be. You can posit then that him being the figurehead is harmful, as he would be one of the first people that the uninitiated would interact with, thus tainting the movement.

Despite this, Stallman being on the board of the FSF and being a figure head for Free software has done more good than harm. Even with him gone, from the FSF board, he was still the defacto head of the movement, as there was no one else who could fill that position. And something like the Free software movement desperately needs a figure head, as its fairly niche, unintuitive to an outsider at first, and needs more people to be drawn to it in order for it to have any weight. So despite Stallman arguably being a problematic figure head, he is still the best option as no one else has stepped up/ could step up, and the lack of a figure head not being an option.

Week 5:

Final week, and final blog post for OSS. Wrapped up some of the code that I had, added in a few sites last minute like imgur. Was working on adding a CLI but hit a snag with running it. Will have to dissect it a bit more before the code can be merged in. I was also looking into adding unit tests, however I dont really know how I could add them without copyright infringement. I can test some, because some of the sites have stuff licensed under creative commons, but not all. Also gave up on the config file idea, that turned out to be untenable because some sites have things you need to write some scripts for. Overall, made pretty good progress this semester.

• reorganize code base
  • Starting to restucture
• Add new sites to the list
  • Pixiv
  • xkcd
  • nh
  • Imgur
  • Archive.org
• Different Download options
  • backburner
• Add CLI interface
  • in progress

Also got distracted with another weeb-poly project but it is open source as well.