implemented multithreading using the multiprocessing module

[virophagesp]

Related Issue/Addition to code

Old-Multi-Threading-Attempt

• Fixes "Multithreading works but too slow"
• Addition to code.

Type of change

• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

Proposed Changes

• utilizing the multiprocessing module to multithread get_comments

Why is this change needed?

It's needed because it speeds up scanning alot

Additional Info

• the threading module does not use multiple cpu cores, which is why the old attempt was too slow

Checklist:

• My code follows the style guidelines of this project and I have read CONTRIBUTING.md
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• Any dependent changes have been merged and published in downstream modules
• I have checked my code and corrected any misspellings

[virophagesp]

What documentation is my checklist referring to?

[virophagesp]

this is currently a basic setup for multi-threading the scanning of comments using only 2 CPU cores
I plan to add for loops for core scaling later
The reason why is because If multiple additional threads are made, large amounts of data shared between them will cause lag, and they share alot of data
If this is merged then this will be a major problem to fix

[virophagesp]

@ThioJoe I think multithreading is done, what do you think?

[virophagesp]

found a bug that causes it to freeze, I will reopen when its fixed and tested for other bugs

[virophagesp]

i also found out i forgot to implement shared memory which might be the cause
this might take a while

[virophagesp]

i implemented shared memory but now there is way too much ram usage and a unrelated bug i am currentlly fixing

[virophagesp]

i fixed the memory leak but there is still the issue of freezing on certain videos after 1 thread finishes

[virophagesp]

I am reopening this since it is mostly fixed
I also found the freezing issue to be uncommon

[virophagesp]

I found a variety of more videos with the freezing issue
I have found it to be caused by certain comments with replies not taking in the pageToken input and causing an infinite loop
What separates them from most comments with replies that dont cause this issue is unclear
I plan to investigate more this week and hopefully fix it

[virophagesp]

I found out that it is for any comment with more than 100 replies
But also it isnt caused by my changes at all

[virophagesp]

@ThioJoe this is ready now

[4yman-0]

Can you explain why there's 1 reference to YOUTUBE[1]

[virophagesp]

It is only for multi threading, I tried using it like it was before but errors were caused so unfortunately there needs to be 2 of them, maybe if other things get multithreaded it will have other uses