More than top 100??

[chapmanjacobd]

This is a very cool idea!

If you could track and not count duplicate lines of code, I believe it would be useful to include as many repos as possible.

[PabloLec]

Hi!

Sorry for the late return. Track duplicate lines of code would cause huge memory consumption. Well, if you track it across entire repos, not single files.
Not sure if it is really necessary, as most of the repos are short in code, they probably do not have any duplicated lines.

[chapmanjacobd]

huge memory consumption

You could hash each line (subtracting whitespace) then save the value into a SQLITE file and count duplicates at the end

[PabloLec]

Yes there could be solutions but I'm not sure it is really needed :/ As most of the repos displayed are quite short, I'm not even sure there is one with what could be considered duplicated code.
And if the goal is to reduce the line count for repos with duplicated code, I don't think let's say a 1000 lines repos with its code duplicated 100x should be considered as a 10 lines repo.
Altogether, I think this would be a feature with too much overhead for something I don't think is needed.

Regarding the title of the discussion, "more than top 100", that is of course possible. There is only two things I see:

1. Some less popular languages do not have a lot of repos with short code and a lot of stars. We'd fall quickly into >100 stars repos but why not.
2. I should manually exclude a lot of repos. As I already did, especially for JS and Python, there is a lot of "false positive", for example "awesome lists" with thousands of stars and a short JS file in it.