Private Docker Registry Scraping

[Skiepp]

Right now, this awesome tool cannot scrape an entire registry for images, you have to add every single image manually or via API (which is not really scalable)

How about the possibility to scrape a registry?
(maybe with some limitations, to avoid "yay! let's scrape dockerhub for fun!")

This is indeed a killer point for introducing the tool in a kinda private business scenario...
I can work on a PR but I'm not 100% sure this feature will be accepted

What do you think guys? 😄

[tegioz]

Hi @Skiepp 👋

Please see this issue for some interesting conversations regarding this topic 🙂

Another problem would be that, for https://artifacthub.io, which is our main priority, this would probably lead quickly to scalability issues (and chaos, if we allow users to scrape entire registries, even with limitations 😅) as we would hit registries' API rate limits (IIRC overall limits in Docker Hub kick in after few thousand of requests per minute, but I think we've seen them earlier sometimes).

We are mostly interested in features that can be made publicly available on artifacthub.io for all users, so that everyone can benefit from them. We also try to help with minor features that can benefit other Artifact Hub deployments, provided they align well with the rest of use cases or aren't likely to cause any problems (like the new option you added recently 😉). But I'm not sure this one does, to be honest.

Hope this helps, thanks anyway!

[Skiepp]

Understood 😄 I suspected something like this, that's why I started a discussion instead of a Issue 👍

I guess for now I can create a simple private job using Artifact Hub API on the local installation or a direct insert into the DB

Happy to come back later on this topic if you change your mind 😉

Thanks a lot

[tegioz]

That sounds like a great plan! 👍

Thanks! 🙂