Some zone walks missing ?

[maaaaz]

Hello there,

Let me first thank you for this great project !

Then, I wonder why some currently walkable zones are missing in the walks folder?
A current simple example, the dz. zone.

Cheers !

[monoidic]

The filters.txt file contains zone names which are explicitly filtered out, though the reasoning is not really explained there at the moment.
At the moment, the reasons are, as follows:

• The zone's contents are already fairly "public" already (root zone, arpa., ee., se., nu.)
• It would simply take more than an hour to perform all of the zone walks if it is included (the results of letting some zone walks run for ~3h are visible here; keeping all of this under an hour would be nice)
• (iirc some zone, maybe sl. or lk., had some issue where most nameservers would provide DNSSEC responses and one wouldn't, which would break zone walks relying on recursive resolvers instead of hammering one authoritative server directly, because some responses would come with NSEC records, some wouldn't; not sure which one it was, can't find it with a quick glance)

The problem is at least somewhat caused by zone walking being a fairly linear thing at the moment. From the start of the zone to the end. I will try to add something to distribute the zone walk into separate sections that can be worked on in parallel.
If that works, I'll see which zones (which aren't "public" anyhow) can be added while still comfortably fitting within the 1h limit, and/or potentially consider making AXFRs and zone walks timed separately, e.g with the latter being bihourly instead of hourly, but with all* zones.

Anyways, some extra info.
The AXFR method already provides the root zone and arpa. anyhow.

There are fairly straight-forward instructions for getting a couple of ccTLD zone files from official sources:
https://www.internet.ee/domains/ee-zone-file
https://internetstiftelsen.se/en/domains/tech-tools/access-to-zonefiles-for-se-and-nu/
https://portal.switch.ch/pub/open-data/

There are also simple domain lists:
https://github.com/cisagov/dotgov-data
https://www.afnic.fr/en/products-and-services/fr-and-associated-services/shared-data-reuse-fr-data/
https://sk-nic.sk/subory/domains.txt (linked on the main page)
https://www.domaine.nc/whos?who=A* (might have to, ahem, parse the HTML a bit)

And most zone files are accessible from Icann's CZDS, given you have a good enough reason for this access.

[maaaaz]

Thanks, very clear.
I am starting to face all these issues also: flotwig/TLDR-2#1

I thought it would be easier, I was wrong !