.se .nu .ch .li .sk are also transferable

[maaaaz]

Hello there,

Thank you for this interesting project.

I have to mention that the titled zones are also transferable:

• https://internetstiftelsen.se/en/zone-data/
• https://www.switch.ch/open-data/

Cheers

[maaaaz]

And also .sk:

• https://sk-nic.sk/en/home/
• https://sk-nic.sk/subory/domains.txt

[flotwig]

Thanks for taking a look!

Currently this tool doesn't have any knowledge of the specific AXFR dance you have to do with .se and .nu. Right now TLDR-2 just sends the same AXFR to every TLD. I don't plan to work on this soon but I'd accept a PR.

This project's scope is really only around doing AXFRs, so .ch and .sk and other TLDs that offer downloads of the zone via HTTP or other mechanism don't belong here.

Eventually, I want to aggregate TLDR-2's lists along with other lists in a unified fashion, since there are so many ways that zones can be enumerated. Work in progress 😃

You should check out another project I've been working on, it has .se and .nu even: https://github.com/flotwig/zone-walks Sadly, those two specific zones are not currently auto-updated because of some issue with the zone walker I wrote causing it to take too long on GitHub Actions, along with some other zones: https://github.com/flotwig/zone-walks/blob/main/ignore-tld-list.txt Also a work in progress.

Previous relevant issue on TLDR: mandatoryprogrammer/TLDR#12

[maaaaz]

Clear, thank you.

By the way, the ch. and .li zonez are AXFRable:

dig -y hmac-sha512:tsig-zonedata-ch-public-21-01:stZwEGApYumtXkh73qMLPqfbIDozWKZLkqRvcjKSpRnsor6A6MxixRL6C2HeSVBQNfMW4wer+qjS0ZSfiWiJ3Q== @zonedata.switch.ch +noall +answer +noidnout +onesoa AXFR ch.

dig -y hmac-sha512:tsig-zonedata-li-public-21-01:t8GgeCn+fhPaj+cRy1epox2Vj4hZ45ax6v3rQCkkfIQNg5fsxuU23QM5mzz+BxJ4kgF/jiQyBDBvL+XWPE6oCQ== @zonedata.switch.ch +noall +answer +noidnout +onesoa AXFR li.

[maaaaz]

Also, why don't you use "ldns-walk" for the zones you have troubles to grab ?

I tried this on my side and it took few seconds, far from the 6-hour maximum Github Action duration:

$ time ldns-walk game > /tmp/test

real    0m38.872s
user    0m0.266s
sys     0m0.133s

$ tail -f /tmp/test
zepto.game. NS RRSIG NSEC
zimbabet.game. NS RRSIG NSEC
zitro.game. NS RRSIG NSEC
zitrogames.game. NS RRSIG NSEC
zlotxo.game. NS RRSIG NSEC
zoa.game. NS RRSIG NSEC
zone.game. NS RRSIG NSEC
zypher.game. NS RRSIG NSEC

$ wc /tmp/test
  4065  16346 124262 /tmp/test

[flotwig]

Also, why don't you use "ldns-walk" for the zones you have troubles to grab ?

@maaaaz It's a good idea. I didn't do this because I want zone-walker to eventually work with all walkable zones, so I intend to fix the bugs with the currently ignored TLDs in zone-walker. I'd accept a PR that uses ldns-walk as a fallback in the interim, but the goal is to fix the bugs in zone-walker ultimately.

[maaaaz]

Well, I am starting to encounter your issues.

I try to zone-walk from Github Action (ubuntu) and :

• ldns-walk gives some weird output: for instance, Error trying to resolve: \000. when walking the root zone

• ldns-walk (version 1.7.1) also fails with a different error when enumerating:
  'Error: Label length overflow' while ldns-walking the root zone  NLnetLabs/ldns#218

• n3secmap also fails to complete a simple zone-walk (root zone), it seems to be rate-limited (by Github ? by root NS ?) and gets timeout at xn--kput3i. (~90 % of the root zone)

For ldns-walk, I will try to use the latest version on Github Action to see if it is the root cause.

Cheers!

[maaaaz]

Hello @flotwig,

A quick update :

• I tried several methods to make NSEC enumeration work from Github Actions, like using an updated-and-myself-compiled version of ldns-walk etc. They all failed for unknown reasons, there should be a rate limitation somewhere but I don't know where, and don't know how to bypass. And when ldns-walk deigns to work, its results are not consistent (cf. issue link above)

• So I ended up doing the enumeration with n3map somewhere else from Github Actions, and just collecting results with Github Actions. It's all there https://github.com/maaaaz/dnsdumps

Definitely, an harder and deeper problem than I thought. I do then fully understand the technical issues you also face :)

Cheers!

[maaaaz]

I just discovered this: https://github.com/trickest/zip

[flotwig]

I just discovered this: trickest/zip

@maaaaz Thank you for sharing another resource. I also recommend that you check out the ICANN Centralized Zone Data Service (CZDS) if you haven't yet. As long as your use case fits ICANN's ToS, you can access over 1,100 zone files provided by the gTLD operators, making it trivial to enumerate domains for OSINT. You can even get .zip zonefiles from the CZDS; although, according to trickest/zip#2, it's not an exact match with trickest/zip's collected data.

AFAIK, existing gTLDs like .com, .net, .org, and .gov participate in CZDS voluntarily. However, participation in CZDS is mandatory for new gTLDs:

New gTLD Registries are required to start providing their zone data upon delegation
- Centralized Zone Data Service Frequently Asked Questions

[maaaaz]

Thank you for this information @flotwig !