Add other domains owned by admiral?

[KeenRivals]

Many other domains were found that are owned by Admiral and point to the same IP as #1. There's a list at https://pgl.yoyo.org/adservers/admiral-domains.txt

[paulgb]

Thanks for this! I'll have a look when I'm at a computer

…

On Aug 12, 2017 11:55 AM, "KeenRivals" ***@***.***> wrote: Many other domains were found that are owned by Admiral and point to the same IP as #1 <#1>. There's a list at https://pgl.yoyo.org/adservers/admiral-domains.txt — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#4>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAC0XRsMHcj3UPhAvHkJToJddeujsbYlks5sXcr6gaJpZM4O1cvW> .

[tofof]

That list is woefully incomplete. See my comment here for some analysis. Click the reverse-domain links I provide (e.g. ipv4info for functionalclam.com) and you can see how deep this rabbit hole goes, and that's just for a single ip.

Sample admiral domains not in that list:
btez8.xyz
innocentwax.com
completecabbage.com
4jnzhl0d0.com
h78xb.pw

It's trivial to observe hundreds of Admiral domains, they probably number in the thousands.

[mvasilkov]

Let's kill the whole Admiral thing with fire!

[paulgb]

I've merged #8 which adds more Admiral domains, if there are still some missing I'm happy to add them as they are discovered

[mirague]

I support adding all domains pointing to the same content - it's likely all these domains would eventually have found their way into EasyList in the first place. Might be less likely now.

[JamyDev]

Maybe block that IP too in anticipation of more domains being added?

[tofof]

#8 is still not even close.

It has...

• Some but not all - e.g. breezybath.com - of the domains currently on 104.155.48.223 (the same ip that unknowntray.com is served from)
• Some but not all - e.g. axiomaticalley.com - of the domains currently at 146.148.6.205
• Some but not all - e.g. btez8.xyz - of domains currently at 35.186.249.84

It's missing...

• owlsr.us and 65+ other domains that have been or are currently on 104.196.29.119
• roastedvoice.com and 64 friends from 146.148.79.27
• 17ars.xyz and past domains hosted by 52.39.21.213 but not photos.hollywood.com that's there now
• lewdwind.com and 20+ domains formerly at 104.155.177.112
• spottysense.com and many formerly at 54.187.68.112
• and more
• and more
• and 30 more
• and commoncannon.com and a dozen more
• and more
• and chickenstation.com and 100 others
• and these
• and those
• and frailoffer.com and friends
• and granodiorite.com and others
• and 43 more except there's an innocent one too

... I think that demonstrates my point. (Yes, there are many duplicates once you start putting all of these together).

If someone actually wants to make a serious attempt, which hasn't happened yet, just walk the related domains on a tool (like threatcrowd) that lets you do so easily.

It's trivial to verify these even when you're learning the naming patterns, since they all serve up the same image. But you really have to do the verifications. Bannersnack.com, for example, is NOT an Admiral domain, even though it was hosted alongside them once.

Some starting points that I haven't already exhausted above include...
tzwaw.pw
0D7DK.XYZ
pz37t.xyz
3jsbf5.xyz - beware, there's at least one domain (apstylebook.com) that'd be a false positive.
4jnzhl0d0.com
82o9v830.com
familiarfloor.com

The biggest problem is that they use google/amazon hosting and so you can't trivially blacklist everything that resolves into their IP space, and that tools like ipv4info, threatcrowd, alienvault, tcpiputils, all have incomplete datasets. You really need multiple people using different toolsets walking the same space to root all of these out.

[lol768]

@tofof Given the ones that I have seen seem to use valid SSL certs from Let's Encrypt, do you think crawling the CT logs is a viable way of checking for these?

[anon182739]

What about doing it the other way around? As you said, it's trivial to verify a domain. What about checking all suspicious domains? If one is found, it's sent to some central server to be added to the list. Another way would be to just blacklist the IPs. This would have some false positives, but if it becomes a default filter for ublock/adblock it would have the effect of forcing them to clean up their IP ranges. But really, how often do you have a legitimate reason for loading random .js from a .xyz domain?

@lol768
They issue hundreds of thousands of certificates each day, but if you manage to get the first filtering down it's feasible, then you can use DNS to filter out more.

@mvasilkov
I don't think that's feasible. Maybe by sending abuse notices to the registrars, since they care more about quantity than quality.

[anon182739]

Also, playing cat-and-mouse with domains is not really a problem. It costs at least $1 to register one, and much less to find and block it.

[paulgb]

Actually, @LOL786's suggestion sounds like a great idea for keeping track of new registrations from this company. It could be automated as a nightly script.

[paulgb]

That's just a way of narrowing down the search space; we'd still verify that the domain behaved like the others.

I'd like to keep this to a pure domain blacklist, as opposed to running code on the client, for a few reasons: 1. it's more portable to existing blocking extensions, 2. it's more performant, and 3. I'm more comfortable with the legal defensibility (I am confident that a passive blacklist can never be illegal; I don't want to speculate about the legality of a more active client-side approach)

[anon182739]

@zymase
You can just check the image dimensions, or the length of the body. This works for me:
def isAdmiralDomain(domain): try: response = requests.get('https://' + domain) except: return False if len(response.text) == 179: return True else: return False

Otherwise, how on earth can a list be conceived otherwise than with try-and-succeed/try-and-fail policy? It's just impossible. We are facing networks as dirty as bots.

Start of with the LE cert list. Filter away anything that doesn't end with .com, .xyz, or .pw. Issue DNS queries for everything. If the whois isn't protected, remove it. If they're not using whatever registrars they're using, remove it. Then use tor/proxies (it's kind of counter productive for them to block IPs) for the final verification.

Why not literally treat them as bots? I don't have any, but SMTP accounts are allegedly quite cheap. What if you'd just send bulk abuse notices to the registrars and accuse them of being C2 servers for some botnet? They generally don't care if the notices are valid, they only care about the quantity of notices they're getting.

@paulgb

That's just a way of narrowing down the search space; we'd still verify that the domain behaved like the others.
And this job can be automated easily, as long as the domain can be identified once you can put it in the list forever.

[anon182739]

GitHub stripped away the formatting, so you need to add some newlines and indentation.
The images are identical right now.
CRC32: 8db019c1
MD5: 681e062bb33b9ba28f3427e7283c81a8
SHA1: 3fcf7e14e92043a00926d340d45778b618bc87a9
SHA2: 32afacb9285649aa4af43ea03e7cd9a522aa3e6d0554a2dabe308fac4531be5f

[anon182739]

Another identifying mark is the robots.txt, which is unlikely to change:
User-agent: * Disallow: /

The 404 page seems nonstandard:
404 page not found
(it's not HTML)
Content-Type:"text/plain; charset=utf-8"

The registrar is enom for the .com domains and namecheap for the .xyz and .pw domains.

They all use the same 4 nameservers. You should be able to enumerate from that, it's an uncommon combination.
NS-1212.AWSDNS-23.ORG
205.251.196.188
NS-1627.AWSDNS-11.CO.UK
205.251.198.91
NS-305.AWSDNS-38.COM
205.251.193.49
NS-697.AWSDNS-23.NET
205.251.194.185

[anon182739]

@zymase
You want to block the domain names. You can get a new IP for almost nothing, getting a new domain name costs some money. The picture's name changes, but the site's structure is the same.

[anon182739]

Also worth noting is that you can query however many admiral domains you want with Tor. If they start blocking IPs, then their ads won't work anymore.

[anon182739]

@zymase
I don't understand what you're trying to say. You can look at a domain and see if it has that picture. If it does, it also has the script we want to block.

The domains don't resolve to the picture. The domains resolve to Google IPs, which then serve that picture. The domains all point to different IPs.

The purpose is to block the script they use, the easiest way to do this is to block the Admiral domains so they can't serve the script.

[tofof]

@anon182739 wrote:

Filter away anything that doesn't end with .com, .xyz, or .pw.

I wouldn't use such a filter. First of all, Admiral has at least one .us domain that I've already prominently mentioned in this thread; it's literally the first domain I name as 'missing'. Second, there's no reason to think they won't expand to other TLDs. Nine months ago, noone had spotted any .xyz Admiral domains - I believe they were only using .com and .pw at that time.

[tofof]

@zymase
You've stated that you don't understand and are not a coder. It's acceptable to be interested, to use the emoji-response features, etc, but please don't clutter a single-issue thread with philosophical meanderings, well-wishings, tortuous analogies, and otherwise "laying the obvious," whatever that means.

To address your final point:
No, there is no reason to think that there must be a common resource, accessible to the public, that would identify all such domains.
If instead you mean that there must be a reason for these domains? Yes, the reason is that Admiral owns them and happens, for now, to serve the same content from all of them. They could just as easily serve nothing but a 403 or 204 error code, or just blackhole connections.

The list's criteria are already stated: it is a list of domains who have misused DMCA takedowns to attempt removal from other lists. This issue is for the suggestion that affiliated domains that are owned by the same company and used for the same purpose be included alongside the singular example named on a DMCA takedown thus far. The criteria for inclusion that's being proposed, then, is similarly obvious: Admiral-owned domains that appear to serve the same (lack of) content and presumably host the scripts used in serving advertising at affiliated websites.

Quite contrary to your assertion, such a list, if built, will be built exactly the same way all other advertising-blocking lists are built: on the finds of participants reporting "I found another one". The starting points are found when an Admiral-protected website (e.g. thewindowsclub.com) uses scripts hosted on an Admiral server to display its contents.

I have already outlined the best possible way to find more Admiral domains given a starting domain: by using tools meant for that, i.e. tools that identify spatially- and temporally-related domains.

[paulgb]

@tofof Thanks for the analysis you've been doing, I have only been skimming this conversation while working on providing all the blacklist formats people want but now that that's done I want to take a real stab at automating some of this.

[anon182739]

@zymase

We're not going to quest the whole web, domain after domain to see the ones which point to that picture, right?

You can narrow it down enough so you don't need to check the entire web.
@tofof
What's wrong with scraping DNS/cert lists? They can easily make sure that 1 domain = 1 ip and avoid tainting each other, it's non-trivial to make it harder to verify the domains.
@paulgb
I'm already working on it, I've managed to hack together a python script that does the job. Should I post it here, or is 'security by obscurity' better?

It turns out they only have 159 domains apparently, all the "different starting points" were somehow interlinked.

https://pastebin.com/6mPnXBiR

[paulgb]

Great stuff, is this from the CT log or just from grabbing the IPs already found?

Let's keep the script apart from this, but if you're not a paying GitHub user I can create a private repo and add you to it so we can collaborate.

[anon182739]

This naming scheme is interesting.
If you visit any admiral domain (example: http://abandonedclover.com http://abruptroad.com) you get that image:
http://abandonedclover.com/6f044848f5e9030b6fd409a7e153defd6d8c4e58fb082a44da549ed3e421f9755aedb08132895be1e0d578e7
But each time you refresh the page, you get a new URL:
http://abandonedclover.com/f1e5b5d86bcceb851312e5cc5f7bce26bb10ab951c152cb63a5068954caaa20d196ab3a042d306f561b71c22
You can use it multiple times, and across different domains. I really wonder how this works? Is it a signature of some sort?
@paulgb
This is from scraping one domain (hfc195b.com) and recursively querying the results from threatcrowd. It seems to cover all of the "starting points" listed though, except for 0d7dk.xyz pz37t.xyz 3jsbf5.xyz that weren't reachable. So this should be all of their domains.
Sure, or I can send it in a PM if you want. It's not of much use now though.

[paulgb]

Sure, a PM works for me. My email is [email protected]

Cheers.

[anon182739]

Oh, you can't send GitHub PMs anymore apparently.
Gmail filters any anonymous e-mail addresses since they're used for spam. If you already have github premium, it's probably easier to do it that way.

[paulgb]

Ok, I created a repo.

[anon182739]

I can't see anything. Where do I get the notice?

[paulgb]

Try this link: https://github.com/paulgb/iptool/invitations

[anon182739]

There is no need to block any ASNs. It is sufficient to register a few hundred accounts with fake information and see which domains are being offered.