Consider proposal for robots.txt extension

[voxpelli]

The closest to a specification for the X-Robots-Tag that I can find is: https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag

robots.txt on the other hand has a full rfc: https://www.rfc-editor.org/rfc/rfc9309.html

I think extending robots.txt with something similar to X-Robots-Tag would make sense for easier rollout, eg. something like:

User-Agent: *
Disallow: /example/
Noml: *

Or a more generic (where Rule is just an arbitrary word I picked, intention is to mimic X-Robots-Tag):

User-Agent: *
Disallow: /example/
Rule: /images/: noml, nosnippet