Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add an option to disable generation of robots.txt #319

Open
CSharperMantle opened this issue Jan 17, 2025 · 0 comments
Open

Add an option to disable generation of robots.txt #319

CSharperMantle opened this issue Jan 17, 2025 · 0 comments

Comments

@CSharperMantle
Copy link

As this is a sitemap plugin, it would be beneficial if an option was added to make it generate sitemaps only. The generated robots.txt contains only a Sitemap: directive and does not have much effect on regulating spiders.

Background. I am using Jekyll Collections + permalink to collect miscellaneous static files scattered at the project root (e.g. Bing & Google site authentication, CNAME, robots.txt, ...) into a folder, so copying could be deferred to build time. This plugin's version of robots.txt is causing an annoying warning:

Conflict: The following destination is shared by multiple files.
          The written file may end up with unexpected contents.
          [root]/_site/robots.txt
           - robots.txt
           - [root]/_static/robots.txt

If the custom robot.txt is created directly at the project root, no warnings would be generated, and this plugin's version would be overwritten correctly. Yet in my scenario, this approach of piling everything at the root is unluckily what I am avoiding.

A similar request was put forward some time ago:

Same problem here. It would be beneficial to make a parameter to this plugin with the option to disable robots.txt creation.

Originally posted by @FrancescoBonizzi in #292

Yet it had received no responses, and the issue was closed as stale afterward.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant