Robots.txt File Validator
Learn how to make a robots.txt file to block certain bots from scanning your website. Validate your robots.txt file and enhance it.
A robots.txt file is a plain text file that is placed at the root level of a website. Its purpose is to ask bots, spiders and crawlers to leave the website (or parts of the website) alone. You may want to do this if you think your web server will be overwhelmed by requests from a bot crawling your website. To learn more about the inner workings of this file, read our Robots.txt Explanation article.
Import Existing Robots.txt File
Upload or paste your robots.txt file to validate itWant to block AI bots?
AI bots will slurp up all the information on your website and use it to train Large Language Models (LLM). These LLMs can be used to generate new text and images based upon your content. If you believe that this constitutes plagiarism then you may want to prevent it. Copy and paste the lines below into your robots.txt file to block the most common AI bots.
User-agent: anthropic-ai
User-agent: Applebot-Extended
User-agent: Bytespider
User-agent: CCBot
User-agent: Claude-Web
User-agent: ClaudeBot
User-agent: cohere-ai
User-agent: Diffbot
User-agent: FacebookBot
User-agent: Google-Extended
User-agent: GPTBot
User-agent: Meta-ExternalAgent
User-agent: omgili
User-agent: Timpibot
Disallow: /
Test Your Website
Once you have made changes to your robots.txt file, enter your domain name into the box below and run a free ValidBot Test to see if everything is correct. Look in the "Common Files" section of the report to see if your robots.txt is working correctly.