Robots Meta Tag
Tells bots how to handle the indexing of the webpage.
The robots meta tag is used on a page to tell search engine bots that crawl your webpage how they should handle that page. For example, if you don't want a page to appear in search engine results, you can use the meta tag to instruct search engines not to index it (example below). You may want to do this for pages used exclusively for testing or administrative purposes, or for pages that have little content or duplicate content to other pages on your website.
<meta name="robots" content="noindex" />
The above code will target all bots that understand this meta tag. If you want to target specific bots, you can do this by making multiple meta tags and specifically naming them, like this:
<meta name="googlebot" content="noindex" />
In the content field of the meta tag you can place more instructions besides "noindex". Here is a list of instructions that you can use:
- index - The bot is allowed to index the page. This is the default value that a bot assumes if this meta tag is omitted.
- noindex - The bot should ignore this page and not index it.
- follow - The bot is allowed to follow links found on this page.
- nofollow - The bot should ignore links found on this page.
- noimageindex - The bot should ignore images found on this page.
- noarchive - The bot should not archive the page to provide a cached version on search results pages.
- nocache - The same as noarchive.
- nositelinkssearchbox - The bot should not show a search box for the site on search results pages.
- nopagereadaloud - Prevents text to speech software from reading the page aloud.
- notranslate - Prevents services from translating the page into other languages.
- nosnippet - Prevents text snippets or video previous to appear in search results.
- indexifembedded - Indexing is only allowed if the page is embedded inside an iframe.
- max-snippet: - Limits the length of text snippets about this website on search engine results pages.
- max-image-preview: - Tells the search engine what size of an image preview it can display.
- max-video-preview: - Tells the search engine the maximum length of video preview it can display.
- unavailable_after: - Tells the bot when the page should be removed from indexes.
- none - Shortcut for 'noindex, nofollow'
- all - Shortcut for 'index, follow'
Multiple values can be specified by separating them with a comma. For example:
<meta name="badbot" content="noindex, nofollow, noimageindex" />
<meta name="googlebot" content="index, follow, noarchive, unavailable_after: Monday, 01-Jan-24 00:00:00 GMT" />
For a detailed explanation of the values that Google understands, you can read their documentation.
Ready to validate your website to check for this meta tag and 100+ others important tests?