Robots Meta Tag

The Robots Meta Tag tells search engines whether it can crawl the webpage and index the results. Usually, the website owner would want their website be listed in search engines. This test will trigger an error if the robots meta tag exists and if it contains either a "noindex" or "nofollow" instruction that would block search engines.

The preferred method for managing bots is to define rules in a robots.txt file placed at the root level of the website, however the meta tag can also be used. Our recommendation is to omit the robots meta tag entirely and instead use the robots.txt file to manage the instructions.