Filesize

The filesize of a robots.txt file should be less than 500KiB.

Google currently enforces a size limit of 500KiB on the robots.txt file. If you need to reduce the size of your robots.txt file, try combining allow and disallow directives by reorganizing your directory structure.