Check robots.txt file of a website for errors and typos

If you own a website and are obsessed with your website being search engine friendly you will probably be aware of a file called robots.txt which often has the site-map to your website. This site-map is immediately picked up by search engines web crawlers which is how search engines end up indexing billions of web […]