The robots.txt file can help search engines like yahoo prioritize crawling and indexing the internet pages that subject most to your web site, in the end improving your internet site’s visibility and natural site visitors. Don't forget if you proper click any website you could look at its source code. https://rankingseotoolz.com/robots-txt-generator