A robots.txt is a file that tells search engine robots which pages they should and shouldn’t crawl.