A robots.txt file tells search engine bots which pages they should and shouldn’t crawl. Learn more.