Look Up Another Term

Redirected from: robots exclusion

Definition: robots.txt


A text file placed in the root directory of a website that prohibits search engine spiders from indexing all or specific pages of the site. The Robots Exclusion protocol provides a format for designating which directories and files are off limits to the spidering program. See spider.