A website that is available to search engine crawlers that index all the pages on the Web. An entire website or just specific pages of a website can be coded as not crawlable if the site owner does not want the content to appear on the search engine's results pages.
Why Not Crawlable?
Although websites are typically created for the most visibility on a search engine, a new site under development or pages with less importance may be marked as not crawlable. In addition, private content may also be marked as such. See
robots.txt and
crawler.