Hey Cuil, Use a Robots.txt Cuil.com is flooding my newsfeeds this morning. Reid already reported that their home page links are only periodically working. I, too, don't see my site on the first three pages of a vanity search. Also, looks like they need to add a robots.txt file to block other search engines from indexing their search results. This should work:

User-agent: *
Disallow: /search

Honestly, who searches for search results? When I hear about a new website, the first thing I do, is a site: search. The second thing I do is check the robots.txt. Isn't that what all normal people do?

Labels: Google, tech

  ¶ 7:09 AM