All the search engines follow these robot directives, which, while not a perfect security solution, limit the entry points used by the bots and spiders that index the Web.
If I am correct, a choppy market this month this will make it difficult for those who invest in index-based products while stock pickers who concentrate on entry levels and risk should do much better.