Robots.txt Not Cumulative

Yesterday, gs1md wrote a meaty post on WMW regarding how Googlebot responds to robots.txt containing directives for both User-Agent:* and User-Agent: Googlebot. Both Googleguy and Vanessa Fox dropped by to clarify the situation: When you use both specifications, Googlebot will go with User-Agent: Googlebot.

Related links:

Related Posts

What's Your Take?





Fatal error: Call to undefined function error_msg() in /web/sites/halfdeck/seo4fun.com/php/cnt-include.php on line 26