# robots txt file is used as an access # list for web crawlers that comply # with the robots.txt file. User-agent: Google #allow Google Disallow: User-agent: Mediapartners-Google* Disallow: User-agent: Yahoo #allow Yahoo Disallow: User-agent: * # deny everthing else Disallow: /