# robots.txt exists so the log doesn't fill with 'file not found' # if spiders start indexing test sites then we should # have # robots.txt for production that allows visits # robots.txt for developers/testers that doesn't User-agent: * Allow: *