bascht.com/content/tech/2013-06-20-emulate-robotstxt-with-simple-nginx-directive.md

title	date	tags
Emulate a robots.txt file with a simple nginx directive	2013-06-20	nginx seo google robots robotstxt

If you use nginx as a reverse proxy, you might want to emulate a simple robots.txt file just to be sure that Googlebot doesn't traverse into each and every location.

So why bother serving those bytes from a file when you can answer requests directly from nginx:

location /robots.txt {
    return 200 "User-agent: *\nDisallow: /";
}