made a robots.txt, I think

src/robots.txt

@@ -0,0 +1,16 @@
+# Group 1
+User-agent: *
+Disallow: /box/
+
+# Group 2
+User-agent: AdsBot-Google
+User-agent: Mediapartners-Google
+User-agent: AdsBot-Google-Mobile
+User-agent: AdsBot-Google-Mobile-Apps
+Disallow: /
+
+# Group 3
+User-agent: *
+Allow: /
+
+Sitemap: http://www.example.com/sitemap.xml