# The marketing site is fully open to crawlers. The named per-bot groups that # used to be here (Googlebot, Bingbot, Twitterbot, facebookexternalhit) all # said nothing but "Allow: /", which is already the default — and they were # actively harmful to keep, because robots.txt applies only the single # best-matching group per crawler. Any Disallow added below would have been # ignored by exactly the four crawlers that mattered most. User-agent: * Allow: / # Build artefacts and internal previews. These are reachable over HTTP but are # not pages anyone should arrive at from a search result: # /emails/ hosted "view in browser" copies of campaign email — near- # duplicates of content that is also on the site, with no canonical # /decks/ internal TMS report and export pages # /banners/ HTML templates used to render social banner PNGs # Left crawlable on purpose: /og/ (holds skills.png, referenced as a real OG # image — blocking it breaks link previews) and /deck/ (the public overview PDF). Disallow: /emails/ Disallow: /decks/ Disallow: /banners/ # Behind a flow; these serve the SPA shell, so indexing them yields a thin # duplicate of the homepage markup. Disallow: /auth/ Disallow: /billing/ Sitemap: https://robotactions.com/sitemap.xml