# Web2MD — AI-search-friendly, scraper-hostile robots.txt User-agent: * Content-Signal: ai-train=yes, search=yes, ai-input=yes Allow: / Disallow: /dashboard Disallow: /auth Disallow: /api Disallow: /login Disallow: /welcome Disallow: /uninstall Disallow: /survey Disallow: /pricing/success Crawl-delay: 10 User-agent: Googlebot Allow: / Disallow: /dashboard Disallow: /auth Disallow: /api Disallow: /login Disallow: /welcome Disallow: /uninstall Disallow: /survey Disallow: /pricing/success User-agent: Bingbot Allow: / Disallow: /dashboard Disallow: /auth Disallow: /api Disallow: /login Disallow: /welcome Disallow: /uninstall Disallow: /survey Disallow: /pricing/success User-agent: GPTBot Allow: / Disallow: /dashboard Disallow: /auth Disallow: /api Disallow: /login Disallow: /welcome Disallow: /uninstall Disallow: /survey Disallow: /pricing/success Crawl-delay: 10 User-agent: OAI-SearchBot Allow: / Disallow: /dashboard Disallow: /auth Disallow: /api Disallow: /login Disallow: /welcome Disallow: /uninstall Disallow: /survey Disallow: /pricing/success Crawl-delay: 10 User-agent: ChatGPT-User Allow: / Disallow: /dashboard Disallow: /auth Disallow: /api Disallow: /login Disallow: /welcome Disallow: /uninstall Disallow: /survey Disallow: /pricing/success Crawl-delay: 10 User-agent: ClaudeBot Allow: / Disallow: /dashboard Disallow: /auth Disallow: /api Disallow: /login Disallow: /welcome Disallow: /uninstall Disallow: /survey Disallow: /pricing/success Crawl-delay: 10 User-agent: anthropic-ai Allow: / Disallow: /dashboard Disallow: /auth Disallow: /api Disallow: /login Disallow: /welcome Disallow: /uninstall Disallow: /survey Disallow: /pricing/success Crawl-delay: 10 User-agent: Claude-Web Allow: / Disallow: /dashboard Disallow: /auth Disallow: /api Disallow: /login Disallow: /welcome Disallow: /uninstall Disallow: /survey Disallow: /pricing/success Crawl-delay: 10 User-agent: PerplexityBot Allow: / Disallow: /dashboard Disallow: /auth Disallow: /api Disallow: /login Disallow: /welcome Disallow: /uninstall Disallow: /survey Disallow: /pricing/success Crawl-delay: 10 User-agent: Perplexity-User Allow: / Disallow: /dashboard Disallow: /auth Disallow: /api Disallow: /login Disallow: /welcome Disallow: /uninstall Disallow: /survey Disallow: /pricing/success Crawl-delay: 10 User-agent: Google-Extended Allow: / Disallow: /dashboard Disallow: /auth Disallow: /api Disallow: /login Disallow: /welcome Disallow: /uninstall Disallow: /survey Disallow: /pricing/success Crawl-delay: 10 User-agent: Amazonbot Allow: / Disallow: /dashboard Disallow: /auth Disallow: /api Disallow: /login Disallow: /welcome Disallow: /uninstall Disallow: /survey Disallow: /pricing/success Crawl-delay: 10 User-agent: Applebot Allow: / Disallow: /dashboard Disallow: /auth Disallow: /api Disallow: /login Disallow: /welcome Disallow: /uninstall Disallow: /survey Disallow: /pricing/success Crawl-delay: 10 User-agent: Applebot-Extended Allow: / Disallow: /dashboard Disallow: /auth Disallow: /api Disallow: /login Disallow: /welcome Disallow: /uninstall Disallow: /survey Disallow: /pricing/success Crawl-delay: 10 User-agent: Cohere-ai Allow: / Disallow: /dashboard Disallow: /auth Disallow: /api Disallow: /login Disallow: /welcome Disallow: /uninstall Disallow: /survey Disallow: /pricing/success Crawl-delay: 10 User-agent: cohere-ai Allow: / Disallow: /dashboard Disallow: /auth Disallow: /api Disallow: /login Disallow: /welcome Disallow: /uninstall Disallow: /survey Disallow: /pricing/success Crawl-delay: 10 User-agent: CCBot Allow: / Disallow: /dashboard Disallow: /auth Disallow: /api Disallow: /login Disallow: /welcome Disallow: /uninstall Disallow: /survey Disallow: /pricing/success Crawl-delay: 10 User-agent: Bytespider Disallow: / User-agent: AhrefsBot Disallow: / User-agent: SemrushBot Disallow: / User-agent: MJ12bot Disallow: / User-agent: DotBot Disallow: / User-agent: DataForSeoBot Disallow: / User-agent: BLEXBot Disallow: / User-agent: PetalBot Disallow: / User-agent: Scrapy Disallow: / User-agent: meta-externalagent Disallow: / User-agent: FacebookBot Disallow: / User-agent: ImagesiftBot Disallow: / User-agent: Diffbot Disallow: / User-agent: Timpibot Disallow: / Sitemap: https://web2md.org/sitemap.xml