Sitemap: https://outliermedia.org/sitemap.xml Sitemap: https://outliermedia.org/news-sitemap.xml Sitemap: https://outliermedia.org/sitemap_index.xml # --------------------------- User-agent: * Disallow: /?s= Disallow: /page/*/?s= Disallow: /search/ Disallow: /wp-json/ Disallow: /?rest_route= User-agent: adidxbot Disallow: / # https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0 User-agent: AdsBot Disallow: / # https://developers.google.com/search/docs/crawling-indexing/google-special-case-crawlers User-agent: AdsBot-Google Disallow: / # https://developers.google.com/search/docs/crawling-indexing/google-special-case-crawlers User-agent: AdsBot-Google-Mobile Disallow: / # https://developers.google.com/search/docs/crawling-indexing/google-special-case-crawlers User-agent: AI2Bot Disallow: / # https://allenai.org/crawler User-agent: Amazonbot Disallow: / # https://developer.amazon.com/amazonbot User-agent: anthropic-ai Disallow: / # https://support.anthropic.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler User-agent: Applebot-Extended Disallow: / # https://support.apple.com/en-us/119829 User-agent: Bytespider Disallow: / # User-agent: CCBot Disallow: / # https://commoncrawl.org/ccbot User-agent: ClaudeBot Disallow: / # https://support.anthropic.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler User-agent: cohere-ai Disallow: / # User-agent: cohere-training-data-crawler Disallow: / # User-agent: Crawlspace Disallow: / # https://crawlspace.dev/ User-agent: DataForSeoBot Disallow: / # https://dataforseo.com/dataforseo-bot User-agent: Diffbot Disallow: / # https://docs.diffbot.com/reference/crawl-introduction User-agent: DuckAssistBot Disallow: / # https://duckduckgo.com/duckduckgo-help-pages/results/duckassistbot/ User-agent: FacebookBot Disallow: / # https://developers.facebook.com/docs/sharing/webmasters/web-crawlers User-agent: Google-CloudVertexBot Disallow: / # https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers User-agent: GPTBot Disallow: / # https://platform.openai.com/docs/bots User-agent: Google-Extended Disallow: / # https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers User-agent: GoogleOther Disallow: / # https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers User-agent: ICC-Crawler Disallow: / # https://ucri.nict.go.jp/en/icccrawler/ User-agent: img2dataset Disallow: / # https://github.com/rom1504/img2dataset/blob/main/README.md User-agent: ImagesiftBot Disallow: / # https://imagesift.com/about User-agent: Kangaroo Bot Disallow: / # https://kangaroollm.com.au/kangaroo-bot/ User-agent: magpie-crawler Disallow: / # https://www.brandwatch.com/legal/magpie-crawler/ User-agent: Mediapartners-Google Disallow: / # https://developers.google.com/search/docs/crawling-indexing/google-special-case-crawlers User-agent: Meta-ExternalAgent Disallow: / # https://developers.facebook.com/docs/sharing/webmasters/web-crawlers User-agent: Meta-ExternalFetcher Disallow: / # https://developers.facebook.com/docs/sharing/webmasters/web-crawlers User-agent: omgili Disallow: / # https://webz.io/blog/web-data/what-is-the-omgili-bot-and-why-is-it-crawling-your-website/ User-agent: omgilibot Disallow: / # https://webz.io/blog/web-data/what-is-the-omgili-bot-and-why-is-it-crawling-your-website/ User-agent: PetalBot Disallow: / # https://webmaster.petalsearch.com/site/petalbot User-agent: Scrapy Disallow: / # https://www.zyte.com/blog/how-to-crawl-the-web-politely-with-scrapy/ User-agent: SemrushBot-OCOB Disallow: / # https://www.semrush.com/bot/ User-agent: SentiBot Disallow: / # https://sites.google.com/senti1.com/sentibot-eu/home User-agent: Storebot-Google Disallow: / # https://developers.google.com/search/docs/crawling-indexing/google-special-case-crawlers User-agent: Timpibot Disallow: / # User-agent: TurnitinBot Disallow: / # https://www.turnitin.com/robot/crawlerinfo.html User-agent: VelenPublicWebCrawler Disallow: / # https://velen.io/ User-agent: Webzio Disallow: / # https://webz.io/bot.html User-agent: Webzio-extended Disallow: / # https://webz.io/bot.html User-agent: YouBot Disallow: / # https://about.you.com/es/youbot/ # ---------------------------