Create accountLogin

Recent searches

No recent searches

Search options

Only available when logged in.
ohai.social is one of the many independent Mastodon servers you can use to participate in the fediverse.
A cozy, fast and secure Mastodon server where everyone is welcome. Run by the folks at ohai.is.

Administered by:

Server stats:

1.8K
active users

ohai.social: AboutStatusProfiles directoryPrivacy policy

Mastodon: AboutGet the appKeyboard shortcutsView source codev4.4.0-nightly.2025-04-01

#robotstxt

0 posts0 participants0 posts today
Frontend Dogma<p>Poisoning Well, by <span class="h-card" translate="no"><a href="https://front-end.social/@heydon" class="u-url mention" rel="nofollow noopener" target="_blank">@<span>heydon</span></a></span>:</p><p><a href="https://heydonworks.com/article/poisoning-well/" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="ellipsis">heydonworks.com/article/poison</span><span class="invisible">ing-well/</span></a></p><p><a href="https://mas.to/tags/ai" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>ai</span></a> <a href="https://mas.to/tags/crawling" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>crawling</span></a> <a href="https://mas.to/tags/robotstxt" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>robotstxt</span></a> <a href="https://mas.to/tags/content" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>content</span></a></p>
PPC Land<p>ICYMI: Google outlines pathway for robots.txt protocol to evolve: How the 30-year-old web crawler control standard could adopt new functionalities while maintaining its simplicity. <a href="https://ppc.land/google-outlines-pathway-for-robots-txt-protocol-to-evolve/" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="ellipsis">ppc.land/google-outlines-pathw</span><span class="invisible">ay-for-robots-txt-protocol-to-evolve/</span></a> <a href="https://mastodon.social/tags/Google" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Google</span></a> <a href="https://mastodon.social/tags/RobotsTxt" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>RobotsTxt</span></a> <a href="https://mastodon.social/tags/WebCrawling" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>WebCrawling</span></a> <a href="https://mastodon.social/tags/SEO" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>SEO</span></a> <a href="https://mastodon.social/tags/DigitalMarketing" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>DigitalMarketing</span></a></p>
PPC Land<p>ICYMI: Google outlines pathway for robots.txt protocol to evolve: How the 30-year-old web crawler control standard could adopt new functionalities while maintaining its simplicity. <a href="https://ppc.land/google-outlines-pathway-for-robots-txt-protocol-to-evolve/" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="ellipsis">ppc.land/google-outlines-pathw</span><span class="invisible">ay-for-robots-txt-protocol-to-evolve/</span></a> <a href="https://mastodon.social/tags/Google" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Google</span></a> <a href="https://mastodon.social/tags/RobotsTxt" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>RobotsTxt</span></a> <a href="https://mastodon.social/tags/WebDevelopment" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>WebDevelopment</span></a> <a href="https://mastodon.social/tags/SEO" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>SEO</span></a> <a href="https://mastodon.social/tags/WebCrawler" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>WebCrawler</span></a></p>
Inautilo<p><a href="https://mastodon.social/tags/Development" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Development</span></a> <a href="https://mastodon.social/tags/Techniques" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Techniques</span></a><br>Poisoning well · An effort to dupe nasty AI crawlers with nonsense <a href="https://ilo.im/1632tq" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="">ilo.im/1632tq</span><span class="invisible"></span></a></p><p>_____<br><a href="https://mastodon.social/tags/AI" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>AI</span></a> <a href="https://mastodon.social/tags/ChatBots" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>ChatBots</span></a> <a href="https://mastodon.social/tags/SEO" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>SEO</span></a> <a href="https://mastodon.social/tags/Content" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Content</span></a> <a href="https://mastodon.social/tags/Protection" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Protection</span></a> <a href="https://mastodon.social/tags/RobotsTxt" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>RobotsTxt</span></a> <a href="https://mastodon.social/tags/WebDev" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>WebDev</span></a> <a href="https://mastodon.social/tags/Backend" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Backend</span></a> <a href="https://mastodon.social/tags/Frontend" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Frontend</span></a> <a href="https://mastodon.social/tags/HTML" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>HTML</span></a></p>
rijo<p>Google outlines pathway for robots.txt protocol to evolve <a href="https://ppc.land/google-outlines-pathway-for-robots-txt-protocol-to-evolve/" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="ellipsis">ppc.land/google-outlines-pathw</span><span class="invisible">ay-for-robots-txt-protocol-to-evolve/</span></a> <a href="https://frankfurt.social/tags/Google" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Google</span></a> <a href="https://frankfurt.social/tags/RobotsTxt" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>RobotsTxt</span></a> <a href="https://frankfurt.social/tags/WebCrawlers" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>WebCrawlers</span></a> <a href="https://frankfurt.social/tags/SEO" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>SEO</span></a> <a href="https://frankfurt.social/tags/DigitalMarketing" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>DigitalMarketing</span></a></p>
PPC Land<p>Google outlines pathway for robots.txt protocol to evolve: How the 30-year-old web crawler control standard could adopt new functionalities while maintaining its simplicity. <a href="https://ppc.land/google-outlines-pathway-for-robots-txt-protocol-to-evolve/" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="ellipsis">ppc.land/google-outlines-pathw</span><span class="invisible">ay-for-robots-txt-protocol-to-evolve/</span></a> <a href="https://mastodon.social/tags/Google" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Google</span></a> <a href="https://mastodon.social/tags/RobotsTxt" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>RobotsTxt</span></a> <a href="https://mastodon.social/tags/WebCrawlers" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>WebCrawlers</span></a> <a href="https://mastodon.social/tags/SEO" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>SEO</span></a> <a href="https://mastodon.social/tags/DigitalMarketing" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>DigitalMarketing</span></a></p>
Inautilo<p><a href="https://mastodon.social/tags/Business" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Business</span></a> <a href="https://mastodon.social/tags/Introductions" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Introductions</span></a><br>Meet LLMs.txt · A proposed standard for AI website content crawling <a href="https://ilo.im/16318s" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="">ilo.im/16318s</span><span class="invisible"></span></a></p><p>_____<br><a href="https://mastodon.social/tags/SEO" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>SEO</span></a> <a href="https://mastodon.social/tags/GEO" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>GEO</span></a> <a href="https://mastodon.social/tags/AI" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>AI</span></a> <a href="https://mastodon.social/tags/Bots" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Bots</span></a> <a href="https://mastodon.social/tags/Crawlers" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Crawlers</span></a> <a href="https://mastodon.social/tags/LlmsTxt" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>LlmsTxt</span></a> <a href="https://mastodon.social/tags/RobotsTxt" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>RobotsTxt</span></a> <a href="https://mastodon.social/tags/Development" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Development</span></a> <a href="https://mastodon.social/tags/WebDev" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>WebDev</span></a> <a href="https://mastodon.social/tags/Backend" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Backend</span></a></p>
Winbuzzer<p>AI Crawlers Overwhelm Open-Source Projects, Forcing Developers to Block Entire Countries</p><p><a href="https://mastodon.social/tags/AI" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>AI</span></a> <a href="https://mastodon.social/tags/Web" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Web</span></a> <a href="https://mastodon.social/tags/Robotstxt" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Robotstxt</span></a> <a href="https://mastodon.social/tags/AIScraping" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>AIScraping</span></a> <a href="https://mastodon.social/tags/OpenSource" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>OpenSource</span></a> <a href="https://mastodon.social/tags/Cybersecurity" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Cybersecurity</span></a> <a href="https://mastodon.social/tags/DataScraping" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>DataScraping</span></a> <a href="https://mastodon.social/tags/Scraping" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Scraping</span></a> <a href="https://mastodon.social/tags/WebScraping" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>WebScraping</span></a> </p><p><a href="https://winbuzzer.com/2025/03/26/ai-crawlers-overwhelm-open-source-projects-forcing-developers-to-block-entire-countries-xcxwbn/" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="ellipsis">winbuzzer.com/2025/03/26/ai-cr</span><span class="invisible">awlers-overwhelm-open-source-projects-forcing-developers-to-block-entire-countries-xcxwbn/</span></a></p>
𝕂𝚞𝚋𝚒𝚔ℙ𝚒𝚡𝚎𝚕<p>When the greed becomes automatist, something like this happens here:</p><p>»Open Source World – FOSS infrastructure is under attack by AI companies:<br>LLM scrapers are taking down FOSS projects' infrastructure, and it's getting worse.«</p><p>😐 <a href="https://thelibre.news/foss-infrastructure-is-under-attack-by-ai-companies/" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="ellipsis">thelibre.news/foss-infrastruct</span><span class="invisible">ure-is-under-attack-by-ai-companies/</span></a></p><p><a href="https://chaos.social/tags/foss" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>foss</span></a> <a href="https://chaos.social/tags/opensource" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>opensource</span></a> <a href="https://chaos.social/tags/llm" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>llm</span></a> <a href="https://chaos.social/tags/attack" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>attack</span></a> <a href="https://chaos.social/tags/floss" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>floss</span></a> <a href="https://chaos.social/tags/kde" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>kde</span></a> <a href="https://chaos.social/tags/gnome" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>gnome</span></a> <a href="https://chaos.social/tags/robotstxt" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>robotstxt</span></a> <a href="https://chaos.social/tags/stolendata" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>stolendata</span></a> <a href="https://chaos.social/tags/LLMs" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>LLMs</span></a> <a href="https://chaos.social/tags/freedom" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>freedom</span></a> <a href="https://chaos.social/tags/infrastructure" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>infrastructure</span></a> <a href="https://chaos.social/tags/greed" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>greed</span></a></p>
glyn<p>New release of nginx_robot_access:</p><p><a href="https://github.com/glyn/nginx_robot_access/releases/tag/v0.1.1" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="ellipsis">github.com/glyn/nginx_robot_ac</span><span class="invisible">cess/releases/tag/v0.1.1</span></a></p><p><a href="https://fosstodon.org/tags/nginx" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>nginx</span></a> <a href="https://fosstodon.org/tags/robotstxt" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>robotstxt</span></a></p>
Ross A. Baker<p>Tracked down my Forgejo CPU spikes with pprof: an otherwise acceptable crawler is indexing each commit of my personal weather station data. All 107,980 of them. Blame info, too.</p><p>Many Forgejo paths are nonsensical to crawl, even by good bots. Codeberg's robots.txt is a great start for these.</p><p><a href="https://codeberg.org/robots.txt" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="">codeberg.org/robots.txt</span><span class="invisible"></span></a></p><p>This should both relieve pressure and expose more bad bots.</p><p><a href="https://social.rossabaker.com/tags/Forgejo" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Forgejo</span></a> <a href="https://social.rossabaker.com/tags/RobotsTxt" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>RobotsTxt</span></a></p>
Inautilo<p><a href="https://mastodon.social/tags/Development" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Development</span></a> <a href="https://mastodon.social/tags/Reports" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Reports</span></a><br>Google AI Mode is here · How to access it and control it with robots.txt <a href="https://ilo.im/162o8h" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="">ilo.im/162o8h</span><span class="invisible"></span></a></p><p>_____<br><a href="https://mastodon.social/tags/Business" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Business</span></a> <a href="https://mastodon.social/tags/Google" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Google</span></a> <a href="https://mastodon.social/tags/SearchEngine" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>SearchEngine</span></a> <a href="https://mastodon.social/tags/AnswerEngine" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>AnswerEngine</span></a> <a href="https://mastodon.social/tags/AI" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>AI</span></a> <a href="https://mastodon.social/tags/RobotsTxt" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>RobotsTxt</span></a> <a href="https://mastodon.social/tags/WebDev" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>WebDev</span></a> <a href="https://mastodon.social/tags/Frontend" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Frontend</span></a> <a href="https://mastodon.social/tags/Backend" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Backend</span></a></p>
zeyus :cv_red:‎:ci_floppy:‎:crt_w_prompt:‎:winamp:<p>Hey does anyone know if there's still a working zip bomb style exploit that can be deployed on a static site/JS (or as a asset/resource)? Specifically to target web scrapers and AI bullshit? The second any server goes online now it's immediately bombarded by stupid numbers of requests.</p><p><a href="https://corteximplant.com/tags/hacking" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>hacking</span></a> <a href="https://corteximplant.com/tags/aislop" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>aislop</span></a> <a href="https://corteximplant.com/tags/crawlers" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>crawlers</span></a> <a href="https://corteximplant.com/tags/webscraping" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>webscraping</span></a> <a href="https://corteximplant.com/tags/webcrawler" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>webcrawler</span></a> <a href="https://corteximplant.com/tags/robots" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>robots</span></a> <a href="https://corteximplant.com/tags/zipbomb" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>zipbomb</span></a> <a href="https://corteximplant.com/tags/zipbombing" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>zipbombing</span></a> <a href="https://corteximplant.com/tags/exploit" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>exploit</span></a> <a href="https://corteximplant.com/tags/robotstxt" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>robotstxt</span></a> <a href="https://corteximplant.com/tags/server" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>server</span></a> <a href="https://corteximplant.com/tags/scraper" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>scraper</span></a></p>
100% TAX ✅ ✅<p><span class="h-card" translate="no"><a href="https://mastodon.social/@Hexangon" class="u-url mention" rel="nofollow noopener" target="_blank">@<span>Hexangon</span></a></span> </p><p>They are now back online!</p><p><a href="https://mastodon.social/tags/robotstxt" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>robotstxt</span></a></p>
100% TAX ✅ ✅<p>FediDB is back? FediDB is back!!!</p><p>❗ ❕ </p><p><a href="https://mastodon.social/tags/FediDB" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>FediDB</span></a> <a href="https://mastodon.social/tags/Robotstxt" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Robotstxt</span></a> <a href="https://mastodon.social/tags/FediDBRox" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>FediDBRox</span></a></p>
100% TAX ✅ ✅<p>Update: It's been fixed. FediDB is now back online! </p><p>YES!</p><p>IT HAPPENED!</p><p><a href="https://mastodon.social/tags/FediDB" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>FediDB</span></a> is still working on the <a href="https://mastodon.social/tags/NoRobots" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>NoRobots</span></a>, <a href="https://mastodon.social/tags/RobotsTxt" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>RobotsTxt</span></a>...</p><p><a href="https://mastodon.social/tags/GoodLuck" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>GoodLuck</span></a>!</p>
Fred<p>Website owners are fighting back: <a href="https://arstechnica.com/tech-policy/2025/01/ai-haters-build-tarpits-to-trap-and-trick-ai-scrapers-that-ignore-robots-txt/" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="ellipsis">arstechnica.com/tech-policy/20</span><span class="invisible">25/01/ai-haters-build-tarpits-to-trap-and-trick-ai-scrapers-that-ignore-robots-txt/</span></a></p><p><a href="https://mastodon.social/tags/News" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>News</span></a> <a href="https://mastodon.social/tags/AI" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>AI</span></a> <a href="https://mastodon.social/tags/AntiAI" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>AntiAI</span></a> <a href="https://mastodon.social/tags/Tarpits" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Tarpits</span></a> <a href="https://mastodon.social/tags/Scrapers" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Scrapers</span></a> <a href="https://mastodon.social/tags/robotstxt" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>robotstxt</span></a></p>
InkDesign<p>✨ Desvende os Segredos do robots.txt! <br>📝 Quer dominar a configuração do robots.txt e turbinar seu site? Aprenda como esse simples arquivo pode impactar sua SEO, controlar o acesso de bots e melhorar sua visibilidade online. Não perca as dicas essenciais que o InkDesign preparou para você! Clique e descubra como otimizar sua presença digital agora mesmo! <br>. <br>. <br><a href="https://mastodon.social/tags/SEO" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>SEO</span></a> <a href="https://mastodon.social/tags/WebDesign" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>WebDesign</span></a> <a href="https://mastodon.social/tags/robotsTxt" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>robotsTxt</span></a><br><a href="https://inkdesign.com.br/dominando-a-configuracao-do-robots-txt-guia-essencial/?fsp_sid=7758" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="ellipsis">inkdesign.com.br/dominando-a-c</span><span class="invisible">onfiguracao-do-robots-txt-guia-essencial/?fsp_sid=7758</span></a></p>
Dawn Tåke 🌙 :sparkletrans:<p>Hi, got a question.</p><p>Is there a standard for Anti-AI/Anti-SEO etc robots.txt file? Or a trustworthy site that explains how to build one if prefab isn't available? Is there anything else I should consider? </p><p>Thanks.</p><p><a href="https://tech.lgbt/tags/AskFedi" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>AskFedi</span></a> <a href="https://tech.lgbt/tags/TechHelp" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>TechHelp</span></a> <a href="https://tech.lgbt/tags/RobotsTXT" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>RobotsTXT</span></a> <a href="https://tech.lgbt/tags/RobotsDotTXT" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>RobotsDotTXT</span></a></p>
Inautilo<p><a href="https://mastodon.social/tags/Development" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Development</span></a> <a href="https://mastodon.social/tags/Releases" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Releases</span></a><br>AI Insights · Cloudflare Radar brings deeper insights into AI trends <a href="https://ilo.im/1626sk" rel="nofollow noopener" translate="no" target="_blank"><span class="invisible">https://</span><span class="">ilo.im/1626sk</span><span class="invisible"></span></a></p><p>_____<br><a href="https://mastodon.social/tags/Cloudflare" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Cloudflare</span></a> <a href="https://mastodon.social/tags/CloudflareRadar" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>CloudflareRadar</span></a> <a href="https://mastodon.social/tags/Trends" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Trends</span></a> <a href="https://mastodon.social/tags/AI" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>AI</span></a> <a href="https://mastodon.social/tags/AiModels" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>AiModels</span></a> <a href="https://mastodon.social/tags/Bots" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Bots</span></a> <a href="https://mastodon.social/tags/RobotsTxt" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>RobotsTxt</span></a> <a href="https://mastodon.social/tags/WebDev" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>WebDev</span></a> <a href="https://mastodon.social/tags/Frontend" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Frontend</span></a> <a href="https://mastodon.social/tags/Backend" class="mention hashtag" rel="nofollow noopener" target="_blank">#<span>Backend</span></a></p>
ExploreLive feeds
About