Skip to content
hostcomplex
Language: en

Block AI Crawlers

What You Need Before You Start

This guide shows you how to block AI crawlers with robots.txt, the plain text file that asks automated bots to avoid parts of your site. It is for site owners who want to opt out of AI training or reduce AI bot traffic. You need access to your site's root folder, a plain text editor and the crawler names that AI companies publish in their documentation.

Steps to Block AI Crawlers

  1. Find the crawler names. Check each AI company's documentation for the user-agent name its crawler uses. For background, read about AI crawlers. Separate training crawlers from AI search crawlers, since you may want to treat them differently.
  2. Open or create your robots.txt file. The file must sit at the root of your domain, such as https://example.com/robots.txt, and it must load without redirects.
  3. Write one rule group per bot. Start each group with a line such as User-agent: GPTBot, then add Disallow: / to block the whole site, or list specific folders to block only part of it.
  4. Keep search engines allowed. If you add a general rule such as User-agent: * with Disallow: /, it also blocks search engines. Give Googlebot its own group with an empty Disallow: line so regular search crawling continues.
  5. Save and test the live file. Open the address in a browser to confirm the new rules appear. Then check your server logs over the following weeks to see whether the named bots still visit.

Common Mistakes and When to Ask for Help

  • Blocking everything by accident. One Disallow: / under the wrong user agent can stop search engines from crawling your pages.
  • Putting the file in a subfolder. Bots look for robots.txt only at the root of the domain.
  • Relying on it for privacy. The file is public and voluntary. Protect drafts and member areas with passwords or server rules.
  • Forgetting upkeep. Bot names change, so review the file and your logs every few months.

Ask your host's support team if you cannot upload the file, if it does not load at the correct address, or if your logs show bots ignoring the rules and you need server-level blocking.

Will blocking AI crawlers remove my site from Google?

Usually not. Blocking AI-specific bot names does not normally affect regular search indexing, because search engines use their own crawler names. The real risk is a broad rule, such as blocking all bots, which can stop search engines too. Check each user agent name before you publish the file.

Does a robots.txt block stop AI companies from using my content?

Not by itself. robots.txt is a voluntary request, and it does not remove pages that were already collected. Some companies say they honor it for future crawling, but enforcement depends on each company's practices and the law where it operates.

Should I block AI search bots as well as training bots?

It depends on your goals. Blocking training crawlers keeps your content out of model training, while blocking AI search crawlers can reduce citations and visibility in AI answers, which is one factor covered in generative engine optimization. Many owners block training bots, allow search bots and review the choice from time to time.