Configure Blogger robots.txt, robots header tags, Search Console, sitemaps and AI crawler access for cleaner indexing and better search visibility.
Blogger handles much of its technical SEO automatically, but a custom-domain site may benefit from more deliberate crawler controls. This is particularly relevant if you run a technical blog, portfolio or knowledge base and want important articles to remain easy for traditional search engines and newer AI-powered discovery systems to find.
The key is not to block as much as possible. It is to distinguish between pages worth crawling and indexing, pages that should stay out of search results, and crawlers that serve different purposes.
Blogger already provides default crawler and sitemap settings, so customisation is not mandatory. In fact, Blogger support guidance generally recommends leaving the defaults alone unless you have a specific reason to change them. If you do need finer control, however, Blogger exposes both a custom robots.txt file and custom robots header tags.
Understand the difference between crawling, indexing and AI access
Before changing anything, it helps to separate three controls that are often confused.
- Crawling determines whether a crawler is permitted to request a URL.
- Indexing determines whether an eligible page may appear in search results.
- AI crawler permissions can govern particular uses of your content, depending on the crawler and provider.
A robots.txt rule is primarily a crawling instruction. It should not be treated as a reliable way to remove a URL from a search index. Google explicitly recommends using noindex when the objective is to prevent a page from appearing in search results.
Configure robots.txt only when you need custom control
In Blogger, the setting is available under Settings > Crawlers and indexing > Enable custom robots.txt.
If your objective is to allow major search and AI crawlers while reducing crawling of Blogger's internal search and label URLs, a configuration can look like this:
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: *
Disallow: /search
Allow: /
Sitemap: https://www.yourdomain.com/sitemap.xmlReplace www.yourdomain.com with the canonical hostname used by your blog.
OAI-SearchBot
OAI-SearchBot is the OpenAI crawler relevant to discovering content for ChatGPT search. OpenAI advises publishers who want their pages to be discoverable, surfaced and cited in ChatGPT search not to block this crawler.
This is distinct from GPTBot. Allowing OAI-SearchBot is the relevant setting for ChatGPT search visibility; it does not mean that you must also permit model-training access.
See OpenAI's guidance for publishers and developers for the current crawler guidance.
GPTBot
GPTBot relates to OpenAI's potential use of publicly available web content for model development and training. It is therefore a separate decision from whether you want content available to ChatGPT search.
If you want ChatGPT search visibility but do not want to permit GPTBot, you can use different rules for the two crawlers:
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /This distinction is useful because AI search discovery and model-training permissions should not be treated as the same thing.
Google-Extended
Google-Extended is also frequently misunderstood. It is not the crawler controlling ordinary Google Search indexing, nor does allowing it improve your Google Search ranking.
Google describes Google-Extended as a robots.txt product token that publishers can use to manage certain uses of crawled content for future Gemini model training and grounding in specified Gemini products. Google states that this setting does not affect inclusion or ranking in Google Search.
Google's AI features within Search, including AI-powered search experiences, continue to use Googlebot and the normal Google Search crawling and indexing controls.
See Google's crawler documentation for Google-Extended and Google's guidance on AI features and websites.
Why block Blogger's /search URLs?
Blogger uses the /search path for internal search results and other listing-style URLs, including label views. These pages can expose substantially the same articles through multiple listing and query combinations.
The rule:
User-agent: *
Disallow: /searchreduces crawler access to URLs beneath that path while leaving normal posts and static pages accessible.
It should not, however, be described as a universal solution to Blogger duplicate content. A robots.txt rule prevents crawling of matching URLs; it does not itself guarantee removal of URLs that a search engine already knows about. Nor does Disallow: /search automatically cover every possible archive URL Blogger can generate.
Use robots header tags to control indexing
Where robots.txt controls crawler access, robots tags are more appropriate when the objective is to prevent low-value listing pages from appearing in search results.
Go to Settings > Crawlers and indexing > Enable custom robots header tags.
If you deliberately choose to override Blogger's defaults, a common structure for a content-focused site is:
- Home page tags:
all - Archive and search page tags:
noindex - Post and page tags:
all
Using noindex on archive and search pages tells compliant search engines not to include those listing pages in their indexes, while individual posts remain eligible for indexing.
Be careful when combining robots.txt blocking with noindex. A crawler generally needs to retrieve a page before it can see its noindex instruction. Google specifically advises against using robots.txt as the mechanism for preventing indexing.
For this reason, crawler and indexing controls should be designed together rather than assuming that more restrictive settings are automatically better.
Set up Google Search Console correctly
Once the blog itself is configured, Google Search Console provides the tools to monitor discovery, indexing and crawler behaviour.
Use a Domain property for a custom domain
For a Blogger site running on its own domain, a Domain property is useful because it covers the domain across protocols and subdomains rather than restricting reporting to a single URL prefix.
For example, a Domain property for example.com can provide consolidated coverage where different host or protocol variations have been encountered.
Google-hosted properties, including Blogger sites, can use either URL-prefix or Domain properties. See Google's Search Console property documentation for the available options.
Submit Blogger's sitemap
Blogger automatically generates a sitemap for published posts. For a custom domain, its usual location is:
https://www.yourdomain.com/sitemap.xmlYou can submit this URL through the Sitemaps report in Search Console. Blogger also generates sitemap infrastructure automatically, so there is normally no reason to build a hand-written XML sitemap.
A sitemap helps search engines discover URLs, but submission does not guarantee that every URL will be crawled, indexed or ranked.
Keep the site on HTTPS
Under Blogger's HTTPS settings, enable HTTPS availability and redirect HTTP traffic to HTTPS where those controls are available for your configuration.
This gives visitors a secure connection and helps keep the canonical version of the site consistent. Google recommends HTTPS and generally prefers the HTTPS version when equivalent HTTP and HTTPS pages exist.
Do not overestimate crawl budget on a small Blogger site
Reducing unnecessary crawlable URLs is sensible technical housekeeping, but claims about dramatically improving a site's "crawl budget" should be treated cautiously.
For a modest Blogger site, crawl-budget limitations are unlikely to be the primary SEO constraint. Cleaner crawler directives are still useful because they reduce unnecessary URL discovery and make your preferred content structure clearer, but they do not guarantee higher rankings or faster indexing.
The more important objective is to make valuable posts accessible, return successful HTTP responses, provide indexable content and avoid contradictory crawler directives.
A practical configuration
For a technical Blogger site, the resulting strategy is straightforward: keep full articles and static pages accessible, limit unnecessary search and listing URLs where appropriate, use noindex for pages you genuinely do not want in search results, and make separate decisions about AI search and AI training access.
If you want maximum OpenAI access, you can allow both OAI-SearchBot and GPTBot. If you want ChatGPT search discovery without opting into potential training use, allow OAI-SearchBot and block GPTBot.
For Google, keep Googlebot able to crawl the content you want in Search. Treat Google-Extended separately because it does not control normal Google Search indexing or ranking.
Finally, verify the results rather than assuming the configuration is working. Search Console's URL Inspection, Page Indexing, robots.txt and Sitemaps reports can show how Google is actually interpreting the site.
The aim is not to force every crawler towards every URL. It is to give each system clear, technically consistent instructions about the content you want discovered, indexed and used.
COMMENTS