Here is a scenario that plays out more often than people realize. A company spends months building out exactly the kind of content we have talked about in this series \u2013 deep, well-structured, genuinely authoritative articles that answer real questions thoroughly. They do the topic mapping. They build the topic clusters with real depth. They even land some genuine earned media coverage.<\/p>\n
And then none of it shows up in AI answers. Not because the content is not good enough. Because the AI systems that would retrieve and cite it cannot actually get to it in the first place.<\/p>\n
This is the unglamorous, deeply important part of GEO that nobody enjoys talking about, because it does not feel like strategy. It feels like plumbing. But if the pipes are broken, it does not matter how good the water is. This article is about making sure your pipes work \u2013 making sure that when an AI crawler comes looking for your content, it can find it, read it, and understand it without friction.<\/p>\n
I also want to deal head-on with some of the noise in this space, because there has been a lot of advice floating around over the past year that ranges from genuinely useful to actively distracting, and I think it is worth being clear about which is which.<\/p>\n
Step One: Make Sure the Crawlers Can Actually Get In<\/p>\n
This sounds almost too basic to mention, and yet it is the single most common technical issue holding brands back from AI visibility. Many sites \u2013 sometimes deliberately, often by accident \u2013 are blocking the very crawlers that would bring their content into AI systems.<\/p>\n
There are now several AI crawlers you need to be aware of, each associated with a different AI system. OpenAI\u2019s GPTBot, Anthropic\u2019s ClaudeBot, and PerplexityBot are among the most active, with GPTBot alone generating hundreds of millions of monthly requests across the web according to crawler traffic analyses. Google\u2019s AI Overviews largely rely on Google\u2019s existing crawling infrastructure, while other systems use their own dedicated bots.<\/p>\n
Your robots.txt file is the first thing to check. It is a simple text file, but it has outsized importance because it is the first thing many of these crawlers check before doing anything else. If your robots.txt disallows a crawler \u2013 whether through an old blanket rule that predates AI crawlers, or a more recent rule added out of caution about AI training \u2013 that crawler will not retrieve your content, full stop.<\/p>\n
The decision about which AI crawlers to allow is not purely technical. There is a legitimate business question buried in here about whether you want your content used to train AI models versus whether you want it retrieved for real-time answers, and some site owners have understandably mixed feelings about the former while still wanting the latter. Some crawlers are starting to distinguish between these purposes, and there is ongoing work at the standards level \u2013 through bodies like the IETF \u2013 to build clearer distinctions into the robots.txt protocol itself. For now, the practical reality is that if visibility in AI answers matters to your business, blocking the major AI crawlers outright works directly against that goal.<\/p>\n
Beyond robots.txt, check whether your CDN or security configuration is silently rejecting AI bot traffic. This happens more often than people expect, particularly with services like Cloudflare, where bot-protection settings configured for security reasons can inadvertently block legitimate AI crawlers along with the malicious traffic they were designed to stop.<\/p>\n
Step Two: If It Is Hidden Behind JavaScript, Many AI Systems Cannot See It<\/p>\n
This is the issue that surprises people most, because it is invisible if you are looking at your site the normal way \u2013 through a browser, where JavaScript renders everything beautifully and you see exactly the page you intended.<\/p>\n
Many AI crawlers do not render JavaScript the way a browser does. They request the raw HTML of a page, and if your important content \u2013 your article text, your key data, your product information \u2013 is loaded dynamically through JavaScript after the initial page load, a crawler that only reads the raw HTML may see an essentially empty page.<\/p>\n
The practical test is straightforward: view your page\u2019s source HTML directly, before any JavaScript executes, and see what is actually there. If your core content is present in that raw HTML, you are in reasonable shape. If the raw HTML is mostly empty divs and placeholder elements that get filled in by JavaScript afterward, that is a real problem for AI crawlability \u2013 and for that matter, it can be a problem for traditional search crawlability too, though Google has gotten better at handling this over the years than many AI crawlers currently are.<\/p>\n
The fix for this is server-side rendering, or one of its variants \u2013 static site generation, where pages are pre-built as complete HTML, or incremental static regeneration, which combines pre-building with periodic updates. If a full migration to server-side rendering is not realistic for your situation, dynamic rendering tools that serve a pre-rendered version of your page specifically to bots can bridge the gap, though the cleanest long-term solution is making sure your content exists in the HTML from the start.<\/p>\n
Step Three: Schema Markup \u2013 What Actually Helps and What Is Overkill<\/p>\n
Schema markup, also called structured data, is a way of explicitly labeling the parts of your content so that machines do not have to guess what something is. This article is an Article. This is its author. This is a Frequently Asked Questions section, and here are the specific question-and-answer pairs within it.<\/p>\n
There is reasonable evidence that this helps. Some implementation guides report that sites with comprehensive structured data across their key page types see meaningfully more appearances in AI Overview results compared to sites without it. The logic makes sense: when you remove ambiguity about what a piece of content is, you make it easier for any system \u2013 search engine or AI \u2013 to use that content appropriately.<\/p>\n
That said, I want to flag something important here, because the GEO advice ecosystem has occasionally overstated this. Google\u2019s own 2026 guidance on generative AI search optimization has been fairly direct that excessive structured data is not a requirement for appearing in AI Overviews or AI Mode, and that the foundation remains the same things that have always mattered for good SEO \u2013 genuinely useful, well-written content that real people would want to read.<\/p>\n
My honest read on this is that schema markup is a genuinely useful, relatively low-cost technical investment \u2013 particularly Article schema for your content, FAQPage schema for genuine FAQ sections, and Organization schema that clearly establishes who you are. It is not, however, a substitute for the content quality and structural clarity we discussed in earlier articles, and it is not going to rescue thin or poorly organized content. Think of it as a helpful accelerant on top of a fire that is already burning, not a fire on its own.<\/p>\n
The llms.txt Question: Useful Emerging Standard or Overhyped Distraction?<\/p>\n
If you have spent any time researching technical GEO over the past year, you have almost certainly encountered llms.txt \u2013 a proposed standard, originally put forward by Jeremy Howard in 2024, that works something like a robots.txt file but is designed to give AI systems a clean, curated overview of a site\u2019s most important content and structure, often in markdown format.<\/p>\n
The idea has gained real traction, particularly among developer-tool companies. Platforms like Stripe, Vercel, and various documentation providers have been experimenting with llms.txt files, and there is a reasonable case for why it works well in that specific context: when an AI coding assistant is trying to figure out how to help a developer integrate with an API, having a clean, curated map of the documentation is genuinely useful, and the assistant can follow that map directly to the most relevant reference material.<\/p>\n
Here is where I want to be careful, though. Google\u2019s own 2026 guidance explicitly states that you do not need to create llms.txt files, AI-specific content rewrites, or special markdown versions of your content to appear in Google\u2019s generative AI search features. That is a fairly direct statement from the company that, for many sites, controls the AI search experience most of their audience will encounter.<\/p>\n
My honest take, reconciling these different signals: llms.txt appears to have real, demonstrated value for developer-facing sites and technical documentation, where AI coding assistants are a meaningful part of your audience and a curated map of your docs genuinely helps those assistants do their job. For general content sites \u2013 blogs, marketing sites, informational resources \u2013 the evidence that llms.txt meaningfully affects whether ChatGPT or Perplexity cites your content is much thinner, and Google has explicitly said it does not factor into their systems.<\/p>\n
If you run a developer-focused product with substantial documentation, implementing an llms.txt file is a low-cost experiment that fits a pattern of genuine, demonstrated value. If you run a general content site and you are choosing between spending a day on llms.txt versus spending that same day improving the structure and depth of one of your cluster articles, the article is very likely the better investment based on what we currently know.<\/p>\n
Site Speed, Mobile Experience, and HTTPS \u2013 The Boring Fundamentals That Still Matter<\/p>\n
I am going to keep this section relatively short, not because these things do not matter, but because if you have done any serious SEO work in the past several years, you have likely already addressed most of this. The point is simply that none of it has become less important in the AI era \u2013 if anything, it has become slightly more important.<\/p>\n
Page speed affects how thoroughly and how frequently AI crawlers can access your site within whatever crawl budget they allocate to you. A slow site gets crawled less completely. HTTPS is treated as a baseline trust signal by AI platforms, the same way it has been for search engines for years \u2013 if your site is still running on unencrypted HTTP in 2026, that is a problem well beyond GEO. And mobile experience matters because an increasing share of the queries that eventually surface AI-generated answers originate from mobile devices, and AI systems are increasingly factoring mobile-friendliness into how they evaluate sources.<\/p>\n
None of this is exciting. All of it is foundational. If your technical SEO has been neglected, technical GEO is not a separate project \u2013 it is largely the same project, with a slightly expanded set of bots to think about.<\/p>\n
Sitemaps, Internal Linking, and Helping Crawlers Find Your Best Content<\/p>\n
A well-maintained XML sitemap, with accurate last-modified dates, helps any crawler \u2013 AI or traditional \u2013 understand what exists on your site and what has recently changed. This becomes particularly relevant given how much we have discussed the importance of freshness for AI retrieval. If your sitemap accurately reflects when content was substantively updated, you are giving crawlers a direct signal about where the freshness-relevant changes are.<\/p>\n
Internal linking does double duty here. We talked in Article 4 about how internal linking helps establish topical authority by showing AI systems that your content exists within a coherent body of work. It also has a more basic technical function: it is literally how crawlers discover pages, especially pages that are not prominently featured in your navigation. A genuinely valuable piece of content that exists as an orphan page \u2013 not linked from anywhere else on your site \u2013 may simply never be found, no matter how good it is.<\/p>\n
Every site has a crawl budget, meaning there is a practical limit to how much of your site any given crawler will explore in a given period. For most small to medium sites this is rarely the binding constraint, but for larger sites \u2013 particularly e-commerce sites with thousands of product pages \u2013 making sure your most valuable content is not buried many clicks deep in your site architecture, and is referenced from your sitemap with appropriate priority, genuinely affects how completely AI crawlers can explore what you have built.<\/p>\n
A Practical Audit You Can Actually Run This Week<\/p>\n
Given everything above, here is a realistic, prioritized sequence rather than an overwhelming list of everything you could theoretically do.<\/p>\n
Start with robots.txt. Open it, and check explicitly for GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and any other AI crawler names you can identify. Confirm none of them are blocked unless you have made a deliberate decision to block them. This takes five minutes and is the highest-leverage check on this entire list.<\/p>\n
Next, check your CDN and security settings for bot-management rules that might be catching AI crawlers in a net designed for malicious traffic. If you use a service like Cloudflare, review your bot-fight or bot-management configuration specifically.<\/p>\n
Then, view the raw HTML source of your two or three most important pages \u2013 the ones you most want AI systems to cite \u2013 and confirm your core content is actually present in that raw HTML, not loaded in afterward by JavaScript.<\/p>\n
After that, do a spot-check on schema markup. You do not need exhaustive coverage immediately. Prioritize Article schema on your key content pieces, FAQPage schema on pages with genuine FAQ sections, and Organization schema establishing who you are. Google\u2019s Rich Results Test is a free, simple way to verify your markup is implemented correctly.<\/p>\n
If you run a developer-facing product with substantial technical documentation, evaluate whether an llms.txt file makes sense for your specific situation. For most other sites, this can wait.<\/p>\n
Finally, confirm your XML sitemap is current, accurately reflects recent updates, and includes your most important content. If you have any genuinely valuable pages that are not linked from anywhere else on your site, fix that.<\/p>\n
None of this is glamorous. All of it removes friction between the content you have worked hard to create and the systems that might otherwise cite it.<\/p>\n
The Bigger Picture: Technical Work Is the Floor, Not the Strategy<\/p>\n
I want to close this article with the same caution I raised at the start, because I think it matters. There has been a wave of GEO advice over the past year that frames technical implementation \u2013 schema markup, llms.txt, crawler configuration \u2013 as the core of the discipline. Some of this comes from tool vendors who have technical solutions to sell, which is not inherently a problem, but it does shape the emphasis of the advice in predictable ways.<\/p>\n
The reality, based on everything we have covered across this series, is that technical GEO is necessary but not sufficient. It removes obstacles. It does not create value on its own. A perfectly configured robots.txt file, comprehensive schema markup, and a pristine llms.txt file will not make thin, generic content suddenly become citation-worthy. What they will do is make sure that when you have built something genuinely good \u2013 deep, well-structured, authoritative content backed by real third-party credibility \u2013 there is nothing standing between that content and the AI systems that might cite it.<\/p>\n