A Search Crawler Read Our llms-full.txt. Here Is What We Learned.

For the last few days, we have been checking whether anyone accesses Klairia’s llms.txt or llms-full.txt.

Most days, the answer was simple: no.

Then we saw a request to llms-full.txt from Googlebot.

It was only one read. It was not ChatGPT, Claude, Perplexity, Gemini, or another AI assistant. It was a search crawler.

But the signal still matters.

A machine-readable file that describes your company is only useful if crawlers can find it, request it, and process it. This event confirmed three things at once: the file is publicly reachable, our tracking is working, and at least one major crawler requested the AI-readable context we publish.

That does not prove llms.txt is influencing AI answers. It does prove the measurement loop is working.

What happened

Klairia tracks access to two public files:

• llms.txt
• llms-full.txt

These files are designed to give crawlers and AI systems a clearer, more structured summary of the site, product, and important public pages.

In our analytics, we track requests to those files with fields such as:

• the requested path
• the bot name
• the bot operator
• the bot category
• the raw user agent

The first notable read we saw was for llms-full.txt, from Googlebot. We did not see a matching read for the shorter llms.txt file in the same window.

The likely explanation is simple: both files are referenced in our sitemap, so Googlebot may have requested the file through ordinary sitemap discovery. That is useful to know, but it is not evidence that Google treats llms-full.txt as a special format.

That is a small signal, but a useful one.

Why this matters

It is tempting to overread a single crawler event. We should not.

One Googlebot request does not mean Google is using llms-full.txt for ranking. It does not mean AI assistants are reading the file. It does not mean llms.txt has become a standard part of the search or answer-engine pipeline. It may simply mean that a crawlable URL listed in the sitemap was crawled.

But it does mean something practical: these files are not just theoretical assets sitting on a server.

They can be crawled. They can be measured. Their discovery can be monitored over time.

That is important because GEO is still full of vague advice. Teams are told to “optimize for AI” without knowing whether AI systems, search crawlers, or answer engines are even seeing the assets they create.

Before you can improve visibility, you need observability.

llms.txt is not a magic ranking file

The useful way to think about llms.txt is not as a shortcut. It is a structured context layer.

A good llms.txt or llms-full.txt file can help summarize:

• what your company does
• which pages matter most
• what your product is for
• which use cases and categories you want crawlers to understand
• where deeper documentation or source material lives

That can be useful to systems that retrieve, summarize, or classify public web content.

But the file does not replace the fundamentals. Your public pages still need to be crawlable. Your positioning still needs to be clear. Your content still needs to be supported by external evidence. Your product still needs to be described consistently across the web.

Machine-readable context helps most when it reflects a strong underlying site.

The bigger lesson: measure crawler behavior directly

The more interesting part of this finding is not that a crawler read the file. It is that we could see it.

Most teams publish technical SEO or GEO assets and then stop at deployment:

• sitemap submitted
• robots.txt updated
• schema added
• llms.txt published
• blog post shipped

That creates a false sense of completion. The real question is what happens after shipping.

Are crawlers requesting the file? Which ones? How often? Are they reading the short version, the full version, or neither? Are AI-specific bots showing up separately from search crawlers? Do requests increase after publishing new content or adding internal links?

Those are measurable questions.

For llms.txt, the minimum useful tracking setup is simple:

1. Log requests for /llms.txt and /llms-full.txt.
2. Capture the raw user agent.
3. Classify known bots by name, operator, and category.
4. Send the event into analytics.
5. Review reads by file and crawler over time.

That turns “we published an AI-readable file” into “we know who is reading it.”

What we will watch next

This first crawler read gives us a baseline, not a conclusion.

The next signals worth watching are:

• whether the crawler returns regularly
• whether /llms.txt gets read in addition to /llms-full.txt
• whether AI-specific crawlers appear
• whether reads correlate with new content, sitemap updates, or internal links
• whether crawler access eventually lines up with changes in AI visibility or citations

That last point matters most. Crawl events are an input signal. GEO outcomes still need to be measured in the answers themselves: whether a brand is mentioned, cited, described accurately, and recommended in the right contexts.

Crawler telemetry does not replace LLM visibility measurement. It supports it.

The practical takeaway

If you are experimenting with llms.txt, do not treat publication as the finish line.

Publish the file, but also instrument it. Track requests. Classify bots. Keep the raw user agents. Separate search crawlers from AI crawlers and AI assistants. Watch the pattern over weeks, not hours.

A single crawler read is a small signal. But small signals are how early measurement systems start.

The point is not to declare that llms.txt “works.” The point is to stop guessing.

At Klairia, we care about that distinction because GEO is becoming a measurement problem as much as a content problem. You need to know not only what you published, but whether the systems that shape discovery can actually see it.

This time, one of them did.

FAQ

Does this mean llms.txt affects rankings?

No. A crawler request only proves the file was requested. If the file is listed in a sitemap, the crawl may come from ordinary sitemap discovery. It does not prove that the file affects rankings, citations, or AI-generated answers.

Why track llms.txt access at all?

Because publishing the file is only the first step. Tracking access tells you whether crawlers and AI systems are actually seeing the machine-readable context you provide.

Is one crawler request enough to draw conclusions?

No. One request is a baseline, not a trend. The useful signal comes from watching which crawlers return, how often they read the files, and whether that correlates with visibility changes over time.

Should teams create an llms.txt file now?

Yes, if they can keep it accurate. It is a low-friction way to provide structured context, but it should support strong public pages, not replace them.

What should teams measure?

At minimum: reads by file, raw user agents, bot name, bot operator, bot category, and whether the reader is a search crawler, AI crawler, or AI assistant.

What is the main lesson?

Do not stop at publishing AI-readable assets. Instrument them, monitor them, and connect crawler behavior to actual AI visibility outcomes.