Meta's Training Crawler Requested Our llms.txt

On August 19, 2026, Klairia’s server recorded a request to /llms.txt from a user agent matching meta-externalagent/1.1 — the string Meta documents for its Meta-ExternalAgent crawler.

One caveat up front: this identification is based on the reported user agent only. We did not independently verify the request’s source IP, and user agent strings can be spoofed. Everything below assumes the request is what it claims to be.

With that stated, this event is different from the ones we wrote about before. According to Meta’s crawler documentation, Meta-ExternalAgent “crawls the web for use cases such as training foundation AI models or improving products by indexing content directly.”

That makes it the first crawler in our logs whose stated purpose includes model training.

Why the crawler name matters

Meta documents several crawlers with distinct purposes:

• Meta-ExternalAgent crawls content for foundation-model training and product improvement.
• Meta-WebIndexer supports the relevance and accuracy of Meta AI search results.
• Meta-ExternalFetcher retrieves individual links at a user’s request.
• FacebookExternalHit collects titles, descriptions, and images for shared-link previews.

The user agent in our log matched Meta-ExternalAgent, not the search, user-triggered, or link-preview crawlers. If we had only logged “a Meta bot,” that distinction would be invisible — and it is the most informative part of the event.

What was requested

The request was for the shorter /llms.txt, the compact map of Klairia’s public site, app entry points, and API surfaces. We did not record a matching request to /llms-full.txt in the same window.

Both files are public and listed in Klairia’s sitemap, so ordinary sitemap discovery remains a sufficient explanation for how the crawler found the URL. The request does not show that Meta recognizes llms.txt as a formal standard.

Three operators, three stated purposes

This is the third crawler operator we have observed requesting Klairia’s llms files, and each one carries a different documented purpose:

• Googlebot, a conventional search crawler.
• GPTBot, which OpenAI documents as a crawler for improving its models.
• Now a user agent matching Meta-ExternalAgent, which Meta ties explicitly to foundation-model training and product indexing.

The pattern is becoming more specific: these files are not only reachable by search crawlers. Crawlers operated by multiple AI companies have now requested them.

What the pattern still cannot show is what happens after retrieval. This request does not tell us whether Meta indexed the content, retained it, used it in training, or surfaced it in any product output. Those remain separate questions with separate evidence requirements.

A final thought

“AI crawler traffic” is too broad to be a useful measurement on its own.

Meta alone documents distinct crawlers for model development, AI search, user-triggered retrieval, advertising, and social previews. An aggregate bot count would collapse all of that into one number. The crawler name, its stated purpose, the requested file, and repeat behavior are what make the log entry informative.

Today’s evidence is narrow but clear: a request matching Meta-ExternalAgent reached Klairia’s /llms.txt. For the first time, the stated purpose behind a request to that file includes training AI models. What happens after retrieval remains outside what a server log can prove.

FAQ

Does this mean Meta will train Llama on Klairia’s llms.txt?

No. Meta documents Meta-ExternalAgent as a crawler used for purposes including foundation-model training, but a single request does not show the content was retained or used.

Was the request verified as coming from Meta?

No. The classification is based on the reported user agent string, which can be spoofed. We note this limitation explicitly rather than overstating the evidence.

Did the crawler also read llms-full.txt?

Not in the same window. Only /llms.txt was requested in this event.

How is this different from the earlier Googlebot and GPTBot events?

The operator and the documented purpose. Googlebot is a search crawler and GPTBot is OpenAI’s model-improvement crawler. Meta-ExternalAgent is the first crawler in our logs whose documentation explicitly cites training foundation AI models.