A server log can tell you that an AI client knocked on the door. It cannot tell you which sentence the client took home. The mistake begins when crawler activity is treated as proof of AI visibility when it is only proof of a request.
The useful question is not whether an AI-named bot appeared in the log. It is what decision that observation can support without pretending the log knows more than it does.
A fetch is not a reading trace
Request logs are good at recording request facts. Cloudflare's HTTP request log fields include details such as host, method, path, status and user agent. Those fields can help separate crawler families, compare request volume and see which URLs were fetched. They do not record which paragraph, heading or claim a client consumed.
Even the URL has limits. Under RFC 3986, a fragment identifier is separated before a resource is dereferenced and handled by the client. If a system uses /guide#pricing, an ordinary origin request does not expose #pricing as evidence that the pricing section was read. The server sees a request for the page, not a trail through the page.
That makes crawler logs useful for distribution questions: Are AI clients reaching the site? Which clients are requesting which pages? Did request behaviour change after a technical change? The same logs cannot answer comprehension questions: Which argument mattered? What was quoted? Did the page influence an answer?
Treating those two categories as interchangeable turns an observable event into an invented outcome.
Markdown matters only when it is requested
The same discipline applies to AI-facing Markdown. Cloudflare's Markdown for Agents is delivered through content negotiation: the client must include text/markdown in its Accept header. Enabling the feature creates an available representation. It does not prove that a crawler asked for it or used it.
A five-domain field test reported that the monitored frontier crawlers did not negotiate for Markdown during the observation window. That is a useful warning against claiming an automatic visibility lift, but it is not a universal verdict on every agent. A client that does request Markdown could still benefit from receiving it. The header decides whether the feature entered that exchange at all.
Google draws an even narrower boundary for its own search product. Its guidance for generative AI features in Google Search says special AI files, special markup, llms.txt and Markdown are not required for visibility or rankings. It also rejects the idea that content must be broken into tiny chunks for AI. That settles what Google says about Google Search; it does not describe every independent agent on the web.
The tempting counterclaim is that capable clients already parse HTML, so a Markdown version is always pointless. The evidence does not support that absolute either. The responsible conclusion is smaller: format work deserves priority only when the target client requests the format and the resulting behaviour matters to the business.
Measure the decision, not the bot
Start with the request data, but keep the claims proportional to it. Segment crawler requests by user agent. Inspect the Accept header before crediting Markdown. Compare page-level fetching before and after a change. Keep referrals, conversions and citations as separate outcomes that need their own evidence.
This also changes how a dashboard should be labelled. AI crawler requests is an honest metric. AI visibility is not, unless the measurement reaches beyond the server request and shows that the brand actually appeared in an answer or influenced a useful visit.
Crawler activity can justify technical investigation. It cannot, by itself, justify a content strategy, prove a ranking gain or identify the paragraph an agent valued. The log knows who knocked. Make it earn every claim beyond that.