Making Your Website Agent-Friendly with llms.txt and CloudFront

The Problem

Your website looks great to humans. But AI agents see HTML soup — CSS, JavaScript, navigation bars, cookie banners. When an agent tries to read your page, it has to parse through all that noise to find the actual content. Often it gets confused or misses key information.

What if agents could get a clean, structured version of your content — just the text, commands, and links, formatted as Markdown?

The llms.txt Standard

The llms.txt standard is like robots.txt for AI agents. You place a file at /llms.txt on your domain containing a Markdown summary of your site. Agents (or humans prompting agents) can fetch it directly:

curl https://yourdomain.com/llms.txt

Major companies like Anthropic (Claude docs), Cloudflare, and Stripe already have one. The standard is still young — most AI crawlers don't auto-check for it yet — but it costs nothing to set up, and the adoption curve is heading up.

Going Further: Content Negotiation

Here's the real trick. Instead of relying on agents knowing about /llms.txt, you can make any URL on your site return Markdown when an agent asks for it. The mechanism is HTTP content negotiation via the Accept header.

When an agent sends a request like this:

curl -H "Accept: text/markdown" https://yourdomain.com/

...your CDN intercepts the request and rewrites the URI to /llms.txt. The agent gets clean Markdown. Everyone else gets the normal HTML page.

The CloudFront Function

If your site runs on S3 + CloudFront (like this one), here's the complete function:

// CloudFront Function — Viewer Request event
function handler(event) {
  var request = event.request;
  var accept = request.headers['accept']
    ? request.headers['accept'].value : '';

  if (accept.indexOf('text/markdown') !== -1 ||
      accept.indexOf('text/plain') !== -1) {
    request.uri = '/llms.txt';
  }
  return request;
}

That's it. 10 lines. It checks the Accept header on every incoming request: if the client asks for text/markdown or text/plain, the request gets rewritten to /llms.txt. Otherwise it passes through unchanged.

Setup in AWS Console

1. Upload llms.txt to your S3 bucket (the .txt extension sets the right Content-Type automatically).
2. CloudFront → Functions → Create function → paste the code above → name it e.g. accept-markdown-routing → Publish.
3. Your Distribution → Behaviors → Default (*) → Function associations → Viewer request → select the function.
4. Invalidate the cache (/*) so visitors pick up the new behaviour.

What Goes in llms.txt?

Whatever an agent needs to be useful with your content. For this site, the llms.txt contains:

1. Every setup command from the getting-started guide (copy-paste ready)
2. All Kiro CLI slash commands
3. MCP server and Skills configuration steps
4. SOP structure, example SOPs, and the conversational creation flow
5. Every relevant link and resource

Think of it as the "API docs" for your website — structured, complete, and optimised for machine consumption rather than visual design.

Testing It

Once deployed, you can verify both paths work:

# Direct path — always works
curl https://yourdomain.com/llms.txt

# Content negotiation — any URL works
curl -H "Accept: text/markdown" https://yourdomain.com/
curl -H "Accept: text/markdown" https://yourdomain.com/any/page

Both should return the same Markdown content.

Try It on This Site

curl -H "Accept: text/markdown" https://goagentic.ch/

Or tell your AI agent:

> Read https://goagentic.ch/llms.txt and help me set up
  Kiro CLI with MCP servers and SOPs.

AWS CLOUDFRONT AI AGENTS LLMS.TXT