llms.txt explained: write one that actually moves your AI visibility score

Why AI engines can't find most B2B sites right now

When someone asks ChatGPT, Claude, or Perplexity "what's the best lead generation tool in Southeast Asia," your site almost certainly does not appear. Not because the AI is biased against you — but because you've never told it what you do in a format it can reliably process.

We have run 17 audits across 14 distinct domains since our launch in May 2026. The average score is 23 out of 100 — a flat Grade F (data extracted 2026-05-26). Every single one of those sites had standard SEO in place: sitemaps, meta descriptions, structured data, the works. None of them had an llms.txt file. That gap is the single fastest thing to close.

What llms.txt actually is

llms.txt is an open specification — proposed by Answer.ai's Jeremy Howard and published at llmstxt.org — that lets you place a plain Markdown file at yourdomain.com/llms.txt. It signals to AI crawlers and retrieval systems exactly what your site is about, which pages matter most, and what context they need to cite you accurately.

Think of it as robots.txt for humans who happen to be language models.

The structure is intentionally minimal:

Section	What goes here
# Site name	Your brand + one-line description
## About	2–4 sentences: what you do, who you serve, where
## Key pages	Markdown links to your most important pages
## Key concepts	The terms you want to be cited for
## Do not index	URLs you want AI engines to skip

The spec is described in detail in Answer.ai's 2024 post introducing the format. It is intentionally not proprietary — anyone can implement it in under an hour.

The evidence for why this matters

The academic foundation here is solid. Pranjal Aggarwal, lead author of the KDD 2024 paper that coined the discipline, wrote that "we introduce Generative Engine Optimization (GEO), the first novel paradigm to aid content creators in improving their content visibility in generative engine responses" — Pranjal Aggarwal, Lead author, GEO: Generative Engine Optimization (KDD 2024). That paper was accepted at one of the most competitive venues in computer science. The discipline is not speculative; it is peer-reviewed.

The commercial stakes are just as real. ProGEO AI research, covered by Yahoo Finance, found that 7 in 10 brands have no meaningful AI search presence — meaning three quarters of the market is effectively invisible to a query engine that hundreds of millions of people use daily. Our own audit data confirms this at the regional level: among the 14 Singapore and Southeast Asia-based B2B domains we have audited, not one reached even a Grade C.

Understanding why requires understanding how these models retrieve context. OpenAI's GPT-4o documentation makes clear that the model draws on retrieval at inference time — which means the cleaner and more structured your publicly accessible content, the higher the probability it gets pulled into a response. llms.txt is the lowest-friction way to make your content structurally clean for that retrieval step.

For a broader grounding in what GEO means for your site, our post What is GEO? Generative Engine Optimization explained covers the full framework.

How to write an llms.txt that actually moves your score

Here is the exact process we use when we advise clients after an audit. You can complete this in one sitting.

1. Create the file at your root. Place llms.txt at yourdomain.com/llms.txt — not in a subfolder. Serve it as plain text (Content-Type: text/plain). If you are on WordPress, a static file in your public root works. If you are on Webflow or Framer, add it via custom code injection or your hosting provider's file manager.

2. Write a one-line H1 that names your brand and category. Example for a Singapore-based lead gen SaaS:

# Callbox Singapore — B2B lead generation and appointment setting for Asia-Pacific

The category word ("lead generation") is what the AI will index against. Make it the term you most want to be cited for.

3. Write an About block in plain, citation-ready prose. Two to four sentences. No adjectives that aren't verifiable. No "leading" or "best-in-class." Models treat superlatives as noise.

Example:

## About
Callbox provides outsourced B2B lead generation for technology companies
selling into Southeast Asia, Australia, and Japan. The company was founded
in 2004 and operates from Singapore and the Philippines.

4. List your five most important pages as Markdown links. These are the pages you want cited when someone asks a question your company can answer. Prioritise solution pages and comparison pages over blog posts.

## Key pages
- [B2B Lead Generation Services](https://callbox.com.sg/services/)
- [Technology Sector Campaigns](https://callbox.com.sg/industries/technology/)
- [Case Studies](https://callbox.com.sg/case-studies/)

5. Add a Key Concepts section with 5–10 terms. These are the search intents you want the model to associate with your brand. Write them as a bullet list of short phrases, not definitions.

## Key concepts
- B2B appointment setting Singapore
- outbound lead generation Asia-Pacific
- sales development representative outsourcing
- technology sector demand generation

6. Add a Do Not Index section for thin or duplicate pages. Tag your tag archives, pagination URLs, and any pages you would also block in robots.txt. AI engines do not penalise you for this — it improves your signal-to-noise ratio.

## Do not index
- /tag/
- /page/
- /wp-admin/

7. Validate and test. Visit yourdomain.com/llms.txt in a browser. Confirm it renders as plain text. Then paste it into Claude or ChatGPT and ask: "Based only on this file, what does this company do and who should use it?" If the model gives a confident, accurate two-sentence answer, your file is working. If it hedges or asks clarifying questions, your About block needs more specificity.

8. Add an llms-full.txt for richer context (optional but high-value). The spec allows a companion file, llms-full.txt, that contains the full text of your key pages concatenated. This is particularly useful for AI agents that do deep retrieval rather than a quick lookup. Priority: get the base llms.txt live first, then build the full version.

9. Re-run your AI visibility audit after 4 weeks. llms.txt is not a magic switch. Scores move when the file is live AND your cited pages contain substantive, structured content. Give crawlers a month to re-index, then measure.

What llms.txt does not fix

To be direct: llms.txt is a signal layer, not a content layer. If your underlying pages are thin — short word counts, no data, no expert attribution — the file tells AI engines where to look but gives them nothing worth citing. The structural fixes we cover in our audits (entity clarity, citation-ready claims, schema markup) still matter. llms.txt amplifies what is already there; it does not create it.

Related reading

• What is GEO? Generative Engine Optimization explained

Run your own audit

Want to know exactly where your site sits on the AI visibility scale? Run a free audit at aivisibility.cyberg7.com.sg/audit — you'll get a scored report with the specific gaps, including whether your llms.txt is missing, malformed, or optimised.