Getting cited by AI search: a practical checklist for developers

By Published 6 min read

Short answer

Nobody can guarantee that an AI assistant will cite your site; what you control is whether it can fetch, read and understand your pages. Let the right crawlers in and check the CDN or firewall above robots.txt, put your content in the HTML rather than behind JavaScript, write answer-shaped content, add truthful structured data, keep dates honest and publish a clean sitemap. Add an llms.txt if you like, though no major AI provider has confirmed that it reads one, then earn mentions elsewhere and measure by asking the assistants.

Nobody can promise that an AI assistant will cite your site. The systems are opaque and change often. What you can control is whether they are able to fetch your pages, read them and understand what they say. This checklist covers that mechanical part.

1. Let the right crawlers in

AI companies generally run different bots for different jobs: some collect training data, some fetch pages to answer a user's question live. Names include GPTBot, OAI-SearchBot and ChatGPT-User from OpenAI, ClaudeBot from Anthropic, PerplexityBot from Perplexity, and the Google-Extended robots token for Google's generative AI use. Vendors rename and add bots, so confirm against each one's current documentation.

Many sites deliberately block training bots and still allow the bots that serve live answers. That is a legitimate choice. What is not a choice is blocking an answer bot by accident, so read your robots.txt as if you were each bot.

2. Check the layer above robots.txt

A CDN or web application firewall can block bots that robots.txt allows. Bot-protection features sometimes challenge or reject unfamiliar user agents. If your robots.txt is open but a bot gets a 403 or a challenge page, the bot sees nothing. Test with a request that sends the bot's user agent and see what comes back.

3. Put the content in the HTML

The crawler documentation from OpenAI, Anthropic and Perplexity (linked above) lists their bots without saying whether they execute JavaScript, so don't assume they do. Google's crawlers can render JavaScript, but rendering is a separate, queued step. If your page's real content appears only after hydration, a crawler that does not render sees an empty shell. Use server rendering or static generation for anything you want read, and confirm by viewing the raw source.

4. Write answer-shaped content

  • Lead with the answer. Put a one-sentence definition or direct answer near the top of the page.
  • Make passages stand alone. An answer engine may quote one paragraph with no surrounding context, so avoid "as mentioned above".
  • Use descriptive headings, ideally phrased the way people ask the question.
  • Be specific and sourced: names, versions, limits, dates. Vague claims give an assistant nothing to cite.
  • Say what you do not do. Clear boundaries make a page more trustworthy to quote.

5. Add structured data, truthfully

JSON-LD helps systems identify what a page is. Use Organization for your company, Article for posts with a real datePublished, BreadcrumbList for hierarchy, and FAQPage only when the page contains those questions and answers visibly. Do not mark up content that is not on the page (Google's structured data policies say the same).

6. Keep dates honest

A visible published and updated date, matching your structured data, helps both readers and systems judge freshness. Do not bump the date without changing the content.

7. Make a clean sitemap

List the URLs you want discovered, with real last-modified dates, and exclude redirects, noindex pages and anything blocked in robots.txt. A sitemap is also where many crawlers start.

8. Consider an llms.txt, with realistic expectations

llms.txt is a proposed convention for a short, LLM-friendly index of your key pages at /llms.txt. Adoption is growing, but as far as we know no major AI provider has confirmed that its crawlers read it. It costs almost nothing to serve, so it is reasonable to add. It is not a ranking lever.

9. Earn mentions elsewhere

Answer engines draw on the wider web. Clear, consistent descriptions of your product on your own site, in documentation, and in third-party pages that discuss you all give a system more to work with. This is the slow part, and no markup replaces it.

10. Measure by asking

Pick ten questions your customers would ask and run them in the assistants you care about. Note whether your domain is cited, and re-run them monthly. Crawlmend's AI visibility check does a version of this using a live web search for queries and your brand name, and shows whether your domain appears among the sources. Treat any such check as a sample, not a guarantee.

The short version

Be fetchable, be readable without JavaScript, be specific, and be honest in your markup. The first two are engineering problems, and Crawlmend checks for them: bot access at the robots and CDN level, JavaScript-dependent content, missing llms.txt and sitemap, and missing structured data. Connect a repo to see where your site stands.