Why Perplexity leaves your product out of vendor lists
Last updated
What Perplexity documents about how sites get into its answers, and five checks you can run by hand in an afternoon.
What Perplexity says
- Its search crawler is PerplexityBot, which “is designed to surface and link websites in search results on Perplexity”.
- “To ensure your site appears in search results, we recommend allowing PerplexityBot in your site’s robots.txt file.”
- After a change to robots.txt, “it may take up to 24 hours for our systems to reflect changes.”
- A second fetcher, Perplexity-User, fetches a page when a person asks about it. “Since a user requested the fetch, this fetcher generally ignores robots.txt rules.”
All four are from Perplexity’s page on its crawlers, which also gives the addresses its crawlers come from.
Five checks, in order
- Is PerplexityBot allowed? Read your robots.txt. A group that names a crawler replaces the general group for that crawler, so rules written for
*stop applying to it.curl -s https://yourcompany.com/robots.txt - Does your firewall let it through? Ask as the crawler:
A 403, a 429 or a challenge page is worth chasing. A 200 is weaker evidence than it looks, because the request came from your computer. The real test is your CDN’s logs, matched against the addresses Perplexity publishes.curl -s -o /dev/null -w "%{http_code}\n" -A "PerplexityBot" https://yourcompany.com/ - Do your pages have words without JavaScript? Fetch a page and look for a sentence you can see on it. The full check.
- Ask it what your company is, by name. If it describes another company, or mixes you with one, you have a name problem before you have a ranking problem. The fix is one plain sentence saying what you are, on your home page and on every profile you have elsewhere.
- Ask the buyer’s question and open the sources under the answer. A vendor list is put together from those pages. If it came from three roundups and a review site, and you are on none of them, that is the reason, and it is also your list of what to do.
One test proves less than it seems. Paste your address into Perplexity, ask about it, and you may get a good summary. That fetch is made for you by Perplexity-User. It says the page can be read, not that the page is in what Perplexity searches.
Why being readable is not enough
Being readable is the entry ticket. After that, a new product meets the same thing in every assistant we have measured: a short list, built from pages other people wrote, most of them before you launched. Why AI recommends established rivals has our numbers from ChatGPT and Gemini, and how AI search tools decide has what the companies document.
For ChatGPT and Gemini, Wellread asks the questions and counts the answers for you. How we measure.