JavaScript Configurators and AI Crawlers: A Test for B2B Sites

The most precise page on an industrial website is often the one an AI crawler is least likely to read. Configurators, parts finders and price lists hold the bore sizes, pressure ratings and lead times that buyers actually ask about, and on many sites none of those values exist in the HTML until a browser runs the JavaScript that fetches them. This protocol checks, page by page, how AI crawlers and JavaScript get along on your own site, with tools you already have.

The operators say less about this than most people assume. OpenAI's page on its crawlers describes OAI-SearchBot, GPTBot and ChatGPT-User. Anthropic's help article describes ClaudeBot, Claude-User and Claude-SearchBot. Perplexity's documentation describes PerplexityBot and Perplexity-User. When we read all three on 20 October 2026, none of them said whether those agents execute JavaScript. Google is the exception: its Search Central guide to JavaScript, last updated on 4 March 2026, states that a headless Chromium renders the page and executes the JavaScript, and that Google indexes the rendered HTML. That describes Google Search. It does not say which copy of a page Gemini or AI Mode works from.

So the working assumption has to be the cautious one. For every assistant outside Google, treat the raw HTML your server sends as the whole page. Anything that appears only after scripts run is something you have to prove readable, not something you can take for granted. The seven steps below do that.

1. List the pages where your numbers live

A configurator is rarely a single page. Make a short list, ten URLs at most: the product configurator, the spare parts finder, the price list or quotation tool, a product family page whose table loads from a database, and the page with stock or delivery times if you publish them.

Next to each URL, write down one value that appears on screen and that a buyer could plausibly ask an assistant about: a part number, a maximum operating pressure, a flange standard, a price. That value is your probe. Headings and marketing copy are no use here, because they are usually in the HTML anyway. The question is whether the specific facts are.

2. Read the HTML an AI crawler receives without JavaScript

Raw HTML is the document your server returns before any script runs. It is everything a crawler that does not execute JavaScript will ever see of that URL.

There are three ways to look at it. In the browser, use View Source (Ctrl+U, or Cmd+Option+U on a Mac), not the developer tools inspector, which shows the page after scripts have rewritten it. Search the source for your probe. In Chrome's developer tools you can also open the command menu, choose "Disable JavaScript" and reload: whatever is left on screen is roughly what a non-rendering crawler gets. From a terminal:

curl -s -A "GPTBot" https://www.example.com/configurator/ | grep -c "PN16"

A zero means the value is not in the document. Setting the user agent does not make curl behave like GPTBot. It only shows whether your server or CDN sends bots something different from what it sends browsers. Run the command once with a browser user agent and once with a bot name, and compare the size of the two responses.

3. Trace where the values come from

If the probe is missing, open the Network panel, filter by Fetch/XHR and use the configurator as a buyer would. You will usually see requests to an endpoint that returns JSON, something like /api/products/, or calls to a configurator hosted on a supplier's domain. That endpoint is where your facts actually live.

Note three things for each one. First, whether your robots.txt blocks it: a Disallow: /api/ line, common and often sensible, means that a crawler which respects the file will not request the data even if it renders the page. Second, whether it sits on another domain, whose robots.txt you do not control. Third, whether it needs a session cookie or a login. Any one of these puts the values out of reach, whatever a crawler does with JavaScript.

4. Check what the crawlers actually requested

Access logs are the closest thing you have to evidence. Filter the last month by the AI user agents and look at the URLs they asked for. A crawler that executes your scripts will normally also request the data those scripts load. If GPTBot or ClaudeBot fetched /configurator/ but never the JSON endpoint behind it, that is a strong hint for that agent on that page. How to read those lines, and how to tell a declared crawler from an impostor, is covered in what an AI crawler looks like in your server logs.

One caution. On a small site, a month without a visit proves very little, because crawlers come when they choose. Use this step to confirm step 2, not to replace it.

5. Ask a question only the configurator can answer

Now the direct test. Write a question whose correct answer exists on your site only inside the JavaScript: "What is the maximum working pressure of the [model] ball valve from [brand]?" or "Which replacement seal fits the [model] pump?". Ask it in ChatGPT, Claude, Gemini and Perplexity with web search on, in the language of the market you sell to. For each answer, record three things: whether it gives your value, whether it cites a URL, and whose URL that is.

Answers vary between runs and between assistants, so one answer is an observation, not a verdict. Look for a pattern instead: assistants quoting a distributor's static page or a marketplace listing in place of yours, or giving a plausible figure with no source at all. If your value comes back correct with your URL beside it, that page is readable for that assistant, whatever the mechanism.

6. Put the facts in the HTML and keep the configurator

A configurator is a good tool for a buyer and a poor document for a crawler. Removing it fixes nothing. The fix is to make sure that every fact a buyer might ask about also exists as text on a page the server delivers complete.

• One static page per product family or model, with the specifications in an HTML table: dimensions, materials, ratings, standards, part numbers. The configurator links to it and it links back.
• Server side rendering or prerendering for the configurator itself, if your platform supports it, so that the first response already contains the default configuration.
• The same content for every visitor. Serving bots a different page from the one people see is the old cloaking problem, and it leaves you with two versions of your facts that drift apart.
• A deliberate decision on prices. If prices sit behind a login for sound commercial reasons, accept that no assistant will quote them, and publish what you are happy to see repeated: whether you sell direct, minimum order quantities, typical lead times.

The PDF datasheet is a related problem with a different cause, and why your spec sheet PDF is invisible to AI deals with it.

7. Run the test again, and keep running it

After the change, repeat steps 2 and 5 on the same URLs with the same questions. Step 2 gives an immediate answer, because the HTML either contains the value or it does not. Step 5 does not. Assistants pick up changes on their own schedule, a corrected page can take weeks to appear in answers, and nothing in this protocol guarantees that it will.

That second half is the part worth automating. PSentry runs the same set of questions on ChatGPT, Claude, Gemini and Perplexity twice a month, in each language you sell in, and records whether your brand is named and which sources the answers cite. It measures and does not change what the assistants say. For a configurator fix, that means a dated record of whether the answers to your probe questions moved after the HTML did.