Research

Contextual Prompt Modeling

Modulus is an AI research company, not an AEO company. We study how AI systems interpret information, how personal context shapes their responses, and how inaccurate claims and omissions can be identified and corrected. Our research combines patented technology, statistical modeling, and analysis of actual user interactions to evaluate the accuracy and reliability of AI responses under realistic conditions.

Why API results differ from what people see

Many companies use APIs to test AI models, but API results alone cannot reliably establish what people will encounter in actual chat sessions. Even with the same question and model name, an API response may only partially agree with what someone sees in ChatGPT or Claude, or may differ substantially.

Personal context is a major reason. People often interact with chatbots through their own signed-in accounts, where preferences, saved memories, and previous conversations can influence each response. The answer can therefore depend on what the chatbot already knows about the person asking the question.

The chat product’s system instructions and tools also matter. Anthropic, for example, explicitly states that updates to the system prompts used in Claude’s web and mobile applications do not apply to its API.[1] These differences further limit the assumption that testing a model through its API reproduces the experience of using the chatbot itself.

The same question through the API and in a signed-in chat sessionThrough the APIIn a signed-in chat sessionQuestionNo session contextModelAnswer AQuestionProduct system instructions and toolsSaved memories, preferences, conversationsWeb search, when the product uses itModelAnswer Bmay differ
The same question and model name can produce different answers through the API and in a chat product, because the chat session carries context the API call does not.

Scraping and retail accounts

Some companies use web scrapers to collect answers directly from chat interfaces. One of the largest AEO providers has publicly acknowledged collecting responses through the browser and running millions of prompts daily. Scraping can be disrupted by interface changes, and unauthorized automated extraction violates restrictions in major AI providers’ consumer terms.

Some AEO providers use fake, free retail accounts with access limited to basic models and low reasoning settings to run millions of prompts each day. These accounts lack realistic preferences, memories, and conversation histories. Their responses reflect the conditions of those accounts and cannot simply be treated as representative of what your customers actually see.

What each testing method reproducesPersonalizationPrior contextSystem instructionsWeb search useAPI testingScraped retail accountsfree accounts, basic models, low reasoning settingsModulus simulationsynthetic user contextsReproducedPartly, and differently by productNot reproduced
Testing through the API leaves out the chat product’s system instructions and the person’s context. Scraping through free retail accounts reproduces the product but not the person, is limited to basic models and low reasoning settings, and violates provider terms. Modulus simulates all four conditions.

Simulating the chatbot user

To reflect real user experiences, testing must account for personalization, prior context, system instructions, and the availability and use of web search.

That is why Modulus developed and patented a system designed to simulate real chatbot user experiences.

Modulus uses probabilistic generative modeling to create synthetic user contexts designed to preserve contextual resemblance to real conversations. By modeling statistical dependencies among prior knowledge, beliefs, preferences, language patterns, and conversation history, Modulus generates coherent, randomized variations of the conditions that shape AI responses.

Through conversational user simulation and Monte Carlo evaluation, Modulus measures how these variations affect factual accuracy, recommendations, and the interpretation of corrective information. The resulting analysis identifies circumstances in which AI systems repeat inaccurate claims, incorporate verified corrections, or produce inconsistent answers. This provides a systematic basis for evaluating the reliability of AI responses across a broad range of plausible user experiences.

How Modulus simulates a chatbot userObserved promptsnearly 200 million a month,identifiers removedContext modelprior knowledge, beliefs,preferences, language patterns,conversation history, and thedependencies among themSynthetic usercontextscoherent, randomizedvariationsConversationaluser simulationacross AI systems,settings and web searchMonte Carloevaluation of accuracy,recommendations andcorrective informationRepeats an inaccurate claimUses a verified correctionGives inconsistent answersAcross a broad range of plausible userexperiences, the evaluation identifies thecircumstances in which an AI system doeseach of these.
Observed prompts and a probabilistic model of user context produce synthetic user contexts. Simulated conversations run under those contexts, and Monte Carlo evaluation measures how the variation affects accuracy, recommendations and the uptake of corrections.

Modeling the questions as well as the context

Modulus applies statistical modeling to prompt generation as well. AEO companies often use AI to generate prompts without modeling the personal context behind them. Generating plausible questions alone leaves an important part of the chatbot experience unmodeled.

When businesses ask where their monitoring prompts should come from, some AEO providers recommend asking customers what they typed into an LLM to discover the business. Those responses capture individual examples, but they cannot reveal the full range of questions, intentions, and circumstances that influence AI answers.

You can add your own prompts to the Modulus system. However, manually creating and maintaining thousands of prompts to track every relevant dimension of customer demand and brand perception is not practical. Statistical modeling allows us to explore variations in both the questions people ask and the contexts in which they ask them.

Questions crossed with the contexts they are asked inQuestions people askContexts they ask inWhich to chooseIs it reliableWhat it costsAlternativesDoes it fit meFirst-time buyerComparing two vendorsHas used a competitorFollow-up in a longer chatWeb search turned onDifferent wording or languagePrompts customers remember typingVariations the model generates and tests
Each cell is a question asked under a particular context. Asking customers what they typed fills in a few cells. The statistical model generates and tests the rest, so an answer is judged across the circumstances in which the question is actually asked.

Observed prompts at scale

Our analysis also draws on actual prompts typed by users. We obtain this data from clickstream providers, browser extensions, opt-in panels, and other sources. Data is collected with user permission, and personally identifiable information is removed before it ever reaches Modulus.

Modulus analyzes nearly 200 million prompts each month. This dataset provides an empirical foundation for modeling customer demand and extrapolating patterns to a market estimated at 4 billion daily prompts.

For categories with sufficient effective sample coverage, our methodology targets a relative margin of error of approximately 1–2% at a 95% confidence level. The proposed estimation framework combines stratified weighting with user-level bootstrap resampling to adjust for observed differences in panel coverage and account for repeated prompts from the same users.

From panel prompts to a category estimatePanel promptsclickstream providers,browser extensions andopt-in panels, collectedwith permission andidentifiers removedStratified weightingadjusts for observeddifferences inpanel coverageUser-level bootstrapresamples by user toaccount for repeatedprompts from thesame peopleCategory estimaterelative margin of errorabout 1–2% at 95%confidence, wherecoverage is sufficientestimate with its intervalNearly 200 million prompts a month, in a market estimated at 4 billion prompts a day.
Observed prompts are weighted to correct for differences in panel coverage and resampled by user so that repeated prompts from the same people do not count as independent evidence. Where coverage is sufficient, the category estimate targets a relative margin of error of about 1–2% at 95% confidence.

Together, observed prompt data and statistical context modeling help you understand what customers are asking about in your category, so you can prioritize topics by actual demand and brand impact.

Sources

  1. [1]Anthropic • System prompts (opens in a new tab)Release notes for the system prompts used on claude.ai and in the Claude mobile apps, which state that the updates do not apply to the Claude API

Complimentary 30-Day Evaluation

Give AI an authoritative account of your organization, backed by evidence and maintained by senior AI consultants. Modulus will show you how AI represents your organization and how verification improves its answers. Investigate, review the evidence, perform A/B testing, ask questions, and evaluate the results.

Experience Modulus for 30 days
Request a confidential consultation