Modulus is an AI research company, not an AEO company. We study how AI systems interpret information, how personal context shapes their responses, and how inaccurate claims and omissions can be identified and corrected. Our research combines patented technology, statistical modeling, and analysis of actual user interactions to evaluate the accuracy and reliability of AI responses under realistic conditions.
Why API results differ from what people see
Many companies use APIs to test AI models, but API results alone cannot reliably establish what people will encounter in actual chat sessions. Even with the same question and model name, an API response may only partially agree with what someone sees in ChatGPT or Claude, or may differ substantially.
Personal context is a major reason. People often interact with chatbots through their own signed-in accounts, where preferences, saved memories, and previous conversations can influence each response. The answer can therefore depend on what the chatbot already knows about the person asking the question.
The chat product’s system instructions and tools also matter. Anthropic, for example, explicitly states that updates to the system prompts used in Claude’s web and mobile applications do not apply to its API.[1] These differences further limit the assumption that testing a model through its API reproduces the experience of using the chatbot itself.
Scraping and retail accounts
Some companies use web scrapers to collect answers directly from chat interfaces. One of the largest AEO providers has publicly acknowledged collecting responses through the browser and running millions of prompts daily. Scraping can be disrupted by interface changes, and unauthorized automated extraction violates restrictions in major AI providers’ consumer terms.
Some AEO providers use fake, free retail accounts with access limited to basic models and low reasoning settings to run millions of prompts each day. These accounts lack realistic preferences, memories, and conversation histories. Their responses reflect the conditions of those accounts and cannot simply be treated as representative of what your customers actually see.
Simulating the chatbot user
To reflect real user experiences, testing must account for personalization, prior context, system instructions, and the availability and use of web search.
That is why Modulus developed and patented a system designed to simulate real chatbot user experiences.
Modulus uses probabilistic generative modeling to create synthetic user contexts designed to preserve contextual resemblance to real conversations. By modeling statistical dependencies among prior knowledge, beliefs, preferences, language patterns, and conversation history, Modulus generates coherent, randomized variations of the conditions that shape AI responses.
Through conversational user simulation and Monte Carlo evaluation, Modulus measures how these variations affect factual accuracy, recommendations, and the interpretation of corrective information. The resulting analysis identifies circumstances in which AI systems repeat inaccurate claims, incorporate verified corrections, or produce inconsistent answers. This provides a systematic basis for evaluating the reliability of AI responses across a broad range of plausible user experiences.
Modeling the questions as well as the context
Modulus applies statistical modeling to prompt generation as well. AEO companies often use AI to generate prompts without modeling the personal context behind them. Generating plausible questions alone leaves an important part of the chatbot experience unmodeled.
When businesses ask where their monitoring prompts should come from, some AEO providers recommend asking customers what they typed into an LLM to discover the business. Those responses capture individual examples, but they cannot reveal the full range of questions, intentions, and circumstances that influence AI answers.
You can add your own prompts to the Modulus system. However, manually creating and maintaining thousands of prompts to track every relevant dimension of customer demand and brand perception is not practical. Statistical modeling allows us to explore variations in both the questions people ask and the contexts in which they ask them.
Observed prompts at scale
Our analysis also draws on actual prompts typed by users. We obtain this data from clickstream providers, browser extensions, opt-in panels, and other sources. Data is collected with user permission, and personally identifiable information is removed before it ever reaches Modulus.
Modulus analyzes nearly 200 million prompts each month. This dataset provides an empirical foundation for modeling customer demand and extrapolating patterns to a market estimated at 4 billion daily prompts.
For categories with sufficient effective sample coverage, our methodology targets a relative margin of error of approximately 1–2% at a 95% confidence level. The proposed estimation framework combines stratified weighting with user-level bootstrap resampling to adjust for observed differences in panel coverage and account for repeated prompts from the same users.
Together, observed prompt data and statistical context modeling help you understand what customers are asking about in your category, so you can prioritize topics by actual demand and brand impact.