Same Question, Different Answers: The Current Reality of AI Search
Consider a consumer researching the optimal no-fee travel credit card. ChatGPT initially recommends Bank of America's Travel Rewards card.
Hours later, identical queries yield Chase's Freedom Unlimited card, this time sourced from training data rather than web search.
Later, the same question produces a third recommendation: Bilt's Mastercard
Meanwhile, Google's new AI Mode consistently promotes Discover it Miles, citing the usual SERP sources.
Welcome to the new reality of AI search results, where the same question gets you different answers depending on when and who you ask.
We Asked AI 792 Product Questions.
At Amplifying, we built AI Product Bench to systematically track these patterns:
- 33 product categories selected based on high-volume consumer search patterns from Google and Amazon data
- 4 query variations per category: 2 conversational formats and 2 search-style formats
"best laptop under 1000" (search)
"top laptop below $1000" (search)
"what's the best laptop under $1000?" (conversational)
"I'm looking for a laptop under $1000" (conversational)- 132 unique queries representing authentic consumer search behavior
- 3 temporal runs per query to assess consistency over time using the Chrome browser and a Bay Area IP address
- 792 total AI conversations generating over 1,600 product recommendations
Explore the data: amplifying.ai/research/consumer-products
Key Takeaways
Cross-Platform Disagreement
Google AI and ChatGPT demonstrate only 47% agreement rates for identical queries. This means that for more than half of all product searches, these two leading AI systems recommend entirely different products as the optimal choice.
Shopping Graph Bias Towards Product Inventory
Google's Shopping Graph integration creates systematic biases toward available inventory over product quality. Analysis reveals instances where Google recommends lower-spec products based on merchant availability and pricing rather than objective performance metrics. In one case, Google promoted a refurbished 2020 M1 MacBook Air over the superior 2025 M4 version, despite both falling within the same price range.
Output Drift in ChatGPT
ChatGPT maintains high consistency in only 12.1% of repeated query scenarios, with 27.8% Jaccard score. This inconsistency correlates directly with the system's information sourcing methodology:
- Web search-based responses: 14.4% Jaccard consistency rate
- Training data-based responses: 31.2% Jaccard consistency rate
- 64.1% of queries rely solely on training data without external verification
Content Partnership Influence
Analysis of ChatGPT's citation patterns reveals significant influence from content licensing agreements. A publication typically more associated with celebrity breakup content, People Magazine, ranks among the top sources for product recommendations.
OpenAI's documented partnerships with News Corp (New York Post) and Dotdash Meredith (People Magazine) demonstrate how content licensing agreements directly impact the information landscape that shapes AI recommendations.
The Bottom Line
The current state of AI product recommendations represents a major shift in how consumers access purchasing guidance. Traditional assumptions about repeatable search results no longer apply in an AI-mediated search environment.
The documented 47% disagreement rate between AI Mode and ChatGPT, combined with individual platform idiosyncrasies, suggests that consumers should approach AI product recommendations with skepticism and seek multiple sources of information.
For businesses, understanding these recommendation patterns becomes critical for product positioning and marketing strategy. Traditional SEO frameworks remain relevant, but the optimization landscape has fundamentally changed. Success now requires tracking when AI systems rely on web search versus training data, understanding inventory prioritization biases, and recognizing how media partnerships influence recommendations.
Alex Vikati & Edwin Ong