How to run a manual AI-visibility benchmark against your competitors — step by step
Without a paid tool, you can map how your company compares to competitors in the major AIs. Here's the 10-query method any company can run today.
Before investing in any AI-visibility strategy, you need to know where you stand — and where your competitors stand. That's the baseline without which any later effort has no reference to be judged against.
The most complete, automated way to do this is with a monitoring tool like Crowly. But even without a paid tool, you can build a useful initial benchmark in under two hours with the method we'll describe here. This manual analysis doesn't replace continuous monitoring — but it's the perfect starting point for understanding the terrain before deciding where to invest.
The 10-query method: how to define the test set
The first step is selecting 10 queries that represent the most important questions your potential customer asks about the problem you solve. Those queries should:
Be intent queries, not brand queries. "The best fleet-management platform for distributors" is an intent query — your customer would ask it without knowing your name. "Crowly reviews" is a brand query — they already know you exist. For competitive benchmarking, intent queries are more relevant.
Cover different journey stages. Include awareness queries ("how does X work"), consideration queries ("best X for Y profile"), and decision queries ("X vs. Y," "how to choose X"). Performance can vary significantly by stage.
Be specific enough to have a focused answer. "Management software" is too generic — the AI will answer about a universe of hundreds of products. "Time-tracking software for companies with remote workers" is specific enough that a focused answer is likely.
The test protocol: how to run it without bias
With your 10 queries defined, run each one in incognito mode (an incognito browser tab, with no account logged in) on the three main platforms: ChatGPT, Gemini, and Perplexity. Why incognito? Because AIs personalize answers based on user history — and you want the "default" answer, not the one personalized to your profile.
For each query on each platform, document:
- Did your company show up? (yes / no)
- If so, in what position in the answer? (first, second, third mention, etc.)
- Did your main competitors show up? Which ones, and in what position?
- What was the answer's "narrative"? (how is the AI describing your segment?)
- Is there any player showing up that you didn't consider a direct competitor?
The benchmark spreadsheet: how to organize the data
Create a simple spreadsheet with the queries in the rows and the platforms in the columns. For each cell (query × platform), record: "Showed up: Y/N | Position: X | Competitors ahead: [names]".
After filling in the 30 fields (10 queries × 3 platforms), you have an immediate visual map of where you're strong, where you're weak, and where your competitors are gaining space that could be yours.
Add a "priority gap" column where you note the queries competitors show up on and you don't — those are the highest-immediate-impact opportunities for your content strategy.
What to look for beyond presence or absence
Presence or absence is the most basic data point. What turns the benchmark into a strategy instrument are the patterns:
Why do they show up and I don't? For each competitor that shows up on queries where you don't, research what they have that you don't: Wikipedia? G2 reviews? Coverage in specific outlets? Detailed comparisons on the blog? The source of the competitor's advantage is the insight that guides your next move.
Which queries does no one cover well? Sometimes the benchmark reveals relevant queries where no player in the sector shows up — the AI gives generic answers or cites sources that aren't direct competitors. Those are the lowest-competition opportunities: being the first with specific content for those queries creates a first-mover advantage.
Cross-platform consistency counts. A competitor that shows up on ChatGPT and Gemini but not Perplexity has a different profile than one that shows up on all three. That reveals where their strategy is working and where it has a gap.
How often to refresh the manual benchmark
The manual benchmark should be redone every 90 days at a minimum — AIs change, competitors publish new content, the market evolves. Document the date of each benchmark so you can compare the trend over time.
For more frequent tracking without the weekly manual work, Crowly automates this monitoring and delivers the updated benchmark automatically.
How Crowly complements the manual benchmark
The manual benchmark described here is a starting point — useful, free, but labor-intensive and inevitably subjective. Crowly automates and systematizes the process: it defines queries, runs them across platforms consistently, documents results, and tracks the trend over time — something a monthly manual benchmark simply can't do.
Start with Crowly's free diagnostic to get your baseline in 2 minutes. Then decide whether continuous monitoring makes sense. Free diagnostic →
Sources:
