A customer with a leaking water heater doesn’t always search Google anymore. Some open ChatGPT, type “Who’s a good plumber in Clearwater who can come today?” and call whoever it names. Nothing tells you whether you were in that answer, or whether what it said about you was right.
You can find out with a list of questions and a spreadsheet. We covered the short version in what AEO is and how it differs from SEO. This is the full method.
Why does it matter what ChatGPT says about you?
Because some customers now ask an AI tool who to call, before they search or instead of searching. In October 2025, OpenAI said ChatGPT had 800 million weekly users, as reported by TechCrunch. Those looking for a plumber or HVAC company get a short paragraph that often names two or three businesses, with no second page.
Two things can go wrong:
- You’re not in the answer. The customer calls someone else, and you never know the question was asked.
- You’re in the answer, but it’s wrong. An old phone number, last year’s hours, a service you dropped or a town you don’t cover. Being named with the wrong number can be worse than not being named.
Neither shows up in your analytics. AI tools mostly don’t report when they mention you, the way Search Console does for Google, so asking yourself is the main way to see what customers see.
What questions should you ask it?
Ask what your customers ask, in their words. Write down ten to fifteen questions and keep the list, so every round asks the same things and the results compare. Cover three kinds, swapping in your own trade, towns and business name.
Category questions, with your city
Here the customer doesn’t know you yet. They ask for a type of business in a place, and the tool decides who to name. For a plumber or HVAC company:
- “Who’s a good plumber in Clearwater?”
- “My water heater is leaking. Who should I call in Brandon?”
- “What’s the best AC repair company in Riverview for an older house?”
- “Which HVAC companies near Wesley Chapel offer maintenance plans?”
Use the towns you want work from and the services that make you money.
Questions about you by name
These check whether the tool has your facts right.
- “What are the hours for [your business name]?”
- “What’s the phone number for [your business name]?”
- “Does [your business name] install tankless water heaters?”
- “What areas does [your business name] serve?”
- “Is [your business name] licensed in Florida?”
- “Is [your business name] any good?”
Keep your real details beside you. You’re checking every fact, not just whether your name appears.
Comparison questions
Customers also ask the tool to choose for them:
- “Who’s better for drain cleaning in Clearwater, [your business name] or [competitor]?”
- “What are the best-reviewed HVAC companies in Brandon, and why?”
Note the reasons it gives. If it says a competitor offers same-day service and you do too, but your site never says so, that’s a gap you can fix.
Which tools should you check?
You can’t know which tool a given customer opened, so check the main ones:
- ChatGPT. Depending on the question and your settings, it may search the web and show links, or answer without them.
- Perplexity. It’s built around web search, and it usually lists the sources it used.
- Gemini. Google’s AI assistant, separate from Google Search.
- Claude. Anthropic’s AI assistant. It may or may not search the web, depending on your settings.
- Google AI Overviews and AI Mode. AI Overviews are the summaries at the top of some Google searches. AI Mode is Google’s chat-style search, where it’s available to you. Both usually link to the pages they drew on.
Why does the same question get different answers in different tools?
At a high level, an answer about a local business comes from a live web search, from what the model learned in training, or a mix of both. A search-based answer pulls in pages, listings and reviews, then summarizes them, often with links. An answer from training data may be out of date and usually comes without links. Unless sources are shown, you often can’t tell which happened, and the companies don’t publish every detail of how their tools choose. Treat each tool as its own test.
Answers can also differ inside one tool, depending on:
- Whether you’re signed in. Some tools draw on your past chats or saved preferences when you are.
- Which model or mode is selected. Many tools let you pick a model or switch web search on and off.
- Where you are. Google, and some AI tools, may use your approximate location for local questions, so the same question can get a different answer across town.
Note which setup you used, so next month you compare like with like.
How do you run a fair test?
You want roughly what a stranger sees, not what a tool that knows your interests says:
- Start a fresh chat for every question. In a long conversation, the tool uses what came earlier.
- Sign out or use a private window where you can. Where a tool needs an account, turn off memory or personalization for the test, or use a temporary chat if it offers one.
- Put the location in the question. Write “in Clearwater,” not “near me,” so every test asks about the same place wherever you’re sitting.
- Don’t lead with your name in category questions. “Is [your business name] a good plumber in Clearwater?” tells the tool what you want to hear. Ask the category question in its own chat, and the name questions separately.
- Use the exact same wording each time. Copy it from your list. Small changes in wording can change the answer.
- Ask each question more than once, on different days. Two or three tries over a week or two show whether a mention is steady or a one-off.
How do you record the results so you can compare later?
A spreadsheet is enough. Give each answer its own row, with these columns:
- Date: when you asked.
- Tool: which one, plus the model or mode and whether you were signed in.
- Question: the exact wording, copied from your list.
- Mentioned: yes or no.
- Position in the answer: first named, third of five, only after a follow-up question, or blank if you weren’t mentioned.
- Details correct: yes or no, and which detail was wrong.
- Sources shown: the sites or pages it linked to, or “none shown.”
Screenshot each answer too, with the date in the file name. Wording is hard to remember a month later.
Don’t read much into position. An AI answer isn’t a fixed ranking, and the order can shuffle between tries. What matters more is how often you show up and whether what’s said is right.
After a few rounds, patterns show up: named for one service but not another, in one town but not the next, or the same wrong phone number coming back from the same directory. Those tell you what to fix first. Our AEO readiness checklist includes a monthly version of this test with a simple results sheet.
What if it gets your details wrong?
You can’t edit an AI tool’s answer. You can fix what it reads.
Step 1: Trace the source. If the tool showed links, open them and find the wrong detail. It’s often an old directory listing, a forgotten social profile or a page on your own site missed in the last redesign. If no sources were shown, search Google for the wrong detail itself, such as the old phone number in quotes.
Step 2: Fix your own site first. It’s the one source you fully control. Check the footer, contact page and service pages, and state your hours, phone, service areas and services plainly. Take out claims you can’t back up: if a page says “24/7” and you don’t take night calls, an AI tool may repeat it.
Step 3: Fix your Google Business Profile. Hours, phone, categories, services and service areas should all match your site.
Step 4: Fix your directory listings. Bing, Apple Maps, Yelp, the BBB, Facebook and the directories in your trade, with one exact version of your name, address and phone. Mismatched listings hurt you on the map too, as we explain in why your business isn’t showing up in Google Maps. When we worked with a San Antonio plumbing company, cleaning up business details across platforms was part of the job.
Step 5: Re-test in a few weeks. A tool that searches the web may pick up a corrected page once it’s crawled again. An answer from training data may repeat the old detail until the model is updated, and nobody outside the AI company can tell you when that will be.
Some tools have a feedback button, such as a thumbs-down, on each answer. Using it costs nothing, but don’t count on it changing what other people are told.
What if it doesn’t mention you at all?
Start with the checks you’d make for Google. Can your pages be crawled and indexed? A leftover noindex tag can hide a site from Google and AI tools alike (here’s how to check). Is there a clear page for each main service and town? Do your listings agree? Then state your facts plainly, including in structured data. None of this guarantees a mention. It makes you easier to find and harder to get wrong.
Why do answers change from day to day?
Very little about them is fixed:
- Answers are generated fresh each time. The wording, the order and even which businesses get named can shift between two tries a minute apart.
- Search results change. A tool that searches the web works from whatever it finds that day, and new reviews or a competitor’s new page can change that.
- Models get updated. AI companies release new versions often, and an update can change answers across the board.
- Your setup changes. A different model, being signed in or asking from somewhere else can each change the result.
So look for patterns, not single answers. One answer that leaves you out isn’t a crisis, and one that names you first isn’t a win to bank. Watch the trend across your whole list over several months: are you mentioned more often, in more towns, with fewer wrong details? It’s also why nobody can honestly guarantee that an AI tool will mention or recommend you. Anyone who promises that is guessing.
Run the test monthly, and again after anything changes, like a move, a new phone number or new hours.
Want us to run the check for you?
If you’d rather not run it yourself, our free AI visibility check asks ChatGPT, Perplexity and Google’s AI answers about businesses like yours in your city. You get back whether they mention you, whether what they say is accurate, what’s missing from your site that AI tools rely on, and the first fixes, written down.
For help with the fixes, here’s how we handle AEO. We record what the AI tools say before we start and ask the same questions after the work is live, so you can see what changed. We don’t promise mentions or positions, because nobody controls those answers.