How do you audit the UX of an AI product?
Short answer
Audit an AI product in order. Is it solving the right problem, are the main paths clear, does every state exist, can people check and correct the AI, and only then the visual craft. Audits that start with craft miss the things that lose users. Have someone who didn’t build it do the review.
Why is auditing an AI product different?
Because the output changes every time.
A normal product does the same thing for the same input, so you can review a screen and be done. An AI product can be right on Monday and wrong on Tuesday for the same request. It can take a second or a minute. It can act on its own. The audit has to cover how the product behaves when the AI is slow, unsure or wrong, and those moments rarely appear in a design file.
What order should the audit run in?
We review in a fixed order, starting with the biggest questions. There’s no point polishing a screen that shouldn’t exist.
1. Is it the right problem?
What is the user trying to get done, and is AI the best way to do it? Some AI features are a slower route to something a button did well.
2. Are the primary paths clear?
Name the two or three things people come to the product to do. Walk each one from start to finish. Count the steps, the decisions and the places you had to think.
3. Does every state exist?
Loading, empty, failing, partly loaded, offline. For AI features add: still thinking, no good answer, and stopped halfway.
4. How does the AI behave?
What it does alone, what it asks first and what it never does. Whether you can see where an answer came from, tell how sure it is and undo it. What happens after you correct it.
5. Craft and accessibility
Type, spacing, contrast, motion, keyboard and screen reader use, interface writing. Important, and last.
What should you look for in each area?
| Area | Question to ask | Sign of a problem |
|---|---|---|
| Problem | Would this be easier without the AI? | The AI feature is used once and abandoned |
| Primary paths | Can a new user finish the main task unaided? | Everything on the first screen has equal weight |
| States | What shows when there’s no data or no connection? | Blank screens and endless spinners |
| AI behaviour | Can I check and reverse what it did? | “Regenerate” is the only fix |
| Craft | Does it look like one product? | Several versions of the same component |
Who should run the audit?
Someone who didn’t make it. In our own process the reviewer is never the person or the AI agent that made the work. The maker knows what the screen was meant to do and reads it that way.
If it has to be internal, give it to someone from another part of the product and hand them the order above.
What do you do with the findings?
Fix in the same order you reviewed. A problem with the primary paths outranks fifty craft issues. Most audits fail at this step: the long list gets sorted by effort, the easy cosmetic fixes ship, and the product is no easier to use.
What does this look like in real work?
We designed an AI content platform for a Hollywood creative lab. When we started, the team had three competing ideas for how the product should work. That’s a step one problem, and no amount of screen design would have settled it.
So the first thing we made was a design blueprint for the whole system, built on one model: everything starts from the person’s story. The product design and a prototype the client could test came after. Read the case study.
A short checklist to run today
- Describe the product’s main job in one sentence. Does the first screen match?
- Complete the main task as a new user and count the decisions
- Open it with an empty account, then with no connection
- Ask the AI something it can’t answer
- Correct an output, then come back tomorrow and see if it held
Who can audit an AI product for us?
Someone who has shipped AI products and will tell you if the problem is upstream of the interface. Ask what order they review in.
Halaska is an embedded product design partner for funded AI, crypto and fintech teams. Every partnership starts with the direction: what to build first and why. See AI product design or get in touch.
FAQ
What is an AI UX audit? An AI UX audit is a structured review of how an AI product works for its users, including how it behaves when the AI is slow, unsure or wrong.
What should an AI UX checklist cover? The problem being solved, the main user paths, every state of every screen, how the AI acts and can be corrected, then visual craft and accessibility.
Who should audit our product? Someone who didn’t build it. Makers read their own screens the way they intended them.
What do we fix first after an audit? Problems with the main paths, then missing states, then AI behaviour, then craft.
Who is Halaska? Halaska is an embedded product design partner for funded AI, crypto and fintech teams, based in Dubai. It’s founded and directed by Chris Halaska, who has 20+ years designing products, including six at Google.