Nine things to settle before you build an AI product
Product managers who approach AI products like traditional software ship disappointing results. Three assumptions stop holding, and the differences are worth settling before design starts.
Product managers approach AI features the way they approach software features. Requirements, acceptance criteria, a definition of done. That framing produces disappointing products, because three of its assumptions do not hold.
Software behaves the same way twice. AI does not. Software is finished at some point. AI degrades after launch. Software fails loudly. AI fails fluently, producing a wrong answer shaped exactly like a right one.
What I settle before design starts.
Start from the problem
The question is which customer problem AI solves better, faster, or cheaper than what exists. Starting from the technology gives you a feature looking for a user. I have shipped one of those. It demoed well internally and nobody outside the company used it.
Define success past accuracy
A model at 95% accuracy can deliver a worse experience than one at 88%, depending on where the errors land. Concentrated on your highest-value segment, 5% is a crisis. Spread across low-stakes queries, it is noise.
Measure adoption, task completion, time saved, and business impact. Accuracy is an input to those, not a substitute.
Design for uncertainty
Outputs are probabilistic. The interface has to handle wrong, incomplete, and ambiguous responses as normal operation rather than exceptions. This is where most AI UX fails: the happy path gets designed carefully and the failure path gets a toast notification.
Handle data quality first
Model capability is rarely the constraint. The data feeding it usually is. Clean, representative, governed inputs from the start cost less than remediation after users lose confidence.
Keep humans in the loop where stakes justify it
For consequential decisions, users need to review, correct, and override. Design the override path early. Retrofitting it means rebuilding the interaction model.
Earn trust through transparency
Users need to know what the system is doing, why it recommended something, and when its confidence is low. Confidence signalling is a product decision that engineering cannot make for you.
Plan for drift
AI products are never finished. Performance shifts as the world moves and user behaviour adapts to the system. Budget for monitoring, feedback collection, and retraining from the start, or your launch quality becomes your peak quality.
Make governance a requirement
Bias, privacy, security, compliance. Written as requirements they shape the design. Raised in review they become blockers, and late blockers get negotiated down rather than solved.
Prototype fast, scale slowly
Validate that users want the outcome before investing in the architecture that delivers it at scale. The cheap version answers the question the expensive version assumes.
The AI products that work are not running the most advanced models. They solve a real problem, tell the user what they are doing, and hold up when the data underneath them shifts.
Related reading
- Product managementThe AI data product manager role is not a rebrandData products became AI products, and the product manager who owns them now needs to read model behaviour, not just gather requirements and hold a roadmap.
- Data platformBad data does not slow you down. It scales your mistakes.Teams invest in models and underinvest in the data feeding them, then wonder why outputs feel off. AI did not create the problem. It removed the human who used to catch it.
- Product managementUsers tell you what hurts, not what is missingThe largest product opportunities do not arrive as customer requests. They come from watching what users work around and never think to mention.