Someone on your team — or a vendor, or an agency, or a hire eager to make an impression — is going to walk into a room and pitch you on "adding AI" to something. It'll be framed as obviously good: faster, smarter, more automated. As a non-technical founder, your job isn't to evaluate the model. It's to evaluate the pitch. Here's how to do that without pretending to know more about machine learning than you do.
Start with "what problem, specifically"
Every good AI pitch starts with a concrete, bounded problem — not a vague aspiration. "We want AI in the product" is not a problem statement. "Customers wait an average of a day for a response to routine billing questions, and most of those questions are answered the same way every time" is a problem statement. If the person pitching you can't name the specific, recurring task the model is meant to handle, the project isn't ready to be scoped, let alone built.
Push for specificity here even if it feels like you're slowing things down. A vague problem produces a vague, unmeasurable project — and those are the ones that quietly burn budget for months without anyone being able to say whether it worked.
Ask where the model's answers come from
This is the single most useful question a non-technical founder can ask, and it separates people who've thought the project through from people who haven't: is the model answering from its general training, or from your actual, current business data?
A model answering from general training will sound confident and fluent — and will be wrong about anything specific to your business, your policies, your pricing, or your customers. A model that's been connected to your real data (your documents, your product catalog, your support history) can actually be checked against something. If nobody in the pitch can explain how the model gets access to accurate, current information, that's a gap worth flagging before anything gets built.
Ask what happens when it's wrong
Every AI system will occasionally produce a wrong, incomplete, or oddly confident answer. That's not a hypothetical edge case — it's a design constraint you should plan around from day one. The question isn't whether the model will get something wrong. It's what happens next.
Does a person review the output before a customer sees it? Is there a clear escalation path when the model isn't confident? Is the cost of a wrong answer a minor annoyance or a real problem — a refund issued incorrectly, a customer given bad advice, a compliance issue? The higher the stakes of being wrong, the more human oversight the system needs, and the less appropriate it is to let the model act fully on its own. A good pitch addresses this directly instead of treating "it'll usually be right" as good enough.
Ask who owns it after launch
AI features aren't "ship it and forget it" the way a simple static feature might be. Data changes, business rules change, and something that worked well at launch can quietly drift if nobody's watching it. Before you approve a project, ask who's responsible for checking on it in three months — not just who's building it now. If the honest answer is "nobody," that's worth resolving before you commit resources, not after something goes wrong.
Be skeptical of scope creep dressed up as ambition
AI projects have a particular way of expanding mid-build. A tool scoped to answer one category of question quietly grows to handle five. A pitch to summarize internal notes turns into a pitch to auto-respond to customers. Each individual expansion can sound reasonable in the moment — more value, less additional work, they say — but scope creep is exactly how a three-week project becomes a three-month one with a fuzzier definition of "done." Your job is to keep asking whether the new scope is still solving the original, specific problem, or whether it's solving a different, bigger problem nobody scoped, budgeted, or tested for.
The pattern underneath all of this
None of these questions require you to understand how a model works internally. They require you to keep asking for specifics: what problem, what data, what happens when it's wrong, who owns it, and is the scope still what we agreed to. A founder who asks those five questions consistently will catch the vast majority of AI projects that would otherwise quietly go sideways — no technical background required.
If you're being pitched an AI project and want a second, plain-language opinion before you commit budget to it, that's exactly the kind of conversation we're glad to have. Book a free 30-minute call with Kaidon Labs, or reach out through our contact form, and we'll help you think it through.