THE METHOD

I've Sat Through Hundreds of Vendor Pitches. Here's How to Read an AI Tool's Case Studies Like a Buyer

2026-09-20

Years selling enterprise software, and later managing the partnerships behind solutions being resold, meant writing case studies for a living and, just as often, picking apart a competitor's case study in the middle of a live deal. It's a skill that transfers directly to reading what AI tool vendors publish now, because the same tricks show up in the same places.

The first tell: a percentage with no baseline. 40% faster means nothing without knowing what it was 40% faster than, and the vendors who leave that number out usually left it out on purpose, because the honest starting point wasn't impressive.

There's a specific version of this worth naming because it's especially common in AI marketing right now: faster than doing it manually as the implied baseline, when the real comparison should be faster than the next-best tool. Almost anything is faster than a fully manual process. That's a low bar, and a vendor citing it is quietly avoiding the comparison that would actually tell you whether their tool is worth switching to.

The second tell: no named company, or a company so small the case study reads more like a favor than a customer relationship. Real case studies name the customer, because the relationship is worth more to the customer's own reputation than it is to the vendor's marketing page. An anonymous logo, or a suspiciously generic one, usually means the results didn't survive attribution.

Worth adding a caveat here, since we've been on the other side of this exact tradeoff: sometimes a named customer genuinely can't be disclosed for legitimate reasons, particularly in regulated industries like healthcare or government, where procurement rules or competitive sensitivity make attribution impossible even when the result is real. The fix isn't to reject every anonymous case study outright, it's to weight it lower and ask more follow-up questions before trusting it the same way you'd trust a named, verifiable one.

The third, and the oldest trick in the book applied to a new category: measuring activity instead of outcome. Posts published, emails sent, hours saved on a task nobody would have spent that many hours on anyway. None of that is revenue, retention, or a decision that actually got better. It's the easiest number to make look good, which is exactly why it gets used so often.

A fourth tell worth adding, specific to AI tools: a case study that describes the result but never mentions how long the customer had been using the tool before measuring it. A number pulled from someone's first excited week of use behaves very differently from the same number measured six months in, after the novelty wears off and the tool gets used the way it actually gets used day to day. If the timeframe is missing, assume it's missing because the longer-term number was less flattering.

This is a real part of how we vet what goes on this site's Tool Directory: an honest take is supposed to mean we looked past the press-release language to what the numbers actually support. If a case study can't survive these four questions, it goes in as a claim to verify, not a reason to recommend. The discipline isn't complicated, it's just rarely applied, because most people reading a case study want it to be true and stop asking questions the moment the headline number sounds good.

One more habit worth building, easiest to apply once you know to look for it: ask the vendor directly for a reference customer you can talk to, not just read about. A vendor confident in their published results will usually connect you, sometimes with minor friction around scheduling. A vendor who stalls, deflects, or offers only a written quote instead of a conversation is telling you something about how comfortable they are with scrutiny of the actual number.