Vanity Metrics Almost Killed My Best Products

One of the best products I have ever shipped has exactly one client. No growth chart. No user count worth screenshotting. RealEstateCRM has been running in production since 2021, doing the daily heavy lifting for that one agency, with no further involvement from me. By every metric a dashboard likes to show, it looks like nothing happened.

Which metrics tell you a product is working? The one that matches the job the product was hired to do — one client depending on it every day, releases actually shipped, money actually in — not the one that is easiest to put on a slide. Signup counts, page views, and total registered users are the easy ones to show off, and they are the ones most likely to lie.

The danger of a vanity metric is not only that it flatters a dead product. It is that it can make you look at your best work and mistake it for a failure because it never produced a chart that goes up and to the right.

The product with one client

RealEstateCRM did not grow the way a SaaS grows. It could not.

It was an on-premise install for a single real estate agency — a CRM plus a listing website, built solo. There is no funnel. There is no month-over-month curve. If you put it next to a startup dashboard, the "active users" number is small enough to be embarrassing.

But look at what it actually does. It went live as something small — a simple apartments database — inside two weeks, and then grew for years from real daily use rather than from a roadmap. It went live before it was finished and was never finished, because it kept earning changes from the one client who could not stop using it.

That is the metric that mattered: one client relied on it every day, for years, enough that it kept earning changes. Retention of one, if you want to be pedantic about it. But it is retention of the only user the product was ever built for, and it never dropped. A product that one person cannot stop using is worth more than a product a thousand people tried once and forgot.

The product measured in releases, not users

Automator is the other one. A music publishing pipeline — ingest, organize, archive, upload, publish — built solo in 2021. It started with a two-day conversation about the actual problem, which turned out not to be publishing at all but organizing audio and images by release and keeping files consistent across multiple sites. First prototype in three weeks. Two weeks of polishing. Then two years running on its own with no involvement needed.

Ask it the wrong question — how many users does it have? — and the answer is unimpressive. It is not a user-facing product with a signup form.

Ask the right question — what did it produce? — and the answer is 200,000+ music releases published across its lifetime. It still runs today, under ten minutes a day, managing 100 to 150 posts across multiple sites. I later proposed and shipped a full rewrite from a mix of Ruby, Python, Go, and Bash into pure Go in a week, which doubled the processing speed and cut the error rate. The metric for Automator was never "users." It was releases shipped and time not spent. On both, it has been winning quietly for five years.

The graveyard had better-looking numbers

Here is the part that stung. Some of the products I abandoned had numbers that looked far healthier than my two most durable ones.

CheckMVP got used on 500+ startup ideas. That is a bigger, rounder, more presentable number than "one client" or "runs for ten minutes a day." On a dashboard, CheckMVP looked like the winner and RealEstateCRM looked like a rounding error.

I shut CheckMVP down anyway, because the 500 was curiosity and nothing underneath it was commitment. The number was the most impressive thing about it, which was exactly the problem.

I can name my own failure pattern because I have lived every item on the list: chasing signups instead of retention, and building for myself instead of for the person who would actually use the thing. A vanity metric feeds both of those habits at once. It rewards the signup you can count today and hides the retention you would only see in a year. It lets you build for an imaginary crowd instead of the one real user in front of you. For a while it had me half-convinced that the products with the big front-of-funnel numbers were the successes and the two still running in production were minor.

They are the opposite. The big-number products are in the graveyard. The small-number products are still working.

Pick the metric that matches the product's job

The fix is not "ignore metrics." It is to choose the metric before you choose the celebration — the one that measures whether the product is doing its actual job, then refuse to be flattered by any other number.

• If the job is to be depended on, measure dependence. For RealEstateCRM, the real signal was that one client could not run their day without it and kept asking for the next feature. That pull — them telling me what to build — is the metric. Not headcount.
• If the job is to produce output, measure output. For Automator, it was releases shipped and daily time saved, not users. A tool that runs unattended has no signup curve and does not need one.
• If the job is to make money, measure money. Paid, committed, invoiced — not "expressed interest," not registered, not visited.

None of these produces a nice chart to show an investor. That is the point. The metric that is easy to show off and the metric that tells you the truth are rarely the same number, and when you cannot have both, keep the true one.

The test I use now is a single question: if this number doubled tomorrow, would the product actually be more successful, or would I just feel better?

Five thousand CheckMVP analyses instead of 500 would have taught me nothing and changed nothing. One more year of RealEstateCRM running untouched, or 50,000 more releases through Automator, is the whole thing working. Learn to tell those two kinds of numbers apart, and you stop killing your best products for the crime of not looking like a startup.