Enterprise AI dashboards are full of numbers that feel like progress and measure almost nothing: models deployed, users onboarded, queries processed, accuracy scores on a validation set. Every one of these can go up while the business gets zero additional value from AI, and every one of these is far easier to report to a board than the metric that would actually prove something changed.

Why vanity metrics are so easy to fall into

Vanity metrics share one property: they measure activity, not outcome, and activity is what a team directly controls. A data science team can guarantee “models deployed” goes up by deploying more models — it’s entirely within their power, independent of whether any of those models change a business decision. A metric like “decisions made differently because of AI” is much harder to guarantee, because it depends on whether the rest of the organization actually acts on the model’s output, which the data science team doesn’t fully control. Teams gravitate toward the metric they can move, and that metric is almost never the one that proves value.

Accuracy is the most common trap of all, because it looks like an outcome metric while actually being an activity metric in disguise. A model can hit 95% accuracy on a held-out validation set and still deliver zero business value if nobody changes their behavior based on its output, or if the 5% of cases it gets wrong happen to be the highest-stakes ones. Accuracy measures whether the model works. It says nothing about whether the model is used, trusted, or acted on.

What an actual value metric looks like

A real value metric ties directly to a business outcome that existed before AI and can be measured the same way after — revenue, cost, cycle time, error rate against a baseline, customer retention. The test is simple: could this number have moved for reasons that have nothing to do with AI being good? If a metric like “queries answered” can go up purely because more people started using the tool, without any change in decision quality, it’s not a value metric. If a metric like “average time to resolve a support ticket” drops and stays down, that’s much harder to explain away — something real changed in how the business operates.

The other property a real value metric needs is a credible counterfactual — some sense of what would have happened without the AI system, whether from a holdout group, a before/after comparison controlling for other changes, or a documented baseline. Without a counterfactual, a favorable-looking number could just as easily be a seasonal trend or a change nothing to do with the model.

How this shows up in board reporting

Boards and executives who ask “how’s our AI program doing” usually get answered with deployment counts and accuracy scores, because those numbers are easy to produce and easy to make look good. The more useful — and more uncomfortable — answer requires the team to say specifically which business decisions are now being made differently, by how much, and how they know. That’s a harder story to tell, and it’s also the only story that actually justifies continued investment, because it’s the only one that shows the AI program touched something the business cares about.

Making the switch

The practical fix isn’t complicated, but it does require discipline: for every AI initiative on the roadmap, name the business metric it’s supposed to move before building anything, establish how that metric will be measured against a credible baseline, and report progress against that metric specifically rather than against activity counts. This reframing alone tends to kill a surprising number of proposed AI projects at the planning stage — if nobody can articulate which business metric a project is supposed to move, that’s useful information before any budget is spent, not after.

Pull up your own team’s current AI dashboard — how many of the numbers on it are things your team directly controls by doing more, versus things that could only move if the business actually changed?