Every AI rollout starts tracking the same number first: how many people logged in, opened the tool, sent a first message. It is the easiest metric to collect, and it is also the one that tells you the least about whether anything actually improved.
The metric everyone reaches for first
Login counts and message volume feel objective. They come straight out of the product's own logs, no survey required, no judgment call about what counts as "success." That is exactly why they spread so fast inside organizations: they are cheap to report and hard to argue with in a slide.
What it hides
- Someone can open a tool daily and use it for nothing that changes their output.
- A team can barely touch a tool and still route its highest-leverage decisions through it.
- Usage that spikes right after a mandate often collapses once attention moves elsewhere.
None of that shows up in a dashboard built around logins. It shows up in whether work got faster, whether quality held, and whether people kept using the tool once nobody was watching.
A better question
Instead of asking how many people used an AI tool, it helps to ask what specific piece of work moved because of it, and whether the person doing that work would notice if it disappeared tomorrow. That question is harder to answer and much closer to the truth.