The dashboard said adoption was excellent. Technicians opened the app dozens of times a shift. The number was real, and the conclusion drawn from it was exactly wrong: they were opening it constantly because they could not find anything.
The ninety-second window
We rode with eleven technicians and timed every interaction. The useful window between parking and starting work was about ninety seconds, usually one-handed, often in rain, sometimes wearing gloves. Nothing in the product had been designed for that window because nothing in the product had been designed anywhere near a van.
Analytics tell you what happened. They are close to useless at telling you why, and actively misleading when the failure mode looks like engagement.
What observation changes that interviews do not
We asked the same technicians in a room what was wrong with the app, and got a list of feature requests. We watched them use it in a car park and got the actual answer, which was that the first screen answered a question nobody was asking.
- Measure the real window of use, in seconds, on site.
- Note the physical conditions: hands, weather, light, noise.
- Watch for the workaround — paper, a photo, a second app. It marks the failure precisely.
- Bring the numbers back as measurements, not opinions, so they survive the roadmap meeting.
The outcome we were proudest of was not a metric. Two months after rollout the technicians we had ridden with had stopped printing their lists. The product had finally become faster than paper.