What 100M users taught me about production computer vision

Every computer vision project I’ve ever been pitched starts the same way: a demo that works. The model finds the face, the overlay sticks, the metric says 95%. And almost every project that later failed, failed for the same reason: the team believed the demo was most of the work.

It’s about 20% of the work. Here is what the other 80% looks like, from two decades of shipping perception systems — try-on, AR, tracking — to more than 100 million users in 60+ countries.

The lab is a lie

Not maliciously — structurally. A demo runs on chosen hardware, in decent lighting, on cooperative subjects, at whatever frame rate it happens to hit. Production is the opposite of all four, simultaneously.

The real world is adversarial without trying to be. Retail floors have fluorescent lighting that strobes at frequencies your exposure compensation hates. Users hold phones at chin height in dim bedrooms. Faces come with glasses, beards, hijabs, makeup, motion blur, and a thumb over half the lens. A dataset that doesn’t systematically cover skin tones, ages, and accessories isn’t a smaller version of the right dataset — it’s a different product that happens to share your code.

The practical rule: your evaluation set should embarrass your model. If it doesn’t, it’s not measuring anything you’ll meet in production.

Latency is a product decision, not an engineering detail

Real-time AR has a budget of roughly 33 milliseconds per frame — camera to screen, everything included. That budget forces the most consequential architecture decision in the system: where does inference run?

On-device buys you latency, privacy, and freedom from network weather — and costs you model size, thermal throttling, and a support matrix of thousands of GPU variants. Cloud buys you model freedom and a single deployment target — and costs you a round trip that makes true real-time interaction impossible for anything that tracks motion.

The systems that work at scale almost always split: perception (tracking, landmarks, segmentation) runs on-device where the latency budget lives, and heavy understanding (embeddings, recommendation, generation) runs server-side where nobody can feel the round trip. Getting this split wrong is expensive to discover late, because it’s an architecture, not a parameter.

Fidelity has a price curve, and you should know where you sit

We track faces with 500+ landmarks for beauty and eyewear try-on. That number isn’t bragging — it’s a decision with a cost curve. Dense meshes buy you photorealistic contact between the virtual object and the face: glasses that sit on the nose bridge instead of hovering near it. They cost compute, battery, and lower-end device support.

For attention detection, 68 landmarks are plenty. For a game filter, even fewer. The discipline is asking what does the product actually need the machine to see? — and resisting both the engineer who wants the dense mesh everywhere and the accountant who wants the cheap one everywhere.

Production has no ground truth

The nastiest property of deployed perception systems: in production, nobody tells you the right answer. There’s no label on the frame. Accuracy — the number the whole project was sold on — becomes unmeasurable the moment you ship.

So you instrument proxies: tracking-loss events per session, landmark jitter, confidence distributions, session abandonment at the camera-permission prompt, frame rates by device cohort. You build shadow evaluation: sampled, consented frames routed to offline labeling so you can estimate live accuracy weeks before a complaint would surface it. Drift is not an if. New phones ship new cameras with new ISPs every autumn, and your input distribution moves whether your model does or not.

If a vendor shows you a live accuracy number for a deployed CV system, your first question should be: measured how?

What I’d ask before believing any CV demo

After twenty years on the building side, this is my buyer’s checklist:

1. What hardware did this run on, and what’s the worst device you support?
2. Show me the failure reel — every honest team has one. The lighting, the pose, the skin tone, the accessory where it breaks.
3. What’s the end-to-end latency, camera to pixels? Model inference time alone is a vanity metric.
4. How do you know accuracy in production, where no one labels the frames?
5. What happens when the model is wrong? Graceful degradation is a design surface, not an apology.

A demo that works is table stakes. The system around the model — data strategy, latency architecture, monitoring, failure design — is the product. That’s the 80%. Budget for it, staff for it, and ask every vendor where it is, because the model is the only part they’ll show you unprompted.