Real-time AR try-on in the browser — no app, no uploads
Brow try-on was unsolved because it needs bone structure, hair density and skin tone understood together, live. We built it with MediaPipe Face Mesh and WebGL shaders, entirely client-side — and product returns dropped 40%.
Why brow try-on was still unsolved
Beauty retail's dirty secret is the return rate, and brow products were among the worst offenders: customers buy from a model photo, apply at home, and return within days because the shade or shape looks different on their own face. Each return cost the retailer roughly 3–4× the product margin once restocking, logistics and lost lifetime value were counted.
Lip and eyeshadow had digital swatching — hex-code overlays on static photos. Brows resisted that treatment because a believable result needs facial bone structure, hair density and skin tone understood together, and it has to run on a live camera feed. A static-photo tool isn't trustworthy enough to change purchase behaviour.
468 landmarks, one shader, zero uploads
The engine is MediaPipe Face Mesh, which delivers 468 facial landmarks at 30fps in the browser via WebAssembly. We extract the brow region per frame (landmarks 46–55 and 276–285) and render the selected style on a canvas overlay using SVG path masking plus WebGL fragment shaders that blend the product texture with the user's detected skin tone.
Skin tone detection samples a 10×10 pixel patch from the cheek each session and maps it to one of 12 Fitzpatrick-scale buckets, which drive the product colour's opacity and hue shift in real time. A 'dark brown' pencil looks correct on both fair and deep skin tones without separate SKUs.
Everything runs client-side. No video frame ever leaves the device — which is not just a privacy nicety; it was the specific blocker that had killed two previous vendor solutions in legal review. The catalogue is a JSON feed from the retailer's existing Shopify store, so new SKUs appear in the try-on tool the moment they're published, and add-to-cart fires straight from the overlay.
Returns down, conversion up
In the 60 days after launch, return rates for featured SKUs dropped 40% against the prior period. Add-to-cart conversion from the try-on screen ran 25% higher than the standard product page — the tool wasn't just preventing regret, it was accelerating decisions.
The same rendering architecture was extended to the retailer's mobile app (a React Native WebView embedding the identical canvas component) within two months, and two more product categories followed. Build the pipeline once, and every new category is a texture pack.
Full case study: Virtual Try-On — 40% fewer returns