From a choppy experience to smooth, real-time movements
An AR gaming startup speeds its path to market with AI-powered gesture recognition
A startup specializing in interactive augmented reality experiences needed to quickly deliver a playable MVP that would be credible to investors. The core of the product relied on real-time hand gesture recognition, but the existing solution struggled to reach 14 FPS and lacked the stability needed for a compelling gaming experience. In 9 weeks, the Mondrian team worked directly alongside the client’s developers to design, train, and deploy a custom AI solution that achieved 35 FPS with only 27% CPU usage—far exceeding the set targets. The client was able to present a functional and immersive MVP, ready for its first demonstrations.
An AR gaming startup bets on AI to build its MVP
Mobile Games and Augmented Reality
~10 employees
North America
This startup has made it its mission to redefine the way players interact with virtual worlds by replacing traditional controllers with their own hands. With a vision centered on immersive experiences and a strong foundation in Web3 and AR technologies, the team sought to prove, through its alpha version, that natural gesture-based interaction could be precise, fluid, and viable on mobile devices.
A disappointing product: choppy animations and unstable coordinates
The team had an initial implementation of gesture detection, but the results on the actual device were disappointing: choppy animations, unstable coordinates, and a gaming experience that fell far short of the standard needed to win over players or investors.
With an MVP to deliver in order to attract early adopters and validate their business model with investors, every week of delay came at a direct cost. A subpar solution risked not only delaying the launch but also undermining the product’s credibility at the most critical moment.
Gesture recognition on mobile devices imposes severe constraints: low power consumption, limited memory, and response times faster than human perception. Traditional approaches—custom models trained from scratch—proved to be too resource-intensive or too slow for the target devices. Finding the right balance between accuracy, speed, and integration into an existing Unity pipeline required specialized expertise that the in-house team lacked.
A Real-Time Gesture Recognition Pipeline
Mondrian designed and deployed a real-time gesture recognition pipeline optimized for Apple devices. Rather than starting from scratch, the team leveraged Apple Vision as a foundation, adding a custom detection model trained on target data and post-processing methods to stabilize the 3D coordinates of the landmarks. The result: a solution capable of smoothly recognizing key in-game gestures, within the hardware constraints.
The Mondrian team integrated directly with the client’s developers, working in short sprints with rapid cycles of data collection, training, and validation. The client contributed its product expertise and knowledge of the game’s use cases—Mondrian brought its rigor in machine learning and expertise in mobile constraints to turn this vision into a deployable reality.
| Component | Role in the solution |
|---|---|
| Apple Vision | Native hand detection on iOS — a high-performance foundation optimized for Apple hardware |
| YOLOv5s (custom) | A custom-trained detection model designed to improve accuracy under in-game conditions |
| Post-processing 3D | Stabilizes landmark coordinates for smooth, predictable gestures |
| Unity | Game integration environment — the solution was designed to integrate seamlessly with it |
A compelling gaming experience, extended battery life, and a faster time to market,
| Indicator | Result | Business impact |
|---|---|---|
| Frame rate | 14 FPS → 35 FPS | A gameplay experience that holds up for first demos |
| CPU usage | Down to 27 % | Battery life is maintained, and the device remains stable during gaming sessions |
| Memory footprint | < 300 MB (target met) | Compatible with target devices from launch |
| Delivery time | MVP in 9 weeks | Accelerated market entry; investors met within the specified timeframe |
A similar challenge in your industry?
A 30-minute call is all it takes to find out if AI can add value in your specific context.
Planifier un appel