Building a Face Matching System for Identity Verification
In digital lending, verifying customer identity is a critical step. A common requirement is ensuring that the selfie a customer takes during onboarding matches their government-issued ID (like Aadhaar). But before we could even compare faces, we faced a different challenge: making sure the selfie captured at the frontend was of good enough quality for verification.
Customers often clicked pictures that were blurry, poorly lit, tilted, or with their face partially covered. These issues led to false mismatches and frustrated both the customer and the system.
First Challenge: Capturing a Good Selfie
The face matching algorithm is only as good as the input photo. So instead of running expensive corrections later, we decided to guide customers while they were taking their selfies.
To achieve this, we:
- Used Mediapipe to detect face position and landmarks
- Trained a custom edge-device model to classify issues like:
- Face too far from the camera
- Eyes closed
- Head tilted left/right
- Face covered or wearing glasses (when not allowed)
These checks ran directly in the frontend, so customers received real-time guidance like “Move closer to the camera”, “Open your eyes”, or “Adjust your face straight”. Only when all checks were passed did we allow the user to capture the picture.
Face Matching
Once a proper selfie was captured, it was sent to the backend for face matching against the Aadhaar photo. The matching itself used a CNN-based face embedding approach, which handled variations in lighting and minor angles while still being strict enough to prevent impersonation.
Tech Stack
- Frontend: JavaScript + Mediapipe (real-time selfie guidance)
- Edge Models: Custom Mediapipe classifiers for face covering & glasses detection
- Backend: Python + FastAPI for face matching inference
Key Learnings
- Quality input matters: Guiding users at capture time reduced false mismatches significantly.
- Edge-device models improve UX: Running checks in the frontend avoided unnecessary server calls and gave instant feedback.
- Balancing strictness vs usability: Too many warnings frustrated users, so we carefully tuned thresholds for “acceptable” selfies.
Conclusion
By combining real-time selfie guidance with face matching, we built a system that not only verifies customer identity but also ensures a smooth onboarding experience.