Building a Video-Audio Verification System with AI: Our Journey

01 Sep 2025

In digital lending, fraud prevention is just as important as credit risk analysis. One critical challenge is verifying that the person applying for a loan is who they claim to be—and that they are genuinely present during verification.

Our goal was twofold:

• Match a customer’s profile photo with their live video
• Verify that the customer reads a given sentence aloud (as per regulatory guidelines)
• Ensure the spoken text matches the expected script

What sounds straightforward turned out to be a multi-step AI challenge.

Breaking Down the Problem

We realized the system had two main parts:

1. Face Matching – confirm the person in the video is the same as the profile photo.
2. Audio-to-Text Verification – ensure the customer actually speaks the required text.

Face Matching: Getting Identity Right

Face matching isn’t just comparing two pictures. Lighting, angles, and video quality can make the same person look very different. To solve this, we used a CNN-based face recognition model. Instead of comparing raw pixels, the model generated facial embeddings, which made matches more reliable even under real-world noise and variations.

This gave us a strong foundation for the “is this the right person?” step.

Audio-to-Text: The Bigger Challenge

The second step was trickier: checking if the customer actually spoke the expected line.

Attempt 1: Google Speech-to-Text

We first tried Google’s API. Accuracy was decent, but latency was too high. In real-time verification, every extra second hurts the user experience.

The Breakthrough: Whisper

We switched to OpenAI Whisper (small model) deployed on our local server. Suddenly, transcription became faster and more reliable. We then fine-tuned Whisper on our own dataset—banking-related sentences, regional accents, and noisy conditions. Finally, we optimized it with CTranslate2, which cut inference time dramatically. This made the system not just accurate, but fast enough for production.

Text Matching: Final Verification

Once we had the transcribed audio, the last step was simple: compare it against the predefined script. This ensured that:

• The person matched the profile photo
• They were physically present in the video
• They actually spoke the correct verification sentence

With all three checks combined, fraudsters had a much harder time bypassing the system.

Deployment & Learnings

By the end, we had built a pipeline that was:

• Accurate: CNN-based face recognition + Whisper fine-tuning
• Fast: Optimized with CTranslate2 for low latency
• Practical: Integrated seamlessly into customer verification workflows

Key Lessons

• Don’t rely on APIs blindly—latency can make or break user experience.
• Local deployment ensures both speed and data security.
• Fine-tuning on domain-specific data gives big accuracy gains.
• Optimization tools like CTranslate2 turn research models into production-ready systems.

Conclusion

What started as a simple idea—“match a face and check if they said the right thing”—turned into a journey through face recognition, speech-to-text, fine-tuning, and deployment optimizations.

Today, our AI-powered verification system ensures a safer, smoother loan application process for customers while helping businesses fight fraud more effectively.