Khata-Vaani: A Voice Agent That Keeps a Shopkeeper's Ledger, So They Don't Have To

Built for the 10 Days of Voice Agents — VoiceForBharat Edition, by Murf AI

The problem, and who it's for

Most small shopkeepers in India — the kirana store owner, the vendor at the corner — still track udhaar (credit given to regular customers) in a paper notebook, or worse, in their head. They don't sit down at the end of a 12-hour day and open an app to type in what they sold and who owes them money. Typing is friction they don't have time for. Talking is not.

That's the gap Khata-Vaani fills: a voice agent, built for the Financial Services track, that lets a shopkeeper log a sale or a credit entry just by speaking — in the middle of the mixed Hindi-English they already speak — and later ask "who owes me how much" and get a real answer.

Voice isn't a gimmick here. It's the only interface that fits how this user actually works.

What the agent does

At its core, Khata-Vaani:

• Logs a sale or an udhaar entry from a spoken sentence, and always reads it back for confirmation before saving anything
• Recalls what's owed, by customer, across calls — not just within one session
• Checks eligibility for relevant government schemes (like PM SVANidhi) using data it already has, and hands deeper scheme questions to a specialist agent
• Escalates to a human when something is outside what it should handle on its own — a suspected fraud report, or a judgment call like whether to write off a customer's debt
• Calls the shopkeeper back (with consent) when a scheme deadline is approaching

How the system works

Four pieces, wired together with LiveKit for real-time transport:

• Speech-to-text: Deepgram (nova-3), configured for multilingual/code-switched input — shopkeepers mix Hindi and English mid-sentence, and the STT layer has to handle that without being told to pick one language.
• LLM: drives the conversation logic, tool calls, and the handoff decision to the specialist agent.
• Text-to-speech: Murf Falcon, the fastest TTS API I used in this build, speaking in an Indian English voice.
• Real-time transport: LiveKit, for both the browser-based agent and outbound phone calls (via Twilio/SIP).

# Simplified shape of a tool the main agent can call
@function_tool
async def save_customer_entry(name: str, amount: float, entry_type: str):
    """Log a sale or udhaar (credit) entry for a customer.
    Always confirm the details with the shopkeeper before calling this."""
    # writes to SQLite, tied to the shopkeeper's profile
    ...

The most important features

• Guardrails from day one, not bolted on later. The agent will never ask for an OTP, PIN, or account number, under any framing — because a bookkeeping tool that asks for that is indistinguishable from a scam call. It also never confirms a loan or scheme "approval" — only whether published criteria are met, with a clear line that final approval happens through the official channel.
• Memory with consent. Every time it's about to save something, it says out loud what it's saving and asks first. If the shopkeeper says no, nothing is written.
• A specialist handoff. The main agent stays focused on logging and recall. Deeper government scheme questions go to a separate specialist, "Scheme Sahayak," which picks up the same conversation without making the shopkeeper repeat themselves — and hands it back when the topic returns to bookkeeping.
• A call analytics dashboard. A call only counts as successful if it actually logged an entry or answered a balance query — not just "the conversation didn't crash." Failures are categorized (user declined, hung up early, tool error, no response), pulled from real call data, not hardcoded.

The hardest part, and what I got wrong

The first real STT config I shipped set Deepgram's language explicitly to hi-Latn, assuming that was the right way to handle Hindi-English code-mixing. It wasn't — Deepgram's actual supported path for code-switched speech under nova-3 is language="multi", combined with a multilingual turn-detection model. I didn't catch this until a later day's task documentation pointed directly at the correct config, and going back to compare, the hi-Latn setting was never going to reliably parse a sentence like "aaj maine Ramesh ko 300 rupaye ka udhaar diya."

The lesson: don't trust a plausible-looking config value just because it doesn't throw an error. Test the actual failure mode (a real code-mixed sentence) before assuming a language setting works, and re-verify against the current provider docs rather than what seems logically right.

Building your own: where to start

If you want to build something similar:

1. Start from a working scaffold, not from zero. I used murf-livekit-starter, which already wires up LiveKit + Murf Falcon TTS.
2. Set up your environment file first. Copy .env.example to .env.local in both the backend and frontend, and never commit it. You'll need a Murf API key, LiveKit Cloud credentials, and whatever STT/LLM provider keys the starter requires.
3. Run backend and frontend separately — the backend is the LiveKit agent process, the frontend is the browser client that connects to a room. Both need to be running to test end to end.
4. Test with your own voice before you test with a demo script. Guardrails and language handling only prove themselves against messy, real speech — not a clean scripted sentence.

Repository: https://github.com/DheerGupta35959/khata-vaani 

What I'd improve next

The memory layer is still SQLite and single-shopkeeper-per-session — a real deployment would need proper multi-tenant storage and probably a lighter-weight confirmation flow so confirming every single entry doesn't get tedious for a shopkeeper logging twenty sales a day. The scheme-eligibility dataset is also hand-built and needs a real refresh process rather than being static.

Built as part of 10 Days of Voice Agents — VoiceForBharat Edition by Murf AI, using Murf Falcon.