VSSV Aditya Ram
Client POC · built at Tata Consultancy Services · 2026

Case study

One assistant.
Chat or voice.

A TCS client proof of concept for account questions across chat and four spoken languages. I owned the architecture, implementation and Azure deployment, using AI pair programming throughout.

Explore the flow ← Back to work

Follow a question through the system.

An illustrative walkthrough with synthetic examples. Choose a question to see why the route changes.

Example conversation

What is my balance?

Your available balance is R184.50.

Sample source: account balance. One lookup; no multi-step graph.

What happens behind the answer

  1. Route a clear balance question to the approved balance tool.
  2. Fetch account data through the backend-owned customer identity.
  3. Format the returned amount into a deterministic answer.

Illustrative interface, based on the documented POC. The examples are synthetic; this preview does not connect to client systems or record audio.

The shape of it

Deterministic first, a model only where it earns its place.

One FastAPI backend serves both chat and voice, and every real account lookup in either path goes through the same allow-listed tool layer. Five distinct LLM call sites exist in the whole system; everything else — routing thresholds, the tool allow-list, the answer templates — is plain Python with no model involved.

A broad question ("what's my balance") costs one model call and a template. A narrow one ("is my phone 5G?") costs two. Only a question that genuinely needs several lookups at once ("am I eligible for 5G?" depends on both the plan and the device) enters the LangGraph graph: a plan node picks the tools, a deterministic fetch node calls them concurrently, an answer node writes the reply grounded in what came back.

When the router isn't confident, the assistant asks one clarifying question and calls nothing. No telecom API is ever hit on a guess.

RoleSolo — architecture, backend, frontend, Azure deployment, testing; AI pair-programmed throughout
Runs onA 1 GB Ubuntu Azure VM behind nginx; no local models by design
ModelsAzure OpenAI GPT-5-series for routing, planning and answers; a Realtime voice model; a live-transcription model; a TTS model for read-aloud
OrchestrationLangGraph (plan → fetch → answer) in the current build; a self-hosted Microsoft Agent Framework version is still under evaluation
State & searchAzure Cosmos DB for NoSQL (one document per conversation); Azure AI Search over history
Voice transportWebRTC from the browser straight to Azure; a self-hosted TURN relay for networks that block UDP
Verification149 fully-mocked unit tests; a 47-check live smoke suite run against the deployed VM
Languages (voice)English, Hindi, Telugu, Tamil — detected and switched mid-call

Architecture

Two front ends, one gate, five tools.

System architecture Browser chat and voice both reach a FastAPI gateway. Chat goes through an intent router that either calls a tool directly, asks a clarifying question, or runs a LangGraph plan, fetch, answer graph. Voice connects the browser directly to Azure's Realtime API over WebRTC and calls back to the gateway for tools. All tool calls pass through one allow-listed tool layer to the telecom APIs. Cosmos DB stores conversations and Azure AI Search indexes them. VOICE PATH CHAT PATH Azure OpenAI Realtime speech in/out, decides tool calls audio: browser ↔ Azure directly, never via the backend Browser chat + voice, WebRTC FastAPI gateway owns identity chat turns · voice token · voice tool calls Intent router (LLM) intent · confidence · scope Direct tool call high confidence → template Ask one question low confidence → no API hit LangGraph graph plan → parallel fetch → answer Allow-listed tool layer 5 read-only tools, nothing else Telecom APIs voice tool calls → the same tool layer Cosmos DB conversation history Azure AI Search search over history
Redrawn for this page from the project's own architecture document; names of the client, endpoints and hosts omitted.

Five model call sites

Where an LLM actually decides something.

"Agent" here means a point where a model is given a prompt and asked to decide or generate. Everything not in this table is deterministic code.

Call siteFires whenDecides or produces
RouterEvery chat turnIntent, confidence, scope (broad or specific), candidate intents, and a clarifying question if uncertain — all in one call.
Plan nodeOnly when the router says the question is complexWhich of the five tools are relevant.
Answer nodeAfter the plan's tools have been fetched, concurrentlyThe final answer, grounded in the real data from every tool that succeeded.
Narrow synthesizerScope is specific ("sms", "is my phone 5G")A short answer extracting just the one fact asked about from one tool's already-fetched data.
Realtime voice modelFor the whole duration of a callThe spoken conversation: understanding audio, choosing when to call a tool, speaking, matching the caller's language.
TranscriptionEvery voice utteranceCaptions only — no reasoning. Its output drives the client-side language detection below.

Approaches explored

Four ways to build the same assistant.

The requirement was fixed; the agent stack wasn't. Each was built far enough to see how it behaved against the constraints that mattered here: latency per turn, control over what the model can call, and running on a very small VM. The last two are still being weighed against each other.

01

Azure AI Foundry prompt agent with OpenAPI tools

The telecom APIs exposed to a Foundry agent as OpenAPI tools, with the UI rendering the JSON the agent returned according to prompt instructions. Fastest to stand up. For this POC it gave the least control over routing decisions, and every answer's shape lived in a prompt.

02

Foundry hosted agents, built in code with the Microsoft Agent Framework

The same agent defined in code with MAF and hosted inside Foundry. Cleaner than prompt-only, and the hosting handled a lot. In this case the per-turn latency and the platform's own handling of state sat awkwardly with a voice-first experience.

03

Microsoft Agent Framework, self-hosted

MAF without the hosted runtime, on our own gateway. Recovers the control the hosted version gave up, and it's still on the table — explored alongside the LangGraph build rather than ruled out. The call between the two hasn't been made yet.

04

FastAPI + a confidence-gated router, LangGraph only for the hard cases

Plain Python owns routing, clarification and the allow-list; LangGraph runs only the plan→fetch→answer graph for multi-lookup questions; voice goes to Azure's Realtime API directly. Lowest latency, tightest control, and light enough for a 1 GB VM. This is what the current build runs on.

What was actually hard

The parts that took several rounds to get right.

Language switching mid-call

Left to itself, the voice model kept whatever language the previous turn was in — a caller who switched from Hindi to Telugu got Hindi back, once with an explanation of why. Telling it the language after the fact was worse, because the transcript arrives after it has often already started answering. The fix changed who controls turn-taking: server-side voice detection still commits the utterance but no longer auto-replies; the browser waits for the transcript, detects the script by Unicode block (filler words deliberately below the threshold so "haan" can't flip the session), sends the model an explicit current-language instruction, and only then asks for a response. About a second per turn, in exchange for the language being a stated fact instead of a guess.

An Azure API that replaces instead of merges

Enabling transcription at session-creation time returned a deployment-not-found error — a platform quirk, not a config mistake — so transcription had to be switched on after connecting, over the data channel. Then the voice kept silently resetting. Cause: the update event replaces nested objects wholesale rather than merging them, so a partial update that only mentioned transcription wiped the voice setting. Both payloads now come from one shared config function, so nothing can drift between them.

Voice through a corporate network

Calls worked on phones and personal laptops and failed on managed corporate ones — twenty seconds stuck connecting, then dropped. A network capture pinned it on outbound UDP being blocked, which STUN can't route around. The fix is a self-hosted TURN relay on the same VM: the browser opens one outbound TLS connection to it that looks like ordinary HTTPS, and the VM relays to Azure over UDP itself. Credentials are short-lived HMAC tokens minted per call, mirroring how the Azure key never reaches the browser either.

Small box, honest failure modes

Everything runs on a 1 GB VM, so there is no local model and no room for a heavy agent runtime — a real input to the current build running on approach four above. Any model failure (timeout, malformed JSON, API error) degrades to a defined fallback — low confidence, an empty plan, a polite "couldn't fetch that" — never a guess and never a crash. The customer's identity is a field the browser sends and the backend owns; neither model can supply or override it.

Also in the build

Product surface, not just plumbing.

  • Persistent, resumable conversations — a refresh restores the open thread; a recents list and full-text search across history.
  • Rich data cards for balance, plan, device, purchases and offers, with screen-reader text kept alongside them.
  • A personalised first turn that still answers the actual question, with live account figures appended.
  • Thumbs up/down feedback persisted per message; read-aloud on every reply with a browser-speech fallback.
  • A live mic waveform, an in-page debug drawer that logs every raw realtime event (which is how the Azure quirks above were found without a debugger), light/dark theme, no framework, no build step.

Stack

PythonFastAPIAzure OpenAIRealtime APILangGraphAzure AI Foundry (explored)Microsoft Agent Framework (explored)Azure Cosmos DBAzure AI SearchAzure VMWebRTCcoturn (TURN)nginxsystemdLet's EncryptpytestVanilla JS
This was client work. What's shown here is the engineering; the client, its API contracts, hostnames and internal tracking references are intentionally left out. Happy to go deeper in conversation.
Talk about this work All work