01 / 06 · LLM systems
Flagship · in production
A talking avatar that answers from a company's own documents
A conversational avatar on a kiosk and on the web, backed by Claude, with hybrid retrieval over the client's own material and speech going both ways. It runs in two modes, and the difference is who does the work: in full mode the vendor handles the speech recognition, the model and the voice. In lite mode it renders video only, at half the per-minute rate, and everything else is ours. The kiosk runs on lite.
The retrieval is genuinely hybrid: a dense vector search over pgvector alongside Postgres full-text ranking, fused so an exact procedure name and a vaguely worded question both land. Source links travel with each passage, so an answer cites something real.
- Claude
- Bedrock
- pgvector
- LiveAvatar
- Whisper
- ElevenLabs
- Polly
- Rekognition
- FastAPI
Decisions worth showing 8
- The model answers from the documents, except on safetyA document-authority rule with a deliberate carve-out, so it never invents policy but never withholds a safety answer either
- An in-process vector cacheHolds a persona's embeddings in memory to delete a trans-Pacific round trip on every question
- Halving the vendor bill meant building what it stopped doingThe cheaper mode was not a switch to flip. It needed a whole speech stack on our side: the visitor's own phone as the microphone, streaming speech to text, and our own voice. Cheaper is a trade, not a discount
- A visual glitch diagnosed frame by frameThe avatar appeared to cut to a different shot mid-sentence. Scene detection over a screen recording found exactly three single-frame jumps, all lining up with pose commands, which showed the renderer hard-cuts on a pose change rather than blending. Two of the three were ours to remove
- It answers in the language you ask inSpeech in and out across languages, which is the point on a floor where not everyone reads English
- Referential query rewritingBecause "say that again in Hindi" retrieved on its own five words, and a helmet question got answered with the nine signs of burnout
- Found nothing is not the same as could not runA three-state fallback, so only a genuine failure ever dumps the whole corpus at someone
- Authorisation in the API, never the promptShort-lived single-audience tokens for tool calls, on the assumption a model will eventually be talked into trying something it was not granted