# Build a real-time voice agent

> Connect streaming speech to text, your own language model and streaming text to speech into a conversational voice agent. Architecture, audio formats, turn-taking and pitfalls.

## Architecture

```text
microphone --16 kHz PCM--> wss /api/v1/stt/ws/ --final text--> your LLM
speaker   <--PCM audio--- wss /api/v1/tts/ws/ <--reply text--- your LLM
```

Moknah provides the two streams. The language model is yours.

## Requirements

- An API key that allows `stt_ws` and `tts_ws`.
- Credits: speech to text is prepaid by the minute, text to speech is charged per character as text is sent.
- One open stream per account for each of the two services.

## Steps

| Step | Do | Check |
| --- | --- | --- |
| 1 | Open the speech to text stream with `?language=` set to the caller's language and region. | You receive `{"status": "connected", ...}`. |
| 2 | Send microphone audio as raw 16 kHz, 16-bit, mono PCM frames of about 100 ms. | `partial` messages arrive while someone is speaking. |
| 3 | On each `final` message with non-empty text, send the text to your LLM. | |
| 4 | Open the text to speech stream, send `init` with a `voice_id` and `output_format` `pcm_24000`, and wait for `ready`. | You receive `{"status": "ready", ...}`. |
| 5 | Send the reply as `text` messages, sentence by sentence as the LLM produces them, then `stop`. | Binary audio frames arrive. |
| 6 | Play the frames as 24 kHz, 16-bit, mono PCM. Stop at `end_of_stream`. | The reply is heard completely. |
| 7 | While the agent is speaking, do not forward microphone audio, or the agent will hear itself. | No echo loops. |

## Minimal agent

```python
# pip install websockets pyaudio
import asyncio
import json
import pyaudio
import websockets

API_KEY = "YOUR_API_KEY"
HEADERS = {"Authorization": f"Bearer {API_KEY}"}
STT_URL = "wss://moknah.io/api/v1/stt/ws/?language=ar-SA"
TTS_URL = "wss://moknah.io/api/v1/tts/ws/"
VOICE_ID = 1710  # use an id from your own voice list

audio = pyaudio.PyAudio()
microphone = audio.open(format=pyaudio.paInt16, channels=1, rate=16000,
                        input=True, frames_per_buffer=1600)
speaker = audio.open(format=pyaudio.paInt16, channels=1, rate=24000, output=True)
speaking = False


async def call_llm(text):
    return f"سمعتك تقول: {text}"  # replace with your model


async def speak(text):
    global speaking
    speaking = True
    loop = asyncio.get_running_loop()
    try:
        async with websockets.connect(TTS_URL, additional_headers=HEADERS) as tts:
            await tts.send(json.dumps({"type": "init", "voice_id": VOICE_ID,
                                       "output_format": "pcm_24000"}))
            while json.loads(await tts.recv()).get("status") != "ready":
                pass
            await tts.send(json.dumps({"type": "text", "data": text}))
            await tts.send(json.dumps({"type": "stop"}))
            async for message in tts:
                if isinstance(message, bytes):
                    await loop.run_in_executor(None, speaker.write, message)
                elif json.loads(message).get("type") == "end_of_stream":
                    break
    finally:
        speaking = False


async def send_microphone(stt):
    loop = asyncio.get_running_loop()
    while True:
        frame = await loop.run_in_executor(None, microphone.read, 1600, False)
        if not speaking:
            await stt.send(frame)


async def handle_results(stt):
    async for message in stt:
        data = json.loads(message)
        if data.get("type") == "final" and data["text"].strip():
            await speak(await call_llm(data["text"]))


async def main():
    async with websockets.connect(STT_URL, additional_headers=HEADERS) as stt:
        await asyncio.gather(send_microphone(stt), handle_results(stt))


asyncio.run(main())
```

## Latency

- Use `pcm_24000` for playback; there is nothing to decode.
- End each text chunk with sentence punctuation so audio starts at once.
- Keep the default `emotion` mode, or set `fast_mode` for the low-latency model.
- Open the text to speech stream before the LLM answers if every millisecond counts, and send text as it arrives.

## Pitfalls

| Symptom | Cause | Fix |
| --- | --- | --- |
| Close code `4290` | A stream of the same service is already open for the account | Close the previous stream first, and wait about 15 seconds after a crash. |
| Empty or nonsense transcripts | The audio is not raw 16 kHz mono 16-bit PCM | Convert before sending. Do not send MP3 or a WAV header. |
| Close code `4009` | An audio frame was too large | Send frames of about 100 ms. |
| The agent answers itself | Microphone audio was forwarded while the agent spoke | Pause forwarding during playback. |
| Close code `4002` during a call | Credits ran out | Check the balance before starting long sessions. |

Message formats and every close code are in [Stream transcription](../api/stt-stream.md) and [Stream speech](../api/tts-stream.md).
