AI Capabilities: What the Device Does and What the Server Does
The XiaoZhi firmware is the entry point of a voice conversation, not a runtime for large models. The device handles wake-up, audio processing, encoding and decoding, display and tool execution; speech recognition (ASR), LLM dialogue and speech synthesis (TTS) are done by the server. This page follows the firmware v2.5.2 source and upstream documents to show where that line sits, where each switch lives, and a few common misconceptions.
Flow of One Conversation
flowchart LR
A[Wake word or button] --> B[Device: audio capture and processing]
B --> C[Opus encoding, uplink]
C --> D[Server: ASR, LLM, TTS<br>or Realtime end-to-end model]
D --> E[Opus downlink]
E --> F[Device: decode and play]
D -. stt / llm / tts messages .-> G[Device: show text and expression]
D -. MCP calls .-> H[Device: run tools]
- Uplink: the device encodes mono 16 kHz microphone audio as Opus (60 ms frames) and sends it to the server.
- Downlink: the server sends Opus audio; the sample rate in the protocol document’s example is 24000 Hz.
- JSON messages:
sttcarries the recognized text,llmcarries the expression (emotion), andttsmarks the start of speech synthesis, each sentence’s text and the end. See the WebSocket Communication Protocol for the fields. - Dialogue approaches: both the traditional streaming ASR + LLM + TTS pipeline and a Realtime end-to-end speech model are supported, and both are implemented on the server.
What the Device Handles
| Stage | What the device does | Conditions and switches |
|---|---|---|
| Wake word | Offline detection based on ESP-SR: WakeNet, and MultiNet for custom wake words; the ESP32-S3 default config uses the “你好小智” model | Kconfig WAKE_WORD_TYPE; a custom wake word needs ESP32-S3 / P4 / S31 with PSRAM; write it in pinyin separated by spaces, threshold 1–99 (smaller is more sensitive, default 20) |
| Echo cancellation and VAD | ESP32-S3 / P4 / S31 use AFE, with AEC and VAD when audio processing is on; ESP32 / C3 / C5 / C6 output raw mono PCM without AEC or VAD | USE_AUDIO_PROCESSOR is on by default; device-side AEC USE_DEVICE_AEC is off by default and selectable only on the boards listed in Kconfig; server-side AEC is marked unstable in Kconfig |
| Coding and transport | Opus encoding and decoding; WebSocket or MQTT + UDP | The server picks the protocol in the OTA response, see First Boot |
| Listening mode | Auto stop, manual stop, Realtime; Realtime requires AEC support | The mode of the listen message is auto, manual or realtime |
| Wake-word audio upload | After wake-up, the most recent ~2 seconds of audio are sent to the server as the first data of the conversation | SEND_WAKE_WORD_DATA is on by default, available only with the AFE wake word and custom wake word |
| Display and expressions | Switches expressions according to emotion in the llm message | See Emoji Emotion Display |
Which wake word types, audio engines and PSRAM requirements apply to each chip is in Chip Support and Storage Specs; extension entry points are in Advanced Extension Points.
Device Control and the Camera
Device-side MCP lets the LLM call tools on the device (volume, lights, motors, GPIO and so on); cloud-side MCP extends the LLM itself. For the protocol and usage see the MCP documentation.
Boards with a camera register the tool self.camera.take_photo, whose parameter is question. When the LLM decides to call it, the device takes a photo, encodes it as JPEG, and POSTs it as multipart/form-data (fields question and file) to an address given by the server, then returns the response text as the tool result. The address and token are delivered by the server in capabilities.vision of the MCP initialize request, see the MCP protocol; without them the tool returns “Image explain URL or token is not set”. Image understanding is done on the server.
Voiceprint Recognition
The upstream README lists voiceprint recognition based on 3D Speaker as an implemented feature. The v2.5.2 firmware source contains no voiceprint code and mentions it only in the README and other documents, so whether it is available depends on the documentation of the server you connect to.
Interface Languages
The firmware offers 40 options under Default Language in menuconfig, and main/assets/locales also holds 40 language directories; the upstream README text says 38. A language directory (for example en-US) contains OGG audio for digits, welcome, activation, Wi-Fi setup, upgrade, PIN and registration errors, plus language.json. Voice prompts prefer localized resources and fall back to English when they are missing. Wake words, fonts, expressions and chat backgrounds can be customized online with the upstream Assets generator, see the Custom Assets Guide.
Common Misconceptions
- “The device runs a large model or TensorFlow Lite locally”: the v2.5.2 repository has no TensorFlow Lite files, and the firmware source and dependency list contain no ESP-DL either. The only on-device models are the ESP-SR wake word and AFE audio processing; the conversation is handled by the server.
- “The firmware ships integration for a particular LLM or speech vendor”: the firmware only implements the protocols for talking to a server (WebSocket, MQTT + UDP). Which model is used depends on the server configuration; for the difference between the official server and a self-hosted one see Self-Hosted Server.
- “Recognition accuracy, latency and power have standard numbers”: upstream publishes no device-level accuracy, latency or power figures; they depend on the server, the network and the board. This site does not provide numbers without an upstream source.
Keep Reading
- Feature Usage Tutorial: what the firmware implements;
- Self-Hosted Server: run recognition, dialogue and synthesis on your own server;
- WebSocket Communication Protocol and MQTT + UDP Protocol: the messages between device and server;
- Chip Support and Storage Specs: audio engine and wake word options per chip.
Source: 78/xiaozhi-esp32 v2.5.2: main/audio/README.md, main/Kconfig.projbuild, main/mcp_server.cc, main/boards/common/esp32_camera.cc, sdkconfig.defaults.esp32s3 and README.md.