audio8-tts-server

CPU-only ONNX Runtime inference server for Audio8-TTS-Preview-0.1B-ONNX-INT8 (and sibling ONNX checkpoints with the same per-token graph layout).

Unlike upstream's onnx_runtime, which targets the 0.6B INT4 export, this server drives the 0.1B per-token slow-AR graph (slow_ar_int8.onnx) that passes attention and Mamba (conv/SSM) states explicitly, and re-streams the prompt token-by-token for prefill (a batched/static prefill contaminates the recurrent states and produces garbage — see the shared insight from the earlier ONNX experiments).

Features

• OpenAI-compatible POST /v1/audio/speech (wav / pcm responses).
• pcm responses carry X-Sample-Rate (and channels/format) so clients can open AudioTrack at the right rate instead of assuming 24 kHz.
• Voice cloning from an inline ref_audio (base64 data-URL WAV) + ref_text, or from an already-registered voice profile by name. Accepts arktts, tts-1, and qwen3-tts* model names.
• Encoded references are cached (keyed by audio hash); the reference+prefix slow-AR prefill is snapshotted once and reused for every sentence, removing the ~2s fixed prefill cost from each subsequent synthesis.
• Persistent voice profiles live in voices/; register via POST /api/voices/register (multipart) or the web UI at /.

Model layout

The model files are not bundled. Point ARKTTS_MODEL_DIR at a directory with:

model/
├── slow_ar_int8.onnx(.data)
├── fast_ar_int8.onnx(.data)
├── codec_decoder_fp16.onnx(.data)
├── runtime_manifest.json
├── tokenizer/tokenizer.json
└── registration/
    ├── codec_encoder_fp16.onnx(.data)
    └── registration_manifest.json

Download once with:

python3 -m pip install -U "huggingface_hub[cli]"
hf download Audio8/audio8-TTS-0.1B-ONNX-INT8 --local-dir model

Run

bash setup.sh               # creates .venv with pinned deps
bash run_server.sh          # serves http://127.0.0.1:8024

Environment:

Variable	Default	Purpose
ARKTTS_MODEL_DIR	./model	Downloaded Hugging Face model
ARKTTS_VOICES_DIR	./voices	Registered voice profiles
ARKTTS_THREADS	5	ONNX Runtime CPU threads (more is slower here)
HOST	127.0.0.1	Listen address
PORT	8024	Listen port

setup.sh installs the upstream requirements.txt pins into a .venv.

Notes

• NixOS: the prebuilt manylinux wheels need libstdc++.so.6 / libz.so.1. shell.nix provides them via a dev shell; a Nix package built against nixpkgs' own python3Packages.onnxruntime gets them automatically and needs no LD_LIBRARY_PATH.
• Threads beyond ~5 make the per-token graph slower (inter-op overhead on a tiny sequential graph), so 5 is the sweet spot on a 32-thread Zen 5.

License

Apache-2.0, as the upstream Audio8-TTS repository.