agent-presentation

An agent that does not write its answer back: it presents it. The model answers each turn with a short beat score (say + show + wait), and the browser performs it with per-beat speech and catalog visuals. A separate judge decides whether speech during the presentation is a real interruption. When it is, the next turn learns where the presentation stopped.

Planning: openspec/changes/add-narrated-presentation-turns/.

Running locally

All models are reached through an OpenAI-compatible litellm endpoint (litellm_url includes /v1).

cp dev.example.json dev.json            # dev.json is git-ignored; edit as needed
# LITELLM_MASTER_KEY (and HERMES_API_KEY) come from .envrc via direnv: ssh-tresor
# blobs in ~/.config/agent-presentation/, sealed to your SSH agent key. The
# first `direnv allow` explains how to create them from nixcfg's sops secret.
nix develop -c python -m agent_presentation.server --config dev.json --port 8765
# or, from the built package:
nix run . -- --config dev.json --port 8765

Then open http://127.0.0.1:8765/ (the microphone needs localhost/127.0.0.1 or HTTPS).

Tests:

nix develop -c pytest -q
nix develop -c node --test 'tests/web/*.test.mjs'

Browser smoke test with fake models (no key needed): loads the page in headless Chromium, checks the console for CSP violations, plays a typed turn and interrupts a second one with Escape.

nix develop .#smoke -c python scripts/browser_smoke.py /tmp

Measured against sgx litellm (2026-10-08)

contract.sh with dev.example.json (Gemma deutsch, qwen3-asr, qwen3-tts voice eddie):

Check	Latency
TTS, one sentence (2.4 s of audio)	396 ms
STT, 2.4 s utterance	86 ms
Presenter, whole 12-beat score (non-streamed)	1280 ms
Judge, 4-token verdict	105 ms

Gemma follows the score syntax well: cards, bars, tables, bullets, facts and images in sensible places. It does invent image URLs (handled by the media placeholder) and copies the example's wait 1s after most beats.

scripts/judge_eval.py on tests/judge_cases.jsonl (24 German utterances), judge = deutsch: 21/24 verdicts correct, 24/24 on interrupt vs. keep presenting. The misses confuse noise and backchannel, which behave the same. Median model-call latency is 45 ms (max 71 ms). This settles the design's open question: Gemma is fast and accurate enough as the judge, so no separate model is needed. The judge prompt leans towards interrupt when in doubt. Without that rule, a correction ("Nein, ich meinte …") and a question about a playing clip were dismissed as noise.

scripts/e2e.py against the live models, with espeak recordings: the spoken question became a turn with the transcript 102 ms, the first beat 308 ms and the first audio 1.17 s after the end of speech. A spoken "Warte, wer hat 2023 gewonnen?" during beat 2 ducked playback within 80 ms. The judge answered interrupt, and the next turn received the note and answered directly ("Das war der Norweger Jon Fosse."). The verdict came 4.6 s after the duck, most of it the 3.5 s utterance itself plus the 700 ms endpoint silence. The first run exposed an echo-check false positive (single-word overlap), now fixed by comparing word pairs.

Research through hermes (add-research-tool)

scripts/research_probe.py through the pass-through http://sgx:4001/hermes-agent/v1, three-day weather question: first event (memory recall status) 3.6 s, tool progress (terminal, a curl to open-meteo) 5.0 s, first content 7.3 s, first beat 7.6 s, done 9.4 s. 4 beats, all with visuals (card, table, bars, fact), no validation issues.

scripts/e2e_research.py against the live models (real Session, VAD, STT, Gemma, judge, TTS and hermes in-process):

• Spoken "Wie wird das Wetter morgen in Erlangen?": Gemma announced and asked for research. hermes recalled memories, ran curl against open-meteo, and delivered 5 beats (card, fact, table, bars, speech), all synthesized. History holds Gemma's announcement, the research line and hermes's score.
• Typed follow-up "Und wie sieht es am Sonntag aus?": Gemma researched again with the city from history. Research start to first hermes activity 3.5 s, to first hermes beat 7.4 s, done 9.2 s.
• News question interrupted by voice while hermes was reading tagesschau.de: judged interrupt, research cancelled, no hermes beat presented, and the next turn received "… während der Recherche zu „Was sind die wichtigsten Nachrichten aus Deutschland von heute?" unterbrochen … Das Ergebnis wurde verworfen."

By-ear session (2026-10-09, Firefox 157, RØDE NT-USB Mini, HDMI speakers)

• 11 spoken turns in about 5 minutes, including two research runs and spoken interruptions judged correctly. Transcript 57–249 ms and first audio 283–729 ms after the end of speech. duck_level 0.25 and endpoint_silence_ms 700 kept as they are.
• USB mic behind a hub stalls. With the RØDE (Full Speed) on a USB 2.1 hub, the kernel stopped delivering capture frames after about 26 s (ALSA hw_ptr frozen while RUNNING), so the page sent only digital silence. Plugged directly into the PC it ran without problems. The mic meter now shows the input level and warns after 3 s of pure silence.
• Fixed during the session: the mic button now names its action ("Zuhören starten"), and a voice interruption no longer leaves the next presentation ducked to 25 %.

Judge fixes after the by-ear session (2026-10-09)

Two misjudgements from the live debug panel, reproduced with the live judge:

• "Du sollst mir nur das Foto zeigen, bitte, sonst nichts." was judged noise whenever nothing was being spoken (clip playing, research running, silent beat) and the request did not match the previous question: the judge took it for side talk. The prompt now counts requests to the presentation as interrupt even when they start a new topic. Fixed (the cases are in tests/judge_cases.jsonl).
• Echo fragments from the speakers ("getroffen.", "wohl die Politik und so.") were judged interrupt while the presentation was speaking. Prompt rules and worked examples both made the judge worse, so the fix is deterministic: a short fragment (≤ 4 words, not a question) made only of words just spoken counts as echo before any model call.
• Still open: misrecognized echo whose words the presentation never said ("das Opizero.", "Ich mag dich.") cannot be told from the user by text. Three eval cases document it. It needs better acoustic echo cancellation in the browser.

judge_eval.py now: 30/33 on interrupt vs. keep presenting, every real interruption detected. The 3 misses are the misrecognized-echo cases above.

tune-interruption-judge (2026-10-09)

Live, while the presentation waited for research, "Wie wird denn das Wetter diese Woche?" was judged noise: the judge read a new topic as side talk. Changes, measured on the 35-case set (interrupt vs. keep presenting):

Judge	Result
before	30/35
without the user's last turn (fixed "-" placeholder kept)	33/35
plus "silence favours the user" (noise → interrupt while nothing is spoken)	34/35

The one miss is the accepted trade-off: "Schatz, kommst du mal?" during research now cancels the research. The debug panel shows such verdicts with reason silent. Median judge latency 46–49 ms.

Presenter and small talk (5 samples per sentence at temperature 0.7): before the rule, 1/15 greetings, thanks and small-talk replies started research ("Guten Morgen" → news). After "Nie für Begrüßungen, Dank oder Smalltalk", 0/30 over two runs, while "Wie wird das Wetter morgen in Erlangen?" still researched 10/10.

add-session-context (2026-10-09)

Both lanes get a line like Heute ist Freitag, 9. Oktober 2026, 12:47 Uhr (Europe/Berlin). Standort: Vaterstetten (48.10, 11.77). It is computed per turn from the browser's time zone and location. The page asks for the location once, on the first action, never on load. Coordinates are rounded to about 1 km and turned into a place name via Nominatim. Without a browser location, the configured one applies (default Vaterstetten). Live: "Wie wird das Wetter morgen hier?" became the research question "Wie wird das Wetter morgen in Vaterstetten bei München?", and hermes answered with a forecast for Vaterstetten.

add-profiles: judge on Qwen3.8-Flash-Next (2026-10-09)

Both profiles judge with qwen3.8-flash-next without thinking (the user's choice). On the 35-case set it matches Gemma on the decision that matters:

Judge	Interrupt vs. keep presenting	All verdicts	Median latency
deutsch (Gemma 4 26B)	34/35	31/35	48 ms
qwen3.8-flash-next, no thinking	34/35 (two runs)	29/35	115 ms

Qwen's extra misses rate echo fragments as backchannel instead of noise, which behaves the same (the presentation continues). The one decision miss is the accepted silence trade-off ("Schatz, kommst du mal?" during research). The judge also no longer waits on Gemma's single slot while Eddie generates.

Profiles compared live (2 runs each, same two questions, session context Friday 9 October 2026, Vaterstetten):

Profile	First beat	Score issues per run	"Wie wird das Wetter morgen hier?" → research question
eddie (Gemma 4 26B)	0.15–0.29 s	0, 9, 1, 0	"morgen in Vaterstetten bei Koordinaten …", "am 10. Oktober 2026 in Vaterstetten"
neutral (Qwen3.8-Flash-Next, no thinking)	0.30–0.45 s	0, 0, 0, 0	"morgen in Vaterstetten", "morgen, den 10. Oktober 2026, in Vaterstetten"

The neutral profile is about 0.15 s slower to the first beat and follows the score format more cleanly. Both resolved "morgen" correctly in these runs.

reasoning_effort "low" for Qwen (refine-from-live-use, 2026-10-09)

Qwen3.8-Flash-Next via SGLang/litellm returns its reasoning as reasoning_content (never spoken). Measured against the no-thinking setup:

Role	No thinking	reasoning_effort: "low"
Judge, interrupt vs. keep presenting (35 cases)	34/35	28/35 (two runs)
Judge, median latency	~115 ms	~900 ms (max 2.4 s)
Judge, verdicts lost to the 400-token limit (unparsable)	0	4–5 per run
Neutral presenter, time to first beat	0.30–0.45 s	0.8–3.7 s
Neutral presenter, score issues / date for "morgen"	0 / correct	0 / correct and explicit ("Samstag, 10. Oktober 2026")

With reasoning, the judge gets worse: it overthinks short echo fragments and side talk into interrupt, and several times its reasoning used up all 400 tokens without an answer. The presenter stays correct but becomes noticeably slower to start.

Painted pictures with flux2_klein (add-generated-pictures, 2026-10-09)

scripts/picture_probe.py against flux2_klein (FLUX.2 klein 9B, ComfyUI on amd, through litellm), size=1024x768, five pictures in a row:

time per picture	3.0–3.1 s, steady
response	b64_json (also without response_format)
image	PNG, 1024×768, 0.4–1.4 MB

The pictures are good: a children's-book whale next to a school bus, a clean water-cycle diagram, a watercolour fox, a minimal book cover. Text inside a picture often comes out as invented letters ("SOSIO" on the bus, a line of pseudo-script on the cover). The presenter rules therefore say: no text in the picture, labels go into caption. Relative sizes are not reliable (the whale came out smaller than the bus), so comparisons by size still need a bars or fact beat. At 3 s, a picture usually arrives within its own beat; the 30 s timeout only covers a busy amd.

Size comparison (same five prompts): 512×512 and 768×576 both take 1.5 s per picture once the model is warm (0.15–0.5 MB and 0.2–0.9 MB), against 3.0 s for 1024×768. The server uses 768×576: as fast as 512×512, the 4:3 format of the stage, and sharper on a TV than an upscaled square.

Live on agent.hoyer.world (Eddie, Gemma with thinking "low"): "Erklär einem Fünfjährigen, wie groß ein Blauwal ist. Zeig ihm ein gemaltes Bild dazu." gave nine beats, with the picture as the third beat (prompt in English, as the rules ask). The picture arrived 3.0 s after its beat was released, while its own sentence was still being spoken: a cheerful whale next to two school buses.