Web-voice mode

The flagship shape: Cicero runs on a headless box (typically a GPU machine) and you talk to it from a browser — your laptop, your phone, anything on the LAN or your VPN.

Setup

Follow setup for prerequisites, the setup wizard, checks, startup, pairing, and the first spoken test. This guide begins with the web-voice-specific credential and connection behavior.

When web_voice.token is omitted or blank, the one-run credential is printed to startup stdout. It is deliberately excluded from the dashboard, dashboard event stream, and Cicero's application logger, but a shell redirect or service manager can retain stdout. Configure a stable token before daemonizing Cicero. For one stable credential across restarts, run openssl rand -hex 16 and paste only its output as web_voice.token; placeholder text is rejected.

For the fast path, start the daemon and run cicero pair. It creates a stable token when the configured value is missing or invalid, prints a pairing URL, and renders that same URL as a terminal QR code. Restart the daemon and re-run the command if it reports that it wrote a new token. The default QR contains a live credential; cicero pair --no-token-in-qr omits it and prints the token separately for manual entry. If tunneling is not configured, pair prints the exact tunnel: { provider: auto } line to add under web_voice without editing it. A live tunnel URL is preferred over the LAN address. Cloudflared quick-tunnel URLs change each daemon run, so re-run pair after a restart; Tailscale hostnames are stable. The manual URL/certificate flow below still works unchanged.

Open https://<box-ip>:8090/?token=<token> in a browser. TLS is on by default (a self-signed cert is generated under ~/.cicero/web-voice/ — accept it once per device; the browser mic requires a secure context). Automatic generation needs openssl on PATH the first time; cicero doctor checks this and prints the platform-specific install command, including scoop install openssl on Windows. The completed pair is selected by one atomic manifest, so concurrent starts and a killed generator cannot expose a mismatched half or leave a permanent generation lock. To provide your own files, configure both web_voice.tls.cert_file and key_file; Cicero treats explicit paths as read-only and never fills in or overwrites a missing half. If automatic generation fails, an exposed listener refuses to downgrade to HTTP. Loopback may still use HTTP; tls.enabled: false is the explicit opt-in for an insecure non-loopback listener. A headless daemon also exits if web voice is disabled or cannot bind, instead of reporting ready with no usable interface.

Talking to it

• Push-to-talk (default): hold SPACE or press-and-hold the orb; release to send.
• Hands-free: flip the mode toggle — an adaptive energy VAD detects when you start and stop talking. Talk over Cicero any time to interrupt (the browser's echo cancellation keeps its own voice out of the mic).
• Only speech gets through (default on): a small local VAD model (Silero v5 on onnxruntime-wasm, ~13MB fetched once on first start, served same-origin — no CDN at runtime) confirms an energy trigger is actually human speech before a hands-free utterance opens or a barge-in interrupts. Keyboard clacks, thuds, and music carry energy but no speech probability, so they stop cutting Cicero off — and with auto-start, the dormant page wakes only when words are spoken. Push-to-talk is never gated (a held key is deliberate). Missing assets or a failed model load degrade to the plain energy gate (watch vad: in the debug line); web_voice.speech_gate: false disables it.
• Auto-start (off by default, remembered per browser): reloads reconnect and open the mic with no click — but only when the browser has already granted the mic permission; the page never prompts without a gesture, and if the browser blocks gesture-free audio anyway it falls back to the Start button. Pairs with hands-free: the orb loads dormant and only lights up when you first speak. With auto-start on, the dormant page also blacks out all of its chrome behind a near-black shroud — pure cinema until first speech fades it out with the orb; taps still pass through, and a missing token keeps the shroud off so the "authorization required" status stays visible.
• It knows when you're done (with turn: {enabled: true}): a semantic end-of-turn model (Smart-Turn v3, ~8MB ONNX, runs on CPU) listens to how you stopped, not just that you stopped — a finished sentence sends ~450ms sooner than the silence timeout, while a mid-thought pause ("let me think… ") keeps the mic open instead of cutting you off. Without the model (or if its server is down) the plain silence hangover governs, exactly as before.
• Cancel a stuck turn: press and hold the orb while it's thinking.
• Type instead: the text box under the mic runs the same brain→TTS pipeline without STT — no mic permission needed, and typing over a reply barges in exactly like talking over it.
• Transfer the call: with lanes configured, "let me talk to the coder" hands the conversation to another agent persona — its own memory, its own voice, optionally its own spoken personality — and "back to Cicero" returns you to the front desk. "Roll call" makes every employee check in, each in their own voice. See the office.
• Long answers stay short by default: the TLDR gate speaks the first few sentences plus a generic coda; say "details" to hear the rest. Configure a summarizer endpoint for a model-written coda, or set web_voice.tldr.enabled: false to speak every sentence. Full text always lands in the chat log.
• Interrupt without losing the thread: after you talk over a reply, "continue" / "go on" / "as you were saying" (within five minutes) resumes it — the agent picks up from what you actually heard, opening with "as I was saying…", instead of starting over.
• Tone rides along (with tone: {enabled: true}): a local speech-emotion sidecar (emotion2vec+ base, CPU-only, auto-launched on :8091) classifies each utterance in parallel with STT. A confident non-neutral read — happy, angry, sad — is appended to what the agent sees (never to the transcript you see), so replies can react to how you said it. Adds ~0 ms: the verdict races the transcript and a late one is dropped, never waited on. Utterances under min_ms (default 1.5s) skip classification entirely — SER models are confidently wrong on short clips. Knobs in config.yaml.example (min_score, grace_ms, min_ms).

Replies stream sentence-by-sentence, so speech starts while the brain is still generating; a pre-rendered filler clip covers the agent's first-token latency when it's slow.

The browser captures in 128-frame AudioWorklet quanta and posts bounded 2048-frame batches to the page. Browsers without AudioWorklet use the older ScriptProcessor path; the debug line shows which one is active. The worklet is a same-origin public asset under /capture-worklet-v1.js, like the pinned VAD assets. The v2 server paces production against playback acknowledgements, keeping about 12 seconds of unplayed audio and at most 64 unacknowledged clips in flight; any admitted clip can be sent when that queue is empty. A stalled turn fails only after the expected end of queued playback plus 30 seconds without acknowledgement progress. The browser's 120-second pending-audio cap is a last-resort safety net. The server also stops a turn if its WebSocket has more than 8 MiB of queued outbound audio.

Protocol v2 reply audio uses a CVA2 binary envelope: magic, two little-endian u16 identity lengths, a little-endian u32 sequence, session ID, turn ID, then WAV bytes. The browser sends {type:"audio_ack", sessionId, turnId, sequence, status:"played"} after a clip ends, or status:"interrupted", atMs when it stops; atMs is the position within that clip. Recovery uses fully played chunks only. A partially played clip is recorded as interrupted but its text is not counted as fully heard. Protocol v1 clients, including the Telegram call sidecar, continue to receive raw WAV and have no playback acknowledgements; their recovery remains an emitted-audio estimate. Local mic/speaker playback is outside this browser transport and retains its existing recovery behavior.

Protocol v2 also accepts bounded client_metric frames for end-to-end conversation latency reporting. The browser reports playback start by audio sequence; the server resolves whether that sequence was a reply or filler. Existing outbound frames are unchanged.

With stt.streaming: true and audio.cpp Nemotron configured in "mode": "streaming", the server announces stream_on. Each browser turn then sends ordered CVS2 PCM chunks inside its CVP2 turn envelope during speech. The server returns rate-limited partial_transcript events for live captions and replaces the caption with the final transcript. The complete WAV is still sent at speech end so a failed live stream can retry batch recognition. This opt-in applies to web voice only; the local microphone keeps its existing batch path.

Restarts don't lose the thread

Restarting the daemon doesn't wipe the conversation: the fresh agent session is primed with a recap of the last turns (riding the warmup ping, so it costs nothing). Turns spoken by a transferred-to colleague are attributed to that colleague in the recap, so the front desk never resumes someone else's personality. Tune with web_voice.resume_turns (default 10, 0 disables).

The page itself opens voice-only — the orb is the interface; the Chat button shows the live transcript for the current visit. A browser refresh starts a new display and does not replay prior rows into the built-in page. Completed turns remain persisted server-side in ~/.cicero for daemon/brain resume.

Install it as an app

The page ships a PWA manifest — "Add to Home Screen" on a phone (or "Install" in Chrome) and Cicero opens fullscreen like a native app. The token is remembered from your first visit, so the installed app reconnects without the query string. After that first visit the page immediately removes ?token= from browser history. HTML responses are not cached and carry a no-referrer policy, CSP, and frame restrictions so the credential is not retained by ordinary browser history, referrers, or intermediary caches.

The manifest starts at /app, an unauthenticated HTML shell. That is intentional: an installed PWA does not retain the ?token= from the page it was installed from, and rejecting the shell would prevent its JavaScript from recovering the token from localStorage. The shell grants no access on its own; the WebSocket and every /api/* route still require the bearer token. If the app is installed before a tokened visit, it displays an authorization message instead of repeatedly opening a dead connection.

Transport identity and limits

The built-in page uses WebSocket protocol v2 (/ws?protocol=2). The server sends a fresh sessionId in its hello frame, and the page assigns a unique turnId to every utterance. Both IDs ride every transcript, sentence, control, error, completion, probe verdict, and binary audio frame. A replacement turn invalidates the old turn's sink, so late audio or a late done from an aborted handler cannot be mistaken for the current reply. Turn queues, abort state, speculation, and replay detection are scoped to one WebSocket; one browser cannot abort or drain another browser's transport queue.

Stopping the conversation sends a {"type":"bye"} control frame before closing. A closed socket alone cannot say whether the operator stopped or the network dropped, and the page reconnects by itself (1s → 2s → 5s) in the second case, so the server ends the conversation immediately after a goodbye and otherwise waits out a grace period that a returning client cancels. That matters to a brain holding deferred work — a cold transfer still connecting — which would otherwise be called off by an ordinary blip. The frame carries no authority and is accepted regardless of session id: refusing it would silently downgrade a Stop into a dropped connection.

Protocol-v2 binary frames are CVP2, two little-endian ID lengths, the UTF-8 session and turn IDs, then the existing WAV or PRB2 payload. Clients that do not request protocol=2 remain on the original raw-binary/untagged-JSON format for compatibility. New integrations should use v2; the legacy path cannot provide stale-frame rejection because it has no identities.

Turn audio is capped at 4 MiB and 120 seconds on both WebSocket and POST /api/turn. Ingress accepts only bounded uncompressed PCM/float WAV, so a small compressed upload cannot expand into hours of STT work. Oversized HTTP bodies receive 413 Payload Too Large. WebSocket text controls are capped at 64 KiB; typed turns are capped at 16,384 characters. JSON bodies are read with an absolute deadline and per-route byte caps, health batches accept at most 100 bounded rows, the listener admits at most 32 sockets, and at most eight authenticated jobs run concurrently (429 asks excess callers to retry). HTTP turn callers may send X-Cicero-Session-Id and X-Cicero-Turn-Id; valid values are echoed in the response body and headers. Callers that omit them get generated IDs, preserving existing integrations.

GET /health is process liveness and remains 200 while the listener exists. GET /ready is daemon readiness: it returns 503 during startup/shutdown and 200 only after the full daemon can accept turns. Both probes are unauthenticated and contain no credentials.

Deliberate multi-client boundary

Transport state is isolated per socket, but Cicero still represents one shared assistant: connected browsers reach the same configured brain/switchboard, active lane, and persisted history. This change does not silently invent a separate agent-memory session per device. That would change product semantics (one household/office conversation versus independent rooms) and requires an explicit conversation/session coordinator plus per-session brain lifecycle. Until that decision is made, use one actively speaking client at a time when the selected brain adapter does not itself serialize concurrent prompts.

ACP streaming turns can speak a short tool-start line and a kind-based spoken-confirmation line through the same turn sink as the reply. A line is dropped when its turn is aborted or superseded. The legacy POST /api/turn endpoint returns one completed WAV and cannot deliver a notice while it is waiting; use the streaming WebSocket for live notices (the Telegram call bridge uses it).

Speculative turns and ACP tools

web_voice.speculative.enabled is off by default and requires turn.enabled. When enabled, a confident mid-pause probe starts transcription and an agent turn before the final recording arrives. An ACP brain holds each tool permission request until that recording passes the adoption checks. Adoption applies the normal auto_approve_tools policy and confirm_tools spoken gate. Discard or abort cancels pending requests. The hold also works through ACP-only lanes and fallbacks. An ACP/CLI mix needs allow_tool_brains: true, as do subprocess CLI brains and other tool-capable backends without a permission round-trip. That setting retains its existing opt-in behavior for early tool execution.

The hold covers only tools the agent routes through ACP requestPermission. Tools the agent auto-approves on its own side, such as in a yolo or auto-approve mode, can still run during speculation. Cicero logs this limitation at startup when ACP speculation is enabled. Speculation stays off while the intent judge runs, even with an ACP brain.

Incomplete-turn filter (opt-in)

Smart-Turn judges audio. The optional text filter asks the configured classifier model whether the final transcript is clearly unfinished, such as “so what I want is, um”. Enable it with:

web_voice:
  incomplete_turn:
    enabled: true
    wait_ms: 3000
    classifier_timeout_ms: 250

A separate classifier backend must be configured (see Intent judge). The filter runs after STT and the addressed-to-me veto, before history, brain work, fillers or speech. Only an exact incomplete verdict holds the turn. Complete, malformed, failed and timed-out verdicts proceed. Speculation is disabled while this filter is enabled because it would start brain work before the verdict.

While held, the WebSocket stays silent and sends a hold control frame so the browser reopens hands-free capture while keeping the pending reply live. A continuation on the same socket joins the unfinished text; the combined transcript reaches the brain and history once. After wait_ms (100–10000, default 3000), the pipeline answers the retained text anyway. Further incomplete verdicts never extend that original deadline. Speech onset cancels the old response, but retains the prefix for one bounded capture (up to 120 seconds); if the original deadline has passed by its final transcript, the combined text proceeds immediately. An abandoned capture expires silently. Typed input, an empty/failed/vetoed continuation, socket close and server shutdown discard retained voice text. Each socket owns its own prefix; reconnects do not recover unfinished turns.

The HTTP POST voice endpoint also checks and waits, but separate POST requests cannot join a turn; use the WebSocket for continuations. Typed input and host-mic turns do not use this browser-voice filter.

The default is off: no classifier call or added wait. Enabling it adds the classifier's latency (bounded by classifier_timeout_ms, 1–1000, default 250), but a complete verdict never pays the silence window. Run bun run bench/incomplete-turn-bench.ts for synthetic complete-turn pipeline overhead; injected-clock tests also verify that a 12ms classifier adds exactly 12ms, not 3000ms. This measures software timing, not real model accuracy or acoustic behavior. Measure your local classifier before enabling this experiment.

False interruptions

Hands-free barge-in pauses the current audio clip and keeps the bounded playback queue. If recognition yields no words, or web_voice.false_interruption_ms elapses (default 1500; range 250–10000), playback resumes from the same position. It does not re-synthesize the reply. A backchannel — a transcript that is only "mm-hmm", "uh-huh", "yeah", "right", "okay", "I see" and the like — is treated the same way, so saying it to show you're following resumes the reply instead of replacing it. "Stop", "wait", "yes"/"no" and anything with real content still interrupt; set top-level backchannel_enabled: false to interrupt on any words. The Telegram call bridge negotiates the same behavior with the daemon; an older daemon retains the existing hard-abort path. Push-to-talk cancellation and Stop remain explicit cancellations.

Tentative captures use batch STT on the completed utterance. Real words replace the old turn; a delayed transcript can still replace a reply that has resumed, but cannot replace a newer turn. Long utterances or slow STT can therefore allow playback to resume before recognition finishes. Recognition failures preserve the old reply. The recognition deadline is 15 seconds, with provider work owned until it settles; capture ownership also expires after the maximum utterance window. Paused audio remains bounded. The browser uses its existing playback limits; Telegram retains at most 64 queued clips and 16 MiB of decoded mono PCM (plus the current clip). Its reader never waits behind playback: control frames must remain visible while audio is paused. Overflow discards buffered speech and reconnects, rather than delaying a confirmed interruption behind old clips.

Live smoke check (requires microphone and Telegram call access): start a long reply, make a short noise with no recognized words, and verify it continues from the pause point. Repeat with a real interjection, then with speech after the timeout. Stop/reconnect while paused and verify the old reply never restarts. Run this on both the PWA and a Telegram call; mocked transport tests do not establish acoustic or device behavior.