Companions in the Web App

Status: Audited against code 2026-07-12 · repo TrickBookWebsite (Next.js, pages router)

What the website ships for companions today: a full 3D Kaori stage at /kaori-live, companion DMs on /messages, and the honest parity picture against mobile.

Related pages

Hub: AI Companions · Mobile equivalent: Mobile App Surfaces · Backend brain: Kaori AI Architecture

Kaori Live (pages/kaori-live.js) — the web 3D stage

A single ~1,200-line page containing the entire experience: a real 3D VRM stage in the browser, hand-rolled with raw three.js ^0.183 + @pixiv/three-vrm ^3.5.2 (no react-three-fiber on web). It loads /kaori/Kaori_V3.vrm (19.6MB) with MToon materials intact — the browser compiles what expo-gl can't, which is why web Kaori has the true anime shading and mobile runs the unlit downgrade.

• Scene: stylized snowy mountain set — fog, snow ground plane, four cone mountains with snow caps, rotating brand-yellow floor ring, and a cyan pulse ring whose opacity/scale encode the conversation state.
• Auto-fit: the VRM is bounding-box-scaled to a 1.8m target height and floored, so a swapped model can't break framing.
• Fallback: if the VRM fails to load, a glowing-sphere stage animates instead.
• Auth: logged-out users get an inline "Log in to talk with Kaori" card; the page never redirects.

Procedural animation (shared motion language with mobile)

All animation is procedural — an explicit design choice (the code comments: Mixamo FBX clips override bone rotations and T-pose VRM models). The layers mirror mobile's idle system: breathing/sway/head-drift, hip weight-shift, 3.5s deterministic blink, finger curl, listening/thinking body language, and a 6-pose conversational gesture cycle (rest, explaining, presenting, emphasis, counting, nodding) that rotates every 2s while speaking. Mouth shapes are simulated sine visemes — there is no audio analyser; the voiceLevel/voiceBands reads in the render loop are dead code that never receives data.

Server-driven emotion (emotion_state from Kith) does exactly one thing on web: an emissive tint across all materials (excited = warm gold, calm = ice blue, happy = pink, sad = slate). It never selects gestures.

lib/kaori-animation-presets.js is dead code. It defines the same 6 poses plus a default-talking preset, but is never imported — the page duplicates the poses inline, and the values have already drifted (resting-smile weights differ). Treat the presets file as an unrealized refactor, not live architecture. There is no board, no trick registry, no rider fundamentals on web — trick demos are mobile-only.

Voice on web

• STT: browser SpeechRecognition (Chromium/WebKit), continuous with interim results, auto-submit after 2s of silence, 5s no-speech timeout. en-US hardcoded; unsupported browsers get an alert.
• TTS: the page opens the Kith WebSocket (NEXT_PUBLIC_KITH_VOICE_WS_URL, prod wss://api.thetrickbook.com/kith/ws) — unauthenticated, the same P0 as mobile. Base64 mp3 chunks are decoded via Web Audio and played gaplessly; a 3s auto-reconnect loop runs while the page is open.
• Session wiring: on _ready the page stores the sessionId and attaches x-kith-session to normal DM message POSTs — the text reply arrives over socket.io, the audio over the Kith WS. On _ready it also fetches a relationship-aware greeting (POST /companion/profile/:id/greeting) and has Kith speak it.
• Barge-in: three paths — starting the mic, the stop button while speaking, and server barge_in_detected — all clear the local audio queue.

⚠️ Config foot-gun: the WS URL falls back to ws://localhost:3040/ws — if the Amplify env var is missing, the page silently retries localhost forever.

The stage prompt never reaches web

Even with a live x-kith-session, the web path goes through routes/dm.js, which never passes onStage to the brain — so Kaori never structures her replies for choreography on web. That's currently fine (web has no demo system to cue), but it's the first backend change needed if the web stage ever gets demos.

Chat surfaces

• /messages — the conversation list carries an "AI Companions" avatar row (yellow ring, permanent green presence dot); tapping starts or resumes the bot conversation via POST /dm/bot-conversation (no homie requirement for bots).
• /messages/[conversationId] — companion DMs use the ordinary DM thread: date-grouped bubbles, typing indicator, read receipts, an "AI" badge, and a cosmetic always-"Online" status. No x-kith-session here — the plain DM page is text-only. Voice Kaori = /kaori-live specifically.
• /homies — despite the name, contains zero companion code; companions live on /messages.
• Rich content cards: none on web. Both surfaces render msg.content as plain text. (Mobile ships a card renderer; neither platform receives richContent from the backend yet — see mobile page.)
• Legacy "Kaori voice: *.mp3" messages are regex-filtered — with two different regexes on the two surfaces, so a legacy row can appear on one and not the other.

Discoverability & assets

• /kaori-live has no nav entry point anywhere — no header link, no homepage promo. It is a direct-URL page. The user-facing companion entry point on web is the AI Companions row on /messages.
• public/kaori/ publicly serves ~70MB: three VRM versions (only Kaori_V3.vrm is used) plus the raw model.vroid VRoid Studio source — an asset leak worth cleaning up alongside the planned move of models to CDN.
• The relationship-profile API client exists on web (getCompanionProfile) but no page calls it — relationship/stage UX is mobile territory for now.

Web ↔ mobile parity matrix

Capability	Web	Mobile
3D VRM stage	✅ raw three.js, MToon	✅ r3f + expo-gl, unlit downgrade
Idle/gesture/blink/viseme layers	✅	✅ (same motion language)
Emotion from Kith	✅ emissive tint	✅ facial expressions
Trick demos + board	❌	✅ (FS360)
Rider fundamentals / TRICKS registry	❌	✅
Live voice (TTS out)	✅ Kith WS	✅ Kith WS
STT in	✅ browser API	✅ expo-speech-recognition
Stage demo prompt (onStage)	❌ never passed by dm.js	✅ via botChat.js
Companion chat	✅ DM thread + AI badge	✅ dedicated bot-chat screen
Rich content cards	❌ none	✅ renderer shipped (dormant)
Relationship greeting	✅ kaori-live auto-greet	❌
Relationship profile UI	❌ (client exists, unused)	❌
Nav entry point to stage	❌ hidden URL	✅ widget + chat header cube
Paywall/entitlement gating	❌	❌

The strategic read: web is the materials/lighting showcase (true MToon) and the zero-install demo surface; mobile is the product (demos, cards, widget). Divergence to watch: the two stages share motion values by copy-paste, not by a shared module — the third copy (Tony) is the moment to extract one.