AI Companions
Status: Live in code, one build away from users · Last updated: 2026-07-12
TrickBook's AI Companions are riders, not chatbots. They chat with real personality, act on your TrickBook data through tool calls, and — as of July 2026 — exist as full 3D characters who talk to you out loud and physically demonstrate tricks with their bodies.
This is the hub for the companions documentation. The deep-dives below were written from a full code audit of the mobile app, the web app, and the backend (2026-07-12).
Mobile App Surfaces — roster, widget, bot chat, rich cards, voice stack · Web App Surfaces — the Kaori Live 3D stage, DMs, parity matrix · Animation System — rider fundamentals, trick timelines, speech-synced choreography, scaling audit · The Board Model — how the board is generated and where per-design boards fit · Motion Capture Pipeline — how we'll learn real trick movements from video · RAG & Internal Tools — the knowledge/tool architecture today and the plan to make companions truly smart
Technical architecture: Kaori AI Architecture · Path to users: Companions Launch Audit · Business model: Monetization: Paywall & Tokens
Kaori 🏔️ — the flagship companion
- Character: Japanese snowboarder (homage to SSX's Kaori Nishidake) — 18, from Sapporo, youngest rider on the SSX circuit
- Personality: corpus-mined rider register — dry, understated, short replies, no cheerleader energy. Praise is one specific line ("that back lip was clean"). Trained against a banned-poser-vocabulary list mined from core snowboard media.
- Brain: Gemini Flash via OpenRouter with an 8-tool calling loop (TB-Backend
kaori-ai-response.js+kaori-tools.js) - Memory: unified across every surface — web DMs, mobile chat, and 3D-stage voice conversations merge into one chronological history, plus a relationship profile (stranger → bestie) that grows with interaction count
- Voice: ElevenLabs streamed through Kith (our own OSS voice runtime), gapless sentence-by-sentence playback
The 3D Companion Stage (mobile)
Merged to v2-rebuild in PR #4 (2026-07-08). An interactive VRM stage where Kaori:
- Renders in full 3D — VRM avatar via three.js + react-three-fiber + expo-gl, orbit/pinch camera, physical devices only (simulators can't run the shaders; a fallback card explains)
- Talks live — speak to her with the mic (on-device STT), she answers in her ElevenLabs voice with mouth-sync, gestures, and emotion expressions; barge-in supported
- Demonstrates tricks with her body — speech-synced choreography: say "show me a frontside 360" and she explains it while physically performing it, her own sentences cueing the wind-up, pop, and landing. Built on a reusable rider-motion vocabulary (
riderFundamentals.ts) + a per-trick timeline registry (trickAnimations.ts). Full details: Animation System. - Keeps the chat — translucent chat overlay on the stage, same brain and history as regular chat
Entry points: the cube icon in Kaori's chat header and the Companion widget on Home. Details: Mobile App Surfaces.
The Kaori Live stage (web)
thetrickbook.com/kaori-live is a full 3D VRM stage in the browser (raw three.js, MToon materials intact, stylized snowy-mountain environment) with live voice: browser speech recognition in, Kith/ElevenLabs audio out. It shares the idle/gesture motion language with mobile but has no board and no trick demos — those are mobile-only today. It also has no nav entry point; it's reachable only by direct URL. Details: Web App Surfaces.
What Can Kaori Do? (the shipped tool registry)
The 8 tools actually live in production (kaori-tools.js):
| Tool | What it does |
|---|---|
search_spots | Find approved skateparks / resorts / breaks — "any parks near Brooklyn?" |
search_trickipedia | Look up tricks by name — "how do I do a backside 180?" |
get_user_tricklists | Read the user's real lists with resolved trick names |
create_tricklist | Create a list in the user's account (flagged createdBy: 'kaori') |
add_trick_to_list | Add a trickipedia trick to one of the user's lists |
update_trick_status | Mark a trick Complete / To Do |
lookup_boardsport_knowledge | Curated static knowledge (magazines, events, culture per sport) |
remember_user_info | Persist name / sports / facts to the relationship profile |
Spot submission (create_spot_draft) and rich-card sharing (share_content) were planned but are not shipped — see the original tool-calling plan vs. what landed. The mobile app already ships the rich-card renderer for when the backend starts sending them (details).
Safety & Guardrails
What's actually enforced in code today:
- No deletion tools — create and read only
- Ownership checks — Kaori can only touch lists/tricks belonging to the requesting user
- Approved spots only —
search_spotsfilters toapprovalStatus: 'approved'and instructs the model to disclose when a suggestion isn't in TrickBook - Audit trail — AI-created content flagged
createdBy: 'kaori' - Loop caps — max 3 tool iterations and 400 output tokens per reply
- Stage prompt gating — the trick-demo prompt only activates for live stage sessions (
x-kith-sessionheader), so normal chat stays normal
What's not built yet (open items in the launch audit): per-user rate limiting on the AI endpoints, voice-token metering, and authentication on the voice WebSocket.
The Vision
Companions are TrickBook's differentiator: a coach in your pocket for every action sport, with a body, a voice, and a memory. Two capability tracks make them genuinely smart rather than chat-deep:
- Knowledge (RAG) — companions grounded in our own knowledgebase: Trickipedia, coaching content, the docs. Today's knowledge is a 1,500-word static JSON; the migration plan to real retrieval is in RAG & Internal Tools.
- Action (internal tool calls) — companions operating the whole product: tricklists, spots, videos, feed. The backend already exposes the REST surface for all of it; the tool registry needs to grow into it. Inventory and architecture in RAG & Internal Tools.
And one embodiment track:
- Demonstration — companions that can show every trick of their sport. The procedural foundation shipped; the scaling plan (motion capture →
.vrmaclip library layered over the procedural fundamentals) is in Motion Capture Pipeline.
Monetization — sample for free, tokens for voice
Free users get a sample of the companions (Kaori, with a daily voice allowance); paid tiers unlock the full roster and monthly voice-token allotments. Voice is metered because every spoken reply costs real money (TTS + LLM). Full model: Monetization roadmap.
Unlockables — boards, outfits, environments
- Boards — deck designs and snowboard graphics as unlockable props (easiest, most on-brand — see The Board Model for the technical path)
- Outfit colorways → full outfits — texture swaps first, complete alternate VRMs later
- Per-sport environments — Kaori belongs in a snowy environment, Tony in a street/park scene, the surf companion on water
- Unlocks come from both usage (streaks, tricks landed, XP) and payment tiers
Roster — skateboarding and snowboarding first
| Companion | Sport | Status |
|---|---|---|
| Kaori 🏔️ | Snowboard | ✅ Live (chat + voice + 3D stage) |
| Tony 🛹 | Skateboard | 📋 Next — largest user segment, deepest Trickipedia coverage |
| Rico 🏄 | Surf | 📋 Later (after skate + snow are strong) |
| Max 🚲 | BMX | 📋 Backlog |
| Zoe ⛷️ | Ski | 📋 Backlog |
Stance-aware coaching
One of the first profile questions for boardsports users will be regular or goofy — it changes which direction a frontside rotation goes, which foot leads, and how a companion coaches every trick. Choreography currently assumes regular (why that matters); stance-awareness is a prerequisite for coaching goofy riders correctly.
Implementation Status
| Feature | Status |
|---|---|
| Chat with personality + 8 shipped tools | ✅ Live |
| Rider persona (corpus-mined register) | ✅ Live (prod 2026-07-09) |
| Unified cross-surface memory + relationship profile | ✅ Live (prod 2026-07-09) |
| Live voice pipeline (Kith + ElevenLabs) | ✅ Live (prod 2026-07-09) |
| Web Kaori Live (3D VRM stage + voice) | ✅ Live (hidden — no nav entry) |
| 3D companion stage (mobile) | ✅ Merged — awaiting new EAS build |
| Speech-synced trick demos (FS360) | ✅ Merged — awaiting new EAS build |
| Mobile rich-content card renderer | ✅ Shipped — dormant (backend never sends richContent yet) |
| Voice-endpoint auth + usage metering | 🚧 P0 — see launch audit |
| Real RAG on our knowledgebase | 📋 Planned — architecture |
| Expanded internal tool calls (spots/videos/feed) | 📋 Planned — inventory |
| Mocap-driven trick clip library | 📋 Planned — pipeline |
| Paywall / free-sample gating | 📋 Planned — see monetization |
| Snowy stage environment · board/outfit unlocks | 📋 Planned |
| Tony (skateboard companion) | 📋 Planned |
| Regular/goofy stance onboarding | 📋 Planned |