Rich Content in Conversations — Design & Plan

Status: Proposed · Date: 2026-09-18 · Tracking: #583 · Reserves ADR-0200 (umbrella) — phase ADRs numbered when each phase starts. Extends ADR-0043 (inbound images) and ADR-0196 (outbound text rendering).

The ask: a Puccha conversation should be able to carry every content format a person actually uses in LINE / Messenger / the widget — text, links, images, files, voice notes, video, emoji, stickers — in both directions: the visitor sends them, and the AI (or the human agent) can reply with them. Today the conversation is text + inbound images only.

---

1. Where we are (audited on origin/main @ e2f70d80, 2026-09-18)

Format	Inbound (visitor → Puccha)	Outbound (AI / agent → visitor)	Knowledge base
Text, markdown	✅	✅ widget md(), LINE Flex bubble (ADR-0196)	✅ PDF · DOCX · MD · TXT · web crawl
Link	✅ (as text)	✅ auto-link; Flex uri action; BUTTONS: chips (text only, no URL cards)	✅ crawl
Image	✅ ADR-0043 P1 — widget, LINE, FB, IG, WhatsApp, Google Chat → Haiku vision (aiVision opt-out)	❌ no channel sends type:'image'; widget md() drops ![]; model has nothing to reference	❌ text layer only — scanned PDFs and figures are invisible
File (PDF/DOCX)	❌ ADR-0043 P2 designed, not built	❌	✅
Voice note / audio	❌ logged as “ข้อความเสียง” placeholder, fixed ack, no RAG turn	❌	❌
Video	❌ placeholder + ack	❌	❌
Sticker	✅ LINE only — stored as a CDN-URL attachment so the inbox shows it; keywords from the webhook are ignored, no RAG turn	❌	—
Emoji	✅ Unicode passes through	✅ Unicode passes through; no persona control over emoji use; LINE product emoji unsupported	—

Other facts that shape the design:

• ChannelReply is { text, markdown?, quickReplies?, packChunks? } (channel-turn.ts:63) — a single text payload. Adapters render it per platform (LINE Flex, FB/IG quick_replies, WA text).
• messages.attachments_json is [{ id, key, type, size, hashedName, sensitive }] and the inbox already reads an optional kind: 'sticker' + url from it — the column is the natural carrier for every inbound media kind.
• /uploads/[...path] serves avatars/ publicly and attachments/ only to the owning visitor (X-Visitor-Id) or a member of the tenant. LINE / Meta / WhatsApp fetch outbound media from their servers with no headers → outbound media needs a signed public URL. This is the one shared infra piece every outbound phase depends on.
• The human-agent composer in the inbox is a bare <textarea> — agents cannot send an image, a file, or a sticker even when the visitor is on LINE.
• Workers AI is already bound (AI, used for bge-reranker in-network fallback, ADR-0115). Whisper models live behind the same binding.
• Widget core is at 41.0 / 41 KB gz — anything that touches the widget must go in the lazy messenger chunk (28.1 / 29 KB) or a new lazy chunk, or rebudget deliberately.

---

2. Principles

1. One reply model, many renderers. The AI and the agent produce a channel-neutral list of parts; each adapter renders what its platform supports and falls back gracefully (image → link, sticker → emoji, video → link card). No adapter-specific branching in the chat pipeline.
2. The model never emits a media URL. It references curated response assets by id (like citations); the server validates the id against the retrieved set and expands it to a part. Same invariant as “citations validated server-side against the retrieved chunk set”.
3. Inbound media becomes text before it becomes a query. Voice → transcript, sticker → keywords, document → extracted text, video → transcript + first frame. The RAG pipeline stays text-first; media only adds vision blocks where ADR-0043 already does.
4. Every media byte is a PDPA record. Same consent, retention, erasure (erase-visitor / erase-tenant already walk attachments/{tenantId}/…), audit and ai* opt-out surface as images. Voice is treated as more sensitive than images (identifiable by nature).
5. Widget budget is a hard constraint. Media renderers and the mic button live in lazy chunks.

---

3. Target model

3.1 ReplyPart — channel-neutral outbound content

// $lib/channels/reply-parts.ts (new)
export type ReplyPart =
  | { kind: 'text'; text: string; markdown?: string }              // today's ChannelReply body
  | { kind: 'buttons'; options: string[] }                          // today's BUTTONS: / quickReplies
  | { kind: 'image'; url: string; previewUrl: string; alt: string; assetId?: string }
  | { kind: 'link'; url: string; title: string; description?: string; imageUrl?: string }
  | { kind: 'video'; url: string; previewUrl: string; alt: string; durationMs?: number; assetId?: string }
  | { kind: 'audio'; url: string; durationMs: number; assetId?: string }
  | { kind: 'file'; url: string; name: string; mime: string; size: number; assetId?: string }
  | { kind: 'sticker'; packageId: string; stickerId: string; fallbackEmoji: string; fallbackImageUrl?: string };

export interface ChannelReply {
  text: string;                 // unchanged — plain-text render of the text parts (back-compat)
  markdown?: string;            // unchanged
  quickReplies?: string[];      // unchanged
  packChunks?: readonly SearchResult[];
  parts?: ReplyPart[];          // NEW — additive; adapters that ignore it keep working
}

parts is additive: every existing adapter keeps working with text alone; each adapter opts in by implementing renderParts(parts) → platform messages[].

3.2 Per-channel capability matrix (what each renderer does)

Part	Widget	LINE	Facebook / Instagram	WhatsApp	Google Chat	TikTok / X
text	md()	text / Flex	text	text	text	text
buttons	chips	quick reply	quick_replies	interactive buttons (≤3)	card buttons	text list
image	<img> (lazy chunk)	type:image — HTTPS, JPEG/PNG, ≤10 MB, preview ≤1 MB	attachment.image	image.link	card image	link
link	link card	Flex bubble w/ hero + button	generic template	text + URL preview	card	text
video	<video controls>	type:video — MP4 + preview JPEG	attachment.video	video.link (MP4)	link	link
audio	<audio controls>	type:audio — M4A + duration	attachment.audio	audio.link (OGG/MP3)	link	link
file	download chip	Flex bubble w/ uri	attachment.file (FB only; IG → link)	document.link	link	link
sticker	<img> of fallback image or emoji	type:sticker — official free sets only	fallbackEmoji text	WebP sticker if provided else emoji	emoji	emoji

Every renderer has a fallback ladder — a part never causes a send failure; at worst it degrades to a link line or an emoji.

3.3 Inbound media → attachments_json entry

// extends the ADR-0043 entry shape; all fields optional except id/kind
{ id, kind: 'image'|'audio'|'video'|'file'|'sticker', key?, url?, type?, size?, hashedName?, sensitive?,
  durationMs?, transcript?, transcriptLang?, stickerKeywords?: string[], extractedTextKey? }

kind is what the inbox and the widget switch on; transcript / stickerKeywords / extractedTextKey are what the RAG turn reads. The visitor-facing message content stays the human text (caption, or [voice note 0:12]), never the raw transcript, so PDPA hashing rules for query_log are unchanged (the transcript is hashed like any query).

3.4 Response assets — the tenant media library (source of outbound media)

New table media_assets:

column	notes
id ast_… · tenant_id · workspace_id	tenant-scoped, ADR-0002
kind	image · video · audio · file · link · sticker
r2_key / url	uploaded bytes under assets/{tenantId}/{id}.{ext} or an external URL (YouTube, brochure CDN)
preview_key	JPEG preview for video/file (LINE + FB require one)
mime · size · duration_ms
title · description · alt · tags_json	the retrievable text — what makes the asset show up for the right question
status	active · archived
created_by · created_at · updated_at

Retrieval: each active asset is indexed as a chunk of a synthetic document (sourceType: 'asset', text = title + description + tags) in the same Vectorize + FTS5 path. It flows through hybrid search → RRF → rerank → ACL like any chunk, so an asset surfaces only when relevant. The prompt lists surfaced assets in <context> as [asset ast_x: image — "แผนที่ทางเข้าโรงพยาบาล"]; the model attaches one by ending a reply with

ATTACH: ast_x | ast_y

(same marker family as SUGGESTIONS: / BUTTONS: in answer-markers.ts). The server keeps only ids present in the retrieved set, expands them to ReplyParts with signed URLs, and strips the marker from the text — exactly the citation-validation contract.

Why curated assets rather than “any image in the KB”: the tenant controls what can be sent (brand-safe, PDPA-safe, no leaked internal screenshots), the model can’t hallucinate a URL, and it works today without ingest changes. §5 Phase E extends this to auto-extracted figures.

3.5 Signed public media URLs (shared infra)

GET /media/{token} — token = base64url({key, exp}) + HMAC-SHA256 with a per-env secret (MEDIA_URL_SECRET). 24 h expiry, Cache-Control: public, max-age=3600, immutable ETag. Used for: outbound image / video / audio / file parts to LINE, Meta, WhatsApp, Google Chat; sticker fallback images; asset previews in the widget (so <img> works without X-Visitor-Id). Never used for visitor uploads shown back to the visitor (blob URLs stay, ADR-0043) — only for tenant-owned assets and agent-sent files. Erasure: keys are deleted; a leaked token then 404s.

3.6 Emoji + sticker policy (persona-level)

Persona gains expression: { emoji: 'none' | 'light' | 'expressive'; stickers: boolean } (ADR-0199 already links channel → persona, so this is per-channel for free).

• Emoji — a prompt-builder rule per level (none: “ห้ามใช้อีโมจิ”; light: “อีโมจิได้ไม่เกิน 1 ตัวต่อข้อความ ท้ายประโยค”; expressive: free). Unicode emoji render on every channel including LINE text + Flex spans, so no transport work — only a test that render-text.ts / md() / Flex escaping don’t strip astral-plane code points.
• Stickers out — tenant sticker_set config maps intents → LINE {packageId, stickerId} + fallbackEmoji (+ optional WebP for WhatsApp): greeting · thanks · apology · celebrate · bye. The model ends a reply with STICKER: thanks when stickers: true; the renderer appends the sticker message (LINE) or the emoji (others). LINE bots may only send stickers from the official free packages (LINE “sendable stickers” list) — the config UI offers those only.
• Stickers in — LINE webhooks carry keywords[] (e.g. ["Thank you","Happy","OK"]). Instead of the fixed ack, run a lightweight turn: [สติกเกอร์: Thank you, Happy] as the user text, no retrieval (skip the RAG stages, persona-only prompt), so “ขอบคุณค่ะ 🙏” gets “ยินดีค่ะ 😊” and a thumbs-up sticker after a resolution can count as a positive outcome signal (ADR-0184 panel).

---

4. Inbound formats — per-format design

4.1 Voice notes (audio)

Source	Webhook	Bytes	Format
LINE	message.type === 'audio', duration	api-data.line.me/v2/bot/message/{id}/content (same fetch as images)	M4A/AAC
Facebook / Instagram	attachments[].type === 'audio', payload.url	GET url	MP4/AAC
WhatsApp	audio.id, audio.voice: true	Graph media download (same as fetchWaImageBytes)	OGG/Opus
Widget	mic button (MediaRecorder, lazy chunk) → POST /api/chat/uploads	multipart	WebM/Opus or MP4

Pipeline (in channel-turn.ts, alongside the image branch):

1. Sniff magic bytes (ftyp / OggS / EBML 1A45DFA3 / ID3 / RIFF) — extend image-store.ts sniffImageType into sniffMediaType. Caps: ≤ 60 s, ≤ 5 MB (phase 1).
2. Store under attachments/{tenantId}/{workspaceId}/{visitorId}/{id}.{ext} + visitor_attachments row → existing retention + erasure cover it with zero new code.
3. Transcribe via Workers AI @cf/openai/whisper-large-v3-turbo (Thai supported; language: 'th' hint from tenant locale; returns text + segments). In-network, no third-party sub-processor — simpler DPIA than vision. Fallback: none in phase 1 (return the “ยังไม่รองรับ” ack on failure).
4. content = 🎤 [voice 0:12], attachment entry gets transcript; the RAG turn uses the transcript as the query (sanitizer + injection checks apply — a transcript is untrusted input).
5. Reply prefixes a one-line echo when confidence is low or the transcript is short: ได้ยินว่า "…" — so the visitor can correct a bad transcription. Suppressible per persona.

Compliance: extend the ADR-0043 consent modal copy to voice; widget_config.attachments.aiVoice opt-out (regulated tenants); DPIA addendum docs/compliance/dpia-attachments.md §voice; attachments.allowedKinds gains 'audio'. Cost: Whisper turbo is ~$0.0005 / min on Workers AI — negligible vs the Haiku turn.

4.2 Documents (ADR-0043 Phase 2 — unchanged design, now scheduled)

PDF · DOCX · TXT · MD, ≤ 10 MB, ≤ 2 per turn. Extract at upload (normalize.ts — unpdf and mammoth don’t execute embedded JS/macros, which retires the phase-2 security question), store extracted text as an R2 sidecar (…/{id}.txt, extractedTextKey), cap what enters the prompt at ~6k tokens with a “(truncated)” marker, wrap in <attachment_text untrusted>. Sources: widget picker (extend ATTACHMENT_ACCEPT), LINE file, FB file, WhatsApp document.

4.3 Video

No Workers-side ffmpeg, so video needs a hosted step. Recommended: Cloudflare Stream — upload from the Worker (tus or direct upload), Stream produces thumbnails (/thumbnails/thumbnail.jpg?time=1s) and can generate captions (AI captions, language-tagged); the transcript + first frame become a normal voice+image turn. Stream also solves outbound video hosting (MP4 download URL for LINE). Caps: ≤ 60 s, ≤ 25 MB. Spike first (½ day): confirm Thai caption quality + per-minute cost (Stream storage $5 / 1k min, delivery $1 / 1k min) before committing. Until then videos keep the placeholder ack.

4.4 Links

Inbound links already work as text. Add: when a visitor message is only a URL (common in LINE — “อันนี้คืออะไร” + link), fetch the page title/description (existing crawl fetcher, redirect: 'manual', 5 s, 256 KB cap, private-IP blocked) and append [ลิงก์: <title> — <description>] to the query so the model can answer about it. Same untrusted-content rules as <context>.

4.5 Emoji

Nothing to transport. Verify the sanitizer / query hashing / FTS5 tokenizer don’t choke on astral code points or ZWJ sequences (👩‍💻) — add fixtures to the Thai-first guard tests.

---

5. Knowledge-base media (answers that show something)

Item	Approach	Size
Response assets library (§3.4)	admin page Content → Media (upload / external URL / YouTube), Vectorize indexing, ATTACH: marker, signed URLs	M
Scanned PDFs (OCR)	detect pages with an empty text layer at ingest → render page via Browser Rendering (pdf.js in headless Chromium) → Haiku vision OCR (~$0.003 / page) → normal chunks. Queue job; tenant-visible “OCR’d” badge.	M (spike ½ day)
Figures in PDFs / crawled pages	extract <img> / embedded images ≥ 300 px at ingest → Haiku caption → stored as auto assets (source: 'ingest', linked to the chunk) → citable via ATTACH: like curated assets. Off by default until precision is measured on the promptfoo golden set.	M
Audio / video as knowledge	upload MP3/MP4 or YouTube URL as a document → Whisper (audio) / YouTube caption track (video) → transcript document with timestamped sections; citations render as link parts with ?t= deep links	S–M

---

6. Human-agent composer

Agents on LINE conversations today can only type. Add to the inbox composer:

• Attach (image / file, ≤ 10 MB) → POST /api/conversations/{id}/uploads (agent-owned prefix agent-uploads/{tenantId}/…, member-gated) → parts on the agent message → same renderers.
• Sticker picker (LINE official sets; hidden on non-LINE conversations).
• Asset picker — insert a response asset from the library by search.
• Emoji: native OS picker is enough; no custom picker.

The optimistic-insert path in ConversationThread.svelte:295 already carries attachmentsJson: null — it becomes the agent’s parts.

---

7. Security & compliance checklist (applies to every phase)

• Tenant scoping: media_assets, agent-uploads/, /media/{token} keys all carry tenant_id; cross-tenant tests for every new endpoint.
• No model-emitted URLs: ATTACH: ids validated against the retrieved set; any ![…](…) or bare media URL in model output that is not a retrieved asset is rendered as plain text (widget already does this for unsafe schemes).
• Magic-byte sniffing for every new inbound kind; MIME allowlists per kind; size + duration caps.
• Transcripts are user input: sanitizer + injection patterns run on them; never logged raw.
• Consent + retention + erasure: reuse the ADR-0043 prefix + visitor_attachments index so erase-visitor / erase-tenant / retention cron need no changes — verified by extending their tests with kind: 'audio' rows.
• Opt-outs: attachments.aiVision (exists) · attachments.aiVoice (new) · allowedKinds.
• DPIA: addendum for voice (phase B) and video (phase D); documents (phase C) covered by the ADR-0043 phase-2 note.
• Rate limits: media turns count against the same per-visitor limit (ADR-0185); transcription adds a per-tenant daily minutes cap (voice_minutes_per_day, default 300) to bound Workers AI spend.
• Widget bundle: media renderers + mic in a lazy chunk; core delta ≤ 0.3 KB gz per phase, measured with scripts/check-bundle-size.sh in ci:local.
• Runtime divergence: every new outbound fetch (Stream, Whisper, LINE content) gets a post-deploy dev runtime check (workerd ≠ undici — redirect: 'error' trap).

---

8. Phasing, order, effort

Ordered by value ÷ effort for Thai LINE-first customers. Each phase = 1 ADR + 1–3 PRs, referencing #583. Sizes: S ≤ 2 d · M ≤ 5 d · L > 5 d.

#	Phase	Delivers	Size	Depends on
0	ADR-0200 umbrella + parts model + signed media URLs	ReplyPart, ChannelReply.parts, /media/{token}, renderer skeleton for LINE/FB/IG/WA/widget with fallback ladder, attachments_json.kind	S	—
A1	Expression: emoji policy + stickers both ways	persona expression, LINE sticker-in semantics (keywords), STICKER: marker + tenant sticker set, outcome signal	S	0
A2	Agent composer attachments	attach image/file + sticker from inbox → LINE/FB/WA/widget	S	0
B	Voice notes in	LINE/FB/IG/WA/widget audio → Whisper → RAG turn; consent + aiVoice; DPIA addendum	M	0
A3	Response assets + ATTACH:	media library page, Vectorize indexing, model-attached image/video/file/link cards	M	0
C	Documents in (ADR-0043 P2)	PDF/DOCX/TXT/MD attachments, extracted-text context block	S	0
E	KB media	OCR for scanned PDFs (spike → build), audio/video transcripts as docs, auto-extracted figures (behind flag)	M–L	A3
D	Video in	Stream spike → upload, thumbnail + captions → turn	L	B, spike

Phase 0 + A1 + A2 fit in one week and are visible on LINE immediately (stickers understood, agents can send pictures, personas can be warm or formal). B is the biggest single lift in perceived capability (“ส่งเสียงมาก็ตอบได้”). A3 turns the KB from text-only to “here’s the map / the video / the form”. E and D are spikes first because both need a hosted rendering step.

Out of scope (deliberately): text-to-speech replies, live voice/video calls, LINE product emoji ($ placeholders), animated/custom LINE stickers (not sendable by bots).

---

9. Verification per phase

• Unit through the real driver (drizzle-D1 batch trap) for every new table/query; renderer tests per channel with a golden parts[] → messages[] fixture set incl. the fallback ladder.
• Channel dogfood: seed a test channel on perftest, forge signed webhooks with an audio / sticker / file payload, assert D1 rows + outbound request shape (recipe in memory “channel webhook dogfood”).
• Widget: bundle gate; Playwright on a __qa-* route for media bubbles + mic.
• Promptfoo: A3 and E add golden cases (“มีแผนที่ไหม” → ATTACH: the map asset; a scanned page’s fact answered) — faithfulness ≥ 0.85, citation-recall ≥ 0.80 stay gating.
• Post-deploy dev runtime check for every outbound fetch; then dev → uat → prod promotion.