Puccha's secret sauce

Updated 2026-09-18 · owner natthawat@harmonyx.co · secret-sauce rag prompts eval

Internal only This is what separates Puccha from “an LLM wired to some documents”. Do not take it outside the team, put it in customer material, or add it to the external KM.

Every page in this group (except this one) is rendered from source on main at deploy time, so it is always the current value — not a copy someone wrote once and forgot. The numbers quoted below can be re-read on All tuning knobs.

The ten ingredients

#	Ingredient	What it does and why it matters	Recipe	Rationale
1	Retrieve-then-read	Search with the user’s query before the LLM is called and put the chunks in <context> — the model never decides whether to search. Took p50 from ~10 s to 2–3 s and made retrieval deterministic	search.ts	ADR-0112 · ADR-0117
2	Hybrid search + RRF	Vectorize (bge-m3, meaning) and FTS5 (exact terms) fused with Reciprocal Rank Fusion, RRF_K = 60, plus a substring fallback for Thai that the tokenizer splits badly	search.ts	ADR-0004
3	Reranker chain	Cohere rerank v3.5 first (calibrated 0–1 scores) → Workers AI bge-reranker in-network → LLM as the last resort, 1500 ms timeout	rerank.ts	ADR-0115
4	Scope gate	If the top rerank score is below SCOPE_RERANK_MIN_SCORE (0.30, from the eval corpus), refuse before the LLM. Kills hallucination and 30–50 % of spend on small tenants. Tunable per tenant (scope_min_score), bypassed for purchase/booking intents	scope-gate.ts	ADR-0112 · ADR-0121
5	Headingless chunking + Thai anchors	512-token chunks, 64 overlap, not anchored on headings (many Thai documents have none); Thai word segmentation before indexing; TOC/listing pages captured as a special chunk	chunker.ts · thai-segment.ts	ADR-0112 · ADR-0187
6	A prompt that forces grounding	“Answer only from the reference material, never invent” + <context> declared untrusted + mandatory [n] citations + no preamble + reply in the user’s language. A persona may change only the opener and tone; the grounding rules are fixed	answer-prompt.ts · prompt-builder.ts	ADR-0147 · ADR-0201
7	Answer cache keyed by chunk set	Answers cached in the CF Cache API for 15 min, key = hash(tenant, locale, query, chunk ids, persona). A document edit changes the key by itself — no manual invalidation. Plus a 7-day embedding cache in KV	answer-cache.ts · embedding-cache.ts	ADR-0018
8	Prompt caching on the Anthropic-native provider	Through AI Gateway with the native provider so a cache breakpoint sits on system+tools, cutting input-token cost on repeat turns	prompt-builder.ts	ADR-0173
9	Refusal detection + gap loop	isIdkText() catches refusal language in Thai and English after the LLM replies; a refusal gets no citations and is logged as a gap for the team to fill (the AI can draft an answer from the gap)	answer-markers.ts · gap-noise.ts	ADR-0121 · ADR-0122 · ADR-0186
10	Layered guardrails + rate limits	Regexes against prompt injection and system-prompt leaks before and after the LLM · 40/h per visitor + 300/h IP backstop · duplicate-query dedupe · per-tenant quota · cost cap at AI Gateway	guardrails.ts · rate-limit.ts	ADR-0021 · ADR-0185

What keeps the recipe honest

• Eval gate in CI — every PR touching RAG must pass Promptfoo: faithfulness ≥ 0.85, citation-recall ≥ 0.80 on a real golden set. See the config and a golden set · ADR-0006
• Persona templates — tuned starters per vertical (hospital, insurance, education, …). See templates.ts · ADR-0167
• Compliance packs — regulatory content the platform maintains and tenants subscribe to. See the authoring guide · ADR-0101
• Per-turn observability — cache_status in query_log and the chat.turn outcome in Analytics Engine say which ingredient ended each turn. See How Puccha thinks § 5

Changing the recipe

1. Edit the source file (the “Source” link at the top of each page), never the KM.
2. Run the eval before merging. Every threshold in the table was picked from the eval corpus, not from feel.
3. Write an ADR when you change an ingredient — not only when you change a number.