Q0 "Tell me about this project."
"I built an enterprise knowledge agent platform.
The main idea is simple. Not all questions carry the same risk. Some questions must be answered exactly right, or it becomes an incident. Other questions just need a good direction.
So I don't put everything into one vector database. I split company knowledge into three layers, by how much error each one can tolerate. Each layer has its own pipeline and its own rules.
And the platform is built for accuracy first, not coverage. It is allowed to say 'I don't have a verified answer for this.' For a bank, I think that is the right default."
"Would you like me to go through the use cases first, or go straight into the architecture?"
→ 让对方选，你就不会讲太久，也显得有结构。
Q1 "Why did you build it?"
"Because one company had three very different kinds of question, and one chatbot could not serve all three.
Some answers have to be exact facts. Some just need a good direction. And some are not in any document at all — they live in the database."
Answer shape: give that, then walk the table — people remember examples, not architecture.
| Who asks | Typical question | Risk if wrong |
|---|---|---|
| Sales team | "What is the warranty period on this battery cabinet?" / "What is the certification number?" | High It goes into a quote or a contract. One wrong number is a real problem. |
| Field & support | "How do I commission this inverter?" | Medium A good direction plus the source document is enough. |
| Sales managers | "How much did we ship to New South Wales last month?" | Data The answer is not in any document. It is in the database. |
"The obvious fix is a RAG chatbot. For us it had two gaps.
First, accuracy. A lot of our answers are facts — a warranty period, a certification number, a model code. They go straight into a quote or a contract. One wrong number is a real problem. RAG always gives you a confident answer, but it cannot promise the number is right.
Second, coverage. A data question has no answer in any document. 'How much did we ship to New South Wales last month' lives in the database."
"Then why not just put your approved answers into the same vector store?"
"Then the approved answer is just one more chunk. It competes with hundreds of pages — and some of them are old versions of the same number. Search ranks by similarity, not by authority, so an outdated datasheet looks just as good.
And once it is one chunk among five, the model rewrites it — so we can no longer call it verified."
"I think the same three layers exist in a bank.
Product disclosure wording and rate cards — that is zero tolerance.
Internal procedure documents — that is tolerant.
Portfolio numbers and branch numbers — that is data, not documents."
Q2 "What makes it different?"
"Most systems put everything into one vector store and then hope the model behaves.
We do the opposite. We decide in advance where the model is allowed to speak, and every capability it can use was written and approved by a person."
Answer shape: that line first, then pick two of the cards below — not all six.
Three layers, three pipelines. Where the model is allowed to speak is a design decision, not an accident.
Knowledge is usable only after a human reviews and publishes it. Document search is a capability we switch on per agent — never the default answer.
Saying "I don't know" is a designed state, not a failure. We measure confidently-wrong answers, not answer rate.
Every answer shows which step did what, with latency, tokens, cost, and the real prompt.
An agent is a set of approved knowledge items picked for one scenario. That bundle is also the boundary of what it can reach.
Verified-only (approved answers and templates, nothing rewritten) · Grounded (documents allowed, citation required) · Open. The mode is a promise to the user, not a setting.
"Every capability the agent can use at runtime is listed, reviewed, and signed off by a human. The model's job is to choose between approved options — not to improvise."
→ 这句是全场最值钱的一句，慢一点说。
Q3 "How does it work?"
"In one sentence: we build a short list of approved capabilities, a small planner picks up to three of them, plain code checks that plan, and only then do we run anything."
Answer shape: say that, then walk these five lines slowly. Don't read the diagram out.
A question comes in (the user has already picked an agent) │ ├─ 1 Build a candidate set approved capabilities only │ Q&A items ≤3 · SQL templates ≤3 · document search ×1 (permissive agents only) │ each type is scored on its own; anything weak is dropped — zero is fine │ ├─ 2 Plan a small planner picks 1 to 3 of those capabilities │ ├─ 3 Check plain code: ≤3 steps · every step from the candidate set · │ no invented values → then one model check: does this answer the question? │ ├─ 4 Run Q&A → returned word for word · SQL → approved template · docs → with citations │ └─ 5 Or refuse nothing suitable, or too complex → say so, and offer to save it as a workflow
"The first path is the one people don't expect. When we hit a curated answer, we do not call the generation model at all. We return the human-approved text word for word.
Because for warranty terms or certification numbers, any rewriting is a risk. That is why the answer can carry a 'verified' badge — nothing touched it."
"What I can show you running is a prototype. In the prototype the three paths are tried in a fixed order, and each one has a strict gate.
The production design replaces that order with the candidate set plus a small planner I just described. The reason is testability: a plan of three steps or fewer, taken from an approved list, is a small object I can compare against an expected plan. A free-running agent loop is not."
"Two reasons. Errors multiply — at 95% per step, three steps is about 86%, six steps is about 73%. And a three-step trace is something a human actually reads.
If a question really needs more, we don't improvise. We refuse, and we offer to save the plan as a workflow for someone to approve. Then next time it runs the same way every time."
Q4 "What did it actually achieve?"
"Short answer: the gates behave as designed.
On data questions, twenty out of twenty end to end, and seven out of seven on the cases it should refuse. On documents, re-ranking added about fourteen points.
I should be honest though — these are small, hand-written test sets."
Answer shape: say the number, then say what it cost. That is what makes it believable.
| Area | Result | The honest part |
|---|---|---|
| Governed SQL | 20 / 20 end-to-end correct; out-of-scope refusals 7 / 7; zero failed executions | Small evaluation set. It proves the gates work, not that it scales. |
| Intent matching | Correct capability at top-1 32 / 32; non-data questions rejected 14 / 14 | One of those 14 was a real bug we found, not a lucky pass. |
| Document RAG | Reranking added +3 hits, about +14 points on a 31-question set | Going from top-5 to top-10 added nothing, so we kept the context small. |
| Exact Q&A | On 27 hand-written cases: threshold only 21 → plus a code guard 23 → plus a light model check 26 | And zero correct answers were lost along the way. |
| Engineering | 141 offline tests; one command sets up a fresh machine | — |
"I want to be honest about these. They come from the prototype build, on small hand-written evaluation sets. What they prove is that the gates behave as designed — not that the system is production-scale."
"In the production design the headline number changes. What I would report there is the rate of confidently wrong answers, plus plan accuracy — because a plan of three steps from an approved list is a small object, so I can compare it to an expected plan exactly, with no model judging it."
→ 主动说出局限，反而更可信。
Q5 "Can you go deeper on one part?"
"Of course. Which part would you like — the routing, the approved answers, the SQL, the documents, or the guardrails?"
Answer shape: let them choose. Then give the short version first, and stop. Each one is 60 to 90 seconds.
Q5.1 — "Tell me more about the intent recognition."
The short version (say this first):
"We don't train a classifier. We treat it as a search problem.
Every capability — a Q&A item or a SQL template — is indexed by
the questions a person would ask to trigger it.
A question comes in, we search those, and we hand the planner a short list of approved capabilities."
Why that choice: "Adding a capability is just adding data. No retraining. One example question is enough to start, and the business team can write it."
Then the three things I'd highlight:
① One capability, many phrasings. "We score every phrasing and keep the best one for that capability. We don't average — averaging would punish whoever wrote the most phrasings. And we collapse to the capability first, so one popular item can't take three of the slots."
② We never compare across types. "Q&A and SQL are scored separately, each keeps up to three, and we put them side by side. Their scores aren't on the same scale, so ranking them against each other would decide things by text shape rather than by meaning."
③ An empty list is allowed. "Each type has its own cut-off. If nothing clears it, that type contributes nothing. That's deliberate — a model will use whatever candidates you show it, so showing it three bad ones is worse than showing it none."
The story behind ③, if you have time: "In testing, a warranty question matched a stock-movement query at 0.5183, and the lowest genuine match was 0.4981. So no threshold could separate them — raising it kills real answers first. That's why 'none of the above' has to be a real outcome, not a leftover."
If they push on ranking: "Two stages. A fast search to shortlist, then a re-ranker that reads the question and the candidate together. The decision cut-off sits after the re-ranker — the first-stage score only decides who gets looked at properly."
Then the planner: "The planner can only choose from that list, it outputs a structured plan of at most three steps, and plain code checks it before anything runs."
Q5.2 — "What is the exact Q&A layer?"
The short version (say this first):
"It's a set of question and answer pairs that a person wrote and approved.
When one of them matches, we return that answer word for word —
we don't call the generation model at all."
What goes in there: "Only the facts where being close is not good enough — a warranty period, a certification number, a pricing rule. Things that end up in a quote or a contract."
How an item is created:
1. We parse the source document.
2. A model drafts candidate question-answer pairs — it only drafts.
3. A person reviews them in a console: approve, edit, or reject.
4. Publishing writes an audit record, and the item can carry start and end dates.
The rule I'd emphasise: "Nothing is searchable until a human publishes it. A draft can never be returned to a user."
How matching works: "Each item is indexed by the different ways people ask it. We search phrasings against phrasings, keep the best phrasing per item, and if nothing is good enough, we return nothing rather than the closest guess."
The story that shaped it, if you have time: "A similarity score alone was not enough. In testing, 'what is the mAP at 416 by 416' matched the stored '320 by 320' answer at 0.827 — one digit different, and the score was right in the middle of the genuine matches, so no threshold could catch it. We added a tiny code check — if the user's question contains a number that the stored question does not, we don't treat it as a match — and then one cheap model check asking 'is this answer about the thing they asked about?' On 27 test cases that took us from 21, to 23, to 26, and we lost no correct answers."
Why this layer exists at all: "This is the layer where the model is not allowed to speak. That's what lets us show a 'verified' badge next to the answer, with a link to the source."
Q5.3 — "How does the governed text-to-SQL work?"
The short version (say this first):
"The important part is what the model is not allowed to do.
The model never writes SQL. Every query is a template that a human wrote and approved.
At runtime the model can only make small, limited changes inside that template —
change a filter value, drop a column, drop a grouping. Anything else is refused, with a reason."
The four steps:
1. Connect to a read-only copy of the business database.
2. Sync the schema, then build a semantic layer — plain-English descriptions of tables, columns and code values.
Governance lives here: a switched-off column simply does not exist for the system.
3. Generate candidate questions and SQL templates — a human reviews and publishes them.
4. At runtime: the template arrives as a candidate, the planner picks it, the model adjusts inside the rules,
and then it goes through the execution gate.
The execution gate (this is the line they'll like): "Single SELECT only. Every table and column must be in the approved list. A row limit is forced on. There is a read timeout. And the database account is read-only anyway. So there are four layers, and each one assumes the previous one failed."
The payoff sentence: "Because the model never produces SQL text, SQL injection is not a risk we mitigate — it's a risk we removed by design."
Numbers: 20/20 end-to-end, 7/7 on refusal gates, zero failed executions, about 8 seconds per question.
Q5.4 — "How does the RAG work?"
Say this first — where it sits:
"In the design, document search is one capability the planner can call, and only on agents where we
allow it. The chunks are what it returns — they never compete with the approved answers for routing."
Then the pipeline: "It's hybrid, and the part I care about most is chunking — most RAG quality problems are chunking problems, not model problems."
The pipeline, in plain words:
1. Parse the PDF, then clean it.
2. Chunk by headings, and cut on sentence boundaries. A table or a figure is never split.
3. Use a model to write a description for each figure and table, so images become searchable text.
4. Search two ways at once: vector search for meaning, keyword search for exact terms like product codes.
5. Merge the two rankings, then re-rank the top results with a stronger model.
6. Expand a little context around each hit, and always return citations — document, page, section.
Why two searches: "Vector search is good at meaning but weak on exact strings. Our users type product codes and internal jargon. Keyword search catches those."
The measurement story: "Re-ranking gave +3 hits, about +14 points on a 31-question set. Then I checked top-5 versus top-10 — no extra gain. So I kept it at five. I didn't want to triple the prompt size for nothing."
Honest line: "I deliberately do not use a score threshold here. On my data the good and bad scores overlap, so a threshold would reject real answers. Not every stage deserves a threshold."
Q5.5 — "What guardrails are in the system?"
The short version (say this first):
"I think of guardrails as five positions, not one layer.
The earlier the position, the cheaper and the more certain it is."
1 · Before anything is stored. Nothing is searchable until a human reviews and publishes it. Publishing writes an audit record. "The strongest guardrail is not letting bad data in."
2 · When we build the candidate set. Only approved capabilities, only from this agent's bundle, each type judged on its own cut-off, and an empty list is allowed. The guardrail here is what the planner is even allowed to see.
3 · At planning. Before we run anything: step count is limited, every step must come from the retrieved candidates, and required values must come from the user — the model cannot invent a date or an account. These are plain code checks, not model judgement. Only the last check is a model: does this plan actually answer this question?
4 · At execution. The SQL execution gate and the read-only database, as I described.
5 · After generation. Curated answers are returned word for word, so nothing can drift. Document answers must carry citations. There is an optional check that flags sentences the sources don't support.
The senior line — use it if you have time: "One thing I care about is which way a guardrail fails. If a check breaks or is unsure, anything that decides whether we execute must fail closed — refuse. A checker that fails open is just decoration."
And the cost, if they ask: "The cost of guardrails is false refusals. So that has to be measured too. Nobody raises a ticket for an answer they never received."
Q6 "Anything you would like to add?"
"If I had to compress the whole thing into a few sentences, it would be these."
Answer shape: then read the six lines below slowly. Slower than feels natural.
1. "Not all questions carry the same risk, so I split knowledge into three layers by error tolerance."
2. "Accuracy over coverage. The system is allowed to say it doesn't know."
3. "When we hit a curated answer, we return it word for word — no model call."
4. "The model never writes SQL. It only adjusts values inside an approved template."
5. "Every plan is three steps or fewer, and plain code checks it before anything runs."
6. "Every answer has citations and a trace, so you can see why it answered that way."
"That's a good question — let me think for a second."
"Let me give you the short answer first, and then go deeper if you want."
"I want to be honest here — that part isn't built yet. What we do today is…"
"Sorry, could you rephrase that? I want to make sure I answer the right thing."
→ 这四句比任何专业词都有用。说慢一点，没关系。
Don't list every technology. Say what problem it solved.
Don't say "it's very accurate". Say the number, then say what it cost.