When an enterprise chatbot gives a wrong answer, the cost comes from the answer looking right, not from it being wrong. Someone follows a procedure, passes incorrect information to a customer, or abandons a claim they were entitled to make. All of it happens because the answer sounded certain.
Three things decide whether these systems succeed: finding the right source, tying the answer to that source, and declining to answer when confidence is low.
The retrieval layer sets the ceiling
Knowledge-grounded assistants work by finding passages that contain the answer and passing them to the model. The foundational reference for the approach is Lewis and colleagues, 2020 (Retrieval-Augmented Generation, arXiv 2005.11401).
The simple fact that gets missed: if the right passage is not among the retrieved results, no model can produce the right answer. It will either say it does not know, or construct something plausible from irrelevant text.
In most deployments that arrive with "the model is good but the answers are bad", the model is not the problem. When we measure, the right passage is usually not even in the top ten results.
So evaluation starts with retrieval, not generation. The metric is straightforward: across a set of real questions, how often does the correct passage appear in the top k results. Swapping models while that number is low is wasted effort.
Chunking matters more than it looks
Splitting documents into fixed-length pieces is the easiest method and the one that causes the most trouble. Fixed splitting cuts procedures mid-step, separates a table header from its data, and detaches an exception from the rule it modifies.
What works in practice is respecting the document's own structure. Preserving heading hierarchy and attaching parent headings to each chunk visibly improves retrieval accuracy. A chunk titled "3.2 Exceptions" means nothing alone; the same chunk carrying "Leave Policy > Annual Leave > Exceptions" is both easier to find and correctly interpreted.
If the knowledge base contradicts itself, the model cannot be consistent
This is the problem engineering teams notice last. When the same question is answered differently in two documents, the assistant uses one source sometimes and the other at other times. From the outside this reads as an inconsistent model. In reality the corporate content is inconsistent.
Building an assistant means, in most organisations, reading every document at once for the first time. The contradictions that surface are not a failure of the project; they are its first benefit.
Managing this is a content problem, not a model problem. Every document needs an owner, a validity date and a version, and expired content has to leave the index. Model tuning without that discipline does not produce durable results.
Declining to answer is a capability
An enterprise assistant that can say it does not know is often more valuable than one that answers correctly, because once users learn where the boundary is, they start trusting what is inside it.
Two mechanisms are needed. First, do not generate an answer when retrieval scores fall below a threshold. Second, check whether the generated answer is actually supported by the retrieved passages, and withhold it if not.
There is a trade-off here and it should be stated plainly: raising the threshold reduces wrong answers and increases unanswered questions. The optimum is not in the same place for every organisation. In a bank's internal policy assistant, the cost of a wrong answer is high and coverage can be sacrificed. In an e-commerce shipping assistant, the reverse may hold.
The threshold is therefore a business decision, not a technical parameter, and whoever set it should be on record.
Citations work in both directions
Showing which document and which section an answer came from delivers two benefits.
For the user it makes verification possible. On anything critical, they can open the source and check. That turns trust from "believing the system" into "being able to check it".
For operations it makes correction possible. When a wrong answer is reported and the source is visible, diagnosis takes minutes: the document is wrong, the document is stale, or the wrong document was retrieved. Without citations the same investigation takes days and usually ends with someone changing the prompt.
One caution: permission control applies to citations too. The name of a document a user cannot access can itself carry information. We cover this in the closed-network architecture article.
On the helpdesk side, deflection rate is misleading
The most reported metric in support automation is the share of tickets resolved by the bot. What that metric measures is usually not what people think.
A user who receives an answer and closes the conversation counts as resolved. A user who disliked the answer and gave up, or asked a question, did not understand the response and left, also counts as resolved. The report does not distinguish them.
The more reliable metric is repeat contact rate: the share of users who come back on the same topic within 24 to 48 hours. If deflection is rising and repeat contact is rising with it, the system is deferring tickets rather than resolving them.
A few more worth tracking:
- Rate at which agents request additional information after a handoff, which reflects handoff quality
- Share of answers grounded in a cited source
- Topic distribution of questions the threshold left unanswered
- How often a user rephrases the same question
The third produces a direct work list for the content team. Unanswered questions are the clearest map of the gaps in your knowledge base.
A handoff is a continuation, not a break
What gets passed to an agent should be more than the chat transcript. The package an agent actually needs is: who the user is, what they are trying to do, which sources the assistant consulted, which steps were tried, and what did not work.
Without that package the customer explains everything again, and the gain from automation evaporates at the moment of transfer. It is easy to measure: track how often the agent's first message after transfer asks for information the user already gave. A high rate means the handoff design is not working.
Do not evaluate once
A quality test run at deployment does not describe the system three months later. Documents change, questions change, model versions get updated.
Organisations that keep this under control run a fixed set of 150 to 200 questions drawn from real tickets on every version change, reporting three axes: is the answer correct, is the answer grounded in a source, and did the system answer when it should have declined.
Part of that set should deliberately have no answer in the knowledge base. Without questions the system is supposed to refuse, you cannot measure abstention behaviour at all.
Telling users they are talking to an assistant
For organisations operating in the EU this is now an obligation. Article 50 of the AI Act requires systems interacting directly with people to disclose that fact, and it became applicable on 2 August 2026 (artificialintelligenceact.eu).
Turkey has no equivalent requirement. Our experience is that concealing it gains nothing anyway. Users work it out within a few messages, and having found it hidden, they trust the system less. Where it is stated openly, expectations are set correctly from the start.
Pre-launch checklist
- Has retrieval quality been measured against real questions
- Does chunking respect document structure, is heading context carried
- Does every document have an owner, a version and a validity date
- Who set the abstention threshold, and on what grounds
- Are answers tied to sources, and does citation pass through permission control
- Does the handoff package meet what agents actually need
- Does an evaluation set exist and run on every version
- Is repeat contact rate reported
To see the architecture and the support flows, look at the Chatbot Platform and Helpdesk Solutions.
References
- Lewis, P. et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. NeurIPS. arxiv.org
- EU AI Act, Article 50: Transparency Obligations. artificialintelligenceact.eu
- ISO/IEC 42001:2023, AI management system. iso.org