Are we actually liable for what it says?
Yes, and this is no longer a hypothetical.
In February 2024 the British Columbia Civil Resolution Tribunal decided Moffatt v. Air Canada. A passenger asked the airline's chatbot about bereavement fares after his grandmother died and was told he could buy a full-fare ticket and claim the discount retroactively within ninety days. The airline's actual policy did not allow that, and the correct policy sat on a page the chatbot had itself linked to.
Air Canada argued that the chatbot was a separate legal entity responsible for its own statements. The tribunal rejected that, finding the airline responsible for all the information on its website whether it came from a static page or a chatbot, and awarded CAD $812.02 in damages and fees.
The sum is small and the principle is not. An organisation owns what its answering surface says, whether the statement was authored deliberately or generated to fill a gap. Every control described below exists because of that sentence.
Note the specific shape of the failure, because it is the one your own system will have. Nothing was hacked. The bot did not misread a document. It produced a fluent, plausible, confident policy that had never existed, and it did so while linking to the page that contradicted it.
What separates a safe system from a risky one?
Where the answer comes from. Not the model, not the vendor, not the interface.
A generated system is given a description of your institution and produces answers consistent with that description. Asked something outside it, the system does not know it is outside it, because it has no notion of where its knowledge stops. It answers anyway, in the same confident register it uses for things it does know. That is precisely what happened to Air Canada.
A grounded system answers from specific approved documents and can tell whether it found support. Asked something the documents do not cover, it has an available answer that is not an invention: it can say so.
The distinction is invisible from the outside. Both are conversational, both sound authoritative, both are usually the same underlying model. The only reliable way to tell is to ask a question you know your documents do not answer, and see whether the system admits it.
Make that your acceptance test. Write down ten questions your policies genuinely do not cover, ask them, and count the inventions. That number is your risk, and no amount of assurance substitutes for measuring it.
Why does refusing matter so much?
Because a system that always answers cannot be trusted with the questions that matter.
Institutions receive edge cases by definition. The unusual exemption, the transitional rule, the case that falls between two policies. Those are exactly the questions where documents are silent, and exactly the questions where a confident invention does the most damage, because the person asking is relying on the answer to make a decision.
So the behaviour to require is unglamorous: when support is absent, say so, and route to a human. "I do not have anything on that, here is who can tell you" is a good answer. It is not a failure of the system; it is the system working.
This is worth writing into the specification and into the acceptance criteria, because it will be the first thing a stakeholder asks to soften. A refusal rate above zero looks like a defect on a dashboard. It is the opposite.
What controls should we insist on?
Six. They are ordinary, and their absence is what you are checking for.
- Named sources on every answer. Which document, which passage, which date. Without this you cannot audit an answer, only argue about it.
- An approval step before publication. Somebody with the authority to speak for the institution reviews what the system may draw on.
- Refusal when support is missing, with an escalation path to a person.
- A correction route that fixes the source. When an answer is wrong, you should be correcting the document that produced it, not patching an instruction on top.
- An audit trail. What was asked, what was answered, and what it drew on, retained long enough to investigate a complaint.
- An off switch a named person can reach without a support ticket.
Ask a supplier to demonstrate each of these live rather than describe them. The demonstration takes twenty minutes and is more informative than any document they will send you.
Where should we start?
With one body of knowledge, one audience, and questions whose answers are already written down.
The instinct is to start where the pain is worst, which is usually the highest-stakes, least-documented area. That is the worst possible pilot: the documents are thin, so grounding has little to hold, and the consequences of an error are large.
Start instead where the documentation is good and the questions repeat. Admissions requirements, opening arrangements, process steps, eligibility rules. You learn how the review workflow feels, your reviewers learn what approving a source means, and mistakes are recoverable.
Then measure two things before widening. What proportion of answers were traceable to an approved document, and what proportion of refusals were appropriate. Those two numbers tell you whether it is safe to expand.
How does Eternal Gardens handle this?
Approved sources in, named sources out, refusal in between.
Material is uploaded as sources and reviewed before it counts. Answers are drawn from that material and cite what they used, with the title, the date and the passage, so a wrong answer is traceable to the document that produced it and corrected there. Where nothing supports an answer, the system says so rather than improvising.
Owner control is the part institutions ask about most. You decide what goes in, who may approve it, what is public, and what is withdrawn. Nothing is published to a public surface because a supplier thought it would be useful.
What we do not claim: that it cannot be wrong. A grounded system can still misread a document, quote a superseded policy that is still in the source set, or answer a question adjacent to the one asked. Which is why the audit trail and the correction route are on the list above, and why review is a role rather than a launch task.
Common questions
Can we make it refuse anything outside our documents entirely?
You can require refusal when nothing supports an answer, which is the control that matters, and you should test it rather than assume it. What no supplier can offer is a mathematical guarantee across every possible question, so the sensible posture is a documented refusal behaviour, an acceptance test with questions you know are uncovered, and periodic re-testing rather than a one-time assurance.
Who is responsible if it gives someone wrong advice?
On the evidence so far, the organisation. A Canadian tribunal held an airline to a policy its chatbot invented, rejecting the argument that the bot was a separate entity answerable for its own statements. Plan on the basis that anything your answering surface says is something you said, and put the review and audit controls in place accordingly.
Should staff or the public go first?
Internal first, in almost every case. Staff can recognise a wrong answer, they will tell you when they see one, and the consequences of an error are contained. It also builds the review habit before anything is public. Once internal answers are consistently traceable and refusals look appropriate, a narrow public surface is a much smaller step.
How do we keep it current as policies change?
By making the source set the thing you maintain rather than the system. When a policy changes, the document changes and the system answers from the new one; that is the whole advantage of grounding over a system trained on a description. It does mean somebody owns removing superseded documents, because a grounded system will faithfully quote an out-of-date policy that is still in its sources.
What should we log, given privacy obligations?
Enough to investigate a complaint and no more: the question, the answer, the sources drawn on, and a timestamp. Decide the retention period deliberately and write it down, apply the same data protection rules you apply to any other record of a public interaction, and be careful about logging identifiers alongside questions that may themselves be sensitive.