← All posts
16 min read

The Day a Hallucination Became Bank Policy

bfsigovernanceai-risk

Peter Baumann named the accountability gap. Here is how it closes.


A Tuesday in Lucknow

A loan officer in Lucknow asked an AI a question last month.

The answer was wrong.

By Friday it was bank policy.

Nobody noticed. A cloud model produced a confident paragraph about soybean default patterns in Bundelkhand. A risk analyst pasted it into a memo. A second analyst quoted the memo. A junior policy officer cited both. Inside seventy-two hours, the wrong answer had cleared three desks and entered a section of the credit handbook.

There is no audit trail. There is no model card to subpoena. There is no rejection log. The handbook now contains a hallucination. The bank cannot identify which sentence is the hallucination.

This is not an edge case. AI-driven credit underwriting and straight-through processing for small-ticket loans are now mainstream Indian BFSI deployment.9 The Lucknow scenario sits squarely inside the emerging mainstream of how Indian banks decide credit.

This is the accountability gap.

Peter Baumann named it last week. His diagnosis is the cleanest piece of writing on AI risk in 2026. [1] He stopped one door short of the answer. This post walks through that door.


The Diagnosis Worth Reading Twice

Most AI commentary in 2026 reads like a list of failure modes. Hallucinations. Prompt injection. Agent drift. Citation fabrication. Baumann's essay does something rarer. It asks where the risk actually lives.

His sentence is worth reading twice.

"At the point where an organization deploys a probabilistic system in a context requiring deterministic accountability, without the verification infrastructure that makes those two things compatible." [1]

The gap is not that LLMs hallucinate. The gap is the absence of infrastructure that reconciles probabilistic generation with deterministic accountability. Baumann names four LLM failure modes.

Hallucination. Arithmetic and reasoning errors. No data lineage. Context rot.

His prescriptions are deliberately system-level. LLMs as natural-language interfaces over deterministic backends. A Single Version of Truth. Retrieval grounding. Audit trails. Lineage. Human review checkpoints calibrated to stakes.

He stops short of saying how to build that infrastructure.

That is what this post is about.


The Villain Has a Name

Before the architecture, name the enemy. Most AI commentary sells a better mousetrap. The mouse is bigger than that.

The villain is not "LLMs." LLMs are tools. The villain is the cloud-AI-as-default deployment pattern in Indian regulated enterprises. It has a track record. It has dead solutions in its wake. And it has a closing date.

The villain has a track record. Hallucination rates across leading commercial models run between 15 percent and 52 percent in enterprise benchmarks. The Stanford AI Index 2026 measured rates as high as 94 percent across the worst-performing models in 26 leading commercial systems.[2] Equity research notes ghostwritten by chatbots. Insurance underwriting decisions traced to a model that "thought" a regulation existed. A health system that fed patient histories to a U.S. inference endpoint and called it "anonymization." Each of these is a real conversation in the Indian enterprise market in 2026.

The propagation mechanism has a name. Conformity bias. When one agent makes a confident assertion, the next aligns rather than pushes back. Errors travel as facts. No agent questions upstream outputs.[3] Lucknow is not a story. Lucknow is the documented behaviour of agent chains, compressed into seventy-two hours.

The legal precedent is no longer hypothetical. Damien Charlotin's tracker of AI-generated court errors crossed 1,300 cases globally in April 2026, with over 900 in U.S. courts alone. Lawyers have been sanctioned. Air Canada was held liable in 2024 for misinformation produced by its own chatbot.[4] This is not a future risk. It is a docket of existing rulings.

The villain has dead solutions. Just-add-RAG. Data-residency toggles. "Private endpoints" that route through three foreign data centres. Fine-tuning the cloud model on your own data. None of these closes the gap. They wallpaper over it.

The villain has a closing date. DPDP Act 2023 becomes fully enforceable on May 13, 2027. The penalty schedule is tiered. 
- Failure to implement security safeguards resulting in a personal data breach: up to ₹250 crore. 
- Failure to notify a breach: up to ₹200 crore. 
- Breach of children's data obligations: up to ₹200 crore. 
- Significant Data Fiduciary obligations: up to ₹150 crore. 
- Any other provision: up to ₹50 crore.[5]

An Indian bank that ships a customer's PII to a foreign inference endpoint without adequate safeguards sits in the highest tier. All of these tiers are material. All of them attach by 13 May 2027.

That is the mouse. Now the mousetrap.


The Regulator Already Wrote the Specification

Indian regulated enterprises are not building this on a hostile regulator's blind side. They are building it on a regulator who has already named what good looks like.

On 13 August 2025 the Reserve Bank of India released the Framework for Responsible and Ethical Enablement of AI, known as FREE-AI. An eight-member committee chaired by Prof. Pushpak Bhattacharyya of IIT Bombay, with members drawn from HDFC Bank, Microsoft India, and the former NASSCOM presidency, articulated seven guiding Sutras and twenty-six actionable recommendations across six pillars: infrastructure, policy, capacity, governance, protection, and assurance.[6] The seven Sutras are trust, fairness, accountability, explainability, resilience, safety, and sustainability. The framework is unambiguous on one point. Responsibility for AI deployment rests squarely with the institution implementing it. Not with the model vendor. Not with the cloud provider. With the bank.

The same RBI surveyed 612 regulated entities. Only 20.8 percent are actually deploying AI today. Fewer than 15 percent conduct post-deployment monitoring. Board-level AI governance structures remain scarce.[7]

Read those numbers carefully. The accountability gap is not a future risk distributed across a mature market. It is the present condition of one fifth of the regulated population, plus the structural unpreparedness of the four fifths who have not yet started. Both groups must be DPDP-compliant by May 2027.

FREE-AI tells the bank what to build. DPDP tells the bank when. Neither tells the bank how. That is the door this post walks through.


The Model Is the Least Important Decision in the Chain

We built AdiOS, the Circular Operating System, on a single category bet.

The Circular Operating System.

The architecture follows from that bet. The most consequential implication is that the model itself becomes interchangeable. A loan officer's question does not go to "Claude" or "Sarvam-2 8B" or our own Bharat-LLM-7B.[12] It goes to a deterministic router that resolves an engine from a five-dimensional matrix.

Figure 2: router becomes the accountability surface

That alone is not the answer. It is the precondition for the answer.


Memory Consultation, Before the Model Runs

Baumann calls for retrieval grounding and a Single Version of Truth. AdiOS makes this structural, not advisory.

Before any InferenceEngine::infer(prompt, context) call fires, the router must do four things in order.

  1. Query adios-seedvault. The personal-tier, ephemeral, in-process memory.
  2. Query adios-neuralmesh. The org-tier, sovereign, persisted memory.
  3. Stitch the two views into the inference context.
  4. Emit an InferenceRoute triple carrying cos:memoryConsulted, cos:contextHash, and cos:issuerDID.

Skipping memory is a SHACL violation. Not a warning. Not a soft fail. The InferenceRouteShape rejects the route at SPARQL emit time. The engine call never fires.11

There is no deadline-pressure path that bypasses retrieval grounding because the path does not exist in the type system.

What Baumann calls a Single Version of Truth is, in AdiOS, the sovereign knowledge graph itself. CRDT-merged across the mesh. Queryable in SPARQL by the customer. Signed at every link. The LLM consults it. The LLM does not author it.


The Gate That Refuses to Open

The accountability gap shows up most sharply when sensitive data goes to a cloud model. AdiOS treats this as a structural constraint, not a deployment recommendation.

Figure 3: Sovereignty gate. Empty by construction, not by claim.

A bank's CTO can run their own audit query: "show me every inference call that touched our PII data and ran on a non-AdiOS-controlled surface." The result set is provably empty. Empty by construction. Not empty by claim.

This is what compliant by architecture means in code. Not in marketing copy.


Hallucinations Cannot Become Institutional Memory

Baumann's most pointed observation is about agent chains.

"Hallucinations become premises of another agent's plan, triggering real tool calls, corrupting shared memory, or initiating actions that are difficult to reverse."1

This is exactly the Lucknow incident. A wrong answer became a memo. A memo became a citation. A citation became policy.

AdiOS blocks this by making memory writes physically separate from inference engines. Engines are pull-only. They consume context. They cannot write back.

After every inference completes, four things happen.

  1. The router emits cos:InferenceCompleted to CDIF, the common data ingestion fabric.
  2. A NeuralMesh subscriber consumes the event via the Compounding trait, ADR-067.
  3. The decision of whether the learning, not the raw output, should compound into org-tier memory is gated.
  4. AVIS Gate 5, Agent Vulnerability Intelligence Stage 5, governs LLM-derived learnings before they touch the knowledge graph.

AVIS Gate 5 evaluates three things on every candidate learning: source reliability of the inference path, cross-corroboration count from independent observations, and confidence threshold against the destination tier. A candidate that fails any one is refused. The graph never sees it.

Then the cortex confidence orbit imposes a three-tier human ladder. This is Patent S8. The Circular Knowledge Compounding Loop.

Figure 4: Three-tier confidence orbit.

The probabilistic system can produce the candidate. Only deterministic human review can promote it. This is exactly what Baumann calls human review checkpoints calibrated to stakes. Enforced as a graph constraint. Not as a process document.


Lineage Is a Record on Every Call

Baumann's third failure mode is no data lineage. Every AdiOS inference call produces three artefacts whose existence is mandatory.

The lineage primitive lives outside the LLM. The model does not have to "tell us" where its answer came from. The system already knows. Retrieval was forced before the engine ran. The route was recorded before the engine returned.

This satisfies a very specific 2026 regulatory target. SEBI's June 2025 consultation paper proposed five-year retention of AI input and output data for entities operating in the Indian securities market.8 An append-only C-Box event store, queryable in SPARQL with content-hash integrity, satisfies a five-year retention requirement with structural room to spare. The lineage architecture was not designed for SEBI. SEBI describes a target the architecture already exceeds.


Context Rot Does Not Apply When Context Is Owned by the Graph

Baumann's fourth failure is progressive degradation of coherence as temporal and causal relationships exceed what the model can maintain.

In AdiOS, the model never carries that state. The knowledge graph does. adios-deeproot for facts. adios-neuralmesh CRDT-merged store for org tier. On every call, context is re-stitched from the graph.

The LLM is stateless with respect to organisational memory by design. There is no rot to manage because there is no long-running model state to rot.

The honest acknowledgment is this. Re-stitching context on every call moves the engineering problem. Retrieval quality, CRDT convergence latency, and graph completeness become first-class concerns inside AdiOS. These are tractable disciplines with measurable answers. Context rot is not.


The Proprietary-LLM Question

A reasonable pushback. "If you build your own LLM, surely the accountability story is different."

It is. But in the opposite direction from the cloud-LLM trade.

Our own models inherit the same accountability surface as third-party engines. Plus three additions.

  1. DID-attributed training lineage. Each proprietary model carries a W3C DID.10 Its training corpus, fine-tune steps, and eval gates are signed and replayable. We can answer "what did this model see?" Frontier models structurally cannot.
  2. Same trait, same SHACL, same matrix. A Bharat-LLM-7B running on a branch GPU is a cos:InferenceModel. It is selected by the same five-dimensional matrix. It emits the same InferenceRoute. It hits the same SovereigntyRoutingShape. Owning the model does not earn it accountability privileges.
  3. Reverse-direction compounding. When an Enterprise-tier inference is reversed by a human after promotion, that reversal flows back through ADR-067 into the model's own evaluation set. Our models improve via verified institutional reversal. Not by scraping more web data.

AVIS Gate 5 applies symmetrically. A hallucination from our own model is no more privileged than a hallucination from Claude. It hits the same gate before it can compound. Proprietary status is not a trust signal in the architecture.

That is the whole point.


Where the Diagnosis Stops, the Architecture Picks Up

Baumann's prescriptions are correct. Each one needs a structural counterpart in code. Here is the mapping.

Every row is a Baumann remedy. Every row is an AdiOS component already in the repository. None of this is roadmap.


The Same Question, Asked Again

Return to Lucknow.

A loan officer asks an AI a question. This time, the question runs through AdiOS.

The router checks the surface. Branch terminal, Tier 3 form factor. It checks the exposure. Customer-adjacent. It checks the actor. Human loan officer with a verified DID. It checks the tier. Sovereign. It checks the classification. PII present.

Figure 5: Same officer. Same question. A different ending.

The hallucinated soybean-default paragraph never makes it to a memo. Not because someone caught it later. Because the confidence score sat at 0.62, below the 0.70 floor for personal-tier capture. The system did its job by refusing to remember.

Six months later, the bank's CTO is asked by an RBI examiner: "Show me every AI-influenced credit decision in Q2, the model that produced each, the data it consulted, and the human who signed off."

The CTO runs one SPARQL query. Twelve seconds later, every row appears. With DIDs. With timestamps. With content hashes. With promotion paths. With the names of the two humans who signed each Enterprise-tier promotion.

DPDP becomes enforceable on 13 May 2027. The bank is ready.


What This Means for the Next Twelve Months

The accountability gap Baumann describes is not theoretical. It is a present liability with a deadline.

Indian banks, hospitals, and government departments using cloud AI as it stands today are non-compliant by architecture. The deadline is 13 May 2027. The penalty is up to ₹250 crore for the highest tier of violation, with material penalties of ₹50 to ₹200 crore for every other tier.5 The remediation is not a vendor contract. It is a structural change in how AI is deployed.

RBI's own data shows the scale of the work. Only 20.8 percent of regulated entities are deploying AI today. The remaining 79.2 percent must adopt and comply on the same calendar.7 The 2027 deadline does not distinguish between the two groups.

AdiOS exists for that deadline. Not as a wrapper that calls Claude with a data-residency toggle. Not as a fine-tuned model in a marketplace. As the operating layer that makes any LLM, ours or theirs, accountable in a regulated environment by construction.

The category shift is structural. The structure is what makes it accountable.

The Circular Operating System.

That is the bet. The architecture is the proof.


Coda: The Door Behind the Door

Peter Baumann walked up to the door. He named what was behind it. He stopped because the next step was not his to take.

The next step is infrastructure. The next step is a router that treats the model as interchangeable. The next step is SHACL gates that refuse routes by SPARQL. The next step is a CRDT-merged sovereign knowledge graph that owns context the LLM no longer carries. The next step is a confidence orbit that requires two humans to sign before a probabilistic suggestion becomes institutional truth.

The next step is an operating system.

We built it.


The author is Founder and CTO of AdiOS Platform Private Limited. AdiOS is India's first sovereign AI operating system

Contact: [email protected] · adiosplat.io · https://calendar.app.google/tpqSuHhJC7yemWNL7


References

[1] Peter Baumann, "The LLM Accountability Gap," Data Strategy in a Nutshell (Substack), April 2026. The quoted sentence is from the same essay. Hallucination as premise of another agent's plan is from the same essay. https://peterbaumann.substack.com/p/the-llm-accountability-gap

[2] Stanford Institute for Human-Centered AI, AI Index Report 2026. Hallucination rates across 26 leading commercial models reported between 22 percent and 94 percent. Enterprise BFSI hallucination rates of 15 to 52 percent reported in industry benchmarks across 2025 and 2026.

[3] Redis Engineering, "Conformity Bias in Agentic AI Chains," 22 April 2026. Characterises the propagation mechanism by which one agent's confident assertion locks in false consensus across a downstream chain.

[4] Damien Charlotin, AI Hallucination Cases Database. Cumulative count exceeded 1,300 court cases globally as of April 2026, with over 900 in U.S. courts. Moffatt v. Air Canada (2024) held the airline liable for misinformation produced by its own customer-service chatbot.

[5] Government of India, Digital Personal Data Protection Act 2023. DPDP Rules notified 13 November 2025. Substantive obligations enforceable from 13 May 2027. Penalty schedule: ₹250 crore (failure to implement security safeguards resulting in personal data breach), ₹200 crore (failure to notify breach; breach of children's data obligations), ₹150 crore (Significant Data Fiduciary obligations), ₹50 crore (any other provision).

[6] Reserve Bank of India, Framework for Responsible and Ethical Enablement of AI (FREE-AI), released 13 August 2025. Eight-member committee chaired by Prof. Pushpak Bhattacharyya, IIT Bombay. Seven guiding Sutras: trust, fairness, accountability, explainability, resilience, safety, sustainability. Twenty-six actionable recommendations across six pillars: infrastructure, policy, capacity, governance, protection, assurance.

[7] Reserve Bank of India, internal survey of 612 regulated entities, mid-2025. 20.8 percent currently deploying AI. Fewer than 15 percent conduct post-deployment monitoring. Board-level AI governance structures remain scarce.

[8] Securities and Exchange Board of India, Consultation Paper on Use of AI/ML in Indian Securities Markets, June 2025. Proposed five-year retention of AI input and output data for entities operating in the Indian securities market.

[9] The Economic Times (India), banking and BFSI coverage, "Top Banking Disruptors for 2026." AI-driven credit underwriting, AI scorecards using cash-flow and alternative data, and straight-through processing for small-ticket loans identified as mainstream BFSI deployment patterns.

[10] World Wide Web Consortium (W3C), Decentralized Identifiers (DIDs) v1.0, W3C Recommendation, July 2022. DIDs v1.1, W3C Recommendation, March 2026.

[11] World Wide Web Consortium (W3C), Shapes Constraint Language (SHACL), W3C Recommendation. SHACL constraint violations are themselves RDF triples and are queryable via SPARQL.

[12] Government of India, IndiaAI Mission. India AI Impact Summit 2026, 16 to 20 February 2026, Bharat Mandapam, New Delhi. Sarvam AI selected to build India's sovereign LLM under the IndiaAI Mission, April 2025. Sarvam unveiled 30B and 105B parameter models at the February 2026 summit. Twenty-two scheduled Indian languages covered.


Originally published on LinkedIn on April 27, 2026.