GovernanceSEP 30, 2026Ishaan Mirchandani, Anthony Ombrore

Designing agents that answer to governance at enterprise scale

How an enterprise should decide which decisions an AI agent takes on its own, which it takes within limits, and which always stay with a person. And why the record of what the agent did, and why, matters more than the instructions it was given.

On this page

The situation
What the evidence shows
What that means
How we design for it
What we don't know yet
Sources

The situation

A UAE motor insurer has to tell a claimant within three days which documents are missing, and settle within fifteen days of receiving a complete file [1]. At 11pm a policyholder messages on WhatsApp: "Has my claim been approved?" An agent can read the claim's status from the claims system, list the two documents still outstanding and explain what the policy says about the excess. The one thing it must never do is answer that question with a yes or a no of its own making.

Every enterprise that deploys an agent draws a line like this. A developer decides whether the agent may restructure a payment plan. A hotel decides whether it may issue a goodwill voucher, and up to what value. The hard part is not agreeing that a line exists. It is deciding where it goes, making sure it holds when nobody is watching, and being able to show afterwards exactly what the agent did and on what basis.

What the evidence shows

The frameworks scale oversight to risk. None asks for a person on every decision.

The NIST AI Risk Management Framework, a voluntary US standard, defines risk as a composite of the probability of an event and the magnitude of its consequences, and asks organisations to document how humans oversee a system and to keep mechanisms to "supersede, disengage, or deactivate" it [2]. ISO/IEC 42001, a certifiable management standard, has a single control on logging, which requires event logs to be kept at least while the system is in use [3]. Neither says where a person must sit.

The EU AI Act is the most specific. For systems it classes as high risk, Article 14 requires that the people overseeing the system can monitor it, override or reverse its output, and stop it. It is the only binding text located for this article that names automation bias, the well-documented tendency to accept a machine's output without checking it, as something overseers must be helped to resist [4]. Its criteria for classing a use as high risk include how far the system acts on its own and "the extent to which the outcome produced involving an AI system is easily corrigible or reversible" [5].

Two points of precision matter for a Gulf reader. The high-risk list is organised by topic, and customer service and claims handling are not on it; only life and health insurance risk assessment and pricing are [6]. And the Act reaches a company outside the EU that uses an AI system only where the system's output is used inside the EU [7], so a UAE or Saudi enterprise serving Gulf customers is outside its scope. Its high-risk obligations now apply from December 2027 [8].

Gulf law is keyed to the effect of a decision, not its subject.

UAE federal data protection law gives a person the right to object to decisions "resulting from automated processing, including profiling, particularly those decisions which have legal impact on or adversely affect" them, and requires the controller to "include the human element in reviewing automated processing decisions at the request of the Data Subject" [9]. The law has been in force since January 2022, but its executive regulations had not been issued as of mid-2026 and the federal Data Office never became fully operational, so there is no enforcement apparatus yet [10]. It does not apply in the DIFC or ADGM, which have their own regimes.

The DIFC's rules are the most developed in the region. Regulation 10 permits a system to process personal data commercially only if its purposes are human-defined or human-approved, or, where the system sets its own purposes, it does so "solely within the limits of human-defined constraints"; and Article 38 lets an individual object to a solely automated decision with legal or "other seriously impactful consequences" and require manual review [11][12]. These bind DIFC entities only. In Saudi Arabia, the implementing regulations of the Personal Data Protection Law, enforced since September 2024, require explicit consent for solely automated decisions and an impact assessment where decisions rest on automated processing; this rests on a law firm's summary, as the regulator's English text was not reachable [13].

For insurers licensed onshore in the UAE, the Central Bank's February 2026 guidance note sets out three oversight tiers. "Human-out-of-the-loop", where the system acts without direct human involvement, "should only be utilised for low-risk, non-material processes", and consumers "should be able to request human review or explanation of AI generated decisions" [14]. It is guidance, not regulation, and it applies to licensed financial institutions, not to developers or hotels.

The person checking the machine is a weaker control than it looks.

Human in the loop means a person approves each decision before it takes effect. Human on the loop means the system acts and a person monitors the results. Both depend on the person engaging, and decades of research say that is where it fails.

In a controlled flight-simulation study, participants given a highly but imperfectly reliable automated aid missed 41% of the events it failed to flag, and followed its wrong recommendation 65% of the time even when other instruments contradicted it [15]. In a study of 554 lay participants predicting pretrial outcomes, of the 304 given a risk score only 23.7% did better than the score alone, 64.1% did worse, and their confidence was negatively correlated with their performance [16]. Explaining the machine's reasoning does not fix this: in a study of 1,626 participants, explanations increased agreement with the recommendation whether it was right or wrong [17]. OWASP's 2025 list of the top ten risks in AI agent applications includes human-agent trust exploitation, where confident, polished explanations mislead operators into approving harmful actions [18].

The public failures share one shape. In the Dutch childcare benefits scandal, the civil servants who reviewed flagged applications could not see the information used to assign the risk score [19]. In Australia's Robodebt scheme, staff had previously reviewed around 20,000 high-risk files a year; the automated system sent out notifications at around 20,000 a week [20]. In each case the reviewer lacked at least one of three things: time, information or authority. Where any of the three is missing, the human check is a formality.

Gates that fire often are ignored.

A gate that interrupts every routine action trains people to click through it. In a study of 1.27 million clinical reminders across 112 clinicians, the likelihood that a reminder was accepted fell by 30% for each additional reminder in the same encounter [21]. So a gate should sit only where it is needed. One vendor's analysis of agent traffic on its own platform, not peer reviewed, found that only about 0.8% of agent actions appeared to be irreversible [22]. If that holds in customer communication, concentrating human attention on the irreversible actions is affordable in a way that reviewing everything is not.

The agent's own sense of when to stop cannot stand alone.

Confidence-based escalation means the agent hands to a person when its estimate of being right falls below a threshold. That estimate is a real signal, but a biased one. A 2026 preprint testing eight language models on tasks with real human decision data found that the confidence at which each model chose to escalate ranged from about 53% to above 100%, with two models escalating everything, and that in 66% of model and condition combinations the model overestimated its own accuracy [23]. On a benchmark of 2,000 agent safety tests, no agent scored above 60%, and the authors concluded that "reliance on defense prompts alone may be insufficient" [24]. A threshold has to be measured for the specific model before deployment, and it belongs alongside rules the agent cannot argue with, not in place of them.

What a usable record contains is already settled elsewhere.

A flight data recorder logs what the aircraft did and what the crew did on one time base, so the cockpit voice recorder can be laid over it. Cockpit recorders on new large aircraft now retain 25 hours rather than two, because two-hour recordings kept being overwritten before anyone knew they were needed, including after a 2017 incident at San Francisco [25]. European investment firms must record conversations about client orders "even if those conversations or communications do not result in the conclusion of such transactions", for five to seven years [26]. US securities rules require records kept in write-once storage or with "a complete time-stamped audit trail" of every change and who made it [27]. The AI industry is converging on the equivalent field list: the OpenTelemetry conventions for generative AI, an open observability standard, name the model, the instructions, the input and output messages, each tool called with its arguments and result, and the agent and conversation identifiers [28]. A tamper-evident log chains its entries so each carries a cryptographic fingerprint of the one before, which makes any later edit detectable [29].

The record decides disputes. In February 2024 a Canadian tribunal ordered Air Canada to compensate a passenger after its website chatbot told him he could apply for a bereavement fare within 90 days of buying a ticket, contradicting the airline's policy page. Air Canada argued the chatbot was, in effect, a separate legal entity responsible for its own actions. The tribunal called that "a remarkable submission" and held that "it should be obvious to Air Canada that it is responsible for all the information on its website" [30]. The passenger had a screenshot. A 2025 UK regulator review of 23 home and travel insurers, with no reference to AI, found that in many cases the minutes of key claims committee meetings "lacked sufficient detail to prove meaningful discussion, challenge or decision-making" [31]. The standard of evidence does not change with the kind of agent that made the decision.

What that means

The working hypothesis for this article was that decisions should be sorted by consequence and reversibility rather than by topic, and that governance has to be enforced by what the agent is able to do and by a complete record, not by instructions alone. The evidence supports both, with two corrections.

First, consequence and reversibility are the right axes, but topic still matters. Legislators use consequence and reversibility to decide what to regulate, then regulate by topic [5][6]. Some decisions are reserved whatever their reversibility: by law, such as a medical judgement; by a regulator's expectation, such as a claims decision, which the Central Bank's guidance would not class as a low-risk, non-material process; or by the business itself, such as anything already with a regulator. A sorting scheme needs a rule that removes those from the agent entirely.

Second, the judgement of consequence must be made in advance, per type of action, by people. The agent's runtime sense of risk is too variable to carry it [23][24]. Confidence-based escalation is a second line, not the first.

The enforcement hypothesis holds, with an addition. Capability limits and a complete record are the floor. But the human approval gate is itself a control that fails in predictable ways, so it has to be designed for the reviewer: fire rarely, arrive with facts rather than a persuasive summary, and go to someone with the authority to change the outcome. A record prevents nothing on its own; it makes accountability possible afterwards and, if it is read, makes the system better. The trade-off is real. Fewer gates mean more decisions taken without a person; more gates mean a person who stops reading. The gate should sit where the irreversible actions are, and almost nowhere else.

How we design for it

Actions are sorted before deployment, and the sorting is enforced in what the agent can do. Each action an agent can take in a customer's systems falls into one of four zones. Actions it takes and records: reading a claim's status, listing missing documents. Actions it takes within limits the business sets: a rebooking within fare rules, a voucher up to a value. Actions it prepares for a person to approve. And actions it is never given. In insurance, the agent never assesses liability, sets a settlement amount, or approves or declines a claim, and this holds because those decision actions are not exposed to the agent at all, not because it has been told not to use them. The business sets, and can change, which conversations go to a person, on what triggers, and which topics are off limits. In the second zone, the customer's rules, such as approval limits and entitlements, are enforced in the architecture rather than written as instructions the agent might ignore, and actions the business marks as sensitive wait for a person's approval.

The agent's confidence is a second trigger, and the handover gives the person facts, not a case. A conversation hands to a person when the agent is not confident enough; given the evidence on overconfidence, the threshold is measured per model before deployment and layered on the rules above. The agent asks a clarifying question when it is unsure rather than guessing. The person who takes over receives the full conversation and a summary, and given what explanations do to reviewers, the brief is designed to show what the customer said, what the agent did and where its uncertainty lies, not an argument for an outcome.

The record is built for a dispute in two years, not a dashboard tomorrow. Every call can be recorded, subject to consent, and transcribed as it happens; every conversation is summarised when it ends and its outcome classified; and the agent answers only from the customer's approved material, so an answer can be traced to its source. A full audit log records every action the agent took and why, in a form where any later change is detectable, with version control so every change to an agent is recorded and reversible, and retention windows the customer sets. The model version and the identity of any person who approved a gated action belong in the same record.

Review is for learning, not blame. European aviation pairs its recorders with a reporting regime built on a "just culture", in which people are not punished for honest errors and safety information may not be used "to attribute blame or liability" [32]. Every conversation can be searched. Every conversation is scored against the customer's own standard, so that review is systematic rather than sampled and feeds back into the agent's design.

What we don't know yet

No published study compares a deployment gated by consequence and reversibility with one gated by topic on the harm that actually resulted. The sorting described here follows from the evidence, but it has not been tested against the alternative.

There is no audited data on how often insurance claims reviewers or customer-service supervisors override an automated recommendation, and nobody has published how large a sample of conversations must be reviewed to hold a given error rate. What exists is litigation allegation and industry custom, not measurement.

If human accept or correct decisions are used to tune when an agent escalates, and those humans are themselves subject to automation bias, the escalation signal inherits the bias. No study appears to have measured this.

Gulf-specific evidence is thin. There is no peer-reviewed study of oversight behaviour in Arabic-language or Gulf customer service, and no published court or regulator decision in the UAE or Saudi Arabia concerning an AI system's statements to a customer could be located. Three regulatory questions were open at the time of writing: when the UAE's executive regulations will be issued, whether the DIFC's 2026 consultation on Regulation 10 has been enacted, and whether Saudi Arabia's Insurance Authority has adopted the shorter settlement window for individual claims it proposed in late 2025 [33].

Sources

Get new research as it's published

Occasional emails when we publish.

Product overview

The platform

Specialise

Centralise

Complete

Industries

Industries

Every industry

Product overview

The platform

Specialise

Centralise

Complete

Industries

Industries

Every industry

Designing agents that answer to governance at enterprise scale

The situation

What the evidence shows

The frameworks scale oversight to risk. None asks for a person on every decision.

Gulf law is keyed to the effect of a decision, not its subject.

The person checking the machine is a weaker control than it looks.

Gates that fire often are ignored.

The agent's own sense of when to stop cannot stand alone.

What a usable record contains is already settled elsewhere.

What that means

How we design for it

What we don't know yet

Sources

Get new research as it's published

Read next

Testing agents on conversations that switch language mid-sentence

Why policy belongs in the architecture, not the prompt