32% of organisations now run agentic AI in production. The concern those organisations report most is not model quality or integration cost — it is AI agent accountability. 29% name human oversight, specifically preventing rogue actions by the agents themselves (Omdia by Informa TechTarget, 2026). The sample is 2,050 active gen AI adopters across ten countries, so this is not a fringe signal.
In Brief
- AI agent accountability is the gap the current architecture cannot close. Accountability designs assumed human pace, human visibility, and human reversibility. Agentic AI honours none of them.
- Thirty-two per cent of organisations now run agentic AI in production. Twenty-nine per cent say their leading concern is keeping the agent's actions visible (Omdia by Informa TechTarget, 2026).
- The Australian Government's updated AI policy names an accountable officer per use case. It does not name who answers for an agent's action between reviews (Digital Transformation Agency, 2026).
- The board-level question is whether the architecture can answer for a decision made between reviews. Auditing it afterwards is a different question.
Read that carefully. The people closest to running these systems in production are not nervous about whether the agents work. They are nervous about whether the oversight design around the agents still works. That is a different problem, and it sits one layer above the technology itself.
The architecture assumed a human decision-maker
Most accountability architectures — the committees, the approval thresholds, the audit logs, the escalation routes — were designed for a world where decisions were made by people. Three things were quietly assumed about every decision the architecture covered. It would be made at roughly human pace. It would be visible to a reviewer before it took effect. And if it turned out to be wrong, another human could reverse it before too much had moved. None of these properties were ever written down inside the architecture, because they did not need to be. They were the air the design breathed.
Agentic AI changes the air. An agent acts in milliseconds, not minutes. Its decision usually happens between scheduled reviews, not before one. By the time anyone responsible could read the input, a record has been updated, a customer has been notified, or a transaction has cleared. The Governance Institute of Australia (2026) puts it plainly: governing agentic systems “requires going further to address their autonomy and dynamic behavior”, and the gap between adoption speed and governance speed is now “one of the defining leadership challenges of 2026”.
The architecture itself looks the same as it always did. What has shifted is the kind of actor it is being asked to cover.
From inside, nothing visibly broke
Part of what makes this hard to see is that nothing dramatic happened when the agents went into production. Committees kept meeting. Audit logs kept recording. Approval workflows kept firing on the right triggers. From the inside, everything looks like it is working. Nobody has had to stand up and answer the question that would actually test the design.
When organisations do look closely, what tends to surface is uncomfortable. The architecture’s most important job was always implicit: when a consequential decision goes wrong, the architecture is what tells you who answers for it. That depended on the decision-maker still being identifiable, still available, and still answerable when the question got asked. An autonomous agent fails every part of that test. The audit log will tell you what the agent did. It will not tell you who is accountable for the fact that this is the decision the agent was permitted to make at that moment, on that input, without further review. That accountability does not fall out of the audit log. It has to be designed in deliberately, and at most organisations it has not been.
National AI policy still operates at the use-case layer, not the decision layer
The Australian Government’s updated Policy for the responsible use of AI in government took effect on 15 December 2025, with the first mandatory requirements landing on 15 June 2026 (Digital Transformation Agency, 2026). It is the most current accountability framework in Australian public administration for AI. Agencies must maintain an internal register of all in-scope AI use cases, name an accountable official for each one, complete mandatory foundational AI training, and route high-risk uses through an AI Review Committee.
Read what the policy actually covers. It specifies who answers for the decision to approve a use case. It does not specify who answers for a decision the agent took inside that use case, between reviews, at 11:47 on a Tuesday. The accountable officer is named per use case. The Review Committee looks at use cases tabled for it. Both work at the layer where the use case is designed and approved. Neither is structured to reach inside a use case already in production and assign accountability to a specific autonomous action that has already happened.
If the most current national AI policy in Australia still operates at the use-case layer rather than the decision layer, the corporate governance frameworks designed three, five, or ten years before agentic AI was deployable almost certainly do too. This is not a criticism of the DTA framework. It is the historical shape of accountability design more broadly. Accountability has always assumed the actor would be reviewable in advance. Agentic AI is the first actor at scale where that assumption no longer holds.
The cost compounds across cycles
When the first autonomous agent makes a contestable decision that ends up in front of the board, the board does not ask whether there was a policy. There is always a policy. The question the board asks is who answered for the action the agent took at 11:47 on the 14th of June. If the architecture is built around accountable officers per use case, the answer routes back to someone whose accountability was defined when the use case was approved months earlier, for the design of the use case, not for what the agent did inside it. The architecture holds. The answer it produces fails to satisfy the question being asked.
That gap is not absorbed in a single incident. It accumulates. Each contested decision the architecture cannot cleanly answer for adds to a perception that the oversight design is structurally behind the technology it is governing. Board confidence in AI investment narrows. Regulator patience for “we will tighten the framework as we go” narrows faster. An organisation that arrives at the first hard question with an architecture already redesigned for the new actor is in a meaningfully better position than one still adapting an architecture that was never designed for it.
The question for the board
The diagnostic question here is structural rather than procedural. What would the current architecture have to look like for an agent acting between human reviews to have a named, real-time accountability route that resolves to someone with current information about the action:
- Rather than to the person who signed off the use case eight months ago?
- Where in the architecture as it exists today is that route actually specified, and where is it being assumed?
- If the route runs through the approval of the use case rather than through the action itself, the architecture is doing exactly what it was designed for.
It just is no longer being asked to do only that.
The executive who reads the architecture before the first contested decision shows up has a different conversation available to them than the executive who reads it afterwards. Seeing the gap precisely is what turns the redesign that follows into a proportionate choice instead of a reactive one. The architecture rarely needs to be rebuilt. It needs to be tested against the actor it is now being asked to govern.
References
- Digital Transformation Agency. (2026, January). AI policy update: Strengthening responsible use across government. Australian Government.
- Governance Institute of Australia. (2026, January). AI governance in 2026: From experimentation to maturity. Governance Institute of Australia.
- Omdia by Informa TechTarget. (2026). The ROI of gen AI and agents 2026. Snowflake Inc.
