The shift from AI that answers questions to AI that takes actions has happened faster than most Australian organisations anticipated. Agentic AI governance has not kept pace. The World Economic Forum and Capgemini, in their November 2025 analysis of agent deployment patterns, found that 82% of executives were planning to deploy autonomous agents while the gap between experimentation and mature oversight was widening, not closing (World Economic Forum & Capgemini, 2025). That gap is an accountability architecture problem, and the two are not interchangeable.
In Brief
- Most Australian organisations have deployed agentic AI; fewer than one in four have a mature governance model to manage it.
- The governance gap is not a technology problem — it is an accountability architecture problem that requires executive-level resolution.
- Organisations that defer governance design until after deployment lose the window in which proactive accountability architecture is possible.
- Agent accountability defaults to whoever is nearest the error when no prior decision-rights architecture exists.
- Five governance questions, answered before deployment, determine whether agent autonomy is governable at scale.
Designed for AI that advises
Most organisations approached the first wave of AI deployment — the period dominated by chatbots, summarisation tools, and recommendation engines — with governance frameworks that treated AI as a sophisticated search function. Something that advises, not something that acts. The framework asks the right questions for that context: who can access this, what data does it touch, who reviews the output? These are reasonable questions. They are structurally insufficient the moment AI begins operating autonomously within business workflows rather than waiting to be consulted.
Agentic AI does not wait to be queried. It executes, navigates systems, makes sequenced decisions, and produces outcomes — frequently without a human reviewing each step. The governance model built for the advisory layer simply does not transfer to the execution layer. Applying it as though it does is what tends to produce accountability gaps that only become visible after something has already gone wrong.
The OECD’s AI principles, adopted by 47 countries including Australia and updated in May 2024, make accountability explicit: those who deploy AI are responsible for its outcomes, including those produced by autonomous systems (OECD, 2024). The structural implication is direct. If the organisation cannot identify who is accountable for a given agent decision, the missing element is not a principle — it is an architecture.
Deployment runs ahead of governance
The pattern that tends to emerge as organisations scale agent deployment is predictable. Deployment is driven by technology teams and vendor relationships, moving at the pace that capabilities and internal enthusiasm allow. Governance architecture requires something different: an executive-level decision about what the organisation is willing to delegate to an autonomous system, under what conditions, and who carries accountability when those conditions are not met. These two activities operate on different timelines, with different owners and different incentives. They do not converge without deliberate design.
The Australian Government’s Voluntary AI Safety Standard, published in September 2024, is worth reading in this light. Its ten governance guardrails include accountability, human oversight, and the ability to contest AI decisions — framed explicitly as executive responsibilities rather than IT controls (Department of Industry, Science and Resources, 2024). The intent is clear: governance at this level is a leadership question.
What tends to happen in practice is that the monitoring dashboard gets treated as the governance answer. The dashboard shows which agents are active, what they did, and whether errors were flagged. It does not show who was accountable for authorising the scope of the agent’s decision rights in the first place. A dashboard is a measurement instrument. The accountability structure is a different thing entirely, and the two are regularly confused.
The error finds whoever is nearest
When an agent error surfaces and no governance architecture exists, accountability defaults to proximity — to whoever is nearest when the error appears, rather than to whoever had the authority to prevent it. In practice, a mid-level technology leader or an operational team absorbs consequences for a decision that was effectively made, by default, when the organisation settled the scope of the agent’s authority. The choice was made. It just wasn’t made deliberately.
Gross, writing in CIO in May 2026, identifies the governance deficit as the primary executive-level exposure in organisations that have moved from AI pilots to operational agent deployment. The pattern he describes is consistent: absent decision-rights design is the common factor in governance failures that reach executive leadership attention (Gross, 2026). The NIST AI Risk Management Framework makes the structural requirement explicit in its Govern function — roles, responsibilities, and accountability defined at the organisational level, not left to emerge from technical deployment decisions (National Institute of Standards and Technology, 2023).
The window to fix this closes
There is a timing dimension here worth addressing directly. The governance architecture absent when an organisation deploys its fifth agent is not merely absent — it becomes harder to retrofit with each subsequent deployment. Each agent that goes live without decision-rights design adds to the unresolved accountability position. The exposure compounds.
At five agents, a governance architecture review is a contained exercise. At fifty agents running across procurement, customer service, compliance, and operations, the same review requires reconstructing accountability for decisions already embedded in live workflows. That is a different exercise in kind. The structural complexity of that reconstruction grows faster than the agent count, and the longer the architecture question sits unanswered, the more disruptive the eventual resolution becomes.
Five questions to settle before deployment
The questions governance architecture needs to settle are not technical. They are questions of organisational design and executive decision-making authority — questions that technology teams are not positioned to resolve on behalf of executive leadership.
Before authorising the next wave of agent deployment, executive leadership needs clear answers to five questions. Who is accountable for the decisions an agent makes, specifically, and who carries that accountability when outcomes fall outside expected parameters? What scope of decisions is the agent authorised to make autonomously, and what sits explicitly outside that scope? Under what conditions is the agent required to defer to a human rather than proceed? How can agent decisions be reviewed after the fact — by whom, and to what standard? And who holds the standing to override or halt an agent, and how is that authority exercised in practice?
None of these questions are unfamiliar to governance. They appear, in various forms, in every well-designed authority matrix. What is new is the speed at which agent deployment is generating the need for answers — and the regularity with which that need is being addressed with monitoring tooling rather than governance design.
The executive leadership conversation is already overdue
The trajectory of agentic AI deployment in most organisations suggests the executive leadership conversation on AI governance is not a future event. It is either already scheduled or already past. The question worth settling before that conversation is whether what will be presented is a governance architecture or a status report on what the technology team has deployed. The difference matters to executive leadership, and it matters to the audit committee. A status report describes what the agents are doing. A governance architecture describes who is accountable for what they decide.
The window in which that architecture can be designed before deployment outpaces it is closing. The organisations that close the gap now will give executive leadership a defensible answer. Those that do not will eventually give it an explanation.
References
- Department of Industry, Science and Resources. (2024). Voluntary AI safety standard. Australian Government. https://www.industry.gov.au/publications/voluntary-ai-safety-standard
- Gross, G. (2026, May 28). The AI governance imperative you can’t afford to ignore. CIO. https://www.cio.com/article/4176067/the-ai-governance-imperative-you-cant-afford-to-ignore.html
- National Institute of Standards and Technology. (2023). Artificial intelligence risk management framework (AI RMF 1.0). U.S. Department of Commerce. https://www.nist.gov/publications/artificial-intelligence-risk-management-framework-ai-rmf-10
- OECD. (2024). OECD AI principles. https://oecd.ai/en/ai-principles
- World Economic Forum & Capgemini. (2025, November). AI agents in action: Foundations for evaluation and governance. https://www.weforum.org/publications/ai-agents-in-action-foundations-for-evaluation-and-governance/
