In July 2026, Australia announced sweeping new rules for AI in government decision-making. The most forensically documented automated decision-making failure in recent public sector history — a thousand pages of Royal Commission evidence — shows what those rules cannot prevent, and what can.
In Brief
- Between 2015 and 2019, Australia automated welfare debt recovery, removed human oversight, and issued 453,000 unlawful debt notices.
- Three structural failures produced the harm: no feedback loop, institutional silos, and rules automated without the judgment they required.
- AI deployed within the same institutional structures will reproduce the same class of failure at greater speed.
- The structural alternative is persistent teams that build, run, and continuously govern the systems they create.
An Office of AI will sit within the Prime Minister’s department. A digital duty of care will require AI companies to build safety features in. A second round of privacy law reform is progressing. Attorney-General Michelle Rowland will lead the development of the new framework as AI providers, including Anthropic, OpenAI, Microsoft, and Google, expand their operations in Australia. The response is substantial and genuine, and it addresses real risks. It operates at the compliance layer. The question is whether that layer is where the structural risk actually sits.
The evidence of the risks is documented across three volumes totalling almost 1,000 pages and 57 recommendations. Between 2015 and 2019, the Australian government ran what the Royal Commission into the Robodebt Scheme later described as “a crude and cruel mechanism, neither fair nor legal” (Holmes, 2023). The Robodebt scheme automated welfare debt recovery using an income-averaging algorithm that neither produced accurate results nor complied with the income calculation provisions of the Social Security Act 1991 (Cth). It affected 453,000 people. The net cost to the Commonwealth was $565 million. The failure was not technological — the Robodebt algorithm did what it was told to do. The failure was structural: an institutional design that separated policy from delivery, removed human judgment from legally consequential decisions, and lacked a mechanism for people experiencing harm to surface it to those with the authority to stop it. Those three structural failures carry direct and documented lessons for any government now deploying AI into citizen-facing services.
The failure was not technological — the algorithm did what it was told to do.
Lesson 1: Automation without a feedback loop scales harm at system speed
Prior to Robodebt, data-matching between Centrelink and the Australian Taxation Office was used as a trigger for manual reviews (Holmes, 2023). Manual reviews involved direct engagement between compliance officers, welfare recipients, and — where necessary — employers, to establish actual income earned. If a discrepancy was confirmed against actual evidence, a lawful debt could be raised. The process was time-consuming and resource-intensive. This issue, while policy saw this as a problem, was not its flaw. Humans actually prevented errors from scaling. The algorithm didn’t.
This issue, while policy saw this as a problem, was not its flaw. Humans actually prevented errors from scaling. The algorithm didn't.
Robodebt removed the friction. The automation of the entire process — from income calculation through to debt notice — was new (Priergaard, 2024). By late 2016, the system was issuing more than 20,000 automated debt notifications per week (Holmes, 2023). When the algorithm’s income-averaging method was wrong — and the Royal Commission found it was systematically wrong — it was wrong 20,000 times a week. No feedback channel existed between the people receiving unlawful debt notices and the people with authority to change the system. Recipients were left to navigate the complexities of internal departmental processes unaided (Ruiz Diaz, 2023). The errors did not self-correct. They accumulated for four years until an external legal challenge forced the issue.
The Royal Commission recommended that the Commonwealth consider establishing a body to monitor and audit automated decision-making processes (Holmes, 2023). That recommendation addresses the problem after deployment. The structural question — the one the recommendation does not reach — is what prevents the problem before deployment: whether the system’s design includes a continuous feedback channel from the people it affects to the people who can change it, operating at a cadence that matches the system’s own decision speed rather than through periodic external audit.
The lesson for AI is direct. Any system that automates or assists decisions affecting citizens needs a designed feedback loop — from the people experiencing the decisions to the people accountable for the system — that operates faster than the harm accumulates. An AI system without that mechanism will reproduce Robodebt’s pattern: errors scaling at machine speed while human correction waits for the next committee cycle.
Lesson 2: Institutional silos made the outcome nobody’s responsibility
At the time of Robodebt’s development, the Department of Social Services (DSS) led social policy development. The Department of Human Services (DHS) — since restructured as Services Australia — delivered services, including welfare debt recovery through Centrelink. Each agency carried separate budget obligations and separate savings targets. The policy function designed the measure. The delivery function implemented it. Each discharged its accountability correctly. The outcome — whether the debt notices were lawful, accurate, and proportionate — belonged to neither.
The Royal Commission documented how completely the separation operated in practice. DSS raised legal concerns about income averaging in late 2014. There is no evidence that this advice reached DHS at the time (Holmes, 2023). Once the measure was included in the federal budget, senior DSS staff claimed to have paid almost no attention to the scheme’s implementation (Redden et al., 2025). For DSS, Robodebt was a scheme DHS had designed and pushed for, over the top of their objections. For DHS, it was policy direction from above that they were executing. The accountability for each function was clear. The accountability for the outcome sat in the gap between them.
Australian National University research into Robodebt’s institutional origins found this was structural, not incidental (Priergaard, 2024). Robodebt was the culmination of decades of institutional change. An agency responsible only for service delivery had just two ways to meet annual savings targets: efficiency gains and stricter compliance. Data-matching-driven compliance had been gradually automated over decades as the least harmful path to those savings. The full automation of the process — from calculation through to debt notice — was the logical next step within a structural incentive that had been operating for thirty years. The researchers concluded that the Royal Commission’s recommendations cannot prevent a recurrence without reform to the budget process and restructuring of the portfolio to bridge the divide between policy and service delivery (Priergaard, 2024).
The lesson for AI: institutional separation between policy design, system development, and service delivery does not produce shared accountability. It produces the conditions under which each function can demonstrate compliance with its own obligations, while the outcome experienced by the citizen belongs to no one. AI deployed within these structures does not bridge that gap. It accelerates whatever the silo produces. The question every agency should answer before deploying AI into a citizen-facing service is whether a single accountable person holds end-to-end responsibility for the outcome the system produces — from policy intent, through the system’s decision logic, to the notice that arrives in someone’s inbox.
Lesson 3: Automating rules designed for human judgment removed the thing that made them work
The Social Security Act 1991 (Cth) set out how income was to be calculated for determining welfare entitlements. The income calculation provisions assumed a compliance officer would assess actual income, not that an algorithm would average it across a financial year and treat the result as evidence of overpayment (Holmes, 2023). The rules were written for human interpretation and applied with discretion. A compliance officer assessing a case could recognise that a welfare recipient who worked intermittently did not earn a constant fortnightly wage, could request additional evidence, and could exercise judgment about whether a discrepancy warranted raising a debt. The rules worked because a human was reading each situation and applying the law to its specifics. The discretion was not an inefficiency in the system. It was the mechanism that made the system lawful.
Robodebt automated the rules but removed the judgment. The income-averaging method encoded a calculation that the legislation did not support. The Royal Commission found that human intervention was gradually removed to the point where debt notices were issued without review (Ruiz Diaz, 2023). Each removal was defensible on its own as an efficiency gain within a delivery agency under structural pressure to find savings. Collectively, they produced a system that complied with the encoded process while violating the law the process was meant to serve. The cost-saving motivation was rational within the institutional incentive structure under which the agency operated (Priergaard, 2024). The harm was the predictable consequence of applying that incentive to a system whose lawfulness depended on a property — human discretion — that the efficiency logic treated as overhead.
The mechanism matters because it is not unique to welfare. Most government decision-making systems — visa processing, compliance assessment, regulatory determinations, grant evaluations — were designed around human judgment. The legislation assumes discretion. The policy frameworks assume interpretation. The cost-savings case for AI in government service delivery is structurally identical to the case that produced robodebt: faster, cheaper processing of decisions that were designed to include human discretion. The savings are genuine. What is being removed to capture them is the thing that made the system lawful, proportionate, and correctable.
The governance design question is not whether AI can execute the rules faster. It is whether the rules still function as intended when the judgment they were written to include has been removed. A debt notice to a welfare recipient is a legally consequential, individually irreversible decision. So is a visa refusal. So, a compliance determination triggers a financial penalty. The point at which a human must remain accountable — where a wrong decision cannot be corrected without disproportionate cost — must be identified before the system is built and governed continuously after deployment. The alternative is to discover it after the harm has been scaled, which is what Robodebt was.
What would have prevented all three key AI failures
The three failures share a single structural root: a design that separated the people who understood the problem from the people who experienced the consequences, with no mechanism for the consequences to inform what happened next. The regulatory responses now being assembled — an Office of AI, a digital duty of care, privacy reform, automated decision-making audit bodies — address real risks at the compliance boundary. None of them changes the internal structure through which AI will be deployed within each agency. Three structural properties would have prevented all three failures. They are the documented characteristics of organisations that consistently capture value from technology investment rather than generating technically compliant outputs that fail to produce the intended outcome (McKinsey & Company, 2023; Boston Consulting Group, 2025).
AI needs feedback loops that run at its speed
The first is a feedback loop from the people affected by the system to the people accountable for changing it — operating continuously, not at audit intervals. The team that builds an AI-assisted service also operates it, observes its output in practice, and has the authority to make adjustments. Transparency about what the system is actually doing — its decision patterns, its error rates, its outcomes for the people it serves — is the input to every subsequent decision about where the system needs to change. Robodebt ran for four years without that transparency. An AI system operating at the speed AI operates at would exhaust the same runway in months, not years.
The people who built it also need to run it
The second is that the people who designed and built the system also manage and iterate it through its full lifecycle. There is no handoff from a development function to a separate operations or sustainment function. The team that chose the algorithm, set the decision boundaries, and encoded the business rules lives with the consequences of those choices in production. That accountability is what creates the incentive to detect and correct errors before they scale — because the people observing the errors are the same people who have the knowledge to fix them. Robodebt separated those two groups institutionally. The people who could see the harm had no authority to change the system. The people with authority to change it had stopped looking.
The people who could see the harm had no authority to change the system. The people with authority to change it had stopped looking.
The human-AI boundary must be governed, not inherited
The third is that the boundary between automated and human decision-making is identified before deployment and governed as a living artefact after it. A human with sufficient authority and context is positioned at the irreversibility boundary — the point where a wrong decision cannot be corrected without disproportionate cost (Hodgson & West, 2026). That boundary is not static. It shifts as the team accumulates evidence about where the system is reliable, where it requires human oversight, and where the original design assumptions no longer hold. The governance design treats the boundary as something the team continuously assesses — using the transparency the feedback loop provides as the input to where the line between automated and human judgment should sit, and moving that line as the evidence warrants. The people who built the system are the people learning from it, and that learning is what produces an AI governance posture that improves over time rather than one that degrades the moment the project team moves on.
What this means for senior leaders
Before deploying AI into any citizen-facing service, identify the irreversibility boundary: the point at which a wrong decision cannot be corrected without disproportionate cost. Position a human with authority there. Ensure the team that builds the AI system also operates and iterates it — a handoff to a separate operations function breaks the feedback loop the system depends on for self-correction. Build a feedback channel from people affected by AI-assisted decisions to those accountable for the system, operating at a cadence that matches the system’s decision speed. Treat the boundary between automated and human decision-making as a governed artefact, reviewed as evidence accumulates about what the system actually produces — not set once at project initiation and inherited by whoever is left when the project closes.
Robodebt was a structural failure in an era of simpler automation. The executive who redesigns the structure is solving a different problem from the one who adds compliance on top of the old institutional design. One is governing AI. The other is governing the appearance of governing AI.
References
- Reference ListBoston Consulting Group. (2025). The widening AI value gap: Build for the future 2025. BCG. https://www.bcg.com/publications/2025/are-you-generating-value-from-ai-the-widening-gap
- Hodgson, M., & West, D. (2026). Empiricism at machine speed: Scrum as a governance framework for agentic AI teams. Scrum.org. https://www.scrum.org/resources/empiricism-at-machine-speed
- Holmes, C. (2023). Report of the Royal Commission into the Robodebt Scheme. Commonwealth of Australia. https://robodebt.royalcommission.gov.au/publications/report
- McIlroy, T. (2026, July 19). Government use of automated AI decision-making to be curbed under new Australian rules. The Guardian.
- McKinsey & Company. (2023, December). The bottom-line benefit of the product operating model. McKinsey Digital. https://www.mckinsey.com/capabilities/mckinsey-digital/our-insights/the-bottom-line-benefit-of-the-product-operating-model
- Priergaard, J. (2024). Not my debt: The institutional origins of Robodebt. Australian Journal of Public Administration, 83(4). https://onlinelibrary.wiley.com/doi/full/10.1111/1467-8500.12658
- Redden, J., Dencik, L., Warne, H., & Hintz, A. (2025). Ignoring harm, saving face: Non-knowledge, senior public servants and the Robodebt scheme. Australian Journal of Political Science. https://doi.org/10.1080/10361146.2025.2486108
- Ruiz Diaz, F. (2023, September 14). Privacy insights from the Australian Royal Commission’s Robodebt investigation. International Association of Privacy Professionals. https://iapp.org/news/a/privacy-insights-from-australian-royal-commissions-robodebt-investigation
