Extends Evolve: The Operating Model AI Demands, Chapter 14, “The Pipeline as Investment Decision”, and the two Chapter 14 questions in Appendix A, the Governance Diagnostic (Hodgson, 2026). Written for the program office that prepares an automation investment case and the CFO, COO or investment committee that approves it.
An investment case that automates the work practitioners learn from is not ready for approval until it completes four fields: classification, replacement mechanism, accountability and sequencing. Chapter 14 of Matthew Hodgson’s Evolve sets out the four fields and states the approver’s position plainly: a completed assessment gives the CFO a basis to approve or challenge, and a blank gives the CFO grounds to require one (Hodgson, 2026). This capability pipeline risk assessment sets out what a passing answer looks like in each field, what gets the case returned, and why the assessment repeats each time automation scope expands.
In Brief
- Automating routine work removes the practice through which junior staff build the judgement the organisation will need in five years, and a standard cost-benefit case does not count that loss.
- Evolve requires four fields in any investment case that automates developmental work: classification, replacement mechanism, accountability and sequencing.
- Every replacement mechanism must restore three conditions at once: real accountability, a problem at the edge of the practitioner’s capability, and experienced judgement at the moment of uncertainty.
- Under the product model the assessment applies at every expansion of automation scope, as well as at first approval.
- A blank or weak field is grounds to return the case, and a staged default defers the savings instead of forgoing them.
What the cost-benefit case does not count
Automating a class of routine work improves throughput and error rates, and it removes the practice ground where junior practitioners learned to handle the cases the automation cannot (Hodgson, 2026). Bainbridge (1983) described the same effect in industrial process control: operators left with only the rare abnormal cases had lost the routine practice that built the skill to handle them, and she concluded that the most reliable automated systems may need the greatest investment in operator training. An investment case that assumes development costs fall as automation improves has the relationship reversed.
The loss appears a year or two after approval, beyond the planning horizon of the decision that caused it. Evolve opens Chapter 14 with a compliance function whose processing time fell by 60% in the first year. Eighteen months later the replacement assessors had spent their first two working years supervising AI outputs, and nobody still employed could resolve a novel case that required weighing competing legislative objectives (Hodgson, 2026).
Chapter 14 places the assessment at the moment of approval because that is the only point where the answer can still change the decision (Hodgson, 2026). Under the product model that moment recurs. Automation scope grows release by release as the product moves through its lifecycle, and each expansion is a new commitment of capacity at the portfolio review, so Zen Ex Machina (ZXM) applies the assessment to every expansion as well as the first. Evolve is explicit that the organisations that get this right do not defer AI adoption to protect the pipeline; they accelerate the AI system and the judgement that governs it together, because the system depends on that judgement (Hodgson, 2026).
The test every replacement mechanism must pass
Judgement develops only when three conditions hold at the same time: the developing practitioner carries genuine accountability for a real problem, the problem sits at the edge of their current capability, and experienced judgement is available at the moment of uncertainty (Hodgson, 2026; Lave & Wenger, 1991). Automation removes all three at once. A replacement that restores one or two of them produces attendance records, so the approver applies this test to every mechanism proposed in Field 2.
| Condition | What automation removes | What a replacement must restore |
|---|---|---|
| 1. Genuine accountability for a real problem | The practitioner reviews or approves machine output instead of making the decision | A decision the practitioner owns, on a live case, made before they see the machine's answer |
| 2. A problem at the edge of current capability | Routine cases are cleared automatically, leaving work that is either trivial or beyond the practitioner | A stream of real cases matched to the practitioner's current level and raised as they develop |
| 3. Experienced judgement at the moment of uncertainty | Senior practitioners see fewer cases and are rarely present when a junior colleague meets uncertainty | A senior practitioner available on live cases while the decision is being made |
Kahneman and Klein (2009) found that skilled intuition forms only where the environment has regularities a practitioner has the opportunity to learn, and that a person’s confidence in their own judgement does not reliably indicate its accuracy. Readiness therefore has to be shown by testing practitioners’ decisions against outcomes. Asking them whether they feel ready measures confidence.
Field 1: Classification
Evolve asks: “Which classes of automated work served as the primary development pathway for which capability?” (Hodgson, 2026).
The cost-benefit analysis behind the automation cannot answer this, and asking senior practitioners what they learned is unreliable, because expertise becomes tacit as it matures (Dreyfus & Dreyfus, 1986). The program office works from cases instead, starting with those senior practitioners resolved in the past year that the new system would escalate or get wrong: several objectives in conflict, incomplete information, no precedent. Evolve locates the capability risk at the same point, the irreversibility boundary, where a person must answer for a decision that cannot be taken back and that no system can be held to account for (Hodgson, 2026, Chapters 8 and 14).
Each senior practitioner is then taken through one real case at a time, with questions about the cues they noticed, the options they weighed and what made them hesitate. Incident-based interviewing of this kind recovers knowledge experts cannot state as rules (Klein et al., 1989). For each cue, the interviewer asks where the practitioner first learned to recognise it. Where the answer is a class of routine work now in scope for automation, that work is a development pathway. The completed field records the class of work, the capability it builds, the share of volume the automation removes, and whether a residual stream of ambiguous cases remains and who receives it.
Return the case if:
- the classification comes from the cost-benefit analysis or a skills framework, with no cases examined;
- the field reports no pathway without listing the cases that were examined;
- a residual stream exists and the design routes all of it to the most experienced staff.
Field 2: Replacement mechanism
Evolve asks: “What deliberate structure will recreate those conditions?” (Hodgson, 2026).
Evolve names three mechanisms that each have genuine value and each fail predictably when they are implemented without the conditions that made the original learning work: red-team rotation, structured shadowing and failure-mode analysis (Hodgson, 2026). Table 2 sets out what each must include to pass the three-condition test.
| Mechanism | Design requirement | Conditions restored | Typical failure |
|---|---|---|---|
| Red-team rotation | The developing practitioner works cases where the AI's output is in question and holds real authority to accept, override or escalate it, with an experienced assessor alongside | 1 and 3; condition 2 through case selection | The junior role becomes observation, and the rotation produces attendance instead of judgement |
| Structured shadowing | The senior practitioner explains their reasoning on live cases as they work; the developing practitioner records a prediction of each decision before it is made, and the difference is discussed | 2 and 3; condition 1 in part, through the recorded prediction | Without the recorded prediction, the junior absorbs the senior's routine but builds no pattern recognition |
| Failure-mode analysis | The developing practitioner diagnoses real failures from the organisation's own AI systems before anyone explains them | 1 and 2; condition 3 through the debrief | Hypothetical or cleaned-up examples remove the error signatures the exercise exists to teach |
Two design choices, both ZXM practice beyond the book, strengthen all three mechanisms. The developing practitioner records a decision before the AI’s output or risk score is visible, then compares the two, which keeps condition 1 in place even when the machine handles the case and avoids the situation in Evolve’s opening case, where new assessors spent two years supervising output. The investment case also names senior practitioners’ time as a committed resource, because that time is the first cost cut when savings targets tighten.
A passing Field 2 states the mechanisms chosen, the conditions each restores, the share of real case volume allocated to developing practitioners, the senior time committed and the date each mechanism completes its first full cycle on live cases.
Return the case if:
- the mechanism is a training commitment, a workshop or a simulation built from hypothetical or curated cases;
- developing practitioners review or approve AI output without first recording their own decision;
- case discussion happens after the decision instead of while it is being made;
- senior practitioner time is assumed and not budgeted.
Field 3: Accountability
Evolve asks “which specific person is accountable for whether the capability pipeline for this class of work is producing the judgement the organisation will need at the five-year mark” (Hodgson, 2026).
Evolve requires one person, because a replacement mechanism without a single, accountable owner does not survive competing priorities, and accountability for capability development in general is typically distributed to the point of inaction (Hodgson, 2026). Human resources and learning functions can operate a mechanism but cannot own its outcome, because they control neither the allocation of real cases nor the time of senior practitioners.
ZXM recommends placing the accountability with the Product Manager accountable for the value stream in which the automated work sits, or with the Product Portfolio Executive, the executive who allocates capacity across the portfolio, where the capability serves several value streams. That owner already decides how capacity is allocated, which is the resource the mechanism consumes. Where the product model is not yet in place, the owner is the executive who controls both the affected function’s budget and its staffing. A five-year horizon can outlast the incumbent’s tenure, so the field names the role, the date it was assigned and the handover arrangement.
Return the case if:
- accountability sits with a committee, working group or team;
- the owner is the human resources or learning function;
- the field names a person with no role and no handover arrangement.
Field 4: Sequencing and consequence
Evolve requires that “the replacement mechanism is operational before the incidental one is removed, with the date and the governance consequence if the sequencing is not met” (Hodgson, 2026).
Under the product model there is no single cutover date. Each release that expands automation scope removes another share of the routine volume, so the sequencing test applies at every expansion. Operational means developing practitioners have completed at least one full cycle of the mechanism on live cases before the release that removes the work it replaces. A mechanism in design, in procurement or due to start at the same time as that release is not operational, because practitioners part-way through their development get no transition period.
The consequence is written into the investment case as a condition of approval. ZXM recommends a staged default: if the mechanism is not operational when the next expansion is due, automation scope stays at the level that leaves the Field 1 residual stream intact, and the decision to proceed goes to the portfolio review. The default is inexpensive to enforce because it defers the savings instead of forgoing them. The completed field records the planned sequence of expansions, the date the mechanism became operational and the consequence clause as approved.
Return the case if:
- the field gives one go-live date for automation scope that will expand over several releases;
- the mechanism starts in the same release that removes the work;
- the consequence is stated as an intention and not as a condition of approval.
Worked example: a case returned, revised and approved
In this illustrative example, a retail bank is automating first-line review of transaction-monitoring alerts. Analysts work through alerts raised by monitoring rules, closing false positives and escalating suspicious activity to senior investigators, who decide whether to report it to the regulator. The new system will close most routine alerts and route the rest onward, in three releases over a year.
The first submission proposed a monthly workshop in which analysts would discuss a curated set of closed historical alerts with a senior investigator. The CFO returned it on all three conditions: analysts would decide nothing, the curated cases were chosen to illustrate points rather than matched to each analyst’s level, and senior judgement would arrive a month after the decision.
Classification for the revised case began with case-based interviews with four senior investigators. Each traced their ability to recognise structured or layered transactions to their early years closing routine alerts, where they learned what normal activity looks like for different customer types. The automation removes most of that volume and leaves a residual stream of ambiguous alerts, which the original design routed straight to senior investigators.
The revised case approved two mechanisms. In a red-team rotation, the residual alerts go to developing analysts, who decide to close or escalate each one before they see the system’s risk score; a senior investigator is rostered to the queue while they decide and reviews every call the same day. In fortnightly failure-mode analysis, analysts diagnose alerts the system closed that later proved suspicious, before the investigation team explains them. Investigator time is budgeted in the case as a fixed commitment.
The Head of Financial Crime Operations owns the outcome, controls both the alert queue and investigator rostering, and has the accountability written into the role’s performance agreement with a handover clause.
Both mechanisms ran on live alerts for twelve weeks under the manual process before the first release. The approved consequence clause holds automation to alert types outside the residual stream at any release where the rotation is not running, and sends any further expansion to the portfolio review. Each Sprint Review reports independent resolution and diagnosis accuracy, and depth of coverage goes to the quarterly portfolio review.
The same structure applies in an Australian government compliance function. Routine regulatory assessment is the automated work, discretionary assessment is the residual stream, and the owner is the executive who controls both the assessment workload and the function’s staffing. ZXM’s guide to workforce transition in the Australian Public Service covers the legislative and industrial framework that governs that transition.
What the portfolio inspects after approval
After approval, the Sprint Review inspects whether the pipeline is producing judgement: Evolve treats the capability pipeline as a product governance question, inspected at the same cadence as every other product outcome (Hodgson, 2026). Chapter 15 places the pipeline inside the Sprint cadence itself, because practitioners build the judgement to govern AI systems by taking part in inspecting what those systems produced (Hodgson, 2026). ZXM proposes four indicators as a starting set, to be refined in use:
- Independent resolution: the share of escalated or non-routine cases that developing practitioners resolve without senior intervention, with outcomes confirmed by later review.
- Prediction agreement on hard cases: how often recorded predictions match senior decisions on difficult cases, tracked over time.
- Diagnosis accuracy: the share of AI failures correctly diagnosed in failure-mode analysis before anyone explains them.
- Depth of coverage: the number of people able to resolve the hardest class of case unaided.
All four test decisions against outcomes, and reported beside the four Key Value Areas of Evidence-Based Management (EBM) on the portfolio dashboard, they show the executive the health of the pipeline next to the value the automation is producing (Scrum.org, 2024).
Template for the investment case
Program offices can copy this table into the investment case or committee pack.
| Field | Item | Response |
|---|---|---|
| Scope | Program name and automation scope | |
| Scope | Value stream or function affected | |
| 1. Classification | Class of work automated | |
| 1. Classification | Capability the work develops, with the cues it teaches | |
| 1. Classification | Cases examined, or the finding that the work provides no pathway | |
| 1. Classification | Share of volume removed; residual stream and who receives it | |
| 2. Mechanism | Mechanism(s) approved | |
| 2. Mechanism | Conditions restored by each mechanism (1, 2, 3) | |
| 2. Mechanism | Real case volume allocated to developing practitioners | |
| 2. Mechanism | Senior practitioner time committed | |
| 2. Mechanism | Date of first full cycle on live cases | |
| 3. Accountability | Accountable owner (role, not incumbent) | |
| 3. Accountability | Date assigned; handover arrangement | |
| 4. Sequencing | Planned automation expansions (release, date, share of volume removed) | |
| 4. Sequencing | Date the mechanism was operational before each expansion | |
| 4. Sequencing | Consequence clause as approved | |
| Inspection | Indicators, reporting cadence and forum |
Declaration. The replacement mechanism is operational before each expansion of automation removes the development pathway identified in Field 1, and the consequence clause in Field 4 is a condition of approval.
| Reviewed by (CFO or delegate) | Role | Date |
|---|---|---|
Zen Ex Machina works with executives and program offices to build the capability pipeline assessment into investment governance and the portfolio review. Request a conversation to test an automation case against the four fields before it reaches the committee.
References
- Bainbridge, L. (1983). Ironies of automation. Automatica, 19(6), 775–779. https://doi.org/10.1016/0005-1098(83)90046-8
- Dreyfus, H. L., & Dreyfus, S. E. (1986). Mind over machine: The power of human intuition and expertise in the era of the computer. Free Press.
- Hodgson, M. (2026). Evolve: The operating model AI demands. Zen Ex Machina.
- Kahneman, D., & Klein, G. (2009). Conditions for intuitive expertise: A failure to disagree. American Psychologist, 64(6), 515–526. https://doi.org/10.1037/a0016755
- Klein, G. A., Calderwood, R., & MacGregor, D. (1989). Critical decision method for eliciting knowledge. IEEE Transactions on Systems, Man, and Cybernetics, 19(3), 462–472. https://doi.org/10.1109/21.31053
- Lave, J., & Wenger, E. (1991). Situated learning: Legitimate peripheral participation. Cambridge University Press.
- Scrum.org. (2024). The evidence-based management guide: Improving value delivery under conditions of uncertainty. Scrum.org. https://www.scrum.org/resources/evidence-based-management-guide
