Improving Program Agility using Evidence Based Measures of Agility in OKRs

OKRs have been an established way to set objectives however the associated activities tend to focus on outputs and not outcomes and impacts. To measure Program Agility and use data to drive improvements, we decided to explore Evidence Based Measures of Agility (adapted from Evidence Based Management ) when developing OKRs. 

I was working with a well established Product Management Group for a large program of work and their new Product Manager was keen to improve and evolve his group to take them to the next level on their agile journey. We suggested that using Evidence Based Management (EBM) as a framework for OkRs would help shift metrics towards value of the investment to help them make strategic investment decision based on evidence to improve business outcomes.

They were two years into their agile transformation and the entire organisation was developing enterprise agility.  This program was one of the higher performing programs and had developed to a good standard of competency.

Challenges

My client was a new Product Manager for the group, and he had noticed the following challenges including:

• Large batch sizes of the incoming work which led to long turn around for delivery: some pieces of work were estimated to take over 1 Program Increment (PI) to deliver.
• The bigger the piece of work, the longer the turnaround times for delivery.
• Longer lead times to deliver a feature from idea to release, increased risks on creating more dependencies to manage within the program and between programs.
• Potential impact to quality due to delivery pressures to meet stakeholder targets.
• The stream of incoming work peaked after sprint 3, towards the end of a Program Increment. This put increased pressure on the solution architects and business stakeholders to refine the work and get it “Ready” for teams to start working on.

The Product Manager wanted to understand how to set improvement goals and measure the improvements. he was keen to explore options around using Operational Key Results (OKR’s) to create alignment and focus on outcomes, but he didn’t have experience implementing them.

ZXM suggested incorporating Evidence Based Agility Measures (EBM) such as time to market and ability to innovate (pivot) as two key areas to explore when developing OKRs for the group.

ZXM collaborated with the Product Manager and provided guidance on creating OKR’s that would run as an experiment over the Program Increment (PI). The following two OKR’s were the first experiments we ran to test and address the key challenges above:

• Reducing batch sizes of incoming work by observing design workshops with the Design SME’s to influence slicing down the work into smaller sizes. This should lead to lower cycle time.
• Establishing a continuous steady flow of incoming work by actively engaging in PMG for stakeholders to raise work as soon as it comes up.

Developing OKRs as an Evidence Based Canvas to Measure Agility 

A small working group with Product Management was formed and Zen Ex Machina developed an EBM Canvas to develop objectives and and the key value measures. The canvas helped identify the problem, the hypotheses we proposed to address the challenge and the key values measures we would use to assess the impact and outcome. This would allow us to test the OKRs as a hypothesis and gather evidence to build in continuous learning and improvement. Weekly catch ups were established, progress was discussed and the approach refined.

Key Learnings 

Our key learnings from running OKRs as Evidence Based Measures of Agility  included: 

• OKR’s created focus and transparency on bottlenecks in the Product Management Group (PMG) process. It drove the Product Manager and team to deep dive into the bottlenecks searching for root cause, data and creating experiments.
• Increased transparency on quality of data captured was gained. By actually looking at the available data set, we noticed that it was lacking the right level of quality to make problems and impacts measurable. The data quality had been improving over the last 2 PI’s, but we didn’t have a representative data set to make decisions on.
• We kept the working group deliberately small for this first iteration. Based on the PMG Inspect and Adapt at the end of the PI, we have decided to increase transparency of the OKR’s to the wider PMG community and get broader buy-in from the group including ideas for experiments.
• We will continue weekly check-ins to ensure progress and fine tune experiments.
• COVID-19 had impacted the initiatives coming into the program of work as the organisation needed to pivot and adapt to changes in eth marketplace. We therefore need a longer time period to gather more data and refine experiments.

Outcome and Impact

The main outcomes from the first iteration were that the OKR’s developed using the EBM canvas, led to increased focus and transparency on bottlenecks in the PMG process. This ultimately drove continuous improvement in time to market and ability to innovate. Data quality has been improving and we defined further improvement opportunities for the next iteration to continue refining the approach.

The Product Manager shared the data available around the OKR’s in the Inspect and Adapt at the end of the PI. ZXM continue to work with the Product Management Group to capture and to show metrics that assess the agility of the Program. Capturing OKRs based on evidence measures of agility has really helped to improve and address challenges and show progress towards the objectives to the executives. 

Download the EBM Canvas

ZXM’s EBM canvas aligns strategic planning activities with investment initiatives and Evidence Based Management (EBM). It fits in perfectly with Toyota Kata planning and learning cycles.