What metrics do you use with your large, scaled agile program of work?
Many people turn to OKRs – Objectives and Key Results – to do their reporting and scaled agile frameworks, like SAFe®, fits in quite well with this standardised format.
Identifying Objectives in SAFe®
In essence, the Scaled Agile Framework (SAFe®) has four levels of objectives. These objectives, though, tend to reflect on activities and not the outcomes that are created.
Each Sprint, agile teams deliver a potentially releasable increment of work. If it’s software, it could be released to the public. If it’s marketing, new communications could be made. If it’s data, a report on the next sample of the population could be released. But this is where objectives and their definition fail.
Writing simple OKRs
OKRs are simple enough to write. Most focus on the activity. A good OKR includes its measure.
Wrong
"Deploy the CRM feature by the end of the PI."
Right
"Deploy CRM feature increases availability for users by 10% this PI."
Delivery is everything, except when it’s not
And of course, delivery of Epics, Features and Stories is crucial. We want the benefit of the objective. We want the benefits a group of Features deliver us. People expect teams to deliver.
Executives and management need to know whether a milestone has been achieved, whether a promise to stakeholders and customers has been kept, and who to hold to account when things don’t get done. This style of accountability and reporting helps turns deliverables into contracts to pay vendors and solution partners and encourages governance arrangements to turn Scrum Masters into Delivery Managers to report on that delivery, and to ensure delivery occurs.
Unfortunately, the result is a return to the project management reporting mindset of on-time, on-scope and on-budget at the cost of understanding whether value was delivered. When the market changes and something else is of value, the tendency in this world is to pull out the contract and say “but we delivered on time” or in some cases (which I’ve seen first hand) simply ignore changing priorities (even if a Product Manager or Product Owner has reorganised a backlog to reflect a new set of priorities) and focus on what was contracted.
OKRs and measuring delivery of value
SAFe®, Nexus, XP, Scrum, Kanban, and other agile frameworks attempt to turn the traditional project management paradigm of reporting and accountability on its head by reinforcing that value is paramount and delivering something that isn’t of value is waste. Agile does this by reinforcing that a number of key principles and ethics underpin the mindset of agile product development.
Individuals and interactions over processes and tools. Working software over comprehensive documentation. Customer collaboration over contract negotiation. Responding to change over following a plan.
- Agile Manifesto, 2001
When executives follow these ethics their initiatives are measured differently and people report differently. While we care about contracts and documentation, the focus is delivery of value by encouraging people to collaborate, identify a need, and then respond to it, where the “need” is something of value that can be defined and measured
Evidence Base Management (EBM) solves the difficulty in understanding how to measure value. It separates value into four Key Value Areas and aligns them to customer value or internal organisational value.
Under EBM, the OKR objective turns into an “outcome” against one of the four key value areas. Importantly, an outcome is measurable by leading indicators. Typically, these tend to be:
- Market share trends.
- Increase sales.
- Increase compliance.
- Reduce employee turnover.
- Improve the cost of customer acquisition.
- Agile and innovation culture .
Importantly, all of these are lagging indicators. To understand whether you’re likely to see results against a lagging indicator, you need leading indicators. Some examples of leading indicators are:
- Stakeholder satisfaction.
- Usability.
- Subscriptions.
- Daily active users.
- Software defects.
- Velocity trends.
- Time spent context switching.
- Handovers.
Turn Epics and Features into hypotheses
When documenting an Epic – essentially the lightweight business case – or its Features, remember it’s an hypothesis. You don’t have a crystal ball, so while you think a solution and scope will meet a certain need and deliver an outcome. We could continue to use the OKR format, but an hypothesis format works better because it looks at both the impact the solution will have as well as its metrics.
We hypothesise that Feature X for our internal grants management team will make the CRM more accessible to them. We know that this will be true when we see Usage Index and Product Cost Ratio change.
Identify the impact and leading indicators
Your solution has value if it’s going to have an impact. To have an impact you need to know what metric you’re going to use, what its score is now, and what it’s score is hypothesised to be once the solution is delivered.
During the definition and design stage of writing these items, research and document:
- If they get the feature, what will happen?How will they be impacted? Will they be happier? Will you get the data you need? Can someone more easily use something?
- What’s the metric score right now? How good is the usability now? How bad are the completion rates now? How bad is your data quality now?
- How will the metric change when you deliver the solution? Will usability go up? Will customer satisfaction go up?
Identify the outcome and lagging indicators
Once customer satisfaction improves, once completion rates go up, what happens next? Ultimately, the outcome is the end result. Again, like impacts and leading indicators you must know what the metric is now. You can’t say your solution gave you an outcome if you don’t know what it was before your solution was delivered.
Reporting
EBM promotes reporting of metrics as a way to improve understanding of a wider enterprise agility over whether a specific solution, feature or widget was deployed. It focuses executives on the things that actually matter:
- Are investments having the desired impact?
- Are they giving us the outcome we’re after?
- Do we invest more to get further benefit in this area?
- Do we halt investment and invest somewhere else?
How often do you review the metrics?
Review leading and lagging indicators every Program Increment toward the end of the PI as part of the Inspect/Adapt workshop. Lean Agile Portfolio Management should also review these metrics at the same time. They inform:
- Finalisation of the Program Backlog for an upcoming PI Planning event. The current set of features may have simply not resulted in changes to the leading indicators, so it’s time to re-prioritise the Program Backlog to give new Features for teams this coming PI.
- Re-prioritisation of the Portfolio Backlog. If an initiative, it’s Epics and Features, isn’t delivering the impact or outcomes hypothesised it’s time to pivot and define a new solution.
Conclusions
OKRs work fine with SAFe, but it requires a shift in mindset from activities and deliverable-based milestones to expecting and measuring the impact and outcome of the investment of strategic initiatives.
Evidence Based Management (EBM) provides a framework for getting people out of the weeds of “did you deliver and why not?!” to a perspective of value that expects a result – did it deliver customer value and/or organisational value and how do I know?
Download the EBM Canvas
ZXM’s EBM canvas aligns strategic planning activities with investment initiatives and Evidence Based Management (EBM). It fits in perfectly with Toyota Kata planning and learning cycles.
