Agile Metrics: Using OKRs with SAFe® and Evidence Based Management (EBM)

What metrics do you use with SAFe®? Many people turn to OKRs – Objectives and Key Results – to do their reporting and SAFe® fits in quite well with this standardised format.

Identifying Objectives in SAFe®

In essence, the Scaled Agile Framework (SAFe®) has four levels of objectives.

Each of these objectives, with perhaps the exception of the Epic, is either an activity or a deliverable – release software, integrate solutions, deliver a set of Features. And this is easy to report on through an OKR. Each objective’s progress is measured by a set of Key Result Areas that reflect various milestones of activity that result in delivery – The Feature has been designed, signed-off, planned, and deployed.

Writing simple OKRs

OKRs are simple enough to write. Most focus on the activity. A good OKR includes its measure.

Wrong

"Deploy the CRM feature by the end of the PI."

Right

"Deploy CRM feature increases availability for users by 10% this PI."

Delivery is everything, except when it’s not

And of course, delivery of Epics, Features and Stories is crucial. We want the benefit of the objective. We want the benefits a group of Features deliver us. People expect teams to deliver.

Executives and management need to know whether a milestone has been achieved, whether a promise to stakeholders and customers has been kept, and who to hold to account when things don’t get done. This style of accountability and reporting helps turns deliverables into contracts to pay vendors and solution partners and encourages governance arrangements to turn Scrum Masters into Delivery Managers to report on that delivery, and to ensure delivery occurs.

Unfortunately, the result is a return to the project management reporting mindset of on-time, on-scope and on-budget at the cost of understanding whether value was delivered. When the market changes and something else is of value, the tendency in this world is to pull out the contract and say “but we delivered on time” or in some cases (which I’ve seen first hand) simply ignore changing priorities (even if a Product Manager or Product Owner has reorganised a backlog to reflect a new set of priorities) and focus on what was contracted.

The Project Management Iron Triangle (Dr. Martin Barnes, 1969) vs the Agile Triangle

OKRs and measuring delivery of value

SAFe®, Nexus, XP, Scrum, Kanban, and other agile frameworks attempt to turn the traditional project management paradigm of reporting and accountability on its head by reinforcing that value is paramount and delivering something that isn’t of value is waste. Agile does this by reinforcing that a number of key principles and ethics underpin the mindset of agile product development.

Individuals and interactions over processes and tools. Working software over comprehensive documentation. Customer collaboration over contract negotiation. Responding to change over following a plan.
- Agile Manifesto, 2001

When executives follow these ethics their initiatives are measured differently and people report differently. While we care about contracts and documentation, the focus is delivery of value by encouraging people to collaborate, identify a need, and then respond to it, where the “need” is something of value that can be defined and measured

Evidence Base Management (EBM) solves the difficulty in understanding how to measure value. It separates value into four Key Value Areas and aligns them to customer value or internal organisational value.

Under EBM, the OKR objective turns into an “outcome” against one of the four key value areas. Importantly, an outcome is measurable by leading indicators. Typically, these tend to be:

  • Market share trends.
  • Increase sales.
  • Increase compliance.
  • Reduce employee turnover.
  • Improve the cost of customer acquisition.
  • Agile and innovation culture .

Importantly, all of these are lagging indicators. To understand whether you’re likely to see results against a lagging indicator, you need leading indicators. Some examples of leading indicators are:

  • Stakeholder satisfaction.
  • Usability.
  • Subscriptions.
  • Daily active users.
  • Software defects.
  • Velocity trends.
  • Time spent context switching.
  • Handovers.

Turn Epics and Features into hypotheses

When documenting an Epic – essentially the lightweight business case – or its Features, remember it’s an hypothesis. You don’t have a crystal ball, so while you think a solution and scope will meet a certain need and deliver an outcome. We could continue to use the OKR format, but an hypothesis format works better because it looks at both the impact the solution will have as well as its metrics.

We hypothesise that Feature X for our internal grants management team will make the CRM more accessible to them. We know that this will be true when we see Usage Index and Product Cost Ratio change.

Identify the impact and leading indicators

Your solution has value if it’s going to have an impact. To have an impact you need to know what metric you’re going to use, what its score is now, and what it’s score is hypothesised to be once the solution is delivered.

During the definition and design stage of writing these items, research and document:

  • If they get the feature, what will happen?How will they be impacted? Will they be happier? Will you get the data you need? Can someone more easily use something?
  • What’s the metric score right now? How good is the usability now? How bad are the completion rates now? How bad is your data quality now? 
  • How will the metric change when you deliver the solution? Will usability go up? Will customer satisfaction go up?

Identify the outcome and lagging indicators

Once customer satisfaction improves, once completion rates go up, what happens next? Ultimately, the outcome is the end result. Again, like impacts and leading indicators you must know what the metric is now. You can’t say your solution gave you an outcome if you don’t know what it was before your solution was delivered. 

Example of an Epic written using an outcomes hypothesis with leading and lagging indicators

Reporting

EBM promotes reporting of metrics as a way to improve understanding of a wider enterprise agility over whether a specific solution, feature or widget was deployed. It focuses executives on the things that actually matter: 

  • Are investments having the desired impact?
  • Are they giving us the outcome we’re after?
  • Do we invest more to get further benefit in this area?
  • Do we halt investment and invest somewhere else?

How often do you review the metrics?

Review leading and lagging indicators every Program Increment toward the end of the PI as part of the Inspect/Adapt workshop. Lean Agile Portfolio Management should also review these metrics at the same time. They inform:

  • Finalisation of the Program Backlog for an upcoming PI Planning event. The current set of features may have simply not resulted in changes to the leading indicators, so it’s time to re-prioritise the Program Backlog to give new Features for teams this coming PI.
  • Re-prioritisation of the Portfolio Backlog. If an initiative, it’s Epics and Features, isn’t delivering the impact or outcomes hypothesised it’s time to pivot and define a new solution.

Conclusions

OKRs work fine with SAFe, but it requires a shift in mindset from activities and deliverable-based milestones to expecting and measuring the impact and outcome of the investment of strategic initiatives.

Evidence Based Management (EBM) provides a framework for getting people out of the weeds of “did you deliver and why not?!” to a perspective of value that expects a result – did it deliver customer value and/or organisational value and how do I know?

About the author

Related Posts:

Product Management versus Project Management

Gartner reports that 85% of executives are now turning to agile product management over projects. Why is this shift occurring and what do you need to know about product management to optimise your strategic investments for greater market share, better ability to pivot, or get your products to market faster?

READ MORE

Remote PI Planning – Insights into what worked and what didn’t

Running remote PI planning for the first time 100% virtual was a bit daunting a few weeks ago when the reality set in that with the current crisis, this was now our reality. We prepared, thought through worse case scenarios and had back up plans and felt eerily calm as the day began. It was a great success and here are our tips.

READ MORE

Scaling agile from 5 teams to 27 teams within 12 months

Scaling is challenging and one of the key reasons agile initiatives fail is that the culture of the organisation is at odds with agile mindset and agile principles. But the next key factor is management support. So we knew that this change needed to be driven and supported by the leadership team in order to be successful.

READ MORE

Copyright © Zen Ex Machina® and ™ (2020). All rights reserved. ABN 93 153 194 220

search previous next tag category expand menu location phone mail time cart zoom edit close