Good metrics and data is the lifeblood of great executive decision-making. To understand the accuracy of its delivery metrics, ZXM took a science-based approach to the statistical analysis of its Agile IQ® data on delivery effectiveness, cost reduction, risk and how it relates to agile capability maturity. Specifically: What top behaviours drive organisations to higher performance, lower costs, and reduced delivery risk?
The Approach
The statistical methodology
ZXM used a longitudinal multiple-linear regression (orthogonal transformation) to understand the factors that predicted:
- Reduced costs
- Improved time to market
- Improved quality
Variables that we investigated to understand the accuracy of our results were:
- What is the reliability of the results compared to the assessors (inter-rater reliability)?
- Were the results different for different industries, e.g. government vs private sector?
- Were there any questions that weren’t significant contributors to the final data model?
We used experts to do the assessments
ZXM used agile coaches with 5-15 years’ experience to observe teams and Agile Release Trains (agile programs) behaviours. The Agile Coaches’ credentials were supported by industry certifications, including Scrum.org PST and SAFe SPCs.
We didn’t use self-assessments. It produces biassed outcomes.
When measuring effectiveness, self-reporting is heavily influenced by cognitive bias:
- Self-assessments aren’t accurate.
- Self assessment are subjective
- People over-evaluate their own performance.
- Self-assessments are time consuming.
How we defined agile behaviour
ZXM used a 6-point Likert scale with 80 behaviours defined from:
- Agile Manifesto and its 12 Principles
- Scrum Guide by Jeff Sutherland and Ken Schwaber
- Lean and Kanban by Daniel Vacanti and Yuval Yeret
- Management 3.0 by Jurgen Appelo
- Extreme Programming (XP) by Kent Beck
- Systems Thinking by Peter Senge
Assessors were asked whether the they “strongly agreed” that the behaviour occurred for the team through to “strongly disagreed”. Either the behaviour occurred or it didn’t. There was no “not applicable”.
Who participated?
- 523 teams over a 10 year period, including software- and non-software teams (HR, finance, marketing).
- 341 teams were assessed every quarter, some for over a 5+ year period.
- 23 SAFe® Agile Release Trains.
- 30 organisations.
The Results
The four behaviours that predict agile outcomes
The results showed that four behaviours were strong predictors of agile delivery outcomes:
Self-Organisation
Self-organising teams are more effective than manager-led teams. They achieved 15-20% more productivity this way.
Agile Values
Those organisations that moved beyond just the symbols of agility and worked to change people's mindsets - the way they think about work - saw stronger behavioural change.
Sprinting
Organisations that had a cadence of rapid, short work cycles, had faster feedback to learn, innovate, and pivot to change.
Continuous learning culture
When people were given room to improve and to be active in learning and taking time out for experimentation, innovation increased causing costs to decrease.
Ágile IQ® is the name we gave to the model that emerged from our analysis.
The model predicts ability to pivot
As agile behaviours strengthened an organisation’s ability to pivot to adapt to change increased.
The model predicts cost savings over time
As agile behaviours strengthened, organisations costs reduced.
The model predicts quality improvements over time
As agile behaviours strengthened, teams’ quality improved, defects and rework decreased, resulting in more time to spend on delivering more. Overall, throughput increased for the same cost with higher quality.
Conclusions
Google’s Project Aristotle showed that dependability, clear structure and goal clarity produces high performing, effective teams. This supports decades of research from psychology (Loerakker & Van de Grift, 2015, and others) that highlights the importance of self-management over traditional manger-led teams.
Ultimately, teams that embraced these behaviours saw a $2M+ cost savings per quarter over teams that didn’t.
These four behaviours – self-organisation, agile values, sprinting and continuous learning culture – derived from the Agile Manifesto can help executives create an agile enterprise.
Visit the Agile IQ page to learn more about to assess and monitor your team’s behaviours and get started!
