Agile Metrics Sin#2: Unbalanced Metrics
Why is a Balanced Regimen in Agile measurements so important? And why for an overall success, you should measure productivity, quality, predictability and sustainability of your team.
This is the second article of a 7-part series of “Revisiting seven deadly sins of agile measurement”. Original paper titled “Seven Deadly Sins of Agile Measurement” was authored by Larry Maccherone.
In the previous article, we visited why it is important to focus on getting valuable feedback from metrics to focus on improvements and not letting the measures get used as a lever to influence other’s behaviours. In this article, we’ll visit why it is necessary to have a balanced metrics regimen.
Sin #2: Unbalanced Metrics
From the paper:
“The second sin has to do with the need for a balanced metrics regimen. The need for this is fairly readily apparent. If you focus on one aspect of performance (say productivity), other aspects will go down (quality, customer satisfaction, etc.).”
Continuing our wellbeing theme from earlier article, just measuring and achieving 10,000 steps in a day certainly won’t be enough to be qualified as a full-proof step towards your overall wellbeing. What if you ate 2 chocolates on the gloomy Monday? Or went for a couple of craft beers with your friends on Friday to blow off the steam? You would of course need to keep a measure of your diet, your stress levels, sleep habits etc. as a bare minimum to ascertain that you’re headed towards a better living.
Intuitively we already know, and the fact is also backed by research that developer satisfaction is important for productivity.[1] While the team might seem productive, but their satisfaction might have taken a dip. Maybe they’re burnt out because of the extra work load they had to endure to ship that “important” delivery. Maybe this is a pattern that keeps repeating. This results in short-term productivity gains and long-term loss in work-satisfaction and even leads to greater than optimal churn.
Moreover, can we, with certainty, ascertain that just because the team is productive, the overall quality of the product is also high? Or that customer satisfaction is also maintained well? I bet we can’t. We need observability in other areas too. Hence, the importance of having a balanced regime of measures.
Larry proposes a well rounded measure from four key areas:
“I recommend that you launch a metrics feedback effort with at least one measure from each of these four areas:
1) Do it fast [Responsiveness]
2) Do it right [Quality]
3) Do it when expected [Predictability], and
4) Keep doing it. [Employee Satisfaction]”
“These outcomes dimensions form the foundation of the Software Development Performance Index (SDPI), which quantifies insights about development work and provides feedback on how process and technology decisions impact a team’s performance. Of the metrics in each of the quadrants shown above (productivity, responsiveness, quality, predictability, customer satisfaction, and employee satisfaction), the first four can be extracted from ALM tools […]. The fifth and sixth metrics we recommend you gather via lightweight (maybe even single-question) surveys.”
Do It Fast (Responsiveness)
In the current markets, business model lifetimes have only grown shorter. Naturally, the advantage is with the organisations which are able to adapt changing business goals through software. Inevitably, to be able to ship features and enhancements as quickly as possible, per the changing business models, gives a steep advantage and leverage to make the most out of the dynamic markets.
Measures like lead times and deployment frequencies can give an indicative view of how fast the software can be shipped. A Time in State chart shows you how long your tasks stay in each step of your workflow on their journey to completion. It is a mighty chart to have in hand when you want to understand how quickly your tasks are completed and whether there are bottlenecks in your workflow.
Do It Right (Quality)
What is the point of shipping it faster, if that is not what the customer finds useful enough? The quality of software is not just about less buggy features, but also about the quality of UX and data flows. A software that doesn’t live up to the expectations of its users, or worse makes their job even more cumbersome, might as well never had been made in the first place — no matter how quickly the team was able to ship it and how early the business was able to capture the initial markets.
While the quality in terms of rework can be tracked via majority of leading tools, customer satisfaction can easily be captured with surveys like Net Promoter Score. Change failure rate and Time to restore service are great indicators of stability too.
Do It On Time (Predictability)
Business strategies are future dated. They are seldom siloed to a single business function. It’s highly likely that there is a marketing team working on how to market the product to the target customer, a sales team preparing the demos and sales pitch, a finance team allocating necessary funds for these related activities. All these activities require coordination to effectively mitigate the dependencies. The predictability of being able to ship the software when it was needed to be is of sheer importance.
Although, you need to be cautious about falling for the “estimation is deadline fallacy”. It is important to have accuracy over precision and to incorporate probability when giving forecasts.
Keep Doing It (Employee Satisfaction)
Research shows that developer satisfaction is important for productivity.[1]. It is imperative to build sustainable environments. Not just a balanced work-life, but developers need an environment which fosters learning and knowledge sharing, and gives the team a sense of self-actualisation; and does so repeatedly in a sustainable fashion.
These measures are more qualitative in nature. In fact Spotify uses a pure qualitative measure for its teams — Spotify Squad Health Check Model
In the next article in this series, we’ll discuss Sin #3 — Believing that metrics can replace thinking. We’ll look at why are qualitative measures as important as the quantitative ones, and how well they compliment each other.
Notes:
[1]: Graziotin D, Wang X, Abrahamsson P. 2014. Happy software developers solve problems better: psychological measurements in empirical software engineering. PeerJ 2:e289 https://doi.org/10.7717/peerj.289
If you manage broader goals in your organisation it is worth looking into well rounded measures for organisation - The Balanced Scorecard
What is DORA Metrics and the Research behind it which proves why these metrics work - Accelerate, Measuring DevOps performance.
DevOps Trends in industry and how your team compares to others - State of Devops Report 2022