The Agile Maturity Assessment
The Agile Maturity Assessment
Satisfying our addiction to statuses and command & control
What happens when you tell Scrum Teams that their performance review will depend on their predictability as a team? Easy, within no time you will have teams that are 100% predictable.
That is Goodhart’s Law in action: make any metric into a target and it ceases to be a good metric. People will find a way to game it.
coolest solution I’ve had a team come up with for the predictability game is to simply leave the estimates empty. Jira happily decides 0 commitment divided by 0 delivery is 1, giving you 100% predictability.
Amusing. Until you consider how much this game costs, to both the teams and management. And not just in terms of time or resources, but much more important: in terms of trust.
The search for a way to measure our teams and create sexy dashboards to see at a glance how things stand with them is a difficult one.
Enter Agile Maturity Assessments. An Agile Maturity Assessment is maybe the apex Agile metric: an extensive questionnaire which aims to cover all aspects of Agile in such a way that it can give an objective rating of how mature a team is. Preferably in a way that can be used to report the Agility status of the whole organisation. And with the advantage that by flooding users with questions, it happily avoids Goodhart’s Law.
Agile Maturity Assessments are usually seen as a way to help teams improve. They are popular because they can be useful in making the team aware of hidden dysfunctions, and they can provide a baseline to help teams measure their improvements. I have also seen them presented as a way for management to be able to determine which teams need help. These perspectives have merit.
Agile Maturity Assessments
I have been a member of a couple of Agile Maturity Assessment workgroups, where we analysed various different Agile Maturity Assessments.
Unfortunately, however ambitious these assessments are, covering tonnes of questions over various categories and presenting results in fancy radar graphs, I cannot help but conclude that they are usually quite useless to evaluate the Agility of a team, often telling us more about their makers’ perspective of Agile than about the teams they aim to assess.
You see, the problem is that Agility is really hard to measure. For me, the bottom line is that the level of complexity that is involved when trying to measure teams in an organisation simply cannot be quantified in the clear-cut way most Agile Maturity Assessment attempt to do so. In fact, trying to establish the Agile maturity of a team by asking a bunch of questions is similar to looking at predictability to determine team performance: it hopelessly oversimplifies a very complex subject.
This is quite clear in the way most assessments try to evade dealing with that complexity by focusing on the process - usually Scrum - as a proxy for Agility. The following are typical questions I’ve encountered that deal with one aspect of Scrum, the Daily Scrum:
“do you stay in the 15min timebox?”,
“Do all 3 questions (what-did-you-do-yesterday, what-will-you-do-today, do-you-have-any-impediments) get answered?”,
“Do all team members participate?”,
and my favourite “Is there a manager present?”.
Unfortunately, the problem with this approach is that a team’s process doesn’t say much about how mature that team is. You can have a team doing ‘perfect’ Scrum and being painfully ineffective, and at the same time, you can also have a team that hardly has a process but still manages to outperform other teams.
Designing Assessments
The truth is that designing assessments is really really hard.
First of all, you need to collect the right questions. This is a science in itself, dealing with all kinds of psychological stuff. Yup, you guessed it, it’s called Psychology.
But you’re not there yet. If you thought that was hard, once you have the right questions you then need to figure out how the questions affect each other and how to aggregate them into a meaningful result. You are basically creating a multi-dimensional model involving really complicated maths, where every additional question increases the complexity exponentially.
And if and after you’ve figured that little challenge out, you're still not there. Realise that having come this far, you only have a hypothesis. Before you can use it for anything meaningful, you have to thoroughly test it, using rigorous and complicated statistics to figure out whether it actually fits with reality!
For more details, the Liberators wrote an article about this: How (Not) To Construct A Proper Questionnaire.
The Scrum Team Survey
So now that you know what a huge challenge it is to create a serious assessment, and taking into account the general lack of science in the Agile community, finding a serious Agile Maturity Assessment might seem like a pointless task.
But there is hope. There is one assessment that fits the bill: the Scrum Team Survey, a tool created by, yes, The Liberators.
The survey involves, indeed just like other assessments, a huge load of questions. The difference is that the questions are modeled into a validated scientific model.
Indeed, focused completely on what makes a Scrum Team effective and aiming to help teams improve, it took The Liberators 5 years to put their assessment together, and they are still working on it. They continuously use data from the teams that fill in the survey (5000 teams and counting) to test and tweak it. Testament to their dedication is the fact that their model has led to a scientific paper soon to be published. No mean feat.
All models are wrong, but some are useful
— George Box —
Notice the irony. The Scrum Team Survey is not an Agile Maturity Assessment. It was developed to help teams improve themselves. Fill it in, and it will tell you which of 6 different areas you can focus on to improve team effectiveness, giving you suggestions for workshops you can use.
Sure, You get a score for your team if you fill it in, and you can see how you compare to the average of all teams that have filled in the survey. It wouldn’t be too hard to use this information to create a company-wide Agile maturity dashboard. But that is just a side-effect of the tool.
In fact, I’m sure The Liberators would encourage you not to do that. They take great pains to prevent abuse of the survey by anyone not in the team and will be the first to tell you it is a sense-making tool and not a status overview dashboard.
Conclusion
The Scrum Team Survey is great as a conversation starter for your team. I have used it with various teams, with varying results.
I‘ve had teams that simply were not ready for it. The information was meaningless to them.
I’ve had one team that had a pretty good score and was quite pleased with themselves. A difficult situation, as I was brought in because it was a team that was literally falling apart. They simply knew the Scrum theory well enough that they were able to give all the right answers, irrespective of reality.
I’ve also had teams where we were able to act on the suggestions and improve. However, even though the survey provided us with interesting surprises and inspiration for what to improve, it was our discussions that really brought it to life. And we found that its function as a baseline was limited. Things usually just change too fast and too chaotically in an Agile team.
The important thing to realise is that the value of the tool is in the conversation that it triggers. In that sense, for me, the Scrum Team Survey is not that different from say a workshop where I examine what Scrum is with my team.
The irony of all of this is that as Agile professionals we should know all this already. The first value of the Agile Manifesto tells us:
Individuals & Interactions over Processes & Tools.
An Agile Maturity Assessment is nothing else but a tool that pretends to tell us how Agile individuals and their interactions are.
Sure, Agile Maturity Assessments may serve as a tool that helps us understand how Agile individuals and their interactions are, but that is unfortunately not how it works.
We humans are obsessed with measuring stuff because it makes us feel secure. And it is so much easier than thinking for ourselves. Agile Maturity Assessments are just too easy to use as a stand-in for real Agile understanding. Why go through all the trouble of figuring out a team, when you have a tool that can do it for you? How much cheaper than having to hire those irritating, self-righteous Scrum Masters?
Think about it, why don’t any real Agile organisations proudly show off their Agile Maturity Assessments? I was at a talk recently by a couple of Agile Coaches at what I would call a successful Agile organisation. They described how their teams were able to decide for themselves whether they needed to ask an Agile Coach for help. Asked how they did that, they told us how it had taken years for the teams to reach that level of Agile understanding. No mention of any assessments, even though it would have been a perfect use case.
I think that the risk of Agile Maturity Assessments is just too great. In a setting where Agile is not yet properly engrained, they are too easy to use to compare teams in a more managerial way, for easy status reporting. They are basically an awesome tool for old-fashioned command & control.
If you want to know how your teams are doing, ask the Scrum Master! That’s what he is for. Or even beter, ask the team! And if you still really want to measure how your teams are doing, use the Scrum Survey. Sell it as an Agile Maturity Assessment, with a sexy dashboard where you can see all your Scrum Teams. And when management comes asking you to explain what they are seeing, use that to teach them how to go and talk to teams!