Introduction
One of the most difficult things for managers to do is to figure out how best to measure their organization’s performance. Everyone knows that metrics are designed to measure performance, but more often than not, the metrics managers choose don’t measure what they think they measure. When you introduce Agile into an organization, managers crank their confusion to 11 and really struggle with how to measure the performance of their changing organization. Rather than think, most managers punt, opting to measure their organizational performance using team velocity (for Scrum teams) or throughput (for Kanban teams).
A quick note on terminology
I use the term organization to refer to a grouping of multiple teams. An organization in this context may be as small as a department (e.g. Information Technology) or as large as an entire company.
The problem
Managers are expected to stabilize and grow their respective organizations. Metrics are used to keep tabs on how the organization is performing. The problem with metrics is that any old metric will not do. Many managers of Agile organizations fall into the trap of summarizing team metrics (velocity and throughput) to the organizational level. If you do this, congratulations, you have no idea what you’re measuring nor how to use that information to further your organizational goals. Don’t let your boss read this article.
The purpose of metrics
The purpose of metrics is to give you insight into your organization’s performance. If your metrics are aligned to your organizational goals, then metrics can help you assess whether or not your organization is progressing toward its goals. Metrics are a type of feedback loop, and as such, they should also be a call to action when your metrics report something odd.
Consider the fuel gauge on the dashboard in your car. The fuel gauge is a metric that tells you how much fuel is in the car with respect to the size of your car’s fuel tank. The metric is quite simple. It doesn’t tell you how many gallons of fuel are onboard; rather, it tells you if you have enough fuel or not enough fuel for the car. When the needle is at or above a certain point, everything is good. However, when the needle drops to a certain point or lower, the call to action is: pull over and get fuel.
The Agile organization
Managers within Agile organizations believe they need metrics, but they rarely know which metrics to implement. The same managers know that their teams use these things called velocity and throughput (V&T). Those same teams use V&T to measure their performance and gauge productivity. If you sum(marize) those numbers from all the Agile teams together, then you get a nice metric to gauge the performance and productivity of the entire organization, right? Wrong. Nothing could be further from the truth.
You would immediately know I was insane if I sum(mariz)ed all fuel gauges of all the cars in my neighborhood to gauge if everybody who drove cars has enough fuel to get where they are going. Worse yet, you’d think I was a fool (and you’d be right) if I tried to use that same metric to measure the neighborhood’s fuel efficiency.
The work a team does is unique to that team and that team alone. Sure, Team A may do similar work as Team B. Even if the work seems like it’s the same, it’s not. There are far too many variables between Team A and B to state that the work is exactly the same. Besides, if the work were literally exactly the same, then why do you have two separate teams doing the exact same work? Talk about inefficient. The skillsets, team composition, deadlines, context, etc. are different between the two teams, and that’s why you have two different teams doing similar work.
Value vs busy-ness
The Agile organization shifts attention away from tasks and on to customer value. This is an important concept to remember. The organization that produces greater value for its customers, as compared to the competition, has a greater chance of outliving the competition. Such is the reality of competitive free markets.
Let’s say Team A and Team B deliver one million dollars of customer value, each, in a set period of time. Team A has an average velocity of 8 while Team B has an average velocity of 25. Which team is the better team? What if I told you both teams completed 5 user stories each? Which one is better now? Team A may have pointed stories in their last iteration as 1+1+2+2+2=8; whereas Team B may have pointed their stories as 5+5+5+5+5=25. Yet, despite having completely different velocities, the total value produced is exactly the same ($1,000,000 each).
Gaming the system
When you use team metrics at the organizational level you encourage teams to game the system. People are easily impressed with the teams with the higher point totals. That immediately sends the message to the other teams that a higher point total is good. Don’t be surprised if Team A starts pointing their stories really high. Instead of a velocity of 8 (1+1+2+2+2) you may just see a future velocity like 55 (8+8+13+13+13).
If you measure V&T at the organizational level, then you also have the problem of deciding what to do if your metric shifts out of an acceptable range. What’s your call to action if your summarized metric drops by 25%? If you measure value, then you shouldn’t care if V&T drops or increases. V&T only has meaning at the team level; they are completely meaningless at the organizational level. If your metric is about value and your metric drops by 25%, then that tells you something—something very important to the health of your organization.
Points are meaningless, it’s the value produced that matters.
Ramirez, O. M. (2021). Velocity and throughput are not organizational metrics. omramirez.com