Comparing the performance of agile teams

Recently I got into a conversation about measuring the performance of Engineering teams. The teams in question are following the Scrum framework and have decided to use story points for estimating the complexity of the work.

In the conversation, we were talking about the validity of measuring individual team performance and whether one should use this data to compare the performance across teams.

Now, for anyone that’s even taken more than a passing glance of any of the literature surrounding agile development, Scrum or estimation, the idea of measuring team performance and comparing performance between teams is a clear anti-pattern. Specifically, we’re told, “don’t compare velocity between teams”.

And I broadly agree. In most cases when this topic is raised, the drivers for comparing team performance are dangerous – to discover which teams are delivering as expected and which aren’t, which teams are good and which aren’t, and inevitably, to performance manage those teams and individuals in bad teams.

However, as with all arguments, there’s nuance, and I think there are some excellent reasons why understanding the performance of teams and comparing between them can be useful.

My approach to managing teams of teams has evolved over the past couple of years to consider more greatly the way the system as a whole is operating. I’ve been inspired to change by reading and development in the areas of system thinking. The rest of this post will pull on some ideas from system thinking.

If you’ve not come across the concept, then what follows might not resonate with you. Good background reading on the topic isĀ Thinking in Systems, or for a quicker read, there’s the Introduction to Systems Thinking.

No matter how hard we work to eliminate cross-team dependencies, a group of engineering teams working towards a common goal operate in the same system of delivery. Therefore, it can be said that the performance of this broader system (the Engineering team) will be directly related to the performance of its constituent parts (a Scrum team, for example).

Furthermore, systems theory tells us that the performance of a system will be directly correlated with the performance of that system’s bottleneck. To improve overall system performance, we need to focus our attention on the bottleneck. Improvements elsewhere in the system are likely only to have negligible effects on the overall performance of the system and in the worst cases, some local enhancements will make the system performance worse.

With these thoughts in mind, our attention needs to be drawn to discovering the bottlenecks in our system and applying maximum leverage to these areas.

Although the system is more than the composite of the individual teams, the teams should be one place we look for our bottleneck. Measuring and comparing the performance of teams helps us understand if one of them is a bottleneck in the system of delivery.

Additionally, we can also learn whether there’s a local (team) issue that needs addressing, or whether there’s a systemic issue affecting more than one team needing our attention.

Let’s look at some examples to demonstrate these points. In these examples, I’m going to use throughput as a measure of performance. Throughput is the count of the number of tickets a Scrum team completes in a week. Increasing throughput is one of the goals of improving this system.

Below is an example chart plotting throughput for a team over 10 weeks. In this chart, we can see that this team has a steadily increasing throughput over this time.

Now we add a second team to this chart.

What conclusions can we draw from this? We could say that the blue team is better than the orange team because their throughput is higher. Yet, is this really true? Perhaps the orange team isn’t splitting their tickets down as aggressively as the blue team?

We also see that both teams are improving – so using these two teams as a sample set for the overall system, we may be able to say that the performance of the system as a whole is improving.

So far, comparing teams hasn’t really shown us anything more useful than not doing so.

Let’s add a third team. In this example, the green team has a higher throughput than both the blue and orange teams. We can also see that unlike blue and orange, the green team throughput is decreasing. As two of the three teams are continuing to improve, we might conclude that we have a local issue with the green team that we should look into.

To be more responsible with our charting, we can remove the axis and normalise the throughput counts. Doing this helps us to draw attention to critical differences and focus on the right conversations (it’s also why I used dotted lines for the actual throughput lines). If we look at the revised chart, it’s clear that we need to spend time with one team seeking improvements.

Looking at one more example, we see that every team in this system has decreasing throughput. Seeing this, rather than starting to work with individual teams to improve their throughput, we should focus our energies on developing a better understanding of the system they are operating in.

In this example, efforts to improve a single team’s performance is likely to have limited impact on overall performance. Had we not compared teams, we would not have seen this, and each one would likely have focused on locally optimising their own throughput.

In conclusion, making comparisons between teams isn’t inherently wrong. It can be done in a way which develops insight about the broader system and better understanding within teams about the issues that can and can’t be solved locally.

I have to pass a tremendous amount of credit onto Troy Magennis of Focused Objective for inspiring my thinking on this article. In 2019 he visited the company I worked at to teach us about statistics, estimation, forecasting and reporting. His work inspired my further discovery into this area and my thoughts on team comparisons are built upon the insights and ideas he shared with us.

Running Goals for 2020

Last year I ran in my first race at the Bracknell Samaritans 10k. The race took place in the blazing June heat, right in the middle of a summer heatwave. I finished with a time of 54m48s.

I followed this up with 2 more 10km races – the Brooklands Running GP and Windsor Lakeside 10km – completing both in sub-50 minute times.

After completing the three races, I’d caught the running bug and decided to push myself further in 2020.

Run 3 half-marathons

After completing 3 10km in 2020, I decided to double the distance and signed up for my first set of half-marathons this year. In February, I’ll be competing in my first half-marathon in Wokingham, before running the London Landmarks Half-Marathon to raise money for Prostate Cancer UK in March. I’ll round off the year in September with the Windsor Half-Marathon.

Training is well underway for this challenge, and I’m encouraged by my progress so far. I have no specific times in mind for the runs. That said, judging by the longer training runs I completed at the end of 2019 (approximately 11 miles), I think that with proper training a sub-2 hour time in one of them could be a real possibility.

My longest run – a loop of Bracknell

Run a 10km race under 45 minutes

Alongside my half-marathon challenge, I want to try and get my 10km time down. My fastest race is 3 minutes above the three-quarter hour mark, and I think with the right training plan, I can push it down to a sub-45 minute finish.

Brooklands running GP
Windsor 10k at Dorney Lake

I’ve got two 10km races booked for the year so far – the London Winter Run 10k in February as part of my spring half-marathon training plans and the summer Bracknell Samaritans 10k. The first is likely to come too soon, and the summer run is likely to be impeded by the heat (although it’s Britain so you can never quite predict this). My best chance will therefore probably a late autumn run, so I just need to pick the race to go for this PB.

2000km for the year?

My final running goal for 2020 is to run consistently and regularly. I’d love to break the 2000km barrier for the year. I don’t really know how possible this one is going to be, yet with the race training I like to do, I’d like to think I can give it a good shot!