Comparing the performance of agile teams

Recently I got into a conversation about measuring the performance of Engineering teams. The teams in question are following the Scrum framework and have decided to use story points for estimating the complexity of the work.

In the conversation, we were talking about the validity of measuring individual team performance and whether one should use this data to compare the performance across teams.

Now, for anyone that’s even taken more than a passing glance of any of the literature surrounding agile development, Scrum or estimation, the idea of measuring team performance and comparing performance between teams is a clear anti-pattern. Specifically, we’re told, “don’t compare velocity between teams”.

And I broadly agree. In most cases when this topic is raised, the drivers for comparing team performance are dangerous – to discover which teams are delivering as expected and which aren’t, which teams are good and which aren’t, and inevitably, to performance manage those teams and individuals in bad teams.

However, as with all arguments, there’s nuance, and I think there are some excellent reasons why understanding the performance of teams and comparing between them can be useful.

My approach to managing teams of teams has evolved over the past couple of years to consider more greatly the way the system as a whole is operating. I’ve been inspired to change by reading and development in the areas of system thinking. The rest of this post will pull on some ideas from system thinking.

If you’ve not come across the concept, then what follows might not resonate with you. Good background reading on the topic is Thinking in Systems, or for a quicker read, there’s the Introduction to Systems Thinking.

No matter how hard we work to eliminate cross-team dependencies, a group of engineering teams working towards a common goal operate in the same system of delivery. Therefore, it can be said that the performance of this broader system (the Engineering team) will be directly related to the performance of its constituent parts (a Scrum team, for example).

Furthermore, systems theory tells us that the performance of a system will be directly correlated with the performance of that system’s bottleneck. To improve overall system performance, we need to focus our attention on the bottleneck. Improvements elsewhere in the system are likely only to have negligible effects on the overall performance of the system and in the worst cases, some local enhancements will make the system performance worse.

With these thoughts in mind, our attention needs to be drawn to discovering the bottlenecks in our system and applying maximum leverage to these areas.

Although the system is more than the composite of the individual teams, the teams should be one place we look for our bottleneck. Measuring and comparing the performance of teams helps us understand if one of them is a bottleneck in the system of delivery.

Additionally, we can also learn whether there’s a local (team) issue that needs addressing, or whether there’s a systemic issue affecting more than one team needing our attention.

Let’s look at some examples to demonstrate these points. In these examples, I’m going to use throughput as a measure of performance. Throughput is the count of the number of tickets a Scrum team completes in a week. Increasing throughput is one of the goals of improving this system.

Below is an example chart plotting throughput for a team over 10 weeks. In this chart, we can see that this team has a steadily increasing throughput over this time.

Now we add a second team to this chart.

What conclusions can we draw from this? We could say that the blue team is better than the orange team because their throughput is higher. Yet, is this really true? Perhaps the orange team isn’t splitting their tickets down as aggressively as the blue team?

We also see that both teams are improving – so using these two teams as a sample set for the overall system, we may be able to say that the performance of the system as a whole is improving.

So far, comparing teams hasn’t really shown us anything more useful than not doing so.

Let’s add a third team. In this example, the green team has a higher throughput than both the blue and orange teams. We can also see that unlike blue and orange, the green team throughput is decreasing. As two of the three teams are continuing to improve, we might conclude that we have a local issue with the green team that we should look into.

To be more responsible with our charting, we can remove the axis and normalise the throughput counts. Doing this helps us to draw attention to critical differences and focus on the right conversations (it’s also why I used dotted lines for the actual throughput lines). If we look at the revised chart, it’s clear that we need to spend time with one team seeking improvements.

Looking at one more example, we see that every team in this system has decreasing throughput. Seeing this, rather than starting to work with individual teams to improve their throughput, we should focus our energies on developing a better understanding of the system they are operating in.

In this example, efforts to improve a single team’s performance is likely to have limited impact on overall performance. Had we not compared teams, we would not have seen this, and each one would likely have focused on locally optimising their own throughput.

In conclusion, making comparisons between teams isn’t inherently wrong. It can be done in a way which develops insight about the broader system and better understanding within teams about the issues that can and can’t be solved locally.

I have to pass a tremendous amount of credit onto Troy Magennis of Focused Objective for inspiring my thinking on this article. In 2019 he visited the company I worked at to teach us about statistics, estimation, forecasting and reporting. His work inspired my further discovery into this area and my thoughts on team comparisons are built upon the insights and ideas he shared with us.

Running Goals for 2020

Last year I ran in my first race at the Bracknell Samaritans 10k. The race took place in the blazing June heat, right in the middle of a summer heatwave. I finished with a time of 54m48s.

I followed this up with 2 more 10km races – the Brooklands Running GP and Windsor Lakeside 10km – completing both in sub-50 minute times.

After completing the three races, I’d caught the running bug and decided to push myself further in 2020.

Run 3 half-marathons

After completing 3 10km in 2020, I decided to double the distance and signed up for my first set of half-marathons this year. In February, I’ll be competing in my first half-marathon in Wokingham, before running the London Landmarks Half-Marathon to raise money for Prostate Cancer UK in March. I’ll round off the year in September with the Windsor Half-Marathon.

Training is well underway for this challenge, and I’m encouraged by my progress so far. I have no specific times in mind for the runs. That said, judging by the longer training runs I completed at the end of 2019 (approximately 11 miles), I think that with proper training a sub-2 hour time in one of them could be a real possibility.

My longest run – a loop of Bracknell

Run a 10km race under 45 minutes

Alongside my half-marathon challenge, I want to try and get my 10km time down. My fastest race is 3 minutes above the three-quarter hour mark, and I think with the right training plan, I can push it down to a sub-45 minute finish.

Brooklands running GP
Windsor 10k at Dorney Lake

I’ve got two 10km races booked for the year so far – the London Winter Run 10k in February as part of my spring half-marathon training plans and the summer Bracknell Samaritans 10k. The first is likely to come too soon, and the summer run is likely to be impeded by the heat (although it’s Britain so you can never quite predict this). My best chance will therefore probably a late autumn run, so I just need to pick the race to go for this PB.

2000km for the year?

My final running goal for 2020 is to run consistently and regularly. I’d love to break the 2000km barrier for the year. I don’t really know how possible this one is going to be, yet with the race training I like to do, I’d like to think I can give it a good shot!

9 months! Really?

It’s been almost nine months since my last blog entry. I’ve never been a prolific writer as you can see this in the gaps between posts. I think I’ve found the reason why.

Recently I came across the article Speed Matters by James Somers. Right there, in the second paragraph of the article, Somers summed up exactly my relationship with this blog:

“If every time you write a blog post it takes you six months, and you’re sitting around your apartment on a Sunday afternoon thinking of stuff to do, you’re probably not going to think of starting a blog post, because it’ll feel too expensive.”

James Somers

I feel seen.

Each time I’ve thought about writing a post, the idea comes into my head, I make a start and then… well then it doesn’t get finished. Or in the rare case it does get published it takes me hours or weeks of refinement and editing.

Each time I go round this loop, turning my next idea into text becomes harder to do. 

I’m not about to say I’m a changed man and that you can expect daily posts. However, I will aim to work more quickly, worrying less about getting the content just right before hitting publish.

And as case in point, this post only took me 10 minutes from idea to publishing. Start as you mean to go on.

Iceland – December 2018

Late in 2018, our thoughts turned to what we’d do to celebrate the end of 2018. Having had a couple of recent New Years Eve’s wiped out through illness, myself and Jo were ready to do a little more for the upcoming New Year celebrations.

Rather than spend time in the UK, we decided to book a last minute short break to Iceland. We booked through TUI, taking advantage of their pre-arranged Iceland tour packages to give us a view of the island.

Continue reading “Iceland – December 2018”

Removing Facebook (again)

Late in March 2018 I decided to delete my Facebook account.

Scheduled deletion of my Facebook account. I’d recently removed it from my phone and disabled my account. #DeleteFacebook— Steve Bennett 💬 (@stevebennett) March 20, 2018

This isn’t the first time I’d decided to do this. In 2014, as Facebook turned 10, I decided to put my own usage of the social network on pause.

This didn’t last too long and only a few months later I’d decided to reactivate my account.

In 2015, in another attempt to reduce my usage of Facebook, I removed the apps from my phone.

Purged #linkedin and #facebook from my phone.— Steve Bennett 💬 (@stevebennett) January 29, 2015

Once again, this didn’t stick and the big blue app found it’s way back onto my device. (LinkedIn is still absent though…)

Last year, a combination of factors convinced me to take more drastic action and delete my accounts on Facebook. Firstly, I was becoming aware that I was spending an increasing amount of time on the app and this time was taking me away from real personal interactions.

I was maintaining a semblance of managing relationships with people when in fact, all I was doing was observing only their personal highlights from afar. In turn, I believe that this was leading me to a false conclusion that I still had a connection with these people even though it’d probably been many, many months (years in some cases) since we last spoke. The truth is that at least for these people, any relationship or friendship I once had had now dwindled away.

Around the same time I was thinking about deleting my account, the Facebook-Cambridge Analytica scandal hit with details of over 87 million profiles being shared between the two companies and used in political campaigning.

Zuckerberg and Facebook apologised for the breach, yet (in my opinion) were more intent on shifting the blame to Cambridge Analytica. Both parties were (once again, in my opinion) complicit in the data sharing, however Facebook granted permission for Cambridge Analytica to use the data and therefore shoulders the accountability for ensuring the data is used responsibly.

As I read more about the data breach (was it really a breach?) I became convinced that letting my Facebook account die was the right thing to do. 30 days later, my big blue account had gone.

As the news coverage and hearings surrounding the scandals continued, I looked at my usage of two other properties owned by Facebook – WhatsApp and Instagram.

Instagram was a fairly easy to remove from my life. I’d never been a big user of the service, however, in the weeks and months since deleting my Facebook account, I had replaced some of my daily feed-lurking with Instagram usage. I took the decision that the cost of being part of the Facebook ecosystem (even via Instagram) wasn’t worth it and I deleted my Instagram account later in the year.

The last stage was to remove myself from WhatsApp. This was a lot harder.

I’d been a user of WhatsApp since the paid 99c days and was really impressed with the the service and it’s approach to messaging. I had a small number of contacts which I exchanged messages with regularly and was a member of a small number of groups. Moreover, unlike my usage on Facebook and Instagram, messaging was an interactive experience and the number of people who I knew on the service was growing.

I decided to remove my WhatsApp account to evaluate alternatives. So far, I’ve tried Telegram and Signal (in addition to SMS) and whilst they’re both comparable in terms of service to WhatsApp, they just don’t have the users. I do think that I’ll be returning to WhatsApp in 2019, especially as I predict that peer-to-peer and private group messaging will start to overtake our usage of broad public social network usage.

Of course, I’m never really out of the Facebook ecosystem. My wife, parents, siblings, family members, and many of my friends all still have accounts. I still appear in photos on their streams and I’m sure that Facebook is still able to put a name to my face.

Facebook tracing cookies litter the web, so I’m pretty confident in saying that I still have a profile. However, rather than it being one that I control (or at least claim to control), it’s a darker, shadowy profile that I can’t see or delete.

And there I think lies a problem. Whilst I can do a bunch of things to remove myself from the Facebook ecosystem, I don’t think you can really ever truly escape.

So, what’s life been like Post-Facebook? I’ve found myself using messaging and email more, and I’ve made more effort to reach out to people who I’ve previously lost connection with. Having not seen everything on Facebook, I’m able to have better conversations with people when we meet. I don’t spend as long on my phone, although my Twitter usage has gone up. Overall, whilst the first few months were more difficult, I can honestly say that I don’t miss Facebook or Instagram at all.

Have I missed stuff since leaving Facebook? Undoubtedly.

Do I feel like I’m missing out? Sometimes.

Am I willing to pay the necessary price to deal with this FOMO? Absolutely not.