Measuring Inner-Loop Agility

When I teach teams and managers about the feedback loops of software development, I try to stress the two most important loops – the ones that define agility.

Releases are where the value gets delivered, and – more importantly – where the end user feedback starts, so we can learn what works and what doesn’t and adapt for the next release.

The sooner we can release, the sooner we can start learning and adapting. So agile teams release frequently.

But frequency of releases doesn’t really define agility. I see teams who release every day or every week, but feature requests still take months to get into production.

That feature-to-production lead time is our best measure of how responsive to change we’re really being. How soon can we adapt to customer feedback?

For a portfolio of software products, a client of mine plotted average feature-to-production lead times against the average time it took to build and test the product.

We see a correlation between that feature-to-production lead time and the innermost loop of software delivery – build & test time.

Of course, this is a small data set, and all the usual caveats about “lies, damned lies and statistics” apply (I would love to do a bigger study, if anyone’s interested in participating).

But I’ve seen this distribution multiple times, and experienced it – and observed many, many teams experiencing it – in the field.

Products with slow build & test cycles tend to have much older backlogs. Indeed, backlogs themselves are a sign of slow lead times. I explained the causal mechanism for this in a previous post about Inner-Loop Agility. When we want to optimise nested loops, we get the biggest improvements in overall cycle time by focusing on the innermost loop.

Now, here’s the thing: everything that goes on between releases is really just guesswork. The magic happens when real end users get real working software, and we get to see how good our guesses were, and make more educated guesses in the next release cycle. We learn our way to value.

That’s why Inner-Loop Agility is so important, and why I’ve chosen to focus entirely on it as a trainer, coach and consultant. I can’t guarantee that you’re building the right thing (you almost certainly aren’t, no matter how well you plan), but I can offer you more throws of the dice.

Inner-Loop Agility (or “Why Your Agile Transformation Failed”)

Over the last couple of decades, I’ve witnessed more than my fair share of “Agile transformations”, and seen most of them produce disappointing results. In this post, I’m going to explain why they failed, and propose a way to beat the trend.

First of all, we should probably ask ourselves: what is an Agile transformation? This might seem like an obvious question, but you’d be surprised just how difficult it is to pin down any kind of accepted definition.

For some, it’s a process of adopting certain processes and practices, like the rituals of Scrum. If we do the rituals, then we’re Agile. Right?

Not so fast, buddy!

This is what many call “Cargo Cult Agility”. If we wear the right clothes and make offerings to the right gods, we’ll be Agile.

If we lose the capital “A”, and talk instead about agility, what is the goal of an agile transformation? To enable organisations to change direction quickly, I would argue.

How do we make organisations more responsive to change? The answer lies in that organisation’s feedback loops.

In software development, the most important feedback loop comes from delivering working software and systems to end users. Until our code hits the real world, it’s all guesswork.

So if we can speed up our release cycles so we can get more feedback sooner, and maintain the pace of those releases for as long as the business needs us to – i.e., the lifetime of that software – then we can effectively out-learn our competition.

Given how important the release cycle is, then, it’s no surprise that most Agile (with a capital “A”) transformations tend to focus on that feedback loop. But this is a fundamental mistake. The release cycle contains inner loops – wheels within wheels within wheels. If our goal is to speed up this outer feedback loop, we should be focusing most of our attention on the innermost feedback loops.

To understand why, let’s think about how we go about speeding up nested loops in code.

for (Release release:
releases) {
Thread.sleep(10);
System.out.println("RELEASE");
for (Feature feature:
release.features) {
Thread.sleep(10);
System.out.println("–FEATURE");
for (Scenario scenario:
feature.scenarios) {
Thread.sleep(10);
System.out.println("—-SCENARIO");
for (BuildAndTest buildAndTest:
scenario.buildAndTestCycles) {
Thread.sleep(1);
System.out.println("——BUILD & TEST");
}
}
}
}

Here’s some code that loops through a collection of releases. Each release loops through a list of features, and each feature has a list of scenarios that the system has to handle to implement that feature. For each scenario, it runs a build & test cycle multiple times. It’s a little model of a software development process.

Think of the development process as a set of gears. The largest gear turns the slowest, and drives a smaller, faster gear, which drives an even smaller and faster gear and so on.

In each loop, I’ve built in a delay of 10 ms to approximate the overhead of performing that particular loop (e.g., 10 ms to plan a release).

When I run this code, it takes 1 m 53 s to execute. Our release cycles are slow.

Now, here’s where most Agile transformations go wrong. They focus most of their attention on those outer loops. This produces very modest improvements in release cycle time.

Let’s “optimise” the three outer loops, reducing the delay by 90%.

for (Release release:
releases) {
Thread.sleep(1);
System.out.println("RELEASE");
for (Feature feature:
release.features) {
Thread.sleep(1);
System.out.println("–FEATURE");
for (Scenario scenario:
feature.scenarios) {
Thread.sleep(1);
System.out.println("—-SCENARIO");
for (BuildAndTest buildAndTest:
scenario.buildAndTestCycles) {
Thread.sleep(1);
System.out.println("——BUILD & TEST");
}
}
}
}

When I run this optimised code, it executes in 1 m 44 s. That’s only a 9% improvement in release cycle time, and we had to work on three loops to get it.

This time, let’s ignore those outer loops and just work on the innermost loop – build & test.

Now it finished in just 22 seconds. That’s an 81% improvement, just from optimising that innermost loop.

When we look at the output from this code, it becomes obvious why.

RELEASE
--FEATURE
----SCENARIO
------BUILD & TEST
------BUILD & TEST
------BUILD & TEST
------BUILD & TEST
------BUILD & TEST
------BUILD & TEST
------BUILD & TEST
------BUILD & TEST
------BUILD & TEST
------BUILD & TEST
----SCENARIO
------BUILD & TEST
------BUILD & TEST
------BUILD & TEST
------BUILD & TEST
------BUILD & TEST
------BUILD & TEST
------BUILD & TEST
------BUILD & TEST
------BUILD & TEST
------BUILD & TEST
----SCENARIO
------BUILD & TEST
------BUILD & TEST
------BUILD & TEST
------BUILD & TEST

Of course, this is a very simplistic model of a much more complex reality, but the principle at any scale works just as well, and the results I’ve seen over the years bear it out: to reduce release cycle times, focus your attention on the innermost feedback loops. I call this Inner-Loop Agility.

Think of the micro-iterations of Test-Driven Development, refactoring and Continuous Integration. They all involve one key step – the part where we find out if the software works – which is to build and test it. We test it at every green light in TDD. We test it after every refactoring. We test it before we check in our changes (and afterwards, on a build server to rule out configuration differences with our desktops).

In Agile Software Development, we build and test our code A LOT – many times an hour. And we can only do this if building and testing our code is fast. If it takes an hour, then we can’t have Inner Loop Agility. And if we can’t have Inner-Loop Agility, we can’t have fast release cycles.

Of course, we could test less often. That always ends well. Here’s the thing, the more changes we make to the code before we test it, the more bugs we introduce and then catch later. The later we catch bugs, the more they cost to fix. When we test less often, we tend to end up spending more and more of our cycle time fixing bugs.

It’s not uncommon for teams to end up doing zero-feature releases, where there’s just a bunch of bug fixes and no value-add for the customer in each release.

A very common end result of a costly Agile transformation is often little more than Agility Theatre. Sure, we do the sprints. We have the stand-ups. We estimate the story points. But’s it ends up being all work and little useful output in each release. The engine’s at maximum revvs, but our car’s going nowhere.

Basically, the gears of our development process are the wrong way round.

Organisations who optimise their outer feedback loops but neglect the inner loops are operating in a “lower gear”.

There’s no real mystery about why Agile transformations tend to focus most of their attention on the outer feedback loops.

Firstly, the people signing the cheques understand those loops, and can actively engage with them – in the mistaken belief that agility is all about them.

Secondly, the $billion industry – the “Agile-Industrial Complex” – that trains and mentors organisations during these transformations is largely made up of coaches and consultants who have either a lapsed programming background, or no programming background at all. In a sample of 100 Agile Coach CV’s, I found that 70% had no programming background, and a further 20% hadn’t done it for at least a decade. 90% of Agile Coaches can’t help you with the innermost feedback loops. Or to put it more bluntly, 90% of Agile Coaches focus on the feedback loops that deliver the least impressive reductions in release cycle time.

Just to be clear, I’m not suggesting these outer feedback loops don’t matter. There’s usually much work to be done at all levels from senior management down to help organisations speed up their cycle times, and to attempt it without management’s blessing is typically folly. Improving build and test cycles requires a very significant investment – in skills, in time, in resource – and that shouldn’t be underestimated.

But to focus almost exclusively on the outer feedback loops produces very modest results, and it’s arguably where Agile transformations have gained their somewhat dismal reputation among business stakeholders and software professionals alike.