Ant Colonies, Agile At Scale & The Illusion Of Control

Just a brief thought this morning about how susceptible we are to the illusion of control over complex systems (including organisations), and how that illusion has given rise to the myth of “scaled” software development processes.

Consider an ant colony: there could be millions of individual ants making up such an organisation. Working together, somehow, the colony appears to act as one – making decisions, performing work (like foraging for food) and building complex structures that look for all the world like they were designed by some overall architect.

But this appearance of central control is an illusion. All of that complex behaviour emerges through the millions of individuals doing their own things, but following simple rules that themselves have emerged through the process of evolution. Ant species who didn’t evolve to follow these simple rules were selected for extinction.

But this illusion of control is very seductive, and we seem to be hardwired to believe it: whether we believe that ant nests are designed, or we believe that the rain is controlled by the rain gods. In reality – and it’s ironic that we really only came to understand this in the age of computers, when we could see complex order emerge as the result of simple rules being applied over and over again in front of our very eyes – in truly complex systems, order can only emerge. It cannot be imposed from outside or above. Any attempt to do so will usually produce a short-term perturbation, but the underlying rules of the system will soon enough return it to it’s emergent form – a process called homeostasis.

And it’s homeostasis that explains why top-down “transformations” and re-organisations ultimately don’t work. If we don’t change the underlying – usually unwritten – rules that determine how individuals behave and interact, the old order will reestablish itself not long after the consultants and coaches have left. It therefore also explains why these “transformations” never seem to end, because management find they have to apply continuous never-ending effort to try to stop the old order re-establishing itself. It’s expensive, exhausting and ultimately futile.

To change the behaviour of ant colonies, you must change the DNA of the ants.

But how do we change the “DNA” of the individuals in our organisation? Well, how did nature do it? Evolution by natural selection. The ants in the colony have behaviours that are beneficial to the colony because if they weren’t, the colony wouldn’t survive. So when we see ant colonies today, we only see the ones where the ants have the good DNA. The bad DNA was selected for extinction.

Likewise, in businesses, it matters a great deal what behaviours in individuals are rewarded and what behaviours get selected for extinction. Quite often, we see behaviour being rewarded that harms our organisation. A classic example is bloating teams. Managers get rewarded more for managing larger teams. But in software development, larger teams tend to achieve less (and when they reach a certain size, they tend to achieve nothing) at much greater cost to the organisation.

It’s in the managers’ interests to grow the teams, but it’s in the organisation’s interest that teams be small. Incentives are not aligned, and damage to the organisation tends to result. Sometimes fatal damage. I’m thinking of a software company that had 1200 people in engineering, performing the work of maybe 30-40 good engineers. Their HQ is a supermarket now.

Reward the wrong behaviours, promote people for the wrong reasons, and the individuals in your organisation will receive those signals loud and clear and adapt their behaviour accordingly. Many software organisations have a knack for losing their best people and retaining the ones doing all the damage.

In nature, when an ant colony dies, the ants in it die. When a business dies, the ants move to other colonies, taking their dysfunctions with them. Just as people within an organisation are often rewarded for behaviours that harm the organisation, our hiring processes often reward those same behaviours. “How many people in your last team? How many $millions was in the budget you controlled?” And so on. When I read job specs for the software industry, I despair at just how sought-after harmful behaviours tend to be.

And, at the risk of getting a little meta, nowhere is this more visible than in the spreading of the illusion of control itself. When a top-down transformation finally implodes, the armies of certified-up-to-the-eyeballs coaches and consultants the transformation created are scattered in the wind, spreading this illusion to miriad other organisations. Until – let’s face it – the spreading of these behaviors becomes the only purpose of these behaviours. A virus, essentially. It exists to spread, to consume and re-purpose its hosts, and then spread again.

So company extinction doesn’t filter out the bad DNA. It just spreads it, like a supernova spreads radioactive elements. Not only do we need to change the incentives within our organisations to reward beneficial behaviours, we also need to take a long, hard look at our hiring practices and ask ourselves “What is this filtering out? What is it letting in?”



Author: codemanship

Founder of Codemanship Ltd and code craft coach and trainer

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s