A Vision for Software Development: It’s All About Teams

I’ve thought a lot in recent years about how our profession is kind of fundamentally broken, and how we might be able to fix it.

The more I consider it, the more I think the underlying dysfunction revolves around software development teams, and the way they’re perceived as having only transient value.

Typically, when a business wants some new software, it builds a team specifically to deliver it. This can take many months and cost a lot of money. First, you have to find the people with the skills and experience you need. That in itself usually works out expensive – to the tune of tens of thousands of pounds per developer – before you’ve paid them a penny.

But the work doesn’t end there. Once you’ve formed your team, you then need to go through the “storming and norming” phases of team-building, during which they figure out how to work together. This, too, can work out very expensive.

So a formed, stormed and normed software team represents a very significant investment before you get a line of working code.

And, as we know, some teams never get past the forming stage, being stuck permanently in storming and norming and never really finding a satisfactory way to move forward together as they all pull in different directions.

The high-performing teams – the ones who work well together and can deliver good, valuable working software – are relative rarities, then: the truffles of the software industry.

Indeed, I’ve seen on many occasions how the most valuable end product from a software development effort turned out to be the team itself. They work well together, they enjoy working together, and they’re capable of doing great work. It’s just a pity the software itself was such a bad idea in the first place.

It seems somewhat odd then that businesses are usually so eager to break up these teams as soon as they see the work is “done”. It’s a sad fact of tech that the businesses who rely on the people who make it prefer to suffer us for as short a time as possible.

And this is where I think we got it wrong: should it be up to the customer to decide when to break up a high-performing dev team?

I can think of examples where such teams seized the day and, upon receiving their marching orders, set up their own company and bid for projects as a team, and it’s worked well.

This is very different to the standard model of development outsourcing, where a consultancy is effectively just a list of names of developers who might be thrown together for a specific piece of work, and then disbanded just as quickly at the end. Vanishingly few consultancies are selling teams. Most have to go through the hiring and team-building process themselves to fulfil their bids, acting as little more than recruitment agencies – albeit more expensive ones.

But I can’t help thinking that it’s teams that we should be focusing on, and teams our profession should be organising around:

  • Teams as the primary unit of software delivery
  • Teams as the primary commercial unit, self-organising and self-managing – possibly with expert helps for accounts and HR etc. Maybe it’s dev teams who should be outsourcing?
  • Teams as the primary route for training and development in our profession – i.e., through structured long-term apprenticeships

I have a vision of a software development profession restructured around teams. We don’t work for big companies who know nothing about software development. We work in partnerships that are made up of one or more teams, each team member specialised enough for certain kinds of work but also generalised enough to handle a wide range of work.

Each team would take on a small number of apprentices, and guide and mentor them – investing in training and development over a 3-5 year programme of learning and on-the-job experience – to grow the 10% of new developers our industry needs each year.

Each team would manage itself, work directly with customers. This should be part of the skillset of any professional developer.

Each team would make its own hiring decisions when it feels it needs specialised expertise from outside, or needs to grow (although my feelings on team size are well known), or wants to take on apprentices. So much that’s wrong with our industry stems from hiring decisions being taken by unqualified managers – our original sin, if you like.

And, for sure, these teams wouldn’t be immutable forever and all time. There would be an organic process of growth and change, perhaps of splitting into new teams as demand grows, and bringing in new blood to stop the pond from stagnating. But, just as even though pretty much every cell in my body’s been replaced many times but I’m somehow still recognisably me, it is possible with ongoing change to maintain a pattern of team identity and cohesion. There will always be a background level of forming, storming and norming. The trick is to keep that at a manageable level so we can keep delivering in the foreground.

There’s No Such Thing As “Agile”. There’s Just Software Development.

Okay, so this Sunday morning rant’s been a long time coming. And, for sure, I’ve expressed similar sentiments before. But I don’t think I’ve ever dedicated a whole blog post to this, so here goes. You may want to strap in.

20 years ago, a group of prominent software folk gathered at a ski resort in Utah to fix software development. Undoubtedly, it had become broken.

Broken by heavyweight, command-and-control processes. Broken by unrealistic and oftentimes downright dishonest plan-driven management that tried to impose the illusion of predictability to something that’s inherently unpredictable. Broken by huge outsourced teams and $multi-million – sometimes even $multi-billion – contracts that, statistically, were guaranteed to fail, crushed by their own weight. Broken by the loss of basic technical practices and the influx of low-skilled programmers to fuel to the first dotcom boom, all in the name of ballooning share prices of start-ups – many of which never made a red cent of profit.

All of this needed fixing. The resulting Manifesto for Agile Software Development attempted to reset the balance towards lightweight, feedback-driven ways of working, towards smaller, self-organising teams, towards continuous and rich face-to-face communication, and towards working software as the primary measure of progress.

Would that someone in that room had been from a marketing and communications background. A fundamental mistake was made at that meeting: they gave it a name.

And so, Agile Software Development became known as “another way of doing software development”. We could choose. We could be more Agile (with a capital “A”). Or, we could stick with our heavyweight, command-and-control, plan-driven, document-driven approach. Like Coke Zero and Original Coke.

The problem is that heavyweight, command-and-control, plan-driven, document-driven approaches tend to fail. Of course, for the outsourcing companies and the managers, they succeed in their underlying intention, which is to burn through a lot of money before the people signing the cheques realise. Which is why that approach still dominates today. I call it Mortgage-Driven Development. You may know it as “Waterfall”.

But if we measure it by tangible results achieved, Mortgage-Driven Development is a bust. We’ve known throughout my entire lifetime that it’s a bust. Winston Royce warned us it was a bust in 1970. No credible, informed commentator on software development has recommended we work that way for more than 50 years.

And yet, still many do. (The main difference in 2021 being that a lot of them call it “Agile”. Oh, the irony.)

How does Mortgage-Driven Development work, then? Well – to cut a long story short – badly, if you measure it by tangible customer outcomes like useful working software and end user problems being solved. If you measure it by the size of developers’ houses, though, it works really, really well.

MDD works from a very simple principle – namely that our customer shouldn’t find out that we’ve failed until a substantial part of our mortgage has been paid off. The longer we can delay the expectation of seeing working software in production, the more of our mortgage we can pay off before they realise there is no working software that can be released into production.

Progress in MDD is evidenced by documentation. The more of it we generate, the more progress is deemed to have been achieved. I’ve had to drag customers kicking and screaming to look at actual working software. But they’re more than happy to look at a 200-page architecture document purporting to describe the software, or a wall-sized Gantt chart with a comforting “You are here” to make the customer think progress has actually been made.

Of course, when I say “more than happy to look at”, they don’t actually read the architecture document – nobody does, and that includes the architects who write them – or give the plan anything more than a cursory glance. They’re like a spare tire in the boot of your car, or a detailed pandemic response plan sitting on a government server. There’s comfort in knowing it merely exists, even if – when the time comes – they are of no actual use.

Why customers and managers don’t find comfort in visible, tangible software is anybody’s guess. It could come down to personality types, maybe.

Teams who deliver early and often present the risk of failing fast. I took over a team for a small development shop who had spent a year going around in circles with their large public sector client. No software had been delivered. With me in the chair, and a mostly new team of “Agile” software developers, we delivered working software within three weeks from a standing start (we even had to build our own network, connected to the Internet by my 3G dongle). At which point, the end client decided this wasn’t working out, and canned the contract.

That particular project lives in infamy – recruiters would look at my CV and say “Oh, you worked on that?” It was viewed as failure. I view it as a major success. The end client paid for a year’s worth of nothing, and because nothing had been delivered, they didn’t realise it had already failed. They’d been barking up entirely the wrong tree. It took us just three weeks to make that obvious.

Saving clients millions of pounds by disproving their ideas quickly might seem like a good thing, but it runs counter to the philosophy of Mortgage-Driven Development.

I’ve been taken aside and admonished for “actually trying to succeed” with a software project. Some people view that as risky, because – in their belief system – we’re almost certainly going to fail, and therefore all efforts should be targeted at billing as much as possible and at escaping ultimate blame.

And, to me, this thing called Agile Software Development has always essentially just been “trying to succeed at delivering software”. We’re deliberately setting out to give end users what they need, and to do it in a way that gives them frequent opportunities to change their minds – including about whether they see any value in continuing.

The notion that we can do that without frequent feedback from end users trying working software is palpable nonsense – betting the farm on a proverbial “hole in one”. Nature solved the problem of complex system design, and here’s a heads-up: it isn’t a design committee, or a Gantt chart, or a 200-page architecture document.

Waterfall doesn’t work and never did. Big teams typically achieve less than small teams. Command-and-control is merely the illusion of control. Documents are not progress. And your project plan is a fiction.

When we choose to go down that road, we’re choosing to live in a lie.