Slow Tests Kill Businesses

I’m always surprised at how few organisations track some pretty fundamental stats about software development, because if they did then they might notice what’s been killing their business.

It’s a picture I’ve seen many, many times; a software product or system is created, and it goes live. But it has bugs. Many bugs. So, a bigger chunk of the available development time is used up fixing bugs for the second release. Which has even more bugs. Many, many bugs. So an even bigger chunk of the time is used to fix bugs for the third release.

It looks a little like this:

Over the lifetime of the software, the proportion of development time devoted to bug fixing increases until that’s pretty much all the developers are doing. There’s precious little time left for new features.

Naturally, if you can only spare 10% of available dev time for new features, you’re going to need 10 times as many developers. Right? This trend is almost always accompanied by rapid growth of the team.

So the 90% of dev time you’re spending on bug fixing is actually 90% of the time of a team that’s 10x as large – 900% of the cost of your first release, just fixing bugs.

So every new feature ends up in real terms costing 10x in the eighth release what it would have in the first. For most businesses, this rules out change – unless they’re super, super successful (i.e., lucky). It’s just too damned expensive.

And when you can’t change your software and your systems, you can’t change the way you do business at scale. Your business model gets baked in – petrified, if you like. And all you can do is throw an ever-dwindling pot of money at development just to stand still, while you watch your competitors glide past you with innovations you’ll never be able to offer your customers.

What happens to a business like that? Well, they’re no longer in business. Customers defected in greater and greater numbers to competitor products, frustrated by the flakiness of the product and tired of being fobbed off with promises about upgrades and hotly requested features and fixes that never arrived.

Now, this effect is entirely predictable. We’ve known about it for many decades, and we’ve known the causal mechanism, too.

Source: IBM System Science Institute

The longer a bug goes undetected, exponentially the more it costs to fix. In terms of process, the sooner we test new or changed code, the cheaper the fix is. This effect is so marked that teams actually find that if they speed up testing feedback loops – testing earlier and more often – they deliver working software faster.

This is very simply because they save more time downstream on bug fixes than they invest in earlier and more frequent testing.

The data used in the first two graphs was taken from a team that took more than 24 hours to build and test their code.

Here’s the same stats from a team who could build and test their code in less than 2 minutes (I’ve converted from releases to quarters to roughly match the 12-24 week release cycles of the first team – this second team was actually releasing every week):

This team has nearly doubled in size over the two years, which might sound bad – but it’s more of a rosy picture than the first team, whose costs spiraled to more than 1000% of their first release, most of which was being spent fixing bugs and effectively going round and round in circles chasing their own tails while their customers defected in droves.

I’ve seen this effect repeated in business after business – of all shapes and sizes: software companies, banks, retail chains, law firms, broadcasters, you name it. I’ve watched $billion businesses – some more than a century old – brought down by their inability to change their software and their business-critical systems.

And every time I got down to the root cause, there they were – slow tests.

Every. Single. Time.

Where’s User Experience In Your Development Process?

I ran a little poll through the Codemanship twitter account yesterday, and thought I’d share the result with you.

There are two things that strike me about the results. Firstly, it looks like teams who actively involve user experience experts throughout the design process are very much in the minority. To be honest, this comes as no great surprise. My own observations of development teams over years tend to see UXD folks getting involved early on – often before any developers are involved, or any customer tests have been discussed – in a kind of a Waterfall fashion. “We’re agile. But the user interface design must not change.”

To me, this is as nonsensical as those times when I’ve arrived on a project that has no use cases or customer tests, but somehow magically has a very fleshed-out database schema that we are not allowed to change.

Let’s be clear about this: the purpose of the user experience is to enable the user to achieve their goals. That is a discussion for everybody involved in the design process. It’s also something that is unlikely we’ll get right first time, so iterating the UXD multiple times with the benefit of end user feedback almost certainly will be necessary.

The most effective teams do not organise themselves into functional silos of requirements analysis, UXD, architecture, programming, security, data management, testing, release and operations & support and so on, throwing some kind of output (a use case, a wireframe, a UML diagram, source code, etc) over the wall to the next function.

The most effective teams organise themselves around achieving a goal. Whoever’s needed to deliver on that should be in the room – especially when those goals are being discussed and agreed.

I could have worded the question in my poll “User Experience Designers: when you explore user goals, how often are the developers involved?” I suspect the results would have been similar. Because it’s the same discussion.

On a movie production, you have people who write scripts, people who say the lines, people who create sets, people who design costumes, and so on. But, whatever their function, they are all telling the same story.

The realisation of working software requires multiple disciplines, and all of them should be serving the story. The best teams recognise this, and involve all of the disciplines early and throughout the process.

But, sadly, this still seems quite rare. I hear lip service being paid, but see little concrete evidence that it’s actually going on.

The second thing I noticed about this poll is that, despite several retweets, the response is actually pretty low compared to previous polls. This, I suspect, also tells a story. I know from both observation and from polls that teams who actively engage with their customers – let alone UXD professionals etc – in their BDD/ATDD process are a small minority (maybe about 20%). Most teams write the “customer tests” themselves, and mistake using a BDD tool like Cucumber for actually doing BDD.

But I also get a distinct sense, working with many dev teams, that UXD just isn’t on their radar. That is somebody else’s problem. This is a major, major miscalculation – every bit as much as believing that quality assurance is somebody else’s problem. Any line of code that doesn’t in some way change the user’s experience – and I use the term “user” in the wider sense that includes, for example, people supporting the software in production, who will have their own user experience – is a line of code that should be deleted. Who is it for? Whose story does it serve?

We are all involved in creating the user experience. Bad special effects can ruin a movie, you know.

We may not all be qualified in UXD, of course. And that’s why the experts need to be involved in the ongoing design process, because UX decisions are being taken throughout development. It only ends when the software ends (and even that process – decommissioning – is a user experience).

Likewise, every decision a UI designer takes will have technical implications, and they may not be the experts in that. Which is why the other disciplines need to be involved from the start. It’s very easy to write a throwaway line in your movie script like “Oh look, it’s Bill, and he’s brought 100,000 giant fighting robots with him”, but writing 100,000 giant fighting robots and making 100,000 giant fighting robots actually appear on the screen are two very different propositions.

So let’s move on from the days of developers being handed wire-frames and told to “code this up”, and from developers squeezing input validation error messages into random parts of web forms, and bring these – and all the other – disciplines together into what I would call a “development team”.

Proactive vs Reactive Learning (or “Why Your Company Only Does Easy Things”)

Imagine you lead an orchestra. The word comes down from on high “Tonight, our audience demands you play Rachmaninoff’s Piano Concerto No. 3. The future of the orchestra depends on it. We’re all counting on you.”

But your orchestra has no pianist. Nobody in your orchestra has even touched a piano, let alone taken lessons. You turn to the lead violin: “Quick. Google ‘how to play piano?’ “

Now, of course, there’s absolutely no chance that any human being could learn to play piano to that standard in a day. Or a week. Or a month. It takes a lot of time and a lot of work to get to that level. Years.

The inevitable result is that the orchestra will not be playing Rachmaninoff’s Piano Concerto No. 3 that evening. At least, not with the piano part. And that’s kind of essential to a piano concerto.

I see tech organisations in this situation on a regular basis. They discover a need that they’re simply nowhere near competent enough to deal with – something completely beyond the range of their current capabilities. “The users demand that the software learns from their interactions and anticipates their needs. Quick. Google ‘how to train a machine?'” “The customer demands a custom query language. Quick. Google ‘how to write a compiler?'” And so on.

Have we become so used to looking stuff up on Stack Overflow, I wonder, that we’ve forgotten that some of this stuff is hard? Some of these things take a long time to learn? Not everything is as easy as finding out what that error message means, or how to install a testing framework using NPM?

The latter style of learning is what some people call reactive. “I need to know this thing now, because it is currently impeding my progress.” And software development involves a lot of reactive learning. You do need to be rather good at looking stuff up to get through the typical working day, because there are just so, so many little details to remember.

Here’s the thing, though: reactive learning only really works for little details – things that are easy to understand and can be learned quickly. If the thing that impedes our progress is that we require a road bridge to be built to get us over that canyon, then that’s where we see the limits of reactive learning. It can remove small obstacles. Big problems that takes a long time to solve require a different style of learning that’s much more proactive.

If your orchestra only plays the instruments needed for the exact pieces they’ve played up to that point, then there’s in increased likelihood that there’ll be gaps. If a dev team only has the exact skill set for the work they’ve done up to that point, there are likewise very likely to be gaps.

It’s hard, of course, to anticipate every possible future need and prepare months or years in advance for every eventuality. But some orgs have a greater adaptive capacity than others because their people are skilled beyond today’s specific requirements. That is to say, they’re better at solving problems because they have more ways of solving problems – more strings to their bow (or more keys to their piano, if you like).

Compiler design might sound like the kind of esoteric computer-sciency thing that’s unlikely to arise as a business need. But think of it this way: what’s our code built on? Is the structure of programs not the domain model we work in every day? While I’ve never designed a compiler, I have had numerous occasions when – to write a tool that makes my job easier – it’s been very useful to understand that model. Programmers who understand what programs are made of tend to be more effective at reasoning about code, and better at writing code that’s about code. We use those tools every day, but all tooling has gaps. I’ve yet to meet a static analysis tool, for example, that had all the rules I’d be interested in applying to code quality.

The most effective dev teams I’ve come into contact with have invested in custom tooling to automate repetitive donkey work at the code face. Some of them end up being open-sourced, and you may be using them yourself today. How did you think our test runner’s unit test discovery worked?

Some books about stuff I had no immediate need to know but read anyway

Now, we could of course hire a pianist for our orchestra – one who already knows Rachmaninoff’s Piano Concerto No. 3. But guess what? It turns out pianists of that calibre are really difficult to find – probably because it takes years and years to get to that standard. (No shortage of people who manage pianists, of course.) And now you remember how you voted against all those “superfluous” music education programmes. If only you could have known that one day you might need a concert pianist. If only someone had warned you!

Well, here I am – warning you. Not all problems are easy. Some things take a long time to learn, and those things may crop up. And while nobody can guarantee that they will, this is essentially a numbers game. What are the odds that we have the capability – or can buy in the capability at short notice (which opens the lid on a can of worms I call “proactive recruitment”) – to solve this problem?

Most of the time, organisations end up walking away from the hard problems. They are restricted to the things most programmers can solve. This is not a good way to build competitive advantage, any more than sticking to works that don’t have a piano part is a good way to run a successful orchestra.

Enlightened organisations actively invest in developing capabilities they don’t see any immediate need for. Yes, they’re speculating. And speculation can be wasteful. Just like all uncertain endeavors can be wasteful. But there are usually signposts in our industry about what might be coming a year from now, a decade from now, and beyond.

And there are trends – the continued increase in available computing power is one good example. Look at what would be really useful but is currently computationally too expensive right now. In 1995, we saw continuous build and tests cycles as highly desirable. But most teams still ran them overnight, because the hardware was about 1000 times slower than today. Now coming into vogue – as I predicted it would over a decade ago – more and more of us are building and testing (and even automatically inspecting) our code continuously in the background as we type it. That was totally foreseeable. As is the rise of Continuous Inspection as a more mainstream discipline off the back of it.

There are countless examples of long-established and hugely success businesses being caught with their pants down by Moore’s Law.

Although digital photography was by no means a new invention, its sudden commercial viability 20 years ago over chemical photography nearly finished Kodak overnight. They had not speculated. They had not invested in digital photography capability. They’d been too busy being the market leader in film.

And then there was the meteoric rise of guitar amp simulators – a technology long sneered at (but begrudgingly used) by serious players, and less serious players like myself. The early generations of virtual amps didn’t sound great, and didn’t feel like playing through a real amp with real tubes. (Gotta love them tubes!) But – damn – they were convenient.

The nut they couldn’t crack was making it sound like it was being recorded through a real speaker cabinet with a real microphone. There was a potential solution – convolution, a mathematical process that can combine two signals, so the raw output of a guitar amp (real or virtual) can be combined with an “impulse response” (a short audio sample, like the short reverberation in a room after you click your fingers) of a cabinet and microphone to give a strikingly convincing approximation of what that signal would sound like in the space – or what that guitar amp output would sound like through those speakers recorded with that microphone. Now, suddenly, virtual guitar amps were convenient and sounded good.

But up to that point, convolution had been too computationally expensive to be viable for playing and recording on commercially available hardware. And then, suddenly, it wasn’t. Queue mad dash by established amp manufacturers to catch up. And, to be fair to them, their virtual amp offerings are pretty spiffy these days. Was this on their radar, I wonder? Did the managers and the engineers see virtual amp technology looming on the horizon and proactively invest in developing that capability in exactly the way Kodak didn’t? Not before virtual amp leaders like Line 6 had taken a chunk of their market share, I suspect. And now convolution is everywhere. So many choices, so many market players old and new.

You see, it’s all well and good making hay while the sun shines. But when the weather turns, don’t end up being the ones who didn’t think to invest in a umbrella.

The Software Design Process

One thing that sadly rarely gets discussed these days is how we design software. That is, how we get from a concept to working code.

As a student (and teacher) of software design and architecture of many years, experiencing first-hand many different methodologies from rigorous to ad hoc, heavyweight to agile, I can see similarities between all effective approaches.

Whether you’re UML-ing or BDD-ing or Event Storming-ing your designs, when it works, the thought process is the same.

It starts with a goal.

This – more often than not – is a problem that our customer needs solving.

This, of course, is where most teams get the design thinking wrong. They don’t start with a goal – or if they do, most of the team aren’t involved at that point, and subsequently are not made aware of what the original goal or problem was. They’re just handed a list of features and told “build that”, with no real idea what it’s for.

But they should start with a goal.

In design workshops, I encourage teams to articulate the goal as a single, simple problem statement. e.g.,

It’s really hard to find good vegan takeaway in my area.

Jason Gorman, just now

Our goal is to make it easier to order vegan takeaway food. This, naturally, begs the question: how hard is it to order vegan takeaway today?

If our target customer area is Greater London, then at this point we need to hit the proverbial streets and collect data to help us answer that question. Perhaps we could pick some random locations – N, E, S and W London – and try to order vegan takeaway using existing solutions, like Google Maps, Deliveroo and even the Yellow Pages.

Our data set gives us some numbers. On average, it took 47 minutes to find a takeaway restaurant with decent vegan options. They were, on average, 5.2 miles from the random delivery address. The orders took a further 52 minutes to be delivered. In 19% of selected delivery addresses, we were unable to order vegan takeaway at all.

What I’ve just done there is apply a simple thought process known as Goal-Question-Metric.

We ask ourselves, which of these do we think we could improve on with a software solution? I’m not at all convinced software would make the restaurants cook the food faster. Nor will it make the traffic in London less of an obstacle, so delivery times are unlikely to speed up much.

But if our data suggested that to find a vegan menu from a restaurant that will deliver to our address we had to search a bunch of different sources – including telephone directories – then I think that’s something we could improve on. It hints strongly that lack of vegan options isn’t the problem, just the ease of finding them.

A single searchable list of all takeaway restaurants with decent vegan options in Greater London might speed up our search. Note that word: MIGHT.

I’ve long advocated that software specifications be called “theories”, not “solutions”. We believe that if we had a searchable list of all those restaurants we had to look in multiple directories for, that would make the search much quicker, and potentially reduce the incidences when no option was found.

Importantly, we can compare the before and the after – using the examples we pulled from the real world – to see if our solution actually does improve search times and hit rates.

Yes. Tests. We like tests.

Think about it; we describe our modern development processes as iterative. But what does that really mean? To me – a physics graduate – it implies a goal-seeking process that applies a process over and over to an input, the output of which is fed into the next cycle, which converges on a stable working solution.

Importantly, if there’s no goal, and/or no way of knowing if the goal’s been achieved, then the process doesn’t work. The wheels are turning, the engine’s revving, but we ain’t going anywhere in particular.

Now, be honest, when have you ever been involved in a design process that started like that? But this is where good design starts: with a goal.

So, we have a goal – articulated in a testable way, importantly. What next?

Next, we imaginate (or is it visionize? I can never keep up with the management-speak) a feature – a proverbial button the user clicks – that solves their problem. What does it do?

Don’t think about how it works. Just focus on visualifying (I’m getting the hang of this now) what happens when the user clicks that magical button.

In our case, we imagine that when the user clicks the Big Magic Button of Destiny, they’re shown a list of takeaway restaurants with a decent vegan menu who can deliver to their address within a specified time (e.g., 45 minutes).

That’s our headline feature. A headline feature is the feature that solves the customer’s problem, and – therefore – is the reason for the system to exist. No, “Login” is never a headline feature. Nobody uses software because they want to log in.

Now we have a testable goal and a headline feature that solves the customer’s problem. It’s time to think about how that headline feature could work.

We would need a complete list of takeaway restaurants with decent vegan menus within any potential delivery address in our target area of Greater London.

We would need to know how long it might take to deliver from each restaurant to the customer’s address.

This would include knowing if the restaurant is still taking orders at that time.

Our headline feature will require other features to make it work. I call these supporting features. They exist only because of the headline feature – the one that solves the problem. The customer doesn’t want a database. They want vegan takeaway, damn it!

Our simple system will need a way to add restaurants to the list. It will need a way to estimate delivery times (including food preparation) between restaurant and customer addresses – and this may change (e.g., during busy times). It will need a way for restaurants to indicate if they’re accepting orders in real time.

At this point, you may be envisaging some fancypants Uber Eats style of solution with whizzy maps showing delivery drivers aimlessly circling your street for 10 minutes because nobody reads the damn instructions these days. Grrr.

But it ain’t necessarily so. This early on in the design process is no time for whizzy. Whizzy comes later. If ever. Remember, we’re setting out here to solve a problem, not build a whizzy solution.

I’ve seen some very high-profile applications go live with data entry interfaces knocked together in MS Access for that first simple release, for example. Remember, this isn’t a system for adding restaurant listings. This is a system for finding vegan takeaway. The headline feature’s always front-and-centre – our highest priority.

Also remember, we don’t know if this solution is actually going to solve the problem. The sooner we can test that, the sooner we can start iterating towards something better. And the simpler the solution, the sooner we can put it in the hands of end users. Let’s face it, there’s a bit of smoke and mirrors to even the most mature software solutions. We should know; we’ve looked behind the curtain and we know there’s no actual Wizard.

Once we’re talking about features like “Search for takeaway”, we should be in familiar territory. But even here, far too many teams don’t really grok how to get from a feature to working code.

But this thought process should be ingrained in every developer. Sing along if you know the words:

  • Who is the user and what do they want to do?
  • What jobs does the software need to do to give them that?
  • What data is required to do those jobs?
  • How can the work and the data be packaged together (e.g., in classes)
  • How will those modules talk to each other to coordinate the work end-to-end?

This is the essence of high-level modular software design. The syntax may vary (classes, modules, components, services, microservices, lambdas), but the thinking is the same. The user has needs (find vegan takeaway nearby). The software does work to satisfy those needs (e.g., estimate travel time). That work involves data (e.g., the addresses of restaurant and customer). Work and data can be packaged into discrete modules (e.g., DeliveryTimeEstimator). Those modules will need to call other modules to do related work (e.g., address.asLatLong()), and will therefore need “line of sight” – otherwise known as a dependency – to send that message.

You can capture this in a multitude of different ways – Class-Responsibility-Collaboration (CRC) cards, UML sequence diagrams… heck, embroider it on a tapestry for all I care. The thought process is the same.

This birds-eye view of the modules, their responsibilities and their dependencies needs to be translated into whichever technology you’ve selected to build this with. Maybe the modules are Java classes. Maybe their AWS lambdas. Maybe they’re COBOL programs.

Here we should be in writing code mode. I’ve found that if your on-paper (or on tapestry, if you chose that route) design thinking goes into detail, then it’s adding no value. Code is for details.

Start writing automated tests. Now that really should be familiar territory for every dev team.

/ sigh /

The design thinking never stops, though. For one, remember that everything so far is a theory. As we get our hands dirty in the details, our high-level design is likely to change. The best laid plans of mice and architects…

And, as the code emerges one test at a time, there’s more we need to think about. Our primary goal is to build something that solves the customer’s problem. But there are secondary goals – for example, how easy it will be to change this code when we inevitably learn that it didn’t solve the problem (or when the problem changes).

Most kitchen designs you can cater a dinner party in. But not every kitchen is easy to change.

It’s vital to remember that this is an iterative process. It only works if we can go around again. And again. And again. So organising our code in a way that makes it easy to change is super-important.

Enter stage left: refactoring.

Half the design decisions we make will be made after we’ve written the code that does the job. We may realise that a function or method is too big or too complicated and break it down. We may realise that names we’ve chosen make the code hard to understand, and rename. We may see duplication that could be generalised into a single, reusable abstraction.

Rule of thumb: if your high-level design includes abstractions (e.g., interfaces, design patterns, etc), you’ve detailed too early.

Jason Gorman, probably on a Thursday

The need for abstractions emerges organically as the code grows, through the process of reviewing and refactoring that code. We don’t plan to use factories or the strategy pattern, or to have a Vendor interface, in our solution. We discover the need for them to solve problems of software maintainability.

By applying organising principles like Simple Design, D.R.Y. Tell, Don’t Ask, Single Responsibility and the rest to the code is it grows, good, maintainable modular designs will emerge – often in unexpected ways. Let go of your planned architecture, and let the code guide you. Face it, it was going to be wrong anyway. Trust me: I know.

Here’s another place that far too many teams go wrong. As your code grows and an architecture emerges, it’s very, very helpful to maintain a birds-eye view of what that emerging architecture is becoming. Ongoing visualisation of the software – its modules, patterns, dependencies and so on – is something surprisingly few teams do these days. Working on agile teams, I’ve invested some of my time to creating and maintaining these maps of the actual terrain and displaying them prominently in the team’s area – domain models, UX storyboards, key patterns we’ve applied (e.g., how have we done MVC?) You’d be amazed what gets missed when everyone’s buried in code, neck-deep in details, and nobody’s keeping an eye on the bigger picture. This, regrettably, is becoming a lost skill – the baby Agile threw out with the bathwater.

So we build our theoretical solution, and deliver it to end users to try. And this is where the design process really starts.

Until working code meets the real world, it’s all guesswork at best. We may learn that some of the restaurants are actually using dairy products in the preparation of their “vegan” dishes. Those naughty people! We may discover that different customers have very different ideas about what a “decent vegan menu” looks like. We may learn that our estimated delivery times are wildly inaccurate because restaurants tell fibs to get more orders. We may get hundreds of spoof orders from teenagers messing with the app from the other side of the world.

Here’s my point: once the system hits the real world, whatever we thought was going to happen almost certainly won’t. There are always lessons that can only be learned by trying it for real.

So we go again. And that is the true essence of software design.

When are we done? When we’ve solved the problem.

And then we move on to the next problem. (e.g., “Yeah, vegan food’s great, but what about vegan booze?”)

Will There Be A Post-Pandemic IT Boom?

For billions of people around the world, things are pretty uncertain now. Hundreds of millions have lost their jobs. Businesses of all sizes – but especially smaller and newer businesses, many start-ups – are in trouble. Many have already folded.

The experts predict a recession the likes of which we haven’t seen in anyone’s lifetime. But there may be one sector that – as the dust settles – might even grow faster as a result of the pandemic.

Information and communications technology has come to the fore as country after country locked down, commanding businesses who could to let their employees work from home. This would not have been possible a generation ago for the vast majority. Most homes did not have computers, and almost no homes had Internet. Now, it’s the reverse.

While some household name brands have run into serious difficulties, new brands have become household names in the last 3 months – companies like Zoom, for example. “Zooming” is now as much a thing as “hoovering”.

Meanwhile, tens of thousands of established businesses have had their digital transformations stress-tested for the first time, and have found them wanting. From extreme cases like UK retailer Primark, who effectively had no online capability, to old hands in every sector who’ve invested billions in digital over the last 30 years, it seems most were not quite as “digital” as it turns out they needed to be.

From customer-facing transactions to internal business processes, the pandemic has revealed gaps that were being filled by people necessarily co-located in offices and shops and factories and so on. A client of mine, for example, still doesn’t have the ability to sign up new suppliers without a human being in the accounts department to access the mainframe via one of the dedicated terminals. They are rushing now to close that gap, but their mainframe skills base has dwindled to the point where nobody knows how. So they have to hire someone with COBOL skills to make the changes on that side, and C# skills to write the web front end for it. Good luck with that!

I’m noticing these digital gaps everywhere. Most organisations have them. They were missed because the processes still worked, thanks to the magic of People Going To OfficesTM. But now those gaps have been laid bare for everyone to see (and for customers and suppliers to experience).

Here’s the thing: thing’s aren’t going back to normal. The virus is going to be with us for some time, and even after we’ve tamed it with a vaccine or new treatments, everyone will be thinking about the next new virus. Just as COVID-19 leaves it mark on people it infects, the pandemic will leave its mark on our civilisation. We will adapt to a new normal. And a big component of that new normal will be digital technology. As wars accelerate science and technology, so too will COVID-19 accelerate digital innovation.

And it’s a match made in heaven, because this innovation can largely be done from our homes, thanks to… digital technology! It’s a self-accelerating evolution.

So, I have an inkling we’re going to be very busy in the near future.

Modularity & Readability

One thing I hear very often from less experienced developers is how difficult it can be for them to understand modular code.

The principles of modular design – that modules should:

  • Do one job
  • Hide their inner workings
  • Have swappable dependencies

– tend to lead to code that’s composed of small pieces that bind to abstractions for the other modules they use to do their jobs.

The key to making this work in practice rests on two factors: firstly, that developers get good at naming the boxes clearly enough so that people don’t have to look inside them to understand what they do, and secondly that developers accustom themselves to reading highly-composed code. More bluntly, developers have to learn how to read and write modular code.

Schools, universities and code clubs generally don’t get as far as modularity when they teach programming. Well, they may teach the mechanics of declaring and using modules, but they don’t present students with much opportunity to write larger, composed systems. The self-contained nature of programming problems in education typically presents students with algorithms whose implementations are all laid out on the screen in front of them.

Software at scale, though, doesn’t fit on a screen. They are jigsaw puzzles, and much more attention to how the pieces fit together is needed. Software design grows from being about algorithms and program flow to being about relationships between parts, at multiple levels of code organisation.

In this sense, young programmers leave school like freshly-minted playwrights who’ve only ever written short monologues. They know nothing of character, or motivation, or dialogue, of plotting, or pacing, or the three-act structure, nor have they ever concerned themselves with staging and practical considerations that just don’t come up when a single actor reads a single page standing centre-stage under a single spotlight.

Then they get their first job as a script assistant on a production of Noises Off and are all like “What are all these ‘stage directions’? Why are there so many scenes? I can’t follow this plot. Can we have just one character say all the lines?”

Here’s the thing; reading and writing modular code is an acquired skill. It doesn’t just happen overnight. As more and more young developers flood into the industry, I see more and more teams full of people who are easily bamboozled by modular, composed code.

Readability is about the audience. Programmers have a “reading age” defined by their ability to understand code, and code needs to be pitched to the audience’s reading age. This means that we may have to sacrifice some modularity for teams of less experienced developers. They’re not ready for it yet.

Having said all of that, of course, we get better at reading by being challenged. If we only ever read books that contained words we already know, we’d learn no new words.

I learned to read OO code by reading OO code written by more experienced programmers than me. They simultaneously pitched the code to be accessible to my level of understanding, and also a very little out of my current reach so that I had to stretch to follow the logic.

I know I’m a broken record on this topic, but that’s where mentoring comes in. Yes, there are many, many developers who lack the ability to read and write modular code. But every one of those teams could have someone who has lots of experience writing modular code who can challenge and guide them and bring them along over time – until one day it’s their turn to pay it forward.

The woeful lack of structured mentoring in our profession means that many developers go their entire careers never learning this skill. A lack of understanding combined with a lot of experience can be a dangerous mixture. “It is not that I don’t understand this play. This play is badly written. Good plays have a single character who stands in the centre of the stage under a single spotlight and reads out a 100-page monologue. Always.”

For those developers, a late-career Damascene conversion is unlikely to happen. I wish them luck.

For new developers, though, there’s a balance to be struck between working at a level they’re comfortable with today, and moving them forward to reading and writing more modular code in the future. Every program we write is both a solution to a problem today, and a learning experience to help us write a better solution tomorrow.

10 Things Every *Good* Software Development Method Does

I’ve been a programmer for the best part of four decades – three of them professionally – and, for the last 25 years, a keen student of this thing we call “software development”.

I’ve studied and applied a range of software development methods, principles, and techniques over those years. While, on the surface, Fusion may look different to the Unified Process, which may look different to Extreme Programming, which may look different to DSDM, which may look different to Cleanroom Software Engineering, when you look under the hood of these approaches, they actually have some fundamental things in common.

Here are the 10 things every software developer should know:

  1. Design starts with end users and their goals – be it with use cases, or with user stories, or with the “features” of Feature-Driven Development, the best development approaches drive their solution designs by first asking: Who will be using this software, and what will they be using it to do?
  2. Designs grow one usage scenario at a time – scenarios or examples drive the best solution designs, and those designs are fleshed out one scenario at a time to satisfy the user’s goal in “happy paths” (or to recover gracefully from not satisfying the user’s goal, which we call “edge cases”). Developers who try to consider multiple scenarios simultaneously tend to bite off more than they can chew.
  3. Solutions are delivered one scenario at a time – teams who deliver working software in end-to-end slices of functionality (e.g., the UI, business logic and database required to do a thing the user requires) tend to fare better than teams who deliver horizontal slices across their architecture (the UI components for all scenarios, and then the business logic, and then the database code). This is ffor two key reasons. Firstly, they can get user feedback from working features sooner, which speeds up the learning process. Secondly, if they only manage to deliver 75% of the software before a release date, they will have delivered 75% of end-to-end working features, instead of 75% of the layers of all features. We call this incremental delivery.
  4. Solutions evolve based on user feedback from increments – the other key ingredient in the way we deliver working software is how we learn from the feedback we get from end users in each increment of the software. With the finest requirements and design processes – and the best will in the world – we can’t expect to get it right first time. Maybe our solution doesn’t give them what they wanted. Maybe what they wanted turns out to be not what they really needed. The only way to find out for sure is to deliver what they asked for and let them take it for a spin. And then the feedback starts flooding in. The best approaches accept that feedback is not just unavoidable, it’s very desirable, and teams seek it out as often as possible.
  5. Plans change – if we can’t know whether we’re delivering the right software for sure until we’ve delivered it, then our approach to planning must be highly adaptable. Although the wasteland of real-world software development is littered with the bleached bones of “waterfall” projects that attempted to get it right first time (and inevitably failed), the idealised world of software development methods rejected that idea many decades ago. All serious methods are iterative, and all serious methods tell us that the plan will necessarily change. It’s management who resist change, not methods.
  6. Code changes – if plans change based on what we learn from end users, then it stands to reason that our code must also change to accommodate their feedback. This is the sticking point on many “agile” development teams. Their management processes may allow for the plan to change, but their technical practices (or the lack of them) may mean that changing the code is difficult, expensive and risky. There are a range of factors in the cost of changing software, but in the wider perspective, it essentially boils down to “How long will it take to deliver the next working iteration to end users?” If the answer is “months”, then change is going to be slow and the users’ feedback will be backed up like the LA freeway on a Monday morning. If it’s “minutes” then you can iterate very rapidly and learn your way to getting it right much faster. Delivery cycles are fundamental. They’re the metabolism of software development.
  7. Testing is fast and continuous – if the delivery cycle of the team is its metabolism, then testing is its thyroid. How long it takes to establish if our software’s broken will determine how fast our delivery cycle’ can be (if the goal is to avoid delivering broken software, of course.) If you aspire to a delivery cycle of minutes, then that leaves minutes to re-test your software. If all your testing’s done manually, then a modestly complex system will likely take weeks to re-test. And it’s a double whammy. Studies show that the longer a bug goes undetected, the exponentially greater it costs to fix it. If I break some code now and find out a minute from now, it’s a trifle to fix it. If I find out 6 weeks from now, it’s a whole other ball game. Teams who leave testing late typically end up spending most of their time fixing bugs instead of delivering valuable features and changes. All of this can profoundly impact delivery cycles and the cost of adapting to user feedback. Testing early and often is a feature of all serious methods. Automating our tests so they run fast is a feature of all the best methods.
  8. All work is undo-able – If we accept that its completely unrealistic to expect to get things right first time, then we must also accept that all the work we do is essentially an experiment from which we must learn. Sometimes, what we’ll learn is that what we’ve done is simply no good, and we need to do over. Software Configuration Management (of which version control is the central pillar) is a key component of all serious software development methods. A practice like Continuous Integration, done right, can bring us high levels of undo-ability, which massively reduces risk in what is a pretty risky endeavour. To use an analogy, think of software development as a multi-level computer game. Experienced gamers know to back up their place in the game frequently, so they don’t have to replay huge parts of it after a boo-boo. Same thing with version control and SCM. We don’t want our versions to be too far apart, or we’ll end up in a situation where we have to redo weeks or months of work because we took a wrong turn in the maze.
  9. Architecture is a process (not a person or a thing) – The best development methods treat software architecture and design as an ongoing activity that involves all stakeholders and is never finished. Good architectures are driven directly from user goals, ensuring that those goals are satisfied by the design above all else (e.g., use case realisations in the Unified Process), and applying organising principles – Simple Design, “Tell, Don’t Ask”, SOLID etc – to the internals of the solution design to ensure the code will be malleable enough to change to meet future needs. As an activity, architecture encompasses everything from the goals and tasks of end users, to the modular structure of the solution, to the everyday refactorings that are performed against code that falls short, the test suites that guard against regressions, the documentation that ships with the end product, and everything else which is informed by the design process. Since architecture is all-encompassing, all serious development methods mandate that it be a shared responsibility. The best methods strongly encourage a high level of architectural awareness within the team through continuous visualisation and review of the design. To some extent, everyone involved is defining the architecture. It is ever-changing and everyone’s responsibility.
  10. “Done” means we achieved the customer’s end goal – All of our work is for nothing if we don’t solve the problem we set out to solve. Too many teams are short-sighted when it comes to evaluating their success, considering only that a list of requested features was delivered, or that a product vision was realised. But all that tells us is that we administered the medicine. It doesn’t tell us if the medicine worked. If iterative development is a search algorithm, then it’s a goal-seeking search algorithm. One generation of working software at a time, we ask our end users to test the solution as a fit to their problem, learn what worked and what didn’t, and then go around again with an improved solution. We’re not “done” until the problem’s been solved. While many teams pay lip service to business goals or a business context, it’s often more as an exercise in arse-covering – “We need a business case to justify this £10,000,000 CRM system we’ve decided to build anyway!” – than the ultimate driver of the whole development process. Any approach that makes defining the end goal a part of the development process has put the cart before the horse. If we don’t have an end goal – a problem to be solved – then development shouldn’t begin. But all iterative development methods – and they’re all iterative to some degree – can be augmented with an outer feedback loop that considers business goals and tests working software in business situations, driving everything from there.

As a methodologist, I could spin you up an infinite number of software development methods with names like Goal-Oriented Object Delivery, or Customer Requirement Architectural Process. And, on the surface, I could make them all look quite different. But scratch the surface, and they’d all be fundamentally the same, in much the same way that programming languages – when you look past their syntax – tend to embrace the same underlying computing concepts.

Save yourself some time. Embrace the concepts.

Automate, Automate, Autonomy!

Thanks to pandemic-induced economic chaos, you’ve been forced to take a job on the quality assurance line at a factory that produces things.

The machine creates all kinds of random things, but your employer only sells a very specific subset of those things. All the things that don’t fit the profile have to be rejected, and melted down and fed back into the machine to make more things.

On your first day, you get training. (Oh, would that were true in software development!)

They stand you at the quality gate and start up the machine. All kinds of things come down the line at you. Your line manager tells you “Only let the green things through”. You grab all the things that aren’t green and throw them into the recycle bin. So far, so good.

“Only let the green round things through!” shouts your line manager. Okay, you think. Bit harder now. All non-green, non-round things go in the bin.

“Only let the green round small things through!” Now you’re really having to concentrate, a few green round small things end up in the bin, and a few non-green, non-round, non-small things get through.

“Only let the green round small things with Japanese writing on them through!” That’s a lot to process at the same time. Now your brain is struggling to cope. A bunch of blue and red things with Japanese writing on them get through. A bunch of square things get through. Your score has gone from 100% accurate to just 90%. Either someone will have to go through the boxes that have been packed and pick out all the rejects, or they’ll have to deal with 10% customer returns after they’ve been shipped.

“Only let the green round small things with Japanese writing on them that have beveled edges and a USB charging port on the opposite side to the writing and a power button in the middle of the writing and a picture of a horse  – not a donkey, mind, reject those ones! – and that glow in the dark through!”

Now it’s chaos. Almost every box shipped contains things that should have been thrown in the recycle bin. Almost every order gets returned. That’s just too much to process. Too many criteria.

We have several choices here:

  1. Slow down the line so we can methodically examine every thing against our checklist, one criteria at a time.
  2. Hire a whole bunch of people and give them one check each to do.
  3. Reset customer expectations about the quality of the things they’re buying.
  4. Automate the checking using cameras and robots and lasers and super-advanced A.I. so all those checks can be made at production speed to a high enough accuracy.

Number 4 is the option that potentially gives us the win-win of customer satisfaction and high productivity without the bigger payroll. It’s been the driving force behind the manufacturing revolutions in East Asia for the last 70 years: automate, automate, automate.

But it doesn’t come for free. High levels of automation require considerable ongoing investment in time, technology and training. In the UK, we’ve under-invested, becoming more and more inefficient and expensive while the quality of our output has declined. Shareholders want their return now. There’s no appetite for making improvements for the future.

There are obvious parallels in software development. Businesses want their software now. Most software organisations have little inclination to invest the time, technology and training required to reach the high levels of automation needed to achieve the coveted continuous delivery that would allow them to satisfy customer needs sooner, cheaper, and for longer.

The inescapable reality is that frictionless delivery demands an investment of 20-25% of your total software development budget. To put it more bluntly, everyone should be spending 1 day a week not on immediate customer requirements, but on making improvements in the delivery process that would mean meeting future customer requirements will be easier.

And so, for most teams, it never gets easier. The software just gets buggier, later and more expensive year after year.

What distinguishes those software teams who are getting it right from the rest? From observation, I’ve seen the same factor every time: autonomy. Teams will invest that 20-25% when it’s their choice. They’re tasked with delivering value, and allowed to figure out how best to do that. Nobody’s telling them how to do their jobs.

How did this blissful state come about? Again, from observation, those teams have autonomy because they took it. Freedom is rarely willingly given.

Now, I appreciate this is a whole can of worms. To take their autonomy, teams need to earn trust. The more trust a team has earned, the more likely they’ll be left alone. And this can be a chicken and egg kind of situation. To earn trust, the team has to reliably deliver. To reliably deliver, the team needs autonomy. This whole process must begin with a leap of faith on the business’s part. In other words, they have to give teams the benefit of the doubt long enough to see the results.

And here come the worms… Teams have to win over their customer from the start, before anything’s been delivered – before the customer’s had a chance to taste our pudding. This means that developers need to inspire enough confidence with their non-technical stakeholders – remember, this is a big money game – to reassure everyone that they’re in good hands. And we’re really, really bad at this.

The temptation is to over-promise, and set unrealistic expectations. This pretty much guarantees disappointment. The best way to inspire confidence is to have a good track record. No lawyer can guarantee to win your case. But a lawyer who won 9 of their last 10 cases is going to inspire more confidence than a lawyer who’s taking this as their first case promising you a win.

And we’re really, really bad at this, too – chiefly because software development teams are newly formed for that specific piece of work and don’t have a track record to speak of. Sure, individual developers may be known quantities, but in software, the unit of delivery is the team. I’ve watched teams of individually very strong developers fall flat on their collective arse.

And this is why I believe that this picture won’t change until organisations start to view teams as assets, and invest in them for a long-term pay-off as well as short-term delivery, 20/80. And, again, I don’t think this will be willingly given. So maybe we – as a profession – need to take the decision out of their hands.

It could all start with one big act of collective autonomy.

 

 

Why COBOL May Be The Language In Your Future

Yes, I know. Preposterous! COBOL’s 61 years old, and when was the last time you bumped into a COBOL programmer still working? Surely, Java is the new COBOL, right?

Think again. COBOL is alive and well. Some 220 billion lines of it power 71% of Fortune 500 companies. If a business is big enough and been around long enough, there’s a good chance the lion’s share of the transactions you do with that business involve some COBOL.

Fact is, they’re kind of stuck with it. Mainframe systems represent a multi-trillion dollar investment going back many decades. COBOL ain’t going nowhere for the foreseeable future.

What’s going is not the language but the programmers who know it and who know those critical business systems. The average age of a COBOL programmer in 2014 was 55. No doubt in 2020 it’s older than that, as young people entering IT aren’t exactly lining up to learn COBOL. Colleges don’t teach it, and you rarely hear it mentioned within the software development community. COBOL just isn’t sexy in the way Go or Python are.

As the COBOL programmer community edges towards statistical retirement – with the majority already retired (and frankly, dead) – the question looms: who is going to maintain these systems in 10 years or 20 years time?

One thing we know for sure: businesses have two choices – they can either replace the programmers, or replace the programs. Replacing legacy COBOL systems has proven to be very time-consuming and expensive for some banks. Commonwealth Bank of Australia took 5 years and $750 million to replace its core COBOL platform in 2012, for example.

And to replace a COBOL program, developers writing the new code at least need to be able to read the old code, which will require a good understanding of COBOL. There’s no getting around it: a bunch of us are going to have to learn COBOL one way or another.

I did a few months of COBOL programming in the mid-1990s, and I’d be lying if I said I enjoyed it. Compared to modern languages like Ruby and C#, COBOL is clunky and hard work.

But I’d also be lying if I said that COBOL can’t be made to work in the context of modern software development. In 1995, we “version controlled” our source files by replacing listings in cupboards. We tested our programs manually (if we tested them at all before going live). Our release processes were effectively the same as editing source files on the live server (on the mainframe, in this case).

But it didn’t need to be like that. You can manage versions of your COBOL source files in a VCS like Git. You can write unit tests for COBOL programs. You can do TDD in COBOL (see Exhibit A below).

You can refactor COBOL code (“Extract Paragraph”, “Extract Program”, “Move Field” etc), and you can automate a proper build an release process to deploy changed code safely to a mainframe (and roll it back if there’s a problem).

It’s possible to be agile in COBOL. The reason why so much COBOL legacy code fails in that respect has much more to do with decades of poor programming practices and very little to do with the language or the associated tools themselves.

I predict that, as more legacy COBOL programmers retire, the demand – and the pay – for COBOL programmers will rise to a point where some of you out there will find it irresistible.  And the impact on society if they can’t be found will be severe.

The next generation of COBOL programmers may well be us.

Is Your Agile Transformation Just ‘Agility Theatre’?

I’ve talked before about what I consider to be the two most important feedback loops in software development.

When I explain the feedback loops – the “gears” – of Test-Driven Development, I go to great pains to highlight which of those gears matter most, in terms of affecting our odds of success.

tdd_gears

Customer or business goals drive the whole machine of delivery – or at least, they should. We are not done because we passed some acceptance tests, or because a feature is in production. We’re only done when we’ve solved the customer’s problem.

That’s very likely going to require more than one go-around. Which is why the second most important feedback loop is the one that establishes if we’re good to go for the next release.

The ability to establish quickly and effectively if the changes we made to the software have broken it is critical to our ability to release it. Teams who rely on manual regression testing can take weeks to establish this, and their release cycles are inevitably very slow. Teams who rely mostly on automated system and integration tests have faster release cycles, but still usually far too slow for them to claim to be “agile”. Teams who can re-test most of the code in under a minute are able to release as often as the customer wants – many times a day, if need be.

The speed of regression testing – of establishing if our software still works – dictates whether our release cycles span months, weeks, or hours. It determines the metabolism of our delivery cycle and ultimately how many throws of the dice we get at solving the customer’s problem.

It’s as simple as that: faster tests = more throws of the dice.

If the essence of agility is responding to change, then I conclude that fast-running automated tests lie at the heart of that.

What’s odd is how so many “Agile transformations” seem to focus on everything but that. User stories don’t make you responsive to change. Daily stand-ups don’t make you responsive to change. Burn-down charts don’t make you responsive to change. Kanban boards don’t make you responsive to change. Pair programming doesn’t make you responsive to change.

It’s all just Agility Theatre if you’re not addressing the two must fundamental feedback loops, which the majority of organisations simply don’t. Their definition of done is “It’s in production”, as they work their way through a list of features instead of trying to solve a real business problem. And they all too often under-invest in the skills and the time needed to wrap software in good fast-running tests, seeing that as less important than the index cards and the Post-It notes and the Jira tickets.

I talk often with managers tasked with “Agilifying” legacy IT (e.g., mainframe COBOL systems). This means speeding up feedback cycles, which means speeding up delivery cycles, which means speeding up build pipelines, which – 99.9% of the time – means speeding up testing.

After version control, it’s #2 on my list of How To Be More Agile. And, very importantly, it works. But then, we shouldn’t be surprised that it does. Maths and nature teach us that it should. How fast do bacteria or fruit flies evolve – with very rapid “release cycles” of new generations – vs elephants or whales, whose evolutionary feedback cycles take decades?

There are two kinds of Agile consultant: those who’ll teach you Agility Theatre, and those who’ll shrink your feedback cycles. Non-programmers can’t help you with the latter, because the speed of the delivery cycle is largely determined by test execution time. Speeding up tests requires programming, as well as knowledge and experience of designing software for testability.

70% of Agile coaches are non-programmers. A further 20% are ex-programmers who haven’t touched code for over a decade. (According to the hundreds of CVs I’ve seen.) That suggests that 90% of Agile coaches are teaching Agility Theatre, and maybe 10% are actually helping teams speed up their feedback cycles in any practical sense.

It also strongly suggests that most Agile transformations have a major imbalance; investing heavily in the theatre, but little if anything in speeding up delivery cycles.