Modularity & Readability

One thing I hear very often from less experienced developers is how difficult it can be for them to understand modular code.

The principles of modular design – that modules should:

• Do one job
• Hide their inner workings
• Have swappable dependencies

– tend to lead to code that’s composed of small pieces that bind to abstractions for the other modules they use to do their jobs.

The key to making this work in practice rests on two factors: firstly, that developers get good at naming the boxes clearly enough so that people don’t have to look inside them to understand what they do, and secondly that developers accustom themselves to reading highly-composed code. More bluntly, developers have to learn how to read and write modular code.

Schools, universities and code clubs generally don’t get as far as modularity when they teach programming. Well, they may teach the mechanics of declaring and using modules, but they don’t present students with much opportunity to write larger, composed systems. The self-contained nature of programming problems in education typically presents students with algorithms whose implementations are all laid out on the screen in front of them.

Software at scale, though, doesn’t fit on a screen. They are jigsaw puzzles, and much more attention to how the pieces fit together is needed. Software design grows from being about algorithms and program flow to being about relationships between parts, at multiple levels of code organisation.

In this sense, young programmers leave school like freshly-minted playwrights who’ve only ever written short monologues. They know nothing of character, or motivation, or dialogue, of plotting, or pacing, or the three-act structure, nor have they ever concerned themselves with staging and practical considerations that just don’t come up when a single actor reads a single page standing centre-stage under a single spotlight.

Then they get their first job as a script assistant on a production of Noises Off and are all like “What are all these ‘stage directions’? Why are there so many scenes? I can’t follow this plot. Can we have just one character say all the lines?”

Here’s the thing; reading and writing modular code is an acquired skill. It doesn’t just happen overnight. As more and more young developers flood into the industry, I see more and more teams full of people who are easily bamboozled by modular, composed code.

Readability is about the audience. Programmers have a “reading age” defined by their ability to understand code, and code needs to be pitched to the audience’s reading age. This means that we may have to sacrifice some modularity for teams of less experienced developers. They’re not ready for it yet.

Having said all of that, of course, we get better at reading by being challenged. If we only ever read books that contained words we already know, we’d learn no new words.

I learned to read OO code by reading OO code written by more experienced programmers than me. They simultaneously pitched the code to be accessible to my level of understanding, and also a very little out of my current reach so that I had to stretch to follow the logic.

I know I’m a broken record on this topic, but that’s where mentoring comes in. Yes, there are many, many developers who lack the ability to read and write modular code. But every one of those teams could have someone who has lots of experience writing modular code who can challenge and guide them and bring them along over time – until one day it’s their turn to pay it forward.

The woeful lack of structured mentoring in our profession means that many developers go their entire careers never learning this skill. A lack of understanding combined with a lot of experience can be a dangerous mixture. “It is not that I don’t understand this play. This play is badly written. Good plays have a single character who stands in the centre of the stage under a single spotlight and reads out a 100-page monologue. Always.”

For those developers, a late-career Damascene conversion is unlikely to happen. I wish them luck.

For new developers, though, there’s a balance to be struck between working at a level they’re comfortable with today, and moving them forward to reading and writing more modular code in the future. Every program we write is both a solution to a problem today, and a learning experience to help us write a better solution tomorrow.

10 Things Every *Good* Software Development Method Does

I’ve been a programmer for the best part of four decades – three of them professionally – and, for the last 25 years, a keen student of this thing we call “software development”.

I’ve studied and applied a range of software development methods, principles, and techniques over those years. While, on the surface, Fusion may look different to the Unified Process, which may look different to Extreme Programming, which may look different to DSDM, which may look different to Cleanroom Software Engineering, when you look under the hood of these approaches, they actually have some fundamental things in common.

Here are the 10 things every software developer should know:

1. Design starts with end users and their goals – be it with use cases, or with user stories, or with the “features” of Feature-Driven Development, the best development approaches drive their solution designs by first asking: Who will be using this software, and what will they be using it to do?
2. Designs grow one usage scenario at a time – scenarios or examples drive the best solution designs, and those designs are fleshed out one scenario at a time to satisfy the user’s goal in “happy paths” (or to recover gracefully from not satisfying the user’s goal, which we call “edge cases”). Developers who try to consider multiple scenarios simultaneously tend to bite off more than they can chew.
3. Solutions are delivered one scenario at a time – teams who deliver working software in end-to-end slices of functionality (e.g., the UI, business logic and database required to do a thing the user requires) tend to fare better than teams who deliver horizontal slices across their architecture (the UI components for all scenarios, and then the business logic, and then the database code). This is ffor two key reasons. Firstly, they can get user feedback from working features sooner, which speeds up the learning process. Secondly, if they only manage to deliver 75% of the software before a release date, they will have delivered 75% of end-to-end working features, instead of 75% of the layers of all features. We call this incremental delivery.
4. Solutions evolve based on user feedback from increments – the other key ingredient in the way we deliver working software is how we learn from the feedback we get from end users in each increment of the software. With the finest requirements and design processes – and the best will in the world – we can’t expect to get it right first time. Maybe our solution doesn’t give them what they wanted. Maybe what they wanted turns out to be not what they really needed. The only way to find out for sure is to deliver what they asked for and let them take it for a spin. And then the feedback starts flooding in. The best approaches accept that feedback is not just unavoidable, it’s very desirable, and teams seek it out as often as possible.
5. Plans change – if we can’t know whether we’re delivering the right software for sure until we’ve delivered it, then our approach to planning must be highly adaptable. Although the wasteland of real-world software development is littered with the bleached bones of “waterfall” projects that attempted to get it right first time (and inevitably failed), the idealised world of software development methods rejected that idea many decades ago. All serious methods are iterative, and all serious methods tell us that the plan will necessarily change. It’s management who resist change, not methods.
6. Code changes – if plans change based on what we learn from end users, then it stands to reason that our code must also change to accommodate their feedback. This is the sticking point on many “agile” development teams. Their management processes may allow for the plan to change, but their technical practices (or the lack of them) may mean that changing the code is difficult, expensive and risky. There are a range of factors in the cost of changing software, but in the wider perspective, it essentially boils down to “How long will it take to deliver the next working iteration to end users?” If the answer is “months”, then change is going to be slow and the users’ feedback will be backed up like the LA freeway on a Monday morning. If it’s “minutes” then you can iterate very rapidly and learn your way to getting it right much faster. Delivery cycles are fundamental. They’re the metabolism of software development.
7. Testing is fast and continuous – if the delivery cycle of the team is its metabolism, then testing is its thyroid. How long it takes to establish if our software’s broken will determine how fast our delivery cycle’ can be (if the goal is to avoid delivering broken software, of course.) If you aspire to a delivery cycle of minutes, then that leaves minutes to re-test your software. If all your testing’s done manually, then a modestly complex system will likely take weeks to re-test. And it’s a double whammy. Studies show that the longer a bug goes undetected, the exponentially greater it costs to fix it. If I break some code now and find out a minute from now, it’s a trifle to fix it. If I find out 6 weeks from now, it’s a whole other ball game. Teams who leave testing late typically end up spending most of their time fixing bugs instead of delivering valuable features and changes. All of this can profoundly impact delivery cycles and the cost of adapting to user feedback. Testing early and often is a feature of all serious methods. Automating our tests so they run fast is a feature of all the best methods.
8. All work is undo-able – If we accept that its completely unrealistic to expect to get things right first time, then we must also accept that all the work we do is essentially an experiment from which we must learn. Sometimes, what we’ll learn is that what we’ve done is simply no good, and we need to do over. Software Configuration Management (of which version control is the central pillar) is a key component of all serious software development methods. A practice like Continuous Integration, done right, can bring us high levels of undo-ability, which massively reduces risk in what is a pretty risky endeavour. To use an analogy, think of software development as a multi-level computer game. Experienced gamers know to back up their place in the game frequently, so they don’t have to replay huge parts of it after a boo-boo. Same thing with version control and SCM. We don’t want our versions to be too far apart, or we’ll end up in a situation where we have to redo weeks or months of work because we took a wrong turn in the maze.
9. Architecture is a process (not a person or a thing) – The best development methods treat software architecture and design as an ongoing activity that involves all stakeholders and is never finished. Good architectures are driven directly from user goals, ensuring that those goals are satisfied by the design above all else (e.g., use case realisations in the Unified Process), and applying organising principles – Simple Design, “Tell, Don’t Ask”, SOLID etc – to the internals of the solution design to ensure the code will be malleable enough to change to meet future needs. As an activity, architecture encompasses everything from the goals and tasks of end users, to the modular structure of the solution, to the everyday refactorings that are performed against code that falls short, the test suites that guard against regressions, the documentation that ships with the end product, and everything else which is informed by the design process. Since architecture is all-encompassing, all serious development methods mandate that it be a shared responsibility. The best methods strongly encourage a high level of architectural awareness within the team through continuous visualisation and review of the design. To some extent, everyone involved is defining the architecture. It is ever-changing and everyone’s responsibility.
10. “Done” means we achieved the customer’s end goal – All of our work is for nothing if we don’t solve the problem we set out to solve. Too many teams are short-sighted when it comes to evaluating their success, considering only that a list of requested features was delivered, or that a product vision was realised. But all that tells us is that we administered the medicine. It doesn’t tell us if the medicine worked. If iterative development is a search algorithm, then it’s a goal-seeking search algorithm. One generation of working software at a time, we ask our end users to test the solution as a fit to their problem, learn what worked and what didn’t, and then go around again with an improved solution. We’re not “done” until the problem’s been solved. While many teams pay lip service to business goals or a business context, it’s often more as an exercise in arse-covering – “We need a business case to justify this £10,000,000 CRM system we’ve decided to build anyway!” – than the ultimate driver of the whole development process. Any approach that makes defining the end goal a part of the development process has put the cart before the horse. If we don’t have an end goal – a problem to be solved – then development shouldn’t begin. But all iterative development methods – and they’re all iterative to some degree – can be augmented with an outer feedback loop that considers business goals and tests working software in business situations, driving everything from there.

As a methodologist, I could spin you up an infinite number of software development methods with names like Goal-Oriented Object Delivery, or Customer Requirement Architectural Process. And, on the surface, I could make them all look quite different. But scratch the surface, and they’d all be fundamentally the same, in much the same way that programming languages – when you look past their syntax – tend to embrace the same underlying computing concepts.

Save yourself some time. Embrace the concepts.

Automate, Automate, Autonomy!

Thanks to pandemic-induced economic chaos, you’ve been forced to take a job on the quality assurance line at a factory that produces things.

The machine creates all kinds of random things, but your employer only sells a very specific subset of those things. All the things that don’t fit the profile have to be rejected, and melted down and fed back into the machine to make more things.

On your first day, you get training. (Oh, would that were true in software development!)

They stand you at the quality gate and start up the machine. All kinds of things come down the line at you. Your line manager tells you “Only let the green things through”. You grab all the things that aren’t green and throw them into the recycle bin. So far, so good.

“Only let the green round things through!” shouts your line manager. Okay, you think. Bit harder now. All non-green, non-round things go in the bin.

“Only let the green round small things through!” Now you’re really having to concentrate, a few green round small things end up in the bin, and a few non-green, non-round, non-small things get through.

“Only let the green round small things with Japanese writing on them through!” That’s a lot to process at the same time. Now your brain is struggling to cope. A bunch of blue and red things with Japanese writing on them get through. A bunch of square things get through. Your score has gone from 100% accurate to just 90%. Either someone will have to go through the boxes that have been packed and pick out all the rejects, or they’ll have to deal with 10% customer returns after they’ve been shipped.

“Only let the green round small things with Japanese writing on them that have beveled edges and a USB charging port on the opposite side to the writing and a power button in the middle of the writing and a picture of a horse  – not a donkey, mind, reject those ones! – and that glow in the dark through!”

Now it’s chaos. Almost every box shipped contains things that should have been thrown in the recycle bin. Almost every order gets returned. That’s just too much to process. Too many criteria.

We have several choices here:

1. Slow down the line so we can methodically examine every thing against our checklist, one criteria at a time.
2. Hire a whole bunch of people and give them one check each to do.
3. Reset customer expectations about the quality of the things they’re buying.
4. Automate the checking using cameras and robots and lasers and super-advanced A.I. so all those checks can be made at production speed to a high enough accuracy.

Number 4 is the option that potentially gives us the win-win of customer satisfaction and high productivity without the bigger payroll. It’s been the driving force behind the manufacturing revolutions in East Asia for the last 70 years: automate, automate, automate.

But it doesn’t come for free. High levels of automation require considerable ongoing investment in time, technology and training. In the UK, we’ve under-invested, becoming more and more inefficient and expensive while the quality of our output has declined. Shareholders want their return now. There’s no appetite for making improvements for the future.

There are obvious parallels in software development. Businesses want their software now. Most software organisations have little inclination to invest the time, technology and training required to reach the high levels of automation needed to achieve the coveted continuous delivery that would allow them to satisfy customer needs sooner, cheaper, and for longer.

The inescapable reality is that frictionless delivery demands an investment of 20-25% of your total software development budget. To put it more bluntly, everyone should be spending 1 day a week not on immediate customer requirements, but on making improvements in the delivery process that would mean meeting future customer requirements will be easier.

And so, for most teams, it never gets easier. The software just gets buggier, later and more expensive year after year.

What distinguishes those software teams who are getting it right from the rest? From observation, I’ve seen the same factor every time: autonomy. Teams will invest that 20-25% when it’s their choice. They’re tasked with delivering value, and allowed to figure out how best to do that. Nobody’s telling them how to do their jobs.

How did this blissful state come about? Again, from observation, those teams have autonomy because they took it. Freedom is rarely willingly given.

Now, I appreciate this is a whole can of worms. To take their autonomy, teams need to earn trust. The more trust a team has earned, the more likely they’ll be left alone. And this can be a chicken and egg kind of situation. To earn trust, the team has to reliably deliver. To reliably deliver, the team needs autonomy. This whole process must begin with a leap of faith on the business’s part. In other words, they have to give teams the benefit of the doubt long enough to see the results.

And here come the worms… Teams have to win over their customer from the start, before anything’s been delivered – before the customer’s had a chance to taste our pudding. This means that developers need to inspire enough confidence with their non-technical stakeholders – remember, this is a big money game – to reassure everyone that they’re in good hands. And we’re really, really bad at this.

The temptation is to over-promise, and set unrealistic expectations. This pretty much guarantees disappointment. The best way to inspire confidence is to have a good track record. No lawyer can guarantee to win your case. But a lawyer who won 9 of their last 10 cases is going to inspire more confidence than a lawyer who’s taking this as their first case promising you a win.

And we’re really, really bad at this, too – chiefly because software development teams are newly formed for that specific piece of work and don’t have a track record to speak of. Sure, individual developers may be known quantities, but in software, the unit of delivery is the team. I’ve watched teams of individually very strong developers fall flat on their collective arse.

And this is why I believe that this picture won’t change until organisations start to view teams as assets, and invest in them for a long-term pay-off as well as short-term delivery, 20/80. And, again, I don’t think this will be willingly given. So maybe we – as a profession – need to take the decision out of their hands.

It could all start with one big act of collective autonomy.