Code Craft’s Value Proposition: More Throws Of The Dice

Evolutionary design is a term that’s used often, not just in software development. Evolution is a way of solving complex problems, typically with necessarily complex solutions (solutions that have many interconnected/interacting parts).

But that complexity doesn’t arise in a single step. Evolved designs start very simple, and then become complex over many, many iterations. Importantly, each iteration of the design is tested for it’s “fitness” – does it work in the environment in which it operates? Iterations that don’t work are rejected, iterations that work best are selected, and become the input to the next iteration.

We can think of evolution as being a search algorithm. It searches the space of all possible solutions for the one that is the best fit to the problem(s) the design has to solve.

It’s explained best perhaps in Richard Dawkins’ book The Blind Watchmaker. Dawkins wrote a computer simulation of a natural process of evolution, where 9 “genes” generated what he called “biomorphs”. The program would generate a family of biomorphs – 9 at a time – with a parent biomorph at the centre surrounded by 8 children whose “DNA” differed from the parent by a single gene. Selecting one of the children made it the parent of a new generation of biomorphs, with 8 children of their own.

biomorph
Biomorphs generated by the evolutionary simulation at http://www.emergentmind.com/biomorphs

You can find a recreation and more detailed explanation of the simulation here.

The 9 genes of the biomorphs define a universe of 118 billion possible unique designs. The evolutionary process is a walk through that universe, moving just one space in any direction – because just one gene is changing with each generation – with each iteration. From simple beginnings, complex forms can quickly arise.

A brute force search might enumerate all possible solutions, test each one for fitness, and select the best out of that entire universe of designs. With Dawkins’ biomorphs, this would mean testing 118 billion designs to find the best. And the odds of selecting the best design at random are 1:118,000,000,000. There may, of course, be many viable designs in the universe of all possible solutions. But the chances of finding one of them with a single random selection – a guess – are still very small.

For a living organism, that has many orders of magnitude more elements in their genetic code and therefore an effectively infinite solution space to search, brute force simply isn’t viable. And the chances of landing on a viable genetic code in a single step are effectively zero. Evolution solves problems not by brute force or by astronomically improbable chance, but by small, perfectly probable steps.

If we think of the genes as a language, then it’s not a huge leap conceptually to think of a programming language in the same way. A programming language defines the universe of all possible programs that could be written in that language. Again, the chances of landing on a viable working solution to a complex problem in a single step are effectively zero. This is why Big Design Up-Front doesn’t work very well – arguably at all – as a solution search algorithm. There is almost always a need to iterate the design.

Natural evolution has three key components that make it work as a search algorithm:

  • Reproduction – the creation of a new generation that has a virtually identical genetic code
  • Mutation – tiny variances in the genetic code with each new generation that make it different in some way to the parent (e.g., taller, faster, better vision)
  • Selection – a mechanism for selecting the best solutions based on some “fitness” function against which each new generation can be tested

The mutations from one generation to the next are necessarily small. A fitness function describes a fitness landscape that can be projected onto our theoretical solution space of all possible programs written in a language. Programs that differ in small ways are more likely to have very similar fitness than programs that are very different. Make one change to a working solution and, chances are, you’ve still got a working solution. Make 100 changes, and the risk of breaking things is much higher.

Evolutionary design works best when each iteration is almost identical to that last, with only one or two small changes. Teams practicing Continuous Delivery with a One-Feature-Per-Release policy, therefore, tend to arrive at better solutions than teams who schedule many changes in each release.

And within each release, there’s much more scope to test even smaller changes – micro-changes of the kind enacted in, say, refactoring, or in the micro-iterations of Test-Driven Development.

Which brings me neatly to the third component of evolutionary design: selection. In nature, the Big Bad World selects which genetic codes thrive and which are marked out for extinction. In software, we have other mechanisms.

Firstly, there’s our own version of the Big Bad World. This is the operating environment of the solution. A Point Of Sale system is ultimately selected or rejected through real use in real shops. An image manipulation program is selected or rejected by photographers and graphic designers (and computer programmers writing blog posts).

Real-world feedback from real-world use should never be underestimated as a form of testing. It’s the most valuable, most revealing, and most real form of testing.

Evolutionary design works better when we test our software in the real world more frequently. One production release a year is way too little feedback, way too late. One production release a week is far better.

Once we’ve established that the software is fit for purpose through customer testing – ideally in the real world – there are other kinds of testing we can do to help ensure the software stays working as we change it. A test suite can be thought of as a codified set of fitness functions for our solution.

One implication of the evolutionary design process is that, on average, more iterations will produce better solutions. And this means that faster iterations tend to arrive at a working solution sooner. Species with long life cycles – e.g., humans or elephants – evolve much slower than species with short life cycles like fruit flies and bacteria. (Indeed, they evolve so fast that it’s been observed happening in the lab.) This is why health organisations have to guard against new viruses every year, but nobody’s worried about new kinds of shark suddenly emerging.

For this reason, anything in our development process that slows down the iterations impedes our search for a working solution. One key factor in this is how long it takes to build and re-test the software as we make changes to it. Teams whose build + test process takes seconds tend to arrive at better solutions sooner than teams whose builds take hours.

More generally, the faster and more frictionless the delivery pipeline of a development team, the faster they can iterate and the sooner a viable solution evolves. Some teams invest heavily in Continuous Delivery, and get changes from a programmer’s mind into production in minutes. Many teams under-invest, and changes can take weeks or months to reach the real world where the most useful feedback is to be had.

Other factors that create delivery friction include the maintainability of the code itself. Although a system may be complex, it can still be built from simple, single-purpose, modular parts that can be changed much faster and more cheaply than complex spaghetti code.

And while many BDUF teams focus on “getting it right first time”, the reality we observe is that the odds of getting it right first time are vanishingly small, no matter how hard we try. I’ll take more iterations over a more detailed requirements specification any day.

When people exclaim of code craft “What’s the point of building it right if we’re building the wrong thing?”, they fail to grasp the real purpose of the technical practices that underpin Continuous Delivery like unit testing, TDD, refactoring and Continuous Integration. We do these things precisely because we want to increase the chances of building the right thing. The real requirements analysis happens when we observe how users get on with our solutions in the real world, and feed back those lessons into a new iteration. The sooner we get our code out there, the sooner can get that feedback. The faster we can iterate solutions, the sooner a viable solution can evolve. The longer we can sustain the iterations, the more throws of the dice we can give the customer.

That, ultimately, is the promise of good code craft: more throws of the dice.

 

Standards & Gatekeepers & Fitted Bathrooms

One thing I’ve learned from 10 years on Twitter is that whenever you dare to suggest that the software development profession should have minimum basic standards of competence, people will descend on you from a great height accusing you of being “elitist” and a “gatekeeper”.

Evil Jason wants to keep people out of software development. BAD JASON!

Well, okay: sure. I admit it. I want to keep people out of software development. Specifically, I want to keep people who can’t do the job out of software development. Mwuhahahahahaha etc.

That’s a very different proposition from suggesting that I want to stop people from becoming good, competent software developers, though. If you know me, then you know I’ve long advocated proper, long-term, in-depth paid software developer apprenticeships. I’ve advocated proper on-the-job training and mentoring. (Heck, it’s my entire business these days.) I’ve advocated schools and colleges and code clubs encouraging enthusiasts to build basic software development skills – because fundamentals are the building blocks of fun (or something pithy like that.)

I advocate every entry avenue into this profession except one – turning up claiming to be a software developer, without the basic competencies, and expecting to get paid a high salary for messing up someone’s IT.

If you can’t do the basic job yet, then you’re a trainee – an apprentice, if you prefer – software developer. And yes, that is gatekeeping. The gates to training should be wide open to anyone with aptitude. Money, social background, ethnicity, gender, sexual orientation, age or disabilities should be no barrier.

But…

I don’t believe the gates should be wide open to practicing as a software developer – unsupervised by experienced and competent mentors – on real software and systems with real end users and real consequences for the kinds of salaries we can earn – just for anyone who fancies that job title. I think we should have to earn it. I think I should have had to earn it when I started out. Crikey, the damage I probably did before I accidentally fell into a nest of experienced software engineers who fixed me…

Here’s the thing; when I was 23, I didn’t know that I wasn’t a competent software developer. I thought I was aces. Even though I’d never used version control, never written a unit test, never refactored code – not once – and thought that a 300-line function with nested IFs running 10 deep was super spiffy and jolly clever. I needed people to show me. I was lucky to find them, though I certainly didn’t seek them out.

And who the heck am I to say our profession should have gates, anyway? Nobody. I have no power over hiring anywhere. And, for sure, when I’ve been involved in the hiring process, bosses have ignored my advice many times. And many times, they’ve paid the price for letting someone who lacked basic dev skills loose on their production code. And a few times they’ve even admitted it afterwards.

But I’ve rarely said “Don’t hire that person”. Usually, I say “Train that person”. Most employers choose not to, of course. They want them ready-made and fully-formed. And, ideally, cheap. Someone else can train them. Hell, they can train themselves. And many of us do.

In that landscape, insisting on basic standards is difficult – because where do would-be professional developers go to get real-world experience, high-quality training and long-term mentoring? Would-be plumbers and would-be veterinarians and would-be hairdressers have well-defined routes from aspiration to profession. We’re still very much at the “If You Say You’re A Software Developer Then You’re A Software Developer” stage.

So that’s where we are right now. We can stay at that level, and things will never improve. Or we can do something about it. I maintain that long-term paid apprenticeships – leading to recognised qualifications – are the way to go. I maintain that on-the-job training and mentoring are essential. You can’t learn this job from books. You’ve got to see it and do it for real, and you need people around you who’ve done lots of it to guide you and set an example.

I maintain that apprenticeships and training and mentoring should be the norm for people entering the profession – be it straight of high school or after a degree or after decades of experience working in other industries or after raising children. This route should be open to all. But there should be a bar they need to jump at the end before being allowed to work unsupervised on production code. I wish I’d had that from the start. I should have had that.

And, yes, how unfair it is for someone who blundered into software development largely self-taught to look back and say “Young folk today must qualify first!” But there must have been a generation of self-taught physicians who one day declared “Okay, from now on, doctors have to qualify.” If not my generation, or your generation, then whose generation? We can’t keep kicking this can down the road forever.

As software “eats the world”, more and more people are going to enter the profession. More and more of our daily lives will be run by software, and the consequences of system failures and high costs of changing code will hurt society more and more. This problem isn’t going away.

I hope to Bod that the people coming to fit my bathroom next week don’t just say they’re builders and plumbers and electricians. I hope to Bod they did proper apprenticeships and had plenty of good training and mentoring. I hope to Bod that their professions have basic standards of competence.

And I hope to Bod that those standards are enforced by… gatekeepers.

Digital Is A Process, Not A Project

One sentiment I’m increasingly hearing on social media is how phrases like #NoEstimates and #NoProjects scare executives who require predictability to budget for digital investments.

I think this is telling. How do executives budget for HR or finance or facilities teams? These are typically viewed as core functions within a business – an ongoing cost to keep the lights on, so to speak.

Software and systems development, on the other hand, is usually seen as a capital investment, like building new offices or installing new plant. It’s presumed that at some point these “projects” will be “done”, and the teams who do them aren’t perceived as being core to the running of the business. After your new offices are completed, you don’t keep the builders on for more office building. They are “done”.

But modern software development just isn’t like that. We’re never really done. We’re continually learning and systems are continually evolving as we do. It’s an ongoing process of innovation and adaptation, not a one-off investment. And the teams doing that work are most certainly core to your business, every bit as much as the accountants and the sales people and anyone else keeping the lights on.

I can’t help wondering if what executives really fear is acknowledging that reality and embracing digital as a core part of their business that is never going to go away.

Overcoming Solution Bias

Just a short post this morning about a phenomenon I’ve seen many times in software development – which, for want of a better name, I’m calling solution bias.

It’s the tendency of developers, once they’ve settled on a solution to a problem, to refuse to let go of it – regardless of what facts may come to light that suggest it’s the wrong solution.

I’ve even watched teams argue with their customer to try to get them to change their requirements to fit a solution design the team have come up with. It seems once we have a solution in our heads (or in a Git repository) we can become so invested in it that – to borrow a metaphor – everything looks like a nail.

The damage this can do is obvious. Remember your backlog? That’s a solution design. And once a backlog’s been established, it has a kind of inertia that makes it unlikely to change much. We may fiddle at the edges, but once the blueprints have been drawn up, they don’t change significantly. It’s vanishingly rare to see teams throw their designs away and start afresh, even when it’s screamingly obvious that what they’re building isn’t going to work.

I think this is just human nature: when the facts don’t fit the theory, our inclination is to change the facts and not the theory. That’s why we have the scientific method: because humans are deeply flawed in this kind of way.

In software development, it’s important – if we want to avoid solution bias – to first accept that it exists, and that our approach must actively take steps to counteract it.

Here’s what I’ve seen work:

  • Testable Goals – sounds obvious, but it still amazes me how many teams have no goals they’re working towards other than “deliver on the plan”. A much more objective picture of whether the plan actually works can help enormously, especially when it’s put front-and-centre in all the team’s activities. Try something. Test it against the goal. See if it really works. Adapt if it doesn’t.
  • Multiple Designs – teams get especially invested in a solution design when it’s the only one they’ve got. Early development of candidate solutions should explore multiple design avenues, tested against the customer’s goals, and selected for extinction if they don’t measure up. Evolutionary design requires sufficiently diverse populations of possible solutions.
  • Small, Frequent Releases – a team that’s invested a year in a solution is going to resist that solution being rejected with far more energy than a team who invested a week in it. If we accept that an evolutionary design process is going to have failed experiments, we should seek to keep those experiments short and cheap.
  • Discourage Over-Specialisation – solution architectures can define professional territory. If the best solution is a browser-based application, that can be good news for JavaScript folks, but bad news for C++ developers. I often see teams try to steer the solution in a direction that favours their skill sets over others. This is understandable, of course. But when the solution to sorting a list of surnames is to write them into a database and use SQL because that’s what the developers know how to do, it can lead to some pretty inappropriate architectures. Much better, I’ve found, to invest in bringing teams up to speed on whatever technology will work best. If it needs to be done in JavaScript, give the Java folks a couple of weeks to learn enough JavaScript to make them productive. Don’t put developers in a position where the choice of solution architecture threatens their job.
  • Provide Safety – I can’t help feeling that a good deal of solution bias is the result of fear. Fear of failure.  Fear of blame. Fear of being sidelined. Fear of losing your job. If we accept that the design process is going to involve failed experiments, and engineer the process so that teams fail fast and fail cheaply – with no personal or professional ramifications when they do – then we can get on with the business of trying shit and seeing if it works. I’ve long felt that confidence isn’t being sure you’ll succeed, it’s not being afraid to fail. Reassure teams that failure is part of the process. We expect it. We know that – especially early on in the process of exploring the solution space – candidate solutions will get rejected. Importantly: the solutions get rejected, not the people who designed them.

As we learn from each experiment, we’ll hopefully converge on the likeliest candidate solution, and the whole team will be drawn in to building on that, picking up whatever technical skills are required as they do. At the end, we may not also deliver a good working solution, but a stronger team of people who have grown through this process.

 

Constraints Can Be A Good Thing

Sometimes we get it too easy.

I often think of the experiences I’ve had recording music using software plug-ins instead of real hardware.

Time was that recording music was hard. My home recording set-up was a Tascam 8-track tape recorder, three guitars, a choice of two amplifiers, a bunch of guitar pedals, a microphone for recording guitars (you can’t just plug your amp directly into a tape recorder, you have to mic it) and occasional bad vocals, a bass guitar, a drum machine and a Roland synth module controlled by MIDI from my 486 PC.

Recording to tape takes patience. And, while 8 tracks might sound a lot, it’s actually very limiting. One track is vocals. For rock and metal, you record rhythm guitars twice for left and right to get a stereo spread. Bass guitar makes four tracks. Lead guitar makes five and six, if you want harmonies. Drums take tracks seven and eight for stereo. Then you “bounce” the guitars down to two tracks – basically output a stereo guitar mix and record L and R on just two tracks – to make space for stereo keyboards at the end.

Typically I needed several takes for each guitar, bass and vocal part. And between each take, I had to rewind the tape to exact point where I needed to “punch in” again. If, during mixing, I decide I didn’t like the guitar sound, I had to record it all over again.

Fast forward to the 2010s, and I’ve been doing it all using software. The guitars are real, but the amps are digital – often recorded using plug-ins that simulate guitar amps, speaker cabinets and microphones – and I can choose from hundreds of amp, cabinet and microphone models. If I don’t lie the guitar sound, I can just change it. No need to re-record that guitar part.

Th3-Main

And I can choose from dozens of software synthesizers, offering thousands upon thousands of potential sounds. I can have as many virtual Minimoogs or Roland Jupiter 8s as I like. My i7 laptop can run 10-20 of them simultaneously.

mini-v-image

My drums are now created using a powerful multi-sampled virtual instrument that allows me to choose from dozens of different sampled kits, or create my own custom kits, and I can tweak the recording parameters and apply almost limitless effects like compression and EQ to my heart’s content. And, again, if I don’t like the drum sound, I don’t need to wind back a tape and record them all over again.

sd2

My Digital Audio Workstation lets me record as many tracks – mono and stereo – as my computer can handle (typically, more than 30 in a final mix), and I can route and re-route the audio as many times as I like.

Because I use software plug-ins for effects like echo/delay, reverb, chorus, EQ, compression and more, I have almost limitless production options for every track.

And, most mind-bending of all, I can record take after take after with no rewinding and the audio quality never degrades.

Digital is aces!

Except… About a year ago I invested in my first hardware synthesizer for a very long time. It’s a Korg Minilogue – a proper analog synthesizer (the first analog synth I’ve owned – and to operate it you have to press buttons and twiddle knobs). Unlike my Roland digital synth, it’s not “multi-timbral”. It makes one sound at a time, and can play up to 4 notes at a time. My Roland JV-2080 could make 16 different sounds a time, and play up to 64 notes simultaneously.

minilogue

Compared to the software – and the digital synth hardware – the Minilogue is very limiting. It’s significantly harder recording music with the Minilogue than with a software synthesizer.

But when I listen to the first quick demo track I created using the real analog synth and tracks I created using software synths, I can’t help noticing that they have a very different quality.

Those constraints led me to something new – something simpler, into which I had to put more thought about the design of each sound, about the structure of the music, about the real hardware guitar effects I used on each “patch”.

I’m not necessarily saying the end results are better than using software synths. I’m saying that the constraints of the Korg Minilogue led me down a different path. It changed the music.

I’ve been inspired to experiment more with hardware instruments, maybe even revisit recording guitars through mic’d up tube amps again and see how that changes the music, too. (And this is a great time to do that, as low-priced analog synths are making a comeback right now.)

All this got me to thinking about the tools I use to create software. It sometimes feels like we’ve maybe got a bit too much choice, made things a little too easy.  And all that choice and ease had led us to more complex products: multiple tech stacks, written in multiple languages, with lots of external dependencies because our package managers make that so easy now. etc etc.

Would going back to simple, limited editors, homogenous platforms, limited hardware and so on change the software we tend to create?

I think it may be worth exploring.

 

Code Craft is Seat Belts for Programmers

Every so often we all get a good laugh when some unfortunate new hire or intern at a major tech company accidentally “deletes Google” on their first day. It’s easy to snigger (because, of course, none of us has ever messed up like that).

The fact is, though, that pointing and laughing when tech professionals make mistakes doesn’t stop mistakes getting made. It can also breed a toxic work culture, where people learn to avoid mistakes by not taking risks. Not taking risks is anathema to innovation, where – by definition – we’re trying stuff we’ve never done before. Want to stifle innovation where you work? Pointing and laughing is a great way to get there.

One of the things I like most about code craft is how it can promote a culture of safety to try new things and take risks.

A suite of good, fast-running unit tests, for example, makes it easier to spot our boos-boos sooner, so we can un-boo-boo them quickly and without attracting attention.

Continuous Integration offers a level of un-doability that makes it easier and safer to experiment, safe in the knowledge that if we mess it up, we can get back to the last version that worked with a simple hard reset.

The micro-cycles of refactoring mean we never stray far from the path of working code. Combine that with fast-running tests and frequent commits, and ambitious and improbable re-architecting of – say – legacy code becomes a sequence of mundane, undo-able and safe micro-rewrites.

And I can’t help feeling – when I see some poor sod getting Twitter Heat for screwing up a system in production – that it was the deficiency in their delivery pipeline that allowed it to happen that was really at fault. The organisation messed up.

Software development’s a learning process. Think about when young children – or people of any age – first learn to use a computer. The fear of “breaking it” often discourages them from trying new things, and this hampers their learning process. never underestimate just how much great innovation happens when someone says “I wonder what happens if I do this…” Remove that fear by fostering a culture of “what if…?” shielded by systems that forgive.

Code craft is seat belts for programmers.

Wheels Within Wheels Within Wheels

Much is made of the cycles-within-cycles of Test-Driven Development.

At the core, we do micro-iterations with small, single-question unit tests to drive out the details of our internal design.

Surrounding those micro-cycles are the feedback loops provided by customer tests, which may require us to pass multiple unit tests to complete end-to-end.

User stories typically come with multiple customer tests – happy paths and edge cases – providing us with bigger cycles around our customer test feedback loops.

Orbiting those are release loops, where we bundle a set of user stories and await feedback from end users in the real world (or a simulated approximation of it for test purposes).

What’s not discussed, though, are the test criteria for those release loops. If we already established through customer testing that we delivered what we agreed we would i that release, what’s left to test for?

The minority of us who practice development driven by business goals may know the answer: we test to see if what we released achieves the goal(s) of that release.

feedbackloops

This is the outer feedback loop – the strategic feedback loop – that most dev teams are missing. if we’re creating software with a purpose, it stands to reason that at some point we must test for its fitness for that purpose. Does it do the job it was designed to do?

When explaining strategic feedback loops, I often use the example of a business start-up who deliver parcels throughout the London area. They have a fleet of delivery vans that go out every day across the city, delivering to a list of addresses parcels that were received into their depot overnight.

Delivery costs form the bulk of their overheads. They rent the vans. They charge them up with electrical power (it’s an all-electric fleet – green FTW!) They pay the drivers. And so on. It all adds up.

Business is good, and their customer base is growing rapidly. Do they rent more vans? Do they hire more drivers? Do they do longer routes, with longer driver hours, more recharging return-to-base trips, and higher energy bills? Or could the same number of drivers, in the same number of vans, deliver more parcels with the same mileage as before? Could their deliveries be better optimised?

Someone analyses the routes drivers have been taking, and theorises that they could have delivered the same parcels in less time driving less miles. They believe it could be done 35% more efficiently just by optimising the routes.

Importantly, using historical delivery and route data, they show on paper that an algorithm they have in mind would have saved 37% on miles and driver-hours. I, for one, would think twice about setting out to build a software system that implements unproven logic.

But the on-paper execution of it takes far too long. So they hatch a plan for a software system that selects the optimum delivery routes every day using this algorithm.

Taking route optimisation as the headline goal, the developers produce a first release in 2 weeks that takes in delivery addresses from an existing data source and – as command line utility initially – produces optimised routes in simple text files to be emailed to the drivers’ smartphones. It’s not pretty, and not a long-term solution by any means. But the core logic is in there, it’s been thoroughly unit and customer tested, and it seems to work.

While the software developers move on to thinking about the system could be made more user-friendly with a graphical UI (e.g., a smartphone app), the team – which includes the customer – monitor deliveries for the next couple of weeks very closely. How long are the routes taking? How many miles are vans driving? How much energy is being used on each route? How many recharging pit-stops are drivers making each day?

This is the strategic feedback loop: have we solved the problem? If we haven’t, we need to go around again and tweak the solution (or maybe even scrap it and try something else, if we’re so far off the target, we see no value in continuing down that avenue).

This is my definition of “done”; we keep iterating until we hit the target, learning lessons with each release and getting it progressively less wrong.

Then we move on to the next business goal.