In-Process, Cross-Process & Full-Stack Tests

Time for a quick clarification. (If you’ve been on a Codemanship course, you may have already heard this.)

Ask twelve developers for their definitions of “unit test”, “integration test” and “system test” and you’ll likely get twelve different answers. I feel – especially for training purposes – that I need to clarify what I mean by them.

Unit Test – when I say “unit test”, what I mean is a test that executes without any external dependencies. I can go further to qualify what I mean by an “external dependency”; that’s when code is executed in a separate memory address space – a separate process – to the test code. This is typically for speed, so we can test our logic quickly without hitting databases or file systems or web services and so on. It also helps separate concerns more cleanly, as “unit testable” code has to usually be designed in such a way to make external dependencies easily swappable (e.g., by dependency injection).

Integration Test – a test that executes code running in separate memory address spaces (e.g., separate Windows services, or SQL running on a DBMS). It’s increasingly common to find developers reusing their unit tests with different set-ups (replace a database stub with the real database connection, for example). The logic of the test is the same, but the set-up involves external dependencies. This allows us to test that our core logic still works when it’s interacting with external processes. (i.e., it tests the contracts at both sides).

System Test – executes code end-to-end, across the entire tech stack, including all external dependencies like databases, files, web services, the OS and even the hardware. (I’ve seen more than one C++ app blow a fuse because it was deployed on hardware that the code wasn’t compiled to run on, for example.) This allows us to test our system’s configuration, and ideally should be done in an environment as close to the real things as possible.

It might be clearer if I called them In-Process, Cross-Process and Full-Stack tests.


Standards & Gatekeepers & Fitted Bathrooms

One thing I’ve learned from 10 years on Twitter is that whenever you dare to suggest that the software development profession should have minimum basic standards of competence, people will descend on you from a great height accusing you of being “elitist” and a “gatekeeper”.

Evil Jason wants to keep people out of software development. BAD JASON!

Well, okay: sure. I admit it. I want to keep people out of software development. Specifically, I want to keep people who can’t do the job out of software development. Mwuhahahahahaha etc.

That’s a very different proposition from suggesting that I want to stop people from becoming good, competent software developers, though. If you know me, then you know I’ve long advocated proper, long-term, in-depth paid software developer apprenticeships. I’ve advocated proper on-the-job training and mentoring. (Heck, it’s my entire business these days.) I’ve advocated schools and colleges and code clubs encouraging enthusiasts to build basic software development skills – because fundamentals are the building blocks of fun (or something pithy like that.)

I advocate every entry avenue into this profession except one – turning up claiming to be a software developer, without the basic competencies, and expecting to get paid a high salary for messing up someone’s IT.

If you can’t do the basic job yet, then you’re a trainee – an apprentice, if you prefer – software developer. And yes, that is gatekeeping. The gates to training should be wide open to anyone with aptitude. Money, social background, ethnicity, gender, sexual orientation, age or disabilities should be no barrier.


I don’t believe the gates should be wide open to practicing as a software developer – unsupervised by experienced and competent mentors – on real software and systems with real end users and real consequences for the kinds of salaries we can earn – just for anyone who fancies that job title. I think we should have to earn it. I think I should have had to earn it when I started out. Crikey, the damage I probably did before I accidentally fell into a nest of experienced software engineers who fixed me…

Here’s the thing; when I was 23, I didn’t know that I wasn’t a competent software developer. I thought I was aces. Even though I’d never used version control, never written a unit test, never refactored code – not once – and thought that a 300-line function with nested IFs running 10 deep was super spiffy and jolly clever. I needed people to show me. I was lucky to find them, though I certainly didn’t seek them out.

And who the heck am I to say our profession should have gates, anyway? Nobody. I have no power over hiring anywhere. And, for sure, when I’ve been involved in the hiring process, bosses have ignored my advice many times. And many times, they’ve paid the price for letting someone who lacked basic dev skills loose on their production code. And a few times they’ve even admitted it afterwards.

But I’ve rarely said “Don’t hire that person”. Usually, I say “Train that person”. Most employers choose not to, of course. They want them ready-made and fully-formed. And, ideally, cheap. Someone else can train them. Hell, they can train themselves. And many of us do.

In that landscape, insisting on basic standards is difficult – because where do would-be professional developers go to get real-world experience, high-quality training and long-term mentoring? Would-be plumbers and would-be veterinarians and would-be hairdressers have well-defined routes from aspiration to profession. We’re still very much at the “If You Say You’re A Software Developer Then You’re A Software Developer” stage.

So that’s where we are right now. We can stay at that level, and things will never improve. Or we can do something about it. I maintain that long-term paid apprenticeships – leading to recognised qualifications – are the way to go. I maintain that on-the-job training and mentoring are essential. You can’t learn this job from books. You’ve got to see it and do it for real, and you need people around you who’ve done lots of it to guide you and set an example.

I maintain that apprenticeships and training and mentoring should be the norm for people entering the profession – be it straight of high school or after a degree or after decades of experience working in other industries or after raising children. This route should be open to all. But there should be a bar they need to jump at the end before being allowed to work unsupervised on production code. I wish I’d had that from the start. I should have had that.

And, yes, how unfair it is for someone who blundered into software development largely self-taught to look back and say “Young folk today must qualify first!” But there must have been a generation of self-taught physicians who one day declared “Okay, from now on, doctors have to qualify.” If not my generation, or your generation, then whose generation? We can’t keep kicking this can down the road forever.

As software “eats the world”, more and more people are going to enter the profession. More and more of our daily lives will be run by software, and the consequences of system failures and high costs of changing code will hurt society more and more. This problem isn’t going away.

I hope to Bod that the people coming to fit my bathroom next week don’t just say they’re builders and plumbers and electricians. I hope to Bod they did proper apprenticeships and had plenty of good training and mentoring. I hope to Bod that their professions have basic standards of competence.

And I hope to Bod that those standards are enforced by… gatekeepers.

Digital Is A Process, Not A Project

One sentiment I’m increasingly hearing on social media is how phrases like #NoEstimates and #NoProjects scare executives who require predictability to budget for digital investments.

I think this is telling. How do executives budget for HR or finance or facilities teams? These are typically viewed as core functions within a business – an ongoing cost to keep the lights on, so to speak.

Software and systems development, on the other hand, is usually seen as a capital investment, like building new offices or installing new plant. It’s presumed that at some point these “projects” will be “done”, and the teams who do them aren’t perceived as being core to the running of the business. After your new offices are completed, you don’t keep the builders on for more office building. They are “done”.

But modern software development just isn’t like that. We’re never really done. We’re continually learning and systems are continually evolving as we do. It’s an ongoing process of innovation and adaptation, not a one-off investment. And the teams doing that work are most certainly core to your business, every bit as much as the accountants and the sales people and anyone else keeping the lights on.

I can’t help wondering if what executives really fear is acknowledging that reality and embracing digital as a core part of their business that is never going to go away.