Hopefully I caught your attention with the controversial (and admittedly hyperbolic) title. Good. Let me restate it in a slightly more elegant and less clickbaity form:
In general, you can either have software delivered on time or good software, but you can’t have both*
* except on counted occasions in established, highly-performant teams
The past few months I’ve been reflecting about why delivering good software does not play well with estimates and planning in general. During my career I’ve seen projects ran in many different ways (waterfall, true agile, agile-fall) and all of them shared a thing in common: no matter what kind of project we were working on, if we “did it right” (i.e. we didn’t take hacky shortcuts that would give us nightmares), we always ran past the deadline.
On the other hand, whenever a project was delivered “on time”, either its scope was reduced along the way or we had to cut so many corners that the tech debt accumulated during implementation all but guaranteed that the project would have to be re-written shortly after launch, which made me wonder: Is a project actually “on time” even when the end result is an uglier, less maintainable, bug-ridden and, plainly speaking, suckier version of what we set out to do initially?
I’ve also worked on projects that were not deadline-driven. There were “deadlines”, sure, but far from them being set in stone, it was understood that deadlines were flexible and that quality trumped on-time delivery. Those were the projects which produced the best software, had the happiest developers and were the most successful overall amongst the ones I’ve worked on. But we all know those projects are a rare find, or I wouldn’t be writing this.
So why is it so difficult to plan and deliver good software given a static timeline? I think it has a lot to do with creativity, craftsmanship and unpredictability.
The creative side of coding
I am of the view that software development is an inherently creative undertaking. Sure, there are developers who perform repetitive, trivial tasks, but they will have a job only as long as no one figures out how to automate theirs — not a great place to be in, and not the focus of this essay.
For me, there’s something special about the act of creating something new and looking for original solutions to challenges that calls me to software development, and I don’t think I am the only one. In fact, I believe creativity is the main reason why developers enjoy their work. In my experience, whenever I’ve worked in environments where there were strict and unchangeable “set of practices” (think tech stack, processes, guidelines, etc.) — i.e. the least creative freedom I had — , the less engaged I felt. “After all, if they already have it all figured out, why do they need me?”, I wondered. On the other hand, I felt much more fulfilled, excited and productive at work when there were few top-down directives, I was given creative leeway and was trusted to own up to my technical decisions.
It’s important to note that more creative freedom begets more trial and error to reach the solution — which is fine. Some people think that you can just know the perfect solution ex-ante (i.e. in advance) before writing a single line of code. I contend instead that, for creative activities, the process of discovering a solution to a given problem (not only software) is a process of tinkering: you can’t have perfect knowledge in advance, but instead you learn by doing, iteratively trying new things and keeping what works, and by refining your solution (and possibly shipping it to your customers if you are lean/agile) until you are satisfied with it.
Just think of how many times you have spent time designing a feature on paper, just to have to change the design completely once you actually start implementing it. There will always be unknown unknowns and the only way to uncover and deal with them is ex-post by actually coding your solution, instead of spending a lot of time theorizing and pretending we have perfect information in advance. This process of tinkering does not play very well with estimates.
Further, as it’s the case with other creative endeavors, coding benefits from Strategic Procrastination — A term coined by Adam Grant which asserts that often you can’t be creative on demand but instead creativity comes as a “push event” from a process running in the background of your mind:
“ Employees who procrastinated regularly spent more time engaging in divergent thinking and were rated as significantly more creative by their supervisors. Procrastination didn’t always fuel creativity: if the employees weren’t intrinsically motivated to solve a major problem, stalling just set them behind. But when they were passionate about coming up with new ideas, putting off the task led them to more creative solutions.”
— Grant, Adam. “Originals: How Non-Conformists Move the World.”
Again, not great news for the central planners out there trying to plan and measure every minute of software development projects.
The craft of building software
The best developers I know are crafts[wo]men. Craftsmanship is the hallmark of good software: you don’t only build something that works, but you build it in the best way possible — It’s relatively easy to build something that works, but very hard to build something that works and stands the test of time.
As a crafts[wo]man, the quality of your work defines you. You prime quality over quantity because you don’t want to be known to write crappy software, even if you could satisfy your Product Manager by just making your software look good “on the outside” but full of nasty surprises on the inside — what I call Kinder Surprise Development. You know that investing time in writing good software pays off and you push back on pressures to “execute faster” as you know the more shortcuts you take now, the less life expectancy your code will have and the more problems it will cause down the road.
My eyes bleed
Craftsmanship is about caring: caring to do a great job, caring about the ones who will maintain your code after you, caring about the consumers of your software having an easy time using it, caring about your team mates, and so on. You care because you are not an asshole and because you know that it’s the right thing to do if you want the project to be successful.
In short, good engineers have the difficult task of caring about quality — something that has the long-term consequences — in a world driven by short-term execution.
In practice, that translates into things like:
- Finding the right mix between encapsulation, extensibility, scalability, etc. — again, you will need iterative trial and error, no one will build the best solution on the first try.
- Taking time to refactor when you stumble upon a shamefully bad part of the code
- Writing good, complete tests — maybe even doing TDD
- Code pairing with colleagues
Needless to say, it’s impossible to plan for all these in advance, so this doesn’t help you hit any deadlines either.
Your predictions are wrong
Even with clear requirements — and it seems that they never are — it is still almost impossible to know how long something will take, because we’ve never done it before. If we had done it before, we’d just give it to you.
— Ron Jeffries, The NoEstimates Movement
Software Projects are Complex Systems: They are made possible by human beings and thus are affected by interpersonal interactions, motivation, communication issues and human psychology in general — all those pretty difficult to model and quantify on a spreadsheet, if you ask me. This makes software projects very difficult to model (and hence predict). Nassim Taleb explains it best in his book Antifragile:
“Complex systems are full of interdependencies — hard to detect — and nonlinear responses. “Nonlinear” means that when you double the dose of, say, a medication, or when you double the number of employees in a factory, you don’t get twice the initial effect, but rather a lot more or a lot less. Two weekends in Philadelphia are not twice as pleasant as a single one — I’ve tried.“
— Nassim Nicholas Taleb, Antifragile
What’s worse, given that time can’t be negative, any unplanned “surprises” will very likely increase the completion time instead of reducing it as there is an asymmetry in the outcomes:
“Just as time cannot be negative, a three-month project cannot be completed in zero or negative time. So, on a timeline going left to right, errors add to the right end, not the left end of it. If uncertainty were linear we would observe some projects completed extremely early (just as we would arrive sometimes very early, sometimes very late). But this is not the case.“
— Nassim Nicholas Taleb, Antifragile
This is bad news, as uncertainty is certain and even small errors in estimating individual tasks will compound exponentially at the project level. All this is assuming the best case scenario where deadlines have been set after some careful estimation by the developers, but reality is even more absurd: most of the time “The Business” sets the deadlines arbitrarily and only then Engineering comes up with a plan to deliver the requirements by those randomly chosen points in time, in a flagrant case of start building the house from the roof and then putting the cart before the horse on top.
Good question
Here are some examples to illustrate some of the nonlinearities and feedback loops of writing software:
- That time when you assumed that the API you need to communicate with accepts
accountIdbut it actually only acceptsmemberId. You had to add 4 days to your estimate to refactor the API code — which in turn had to go through a separate review process which added 2 additional days. - A task estimated to 2 days ends up taking a week because during the review process one of your team mates pushes you (and rightly so) to refactor and improve a very ugly part of the code that was long overdue.
- That one-point task where you had to implement some new functionality that happened to require an update to a dependency which ended up taking 4 days as the dependency update triggered a chain reaction of dependencies needing to be updated and that caused a bunch of errors in the build.
Are we screwed?
We keep playing the estimation and planning game out of inertia to reassure ourselves that we know what we are doing, but the truth is that we don’t, and that software projects are empirically unpredictable. Thus, in my opinion, we’d be better off just focusing more on doing rather than planning — #NoEstimates anyone? But of course this will not fly in many organizations: “We can’t just let the engineers run the show, unchecked, there needs to be some accountability!”. I get it.
You don’t say?
What to do then? I think it comes down to bridging the gap between the world of the spreadsheet and the world of the IDE in a way that allows maximum creativity, flexibility and craftsmanship to Engineering while at the same time carefully managing the commitments made to, and the expectations from the project stakeholders. The Engineering Manager is in the best position to bridge and manage this gap and act like a buffer between the two worlds. It’s not an easy job, but a necessary one. Aaron Longwell explains it well in his article:
“Because engineering managers inhabit the border between the business and the technical, they are the ones who need to resolve the tension between estimate and reality. It’s a bit like being the wishbone being pulled at both ends; either side can snap. When the business side ‘wins’, the developers end up in a death march. When development concerns outweigh business ones, you end up blowing the budget and deadline. Either way you’re broken. Successful software managers find ways to be flexible_; to bend without breaking and to resolve the tension gradually. Servant leadership can be a guide to finding this flexibility.”_
— Aaron Longwell, Why Software Development Requires Servant Leaders
It’s also very important to have a strong relationship based on trust between Product and Engineering. If there is trust, you will be able to negotiate timelines honestly and openly with good faith. If you have previously proven that your team delivers good software, there should be enough “social capital” for the stakeholders to trust that if you push back on a given schedule is for good reasons and for the common good.
Another “trick” that I’ve personally used as a Manager is to avoid giving concrete dates, as they will inevitably be used as hard deadlines. Diffuse time ranges work best, something like “between 3 and 5 weeks”. Then, the closer the diffuse deadline gets, the more definition you add to it: “Between April and May” turns to “Between April 15th and May 3rd” at the beginning of April, which turns to “The week of April 20th” around April 10th, and so on. This way you are openly truthful with your peers while allowing your team the flexibility they will need to solve the inevitable unforeseen problems that will arise.
Lastly, remember that it’s up to the developers to stand up for the technical quality of the product delivered, not to the other stakeholders. It’s only natural that tensions will arise between organizations with seemingly divergent incentives on the surface. The key is to point out that, in fact, you are all (presumably) aligned with the same goal: Deliver good quality software to the customers in the fastest way possible — the gist that only good developers seem to understand is that you should avoid taking the deceivingly “fast and easy” way as it’s actually the slowest one in the long run.
In conclusion, it’s a solvable yet complex (and common) problem. I’d say that if you feel like your manager or your organization doesn’t set you up to build good software, it’s up to you to speak up and try to change it or, if that fails, find somewhere else to work.