Process, accountability, chaos and the illusions of progress.
“We don’t need process, we need progress. We’re a small startup. Everything is slow, there are too many gates.” This lament was recently fast-pitched to me by a CEO client. I tried to explain why process is important, and that lack of process is most often replaced by semi-controlled chaos, the illusion of progress, health crashes, and risk profiles that tend to snowball into future time, only to melt during the most inconvenient season in a company’s life.
I’m extremely passionate about process. I got this way by taking too many risks, working too much, failing too much, jeopardizing my health for my career, but above all by developing an incredible respect and empathy for the expert builder. Process is about people; it’s about how we want to live our lives. By extension, if we mandate a process for others, then we are showing them how we think it best for them to live their lives.
Ethics
Process is about ethics. Who am I? What is my worth? What am I willing to compromise for my employer? What part of my life does my employer own? Do they have a right to ruin my health? Do they own my confidence? Do they own my weekends? Do I owe them my evenings as well?
As software builders, our codebases own us. Because our codebases and various infrastructure are running every second of every day, this presents many unique life challenges for builders. The company owns our codebase. Thus, by extension the company owns our life. The company’s internal processes then, are a declaration of ethics about employee productivity, health and well-being.
The inevitability of success
If a company is successful, people will need to work at an uncomfortable pace for extended periods of time, sometimes measured in years. People who work too much get tired. Tired people are less efficient. Tired people are more irritable. Tired people get sick easily. Tired people will operate according to a greater percentage of sub-optimal or cloudy assumptions.
One of the more common things I see is burnout. I’d say that 80+% of the people I’ve known in the startup game (myself included) have worked themselves to complete burnout at some point in their careers. By burnout I mean that after they slow down, exit, or a company folds, they cannot work in a normal productive capacity for 12+ months. Some even wonder if ‘burnout’ should be eligible for addition to the Diagnostic and Statistical Manual of Mental Disorders (DMS). Perhaps it’s a rite of passage, but overwork by any of its names is serious, life altering, existential stuff.
In light of real success and the pace that comes with it, the only thing that can ensure an ethical environment for employees is process — a system of reasonable constraints that dictate organizational workflow, scheduling, cadence, batch size (i.e. the amount of work-in-process at any given moment), automation, and failure mitigation and resolution. These considerations are at the core of things like Lean, Scrum, XP, Agile, Kanban. Any process needs to be co-signed from the top of an organization, and inspired by the bottom of an organization—inspired by the people who have the most to lose from an institutionalized lack of process or semi-productive chaos.
The fallacy of working harder vs. expert efficiencies
Expert builders are said to be up to 10X as productive as junior employees. This means that what takes a junior person 10 years to build, an expert will build that in a year. This idea should never be lost on anyone. The 10X estimate may even be 10X low when considering that the expert will make entirely different decisions that contribute to long-term stability; like building a house on sand instead of concrete.
But why are experts so efficient? They remove waste from ideas/systems in their mind, before speaking, before paper, before design, before code. Their brains are already trained to operate within an enforced system of skill-specific constraints, i.e. these people are highly structured, they follow internal processes. This internal system of constraints will produce cleaner, more realistic ideas that are already fact-checked against a career of wars against complexity.
This is why experts need process. Lack of process destroys efficiency by allowing them to become the organizational filter for whimsical direction and decision making.
*This is not to say that only builders are experts at things, and does not consider creativity. In software systems, builders are almost always the finite resource. Ideas and creativity are quite limitless, while time to code will always be limited by expert throughput.
Accountability of upstream resources
As alluded to above, one of the most redeeming qualities of any well-designed process is that upstream resources are bound and constrained by downstream requirements. In essence, the experts, or the most efficient minds in the system, expect inbound requests to match their level of distillation and critical thinking, and will often set up queueing and requirements for the upstream resources. This is done to help upstream languages speak a closer language to the expert. This allows the experts to keep efficiency extremely high, and will also create a ‘pull’ system, where someone on the team can pull a task at which time it’s already concepted, partially designed, distilled and vetted, i.e. ready. This starts to create accountability for the upstream resources.
The above scenario could be visualized with the concept-design-build pipelines we often seen in the software industry.
The cost of interruptions, multitasking and context switching
Another benefit of good process is the potential for a drastic reduction in interruptions, multitasking, and context switching for employees who require long periods of focus. More people are starting to report on the costs of interruptions at work like here, here and here.
Simply put, employees want to make their managers and executives happy. Technical employees are often unofficially tasked with helping people learn and understand the myriad possibilities. If each day a technical employee fields four requests, that means an acute interruption every two hours.
It has been suggested that after an interruption, it can take an average of ~25 minutes to regain focus. If we allow for four interruptions per day, that sets the potential for 100 minutes (almost two hours) of wasted time. This is catastrophic for efficiency and productivity. With many of our contemporary offices having open and social floor plans, only four interruptions per day might be optimistic, it could be much worse.
Imagine then, these distracted employees will need to work all night, every night, in order to find the quiet and predictable environment in which to complete their work.
Authority, choice, and outcomes
Process must be a top-down initiative. How many execs who talk Agile or Lean take the time read a well-known book about it? Or better yet how many hire a real agile/lean coach who has been actively building software the past five years, instead of just coaching? How many executives simply lack the discipline and foresight to accommodate real process?
The executives that promote zero-process, or fluid-process, or when-I-say-something-you-need-to-seem-like-you’re-immediately-responding-process, they will get the output associated with that type of environment. This progress will likely be short-lived and illusory. Furthermore, these executives will naturally attract people that do not yet understand how destructive long-term process chaos will be for those most affected, i.e. inexperienced people who are much less efficient.
It reminds me of the famous eastern quote: “So long as the bosses pretend to pay us, we will pretend to work.” Except in our version: “So long as the bosses pretend to know what they’re talking about, we will pretend we don’t need process to be successful in business and life.”
Does expecting process mean that we are entitled or lazy?
Absolutely not. While we have it pretty great in the software industry, there are no statutes or norms in the industry to protect people from brutal workloads and long hours. Only our elder statesman have the experience to know better. That’s largely where our contemporary processes came from. It turns out these processes are also very good for the customer and product.
Why can’t we just ‘freestyle’ it?
Well, you can and many people do. However, without a system of constraints on work-in-process, a team will run into the age old conditions of compounding risk profiles, feature scope creep, unpredictable ROI on people and code, and more.
Progress and risk
Is it progress today if it costs us double next week? This is the ultimate question. We are often under dire pressure to make hasty decisions that benefit us today, while incurring much debt far into the future. The debt from these pressures is impossible to quantify, but will come in code, morale, health, relationships and all measure of nuanced organizational cancer.
Process likens to the ‘tortoise and hare’ analogy. It’s a good one. However, with our contemporary tools and methodologies, the tortoise is getting quite a bit slimmer and faster. As such, it’s imperative that executives acknowledge the entire field of play before advocating for less process.
In the end, some teams can and have been successful without established process. Absolutely. I’m just not willing to work with them, because my life, career and craft are about much more than the mythical startup hustle.