How Engineering Managers Should Approach Planning

Throughout my time managing teams, I have evolved my approach to planning, and have been appreciated for being good at it. While I have read and learned intentionally to improve, my approach to integrating these learnings has been intuitive. Consequently, I have not devoted much time to spelling out my thought process, even to myself. However, interviewing people for engineering management roles in my organization has compelled me to crystallize my ideas about the various facets of the job, of which planning is a significant part. This is my attempt at outlining how to be effective at it.

Content overview

• Who is this for?

  • Assessment

• Planning - What does it mean?

  • What makes a plan good?

• Why plan? In other words, why not wing it and learn from mistakes?

• Myths

• Key aspects of planning

  • Clarity of goals or desired outcomes
  • Degree of alignment among stakeholders
  • Framework for work estimation
  • Measuring progress
  • Smoothness of execution
  • Timeliness and consistency of delivery
  • Adaptability
  • Communication

• Parting thoughts

Who is this for?

While the guidelines here apply to planning at every level, this is specifically tailored to quarterly, monthly, and bi-weekly planning - the kind that engineering managers are routinely tasked with. As such, it’s most useful for engineering managers who have a few years under their belt but want to adopt a more methodical approach to planning and those who are starting their journey in the management track and looking to navigate it effectively. This should also come in handy to senior individual contributors who are involved in the planning process.

Since my experience has primarily been in managing product initiatives and product engineering teams, that’s what most examples will be based on. The principles should nonetheless be relevant to other kinds of engineering work, like platform, or infrastructure engineering. The actors and concepts would largely remain the same. For example, customers to you might be other engineering teams, the product owner might be an architect, and user experience might mean developer experience.

Assessment

Think about the following scenario to assess your natural tendencies, and keep that in mind as you read the article. It’ll inform your approach to planning, and your focus areas.

Say you sign up for your first marathon. You needed a challenge, something to kick you out of your slumber and comfort zone. It’s four months away and you signed up on a whim, during a conversation with your friends. What do you do after? Do you show up for the race unprepared? Do you not show up - you signed up on an impulse, after all? Or, do you plan and start training, and give it your best shot? If you’re the kind that plans and trains, how would you go about it?

The answer depends on a lot of factors, which I don’t want to delve into, but out of the three ways you could go about it, there’s exactly one way of maximizing your chances of success.

Planning - What does it mean?

Planning can simply be defined as the process of outlining how to approach a task before executing it, to achieve a specific goal.

What makes a plan good?

Simply put, a plan is good if it increases your chances of achieving the desired outcomes. As is evident from the definition above, there are two main ingredients to defining a plan - the how, which incorporates past experiences, an awareness of the current constraints and limitations, and the what, a clear understanding of the goals.

Why plan? In other words, why not wing it and learn from mistakes?

Planning helps you increase the odds of achieving the goals that you set out. Effective planning helps organizations consistently deliver value to customers and make a positive impact. It minimizes surprises for team members, stakeholders, and leadership and helps navigate potential paths and mitigate risks. Crafting a plan enables you to proactively uncover unknowns and complexities. Moreover, it streamlines the connection between the tactical details of the work and the overarching strategy, goals, and vision of the team or the broader organization. In essence, effective planning underpins a deliberate and successful journey toward organizational goals and can be one of the most effective tools in setting your team or organization apart.

To speak to the second part of the question, learning from mistakes is key regardless. Planning reduces the time and resources required to make progress by eliminating obvious bad choices proactively.

Myths

• Planning slows you down - This arises from misconceptions about the time required to plan and from a myopic view of velocity. Planning unlocks benefits in terms of streamlined execution and long-term value
• Planning kills innovation - Contrary to this, effective planning can foster a culture of innovation by providing a structured framework for creativity and innovation to thrive and aligning them with the strategic goals of the organization
• Planning hinders agility - This myth is unfortunately prevalent due to misguided notions of agility. I’ve worked in organizations where a lack of clarity and hence the ability to plan meant that we were agile. Agility is the openness to change/adapt the plan based on information and feedback
• Planning is overrated - This myth stems from ineffective planning which results in missed opportunities and failures to achieve the desired goals. If people fail to lose weight, or fat, blaming the training plan is hardly useful. A more productive way to approach it would reflect on it, analyze it, and re-assess how you plan and execute

Key aspects of planning

The effectiveness of a plan is not solely about its quality in isolation. A comprehensive assessment of its efficacy involves taking into account factors such as the following -

Clarity of goals or desired outcomes

Sometimes my wife tells me that she wants to watch something nice on a Friday night. With no clue about what nice means on a given night, we end up mindlessly scrolling through the nearly infinite content on the streaming platforms, only to either turn off the TV pissed, or worse yet, watch a bad movie.

The clarity in articulating goals or desired outcomes sets the foundation for a successful plan. It’s important at every level you’re planning for - annual, quarterly, monthly, or even in the scope of a Sprint.

The keywords here are goals and outcomes, in contrast to output. Output is a measure of the work done and is often more concrete, and hence easier to quantify. Outcome is the change or impact it creates and speaks to the purpose or goals. Output is often a means to an end. The outcome is what you’re aiming for.

Redefining work in terms of outcome rather than merely the output is a strategic shift. Say you have a product feature to build, and the team is defining the frontend goals of Sprint in terms of the number of UI components to be completed. While it still indicates and ensures progress, you may not have a coherent piece of the software working at the end of it. Reframing it in terms of the subset of functionality that the team would unlock lets you plan effectively, while also increasing cohesion between the tasks, which means reduced context-switching and hence cognitive load.

User story mapping is a useful exercise to build clarity at the project level, while user stories are useful during the execution phase.

Degree of alignment among stakeholders

To what extent are the stakeholders aligned on the various project dimensions? A high degree of alignment among stakeholders is crucial for the seamless execution of a plan. Here are some dimensions of alignment I think about - resource allocation, goals, tactics, relative priorities, division of responsibilities, and impact.

Let’s explore them in a bit more detail -

Alignment of resource allocation  - In engineering management, resources are typically the people directly responsible for the execution - the engineers. But it could also mean non-human resources like the budget(cost of spinning up a new data center), and infrastructure(compute power, data footprint, and so on.). Alignment here is mainly for the benefit of the leadership, to assess how effectively we’re utilizing the resources, and the engineers, to understand the limits and constraints we operate in, and to make the right trade-offs.

Here are some examples of strategic allocation of human resources -

1. If two engineers are allocated to a workstream but the designs and product requirements are not ready as you head into the quarter, find significant work from the technical excellence bucket. This helps them continue to work on meaningful projects.

2. Pair senior engineers with junior engineers, to help them both level up. This fosters a mutually beneficial dynamic where the senior can grow by mentoring and delegating, and the junior, by taking on more responsibilities, getting constructive feedback, and learning. Maintain the pairing until the engineers have developed mutual trust and a robust working relationship.

3. Find ways to marry individual interests and goals with the organizational needs - If one engineer is performing well and is aspiring for a promotion, look for opportunities for them to shine by having them work on projects that have greater visibility and scope, thus expanding their influence.

4. If you have a new engineer on the team, consider a phased approach to increasing their responsibilities. Begin with tasks like working on bug fixes, or increasing test coverage, which builds their knowledge of the system and the product domain without affecting the critical paths. Then have them work on an enhancement to an existing feature, or functionality, before throwing at them something to build from the ground up. Advancement to the next level should be contingent upon their performance and strengths at each stage. Another way would be pairing them with a seasoned engineer in the team from the get-go.

5. Carve out some engineering time to pursue passion or hobby projects, and/or prepare and plan for long-term product needs. Examples of hobby projects could include exploring new technologies like Gen AI, or ways to enhance existing workflows using automation. Long-term product needs may involve architecting for features that are three to twelve months down the pipeline. These will ensure that the team is innovating, evolving, and well-equipped for the long term rather than being myopic

A few scenarios where strategic allocation of non-human resources is required and hence alignment on it becomes important -

1. If you have strict and limited computing power available in a data center, you have trade-offs to make between running critical services and non-critical services, the data footprint, and the resources consumed by infrastructural needs like logging, monitoring, security, and so on.

2. If you had a legacy system to maintain, the trade-offs would be between continuing to spend money and the engineers’ time on keeping the system alive and shutting it down or rebuilding it in a newer stack, thus unlocking long-term benefits.

3. If you had the budget to build/buy a new data center, one in an expensive region like the San Francisco Bay Area, and another in a relatively cheaper location like Ohio, where would you put your finite budget into?

Goal and impact alignment - When everyone shares a common understanding of what needs to be achieved and why, it fosters a collaborative environment and purpose. Concreteness and specificity are key factors in defining goals and impact. Abstract statements are likely to be interpreted differently by different people. SMART framework to specify goals is useful here, as are other frameworks like OKRs.

An example of a badly defined execution goal is “Make the app more user-friendly, and visually appealing”. While “user-friendly” and “visually appealing” are good dimensions to optimize for, a better way to state the goals would be something like “Reduce page load times of the app by 20%, to make it more user-friendly, and add icons with labels for buttons, to make it more visually appealing”

Tactical alignment - While there is more than one good way to achieve the goals you set, it’s important to align on what a group thinks the best way is. It reduces confusion, disappointment, duplicated effort, and the likelihood of fragmented, or sub-optimal results.

Alignment of the relative priorities - In a complex organization there’s usually more than one thing at any given time that people could be working on, and the priorities are dynamically changing, which makes it critical to align on the relative priorities. While we could ideally work on infinite things, in reality, we’re bounded by constraints, like time, cost, people, and finite abilities.

Aligning the division of responsibilities helps hold people accountable, and accountability is crucial to achieving predictable outcomes. “Assumption is the mother of all f* ups”, as the saying goes.

Framework for work estimation

Getting a realistic sense of how much time, effort, or other resources something would take feeds into the plan, and hence a robust framework for work estimation is essential in providing an accurate picture. It consists of a well-defined thought process(how you think about complexity, and cost, for example), and a standard unit for estimation, for consistency.

How do you estimate the time required for a task on a high level, ensuring accuracy with minimal effort? Although the popular models in this space like COCOMO, and PERT are heavy-handed, they provide a great structure to guide your thinking. I love the idea of “cost drivers” from the COCOMO framework, for example, and the idea of visualizing the various aspects of projects, especially to gain clarity and alignment, from the PERT model.

Before we delve into the exercise of estimation, let’s consider the crucial factors that dictate the accuracy and reliability of estimates.

• Complexity - Not all problems are created equal, and more complexity usually translates to more time and effort
  • Problem complexity - Some problems are inherently more complex to solve than others. Examples - A CRUD application is simpler to build than a reporting application that requires intensive data management. Read-only features are relatively simpler to build than an application with a multi-step management workflow
  • System complexity - Building on top of a shaky foundation adds significant complexity, whether it’s a legacy monolith, a business-critical system, or a particularly fragile system with zero tests, and spaghetti code
  • Dependencies - Whether it’s technical dependencies, like a feature that needs a major overhaul of the existing system, or team/personnel dependencies, where part of the work is expected to be done by a different team, or a third-party vendor. The more the dependencies, the more complex a system gets
• Technical unknowns and learning curve - Has the problem been solved before in the team or the organization? Is there a steep learning curve to building the knowledge and expertise required to solve the problem in the team?
• Product/business unknowns - Clarity of the product and design requirements greatly influence the estimates. Strive to crystallize the desired outcomes before anything else; it’s the basis on top of which other factors rest

There are a few steps I break estimation into - one, estimating the level of effort, done in abstract units, then in relative units of time, followed by timeline estimation, that is, estimating the completion of the work in calendar times. You can relate this to the idea of a basic, intermediate, and detailed estimate from the COCOMO model.

• Level of effort estimation - I use T-shirt sizing(S, M, L, XL), or story pointing(most popularly done in the Fibonacci sequence), depending on the granularity and level at which I’m doing this exercise. T-shirt sizing is more useful in planning activities like quarterly, or yearly, whereas story pointing proves more useful in the context of bi-weekly, or Sprint planning, where the problems have been broken down into manageable chunks quite a bit. Although the LOE units loosely correlate with the time required, especially when carried out in a framework of fixed intervals like Sprints, they don’t have to be. The intent here is to understand the level of complexity and effort, rather than the expected time taken to solve a problem. Once I have an estimate of the LOE, I proceed to the next step, which is measuring them in time.

• Estimates in relative units of time - I find it useful to map the level of effort from the previous step to person-weeks, which is the number of weeks it’d take for one engineer to complete the work if worked full-time. It still doesn’t take into account the realities of a team, which I’ll get into in the next section.

  • Ideal estimates - Assumes that given a piece of work takes the same amount of time to complete regardless of the engineer working on it. Depending on whether you anchor this based on your slowest or fastest engineer, you get the most pessimistic or optimistic estimates, both of which are useful. In reality, it’s been proven that humans routinely underestimate and hence your real completion time will most likely align with your conservative estimates.

  • Pragmatic estimates - Takes into consideration the people working on it, the make-up of your team, including their technical and domain expertise(Are they new to the organization/team? Are they comfortable with the domain? From their track record, are they known to be a fast learner?), communication skills and drive(Can they communicate well and drive projects with other engineers and non-engineering stakeholders with minimal supervision? Do they tend to take initiative and be proactive - do they wait for instructions, or direction, or thrive in uncertainty and chaos?), strengths and productivity(What’s their average Sprint velocity, i.e. how much are they known to achieve in a week or two weeks?)

• Timeline estimation - Following the previous step, this would give you a sense of calendar times in which the problem can be expected to be solved. This requires capacity planning - taking into account people’s availability, organizational constraints like budget, and holidays, and competing priorities. Gantt chart is an invaluable tool for this step. The length of each project/task is the time in relative units from the previous exercise. Stack them up in a way that tries to minimize context-switching for the engineers(it has a significant cost), and delivers the most urgent things first, while also allocating the time to tackle the important, yet non-urgent things. I find the Gannt view in Airtable, and JIRA particularly useful, depending on whether I’m doing quarterly planning or Sprint planning.

A note on getting better at estimation - Since it’s an art more than Science, practice is the key to improving at it. To develop my intuition around it, to make it more natural, I constantly perform estimation myself first, without involving the engineers and experts, then get the estimates from them, and compare how off I was and why. It helps me identify my blind spots and build the required technical and domain expertise. It’s an attempt at “deep practice”(a term I learned from Daniel Coyle’s book, “The Talent Code”) to estimation.

Core principles from the Agile methodology also serve well to improve estimation skills - break down problems into smaller parts, estimate them, analyze and learn from them, and iterate.

Measuring progress

Measuring progress not only ensures alignment with objectives but also serves as a guide to knowing when you’ve successfully achieved your milestones. Furthermore, it helps you re-assess and adapt the plan when needed. But how can you gauge if a project is on track and determine the progress made toward the goals?

Let's use race as a context to systematically consider the metrics needed to better understand progress. In the context of a long-distance race, here are some of the aspects you would measure -

How fast you’re going - Speed(miles/min), and pace(mins/mile) are the most common ones runners use. In software engineering, metrics like sprint velocity, and lead time tell you how fast you’re going, in terms of the productivity of the team. I use JIRA reports to measure both.

The distance you’ve covered so far/the distance yet to be covered - In a race, you’d look at the mile markers to determine the distance you’ve covered. Knowledge of the total distance that needs to be covered tells you how much more there is to go. Burn-up and burn-down charts are useful tools to compare your commitments to your delivery.

Where you’re spending time - Are the uphills killing you? Or, are the downhills burning your knees? Are you spending way too much time in the aid stations? Are you getting lost on the course? Cycle time, and lead time, help you derive insights about the amount of time/cost spent in the various stages of the project lifecycle.

How well you’re doing - What’s your heart rate? Are you sufficiently hydrated? Fueled? Are your knees, calves, or ankles hurting? This is the most subjective of all. Although you can look at your heart rate using sensors, nothing beats paying attention to your body. Knowing how overwhelmed or stressed your team is helps you make better predictions of performance. Some metrics can act as reasonable proxies to measure the well-being of the team, like code churn and defect density, for example, especially if they start to rise during a specific project. However, they are no substitute for direct and open conversations with the team.

Smoothness of execution

The quality of the plan directly influences how smooth the execution is. The identification and mitigation of hiccups, such as confusion, conflicts, delays, missed deadlines, and surprises encountered during execution, provide insights into the plan's effectiveness, guiding future planning efforts.

Timeliness and consistency of delivery

The ability to meet delivery timelines consistently is a key indicator of the plan's effectiveness and the team's overall efficiency.

Adaptability

No amount of planning can predict the real world’s twists and turns. Unexpected events like a market shake-up, people leaving the company, or reorgs, can derail the best-laid plans. That’s where adaptability comes in. Being adaptable doesn’t have to be mutually exclusive to having a plan, but rather having the flexibility to course-correct. If your plan included potential alternatives they come in handy to change the course of action, when needed, without compromising the goals. The previous aspects of planning outlined help with being adaptable by feeding into the plan more information, context, and learnings.

Alpha and Beta releases of products help get feedback and incorporate them in future deliverables. Project, or team retrospectives and Sprint reports help you learn from the previous cycle and improve.

Communication

Lastly, a plan is no good without proper communication, and written communication scales better than verbal. Strive to document things in a written form. Some questions that I have found useful to think about -

• How are things communicated and where - in JIRA, Slack, docs, or emails? I strive to keep the implementation level product and technical decisions in JIRA, closest to the engineersDiscussions are over Slack, but non-trivial decisions have to be moved elsewhereI use emails for a more formal mode of communication, to share it with a relevant, yet broad group of stakeholders. Progress/status updates, for example, make sense over emails
• Where are decisions captured, who is notified, and how frequently do you review them?
• Who do you hold accountable for updating them? is a helpful model for determining this

Parting Thoughts

The post turned out to be longer than I had imagined. But I also didn’t know what I was about to find out by digging into it, to spell out what I had been doing almost unconsciously. While it might seem like a lot of work, it’s surprising how easy it becomes the more you do it. It’s like long-distance biking - you have to maintain good form, ensure that the bike is in good shape, keep an eye on the road for any obstacles, traffic, or rough patches, hydrate, and fuel well, carry layers, wear sunscreen, pay attention to your body and the bike at all times. But as you become adept at it it almost feels natural.

Contact me if you have any feedback, comments, or thoughts on the post.