Waterfall vs Agile
Imagine you are a builder and you’re tasked to build one house and one city. How would you approach each building challenge?
The house seems straightforward enough. To achieve your goal, you choose an architect, create a detailed blueprint of the house with all the measurements and materials, select the contractors, start to build. When the work is completed, your house is ready to move in.
Now consider the challenge of building a city. Technically you could follow the same process as building a single house. In practice, it’s unlikely you would have an all-encompassing master plan of the city covering all the requirements. You may also not be able to freeze the requirements; each new citizen may have their own wants and needs, a new mayor may come along with its own political agenda, etc.
To solve those different building parameters, let’s consider 2 methodologies: Waterfall and Agile.
Agile has conquered the tech world today and makes the life of builders easier, with its new concepts, philosophy, team rituals and built-in flexibility.
In the article below we’ll talk about the shift from Waterfall and a monolithic approach to software, to an agile world of microservices, as well as our reflections at Slash about the challenges of operating Agile.
The history of Agile
It is difficult to imagine, but the first commercial software has been developing since the early 70s. In 1970, the American computer scientist Winston Royce compiled a paper entitled "Managing the Development of Large Software Systems". In his work, Royce argued that software development should not follow other industrial processes, such as the automotive assembly line, consisting of many steps, where each previous step has to be completed before going to the next one.
Instead, Royce proposed a phased approach, where all the project requirements are gathered up front. Only after this initial step, the remaining processes are meant to be done.
The paper led to the adoption of the waterfall method for software development, but the irony is that it was one big misunderstanding. Royce was arguing for an agile process where requirements were gathered before each step of development & testing; and each step was “small enough”.
The early 90’s gave birth to the first flexible software development methods that were able to challenge the dominant “plan-driven” approach of the Waterfall method so revered by other industries as well.
A decade later, in 2001, 17 software developers gathered in the state of Utah (USA). As a result of their meeting, the Agile Manifesto was born. This was the very beginning of the new project management and software development era.
You can read more on the origins of agile, in the following article: 12 principles of Agile Manifesto.
So why Agile?
As we’ve seen in the example of building a house vs a city, the challenge of the Waterfall method is that it contains 2 risky assumptions: requirements are clear upfront and implementation is accurate.
In a dynamic environment, this invites failure. Requirements evolve, customers and end-users have opinions (and change them!), and testing of the development process should occur continuously. Failing to incorporate this in your workflow can have devastating results: a product no one wants to use, budget overrun of 100%+, tech debt, etc.
To better understand why most IT problems are dynamic and not static, let’s consider how IT architecture is evolving.
Monolithic applications vs microservices: why or why not?
Two decades ago traditional IT was based on monolithic applications, nowadays most new IT systems or applications are based on microservice architecture.
While a monolithic application is a single unified unit, a microservices architecture breaks it down into a collection of smaller independent units. These units carry out every application process as a separate service. So all the services have their own logic and the database, as well as perform the specific functions.
Source of table: https://articles.microservices.com/monolithic-vs-microservices-architecture-5c4848858f59
The attractiveness of microservices is easily explained. Developers can work in parallel using the programming language, platforms, and data formats of their choice. They do not need to coordinate with other developers the deployment and scaling of application components built with microservices. And for clients, it enables rapid development, more future-proof tech and scale.
The attractiveness of microservices is easily explained. Developers can work in parallel using the programming language, platforms, and data formats of their choice. They do not need to coordinate with other developers the deployment and scaling of application components built with microservices. And for clients, it enables rapid development, more future-proof tech and scale.
In many ways microservice architecture is made for agile development.
Agile at Slash: Our 5 lessons learned
While we at Slash are big fans of Agile as a philosophy and practical framework to organize our work processes, here are 5 challenges we have faced so far in implementing it.
Challenge 1: clients want to work Agile but pay Waterfall
We found out the hard way that clients want to work in Agile, but prefer to pay for Waterfall. In other words: they want a contract with a defined scope and a fixed budget, but they love to have the flexibility to modify the scope and requirements afterwards.
In a client-vendor relationship, that is a killer and poses the 2 practical challenges:
- Clients need to be educated on the pros and cons of working with Agile.
- Contracts need to be adjusted to allow for a more flexible, agile framework.
We have adopted two strategies to deal with this:
- Strategy 1: clients pay for a retained team with a flexible scope. We define pre-agreed day-rates for the members of the team and a transparent planning and tracking mechanism to guide the daily work.
- Strategy 2: clients pay for a retained team with an ~80% pre-agreed scope and ~20% variable. We install a requirements engineering and approval process to allow for scope to evolve within the team capacity.
Challenge 2: protect the integrity of the agile process
To be effective in an agile process, you need to protect and maintain the integrity of the process at all costs. Agile is opinionated. If you don’t protect the process, you do more harm than good and develop anti-agile patterns.
Challenge 3: kill bad habits
The agile process doesn’t do well with lone rangers or lone geniuses. Agile requires good teamwork to be effective. You need to build an environment where everyone respects and supports the process, including elimination of bad habits, late-night work and weekend work.
Challenge 4: build trust relationships
Agile is based on a collaborative relationship of deep trust, dialog and transparency. If you work with external clients or startup partners like us, you need to transform your relationship into a trusted partner for Agile to work properly. Traditional client-vendor relationships may not have the team work, trust and leadership required.
Challenge 5: factor in overheads
Agile comes with “a lot” of project management overhead on top of the development time, depending on the project 30-60%. This has to be factored in terms of scope, budget, timeline and team. In Waterfall this overhead is more “hidden” in the planning phase, delays etc. In Agile, this overhead is made very explicit.
Agile Survey: soundbites from the Slash team
We surveyed our team to hear about their experience with Agile.
The first experience with Agile:
The majority had no or little experience with Agile prior to Slash. A few worked on Agile projects in industries such as banking, credit card organizations, SaaS, etc. The majority found the switch from Waterfall to Agile quite difficult because of initial misunderstanding and concept misinterpretation.
Changes in individual habits:
The #1 change they had to adopt is “Flexibility” . The #2 is “Teamwork”. Slashers found it challenging to get used to it at first, especially those who came from enterprise backgrounds and were used to operating in waterfall models.
Changes in team behaviour:
Agile affects the team behaviour in many ways.
“The habits change by focusing on sprint as a whole instead of focusing on each individual task,” - as noted one respondent, ”[...] and we will prioritize what benefits the client the most.” Another team member recalled clashes and misunderstandings between the team when switching to Agile, because many of the old habits were no longer needed.
Benefits of Agile:
“Some of the biggest benefits to me come from the simple concept of being able to quickly and frequently put software into the consumer’s hands.” “Sometimes unpolished products cause problems”. “Embrace a fail fast, fail often mindset”.
Lessons learned from the team
All our respondents stressed on the importance of adopting new teamwork techniques and developing the ability to constantly communicate with team members to get work done.
An Agile mindset can help get things done better and faster, however everyone in a team needs to stay focused on the final outcome.
Summary
Agile is probably the most “native” operating model for teams of builders and innovators. It is not, however, a panacea and, like all methodologies, requires careful nurturing and development in order to obtain the benefits. We embarked on the Agile journey across all our tech and non-tech teams in 2017, and every day we’re still learning.