In traditional software operations, software would be "thrown over the fence" to operations teams. Technical operations teams would be aided in operating a service using Standard Operating Procedures (SOPs).
With the advent of practices like DevOps and the growth in hiring SREs (Site Reliability Engineers), development and operations are increasingly unified. This has allowed "chore" work like deployments and maintenance to be automated. Standard Operating Procedures still play a role in this world, but the work of SREs seeks to decrease their necessity over time.
This has meant that product teams have increasingly (if not wholly) operated their own software in production environments, perhaps with the help of SREs embedded in their product teams.
Simultaneously; the ownership of a given service or component of software does not necessarily always align with the necessity to make changes. For example, a Business Intelligence team may need to modify a service owned by a Billing team to extract some additional information. This is also true in engineering organisations with platform teams; for example, the billing team may need to alter code in a component library owned by a UI platform team.
Ownership is a vital trait for maintaining technical systems and mitigating risk. Nevertheless; a one-size-fits-all approach may not suit all scenarios.
Where multiple engineering teams both develop and operate on the same systems; communication becomes a vital part of working together. It is essential to understand the different roles and responsibilities involved.
Here, I wish to raise three different models of ownership that I have worked with in the past and discuss them. These models all assume a traditional engineering organisation structure, so may be applicable differently for those working in companies which have adopted Spotify's "Squads and Tribes model".
Complete Ownership
Complete Ownership is a model where a given service is wholly owned by one engineering team. It remains essential to document this ownership so there is a common understanding of software ownership across the engineering organisation and wider company.
In these circumstances, one team takes responsibility for a service; they will own everything from the development environment to fixing bugs. This model provides a clear understanding of roles and responsibilities, and the team's leadership is able to take full responsibility for managing risk.
Other teams may contribute code into services they don't own, but the team is responsible for setting up the rules of engagement for doing so (such as by reviewing Pull Requests and making deployment decisions).
In the majority of cases, such as where the service only meets one business purpose, this arrangement is highly effective.
Ownership Transfer
From time to time, the business goals of a service may better align to a different team. In such cases, Engineering Managers (EMs) may agree that the service should move to the ownership of a different team. For example; an SMS delivery service originally used for marketing purposes may end up being predominantly used by a team focussed on shipping/delivery notifications.
In my own experience, this is typically formed of a process as follows:
- The two EMs of both teams meet and agree ownership transfer in principle.
- A written handover document is produced outlining the timeline and approach to transfer; including any plans for training, architectural diagrams and any documentation.
- The plan is signed-off by both EMs and any other stakeholders. Relevant documentation and systems are updated to represent these changes.
During the transition timeline of ownership, a period of Joint Care can help ensure a smooth transition; this brings us to our next ownership model.
Joint Care
Joint Care comes about when a service is temporarily or permanently operated by two engineering teams under a Joint Care Agreement.
Whilst a largely undocumented process in the engineering world, Joint Care is used in other professions. In medicine, shared care is used by doctors to proscribe at the recommendation of professional colleagues (in the UK at least, this requires the agreement of all parties, including the patient). Academia is another area where collaboration agreements (namely Lambert Agreements) are used to allow multiple institutions to work together on projects.
Within software engineering; Joint Care allows Engineering Managers to agree to operate their service with another team. In temporary situations, this may be to enable the transfer of a service to a different team or to split a service into distinct microservices. In other situations it can be a lasting agreement where a system serves multiple different engineering functions or there is another operational need (say there needs to be on-call coverage spanning multiple different timezones).
Roles and responsibilities can be confusing in such situations, so Joint Care Agreements are essential. These agreements aren't set in stone and should be adaptive but provide an important mechanism to agree and document essentials. Issues to cover in a Joint Care Agreement can include:
- What are the roles and responsibilities of each of the teams?
- How is responsibility split for supportability work or incident management?
- How do the teams co-ordinate effectively (i.e. communication and decision making)?
- When/if does the agreement end? What happens if the agreement ends?
This model is not without risks; it requires high levels of trust and communication between the teams. Collective code ownership must go beyond one team and span all engineers involved regardless of their team. In any event, engineering professional judgement remains critical even when a Joint Care Agreement exists.
Conclusion
Standard Operating Procedures were designed for an era where the development and operations of software were separate functions. Increasingly, the same teams develop and operate the same software.
Ironically; this can lead to additional complexity where multiple engineering teams need to work on the same software. This blog posts presents three models that I have personally used to handle this conflict.
These models are not perfect but can provide a foundation to work from to formalise the relationships needed to work through these challenges. At the very least, I hope this blog post has provoked some thought around how we handle code ownership with modern engineering practices. Whilst I am normally lightweight on process myself, I have found that software ownership is a foundational concern in ensuring engineering success in growing organisations.
Nevertheless; communication remains the most essential tool that an Engineering Manager has at their disposal to work through these challenges and must be foundational in whichever model is used.
To this end, I have found that component ownership documentation, handover documents and Joint Care Agreements can be vital tools in helping ensure there is clarity when addressing software ownership challenges.
Header Image: Highways England, HM Government. CC BY 2.0. Via Wikimedia Commons.
