Foreword
Seeing this paper from December 2, 1996, published for the first time here — in its original form — is for me like breaking open a time capsule. After joining Dean in the FAST group at Bear Stearns back in January of 1997, he and I soon realized that we shared similar outlooks on what software development, and particularly effective reuse, should look like. Upon introducing him to my first book, Large-Scale C++ Software Design (1996), he showed me this paper. After reading it, I told him that I would likely want to cite it in my, then nascent, next book, Large Scale C++: Process and Architecture — the one that, ironically, I’m only just now about to complete. But because Dean’s work had never been formally published, I wasn’t able to do that… until now.
Largely organizational in nature, the ideas presented in this article transcend programming languages, and he makes no specific references to any. Nearly twenty years later, the engineering ideas we held in common remain an integral part of the software development practices and processes currently employed by the BDE group at Bloomberg in creating and enhancing our highly successful, Hierarchically Reusable BDE foundation libraries, a.k.a. Software Capital. These ideas also helped form the philosophical basis for talks I regularly give worldwide, many of which are available for viewing online (e.g., 1, 2, 3, 4). Indeed, much of the Software Capital we have created at Bloomberg has been open-sourced for the benefit of all.
Dean, who is also still writing software, continues to put these ideas into practice in the development of his ClearFactr product.
— John Lakos, Software Engineering Manager, BDE Group, Bloomberg LP
Summary
To most effectively manage an ever-increasing demand for ever-increasingly complex software, it is vital that an organizational process be established which aims to achieve a central goal: maximizing the creation of, and benefits from, Software Capital. Software Capital, with specific regard to software developed over time in-house, can be defined as the cumulative technology that can be re-deployed to new situations. To best achieve it, two separate development teams with distinct but mutually reinforcing goals and missions need to exist: Application developers and Software Capital developers. Application developers serve the business units directly, whileSoftware Capital developers ensure maximum quality and productivity from the application developers. The separation allows both sides to leverage their respective strengths in business knowledge and good software design.
Software Capital becomes the highly leveraged toolset with which small teams of application developers can more rapidly respond to business needs. It can significantly raise the starting point from which a business specific solution begins, and thus significantly reduce time to market. With many profit opportunities being front-loaded with respect to time, it can help anorganization capture a larger percentage of an opportunity’s profit potential.
Introduction
Application developers can work furiously on a project and produce reams of code that solve this moment’s business need. Another need comes along and the development process begins anew. The critical question is: from where?
Software embodies knowledge and experience. With each response to a new business need, software expands to contain that knowledge. How the knowledge is contained and how it can be accessed directly impacts its usefulness to future problem solving. If a past solution might be applicable to a current problem, but cannot be accessed, any potential value of this knowledge and experience has been lost. Too much incremental effort is then expanded and the best one can hope for is accurate replication. To the extent that yesterday’s solutions can become the starting point for today’s, the software becomes a powerful leverage point. It becomes Software Capital.
Yet the formation of Software Capital does not occur automatically. In fact, powerful organizational and business forces are typically in place that actively discourage its creation.
The problem, its sources and its symptoms
Two common and coexisting circumstances lead to the lack of a Software Capital base:
First: Responding rapidly to end user requests with inadequate system design.
Experienced developers know that simply responding “as fast as possible” to end-user requests will typically result in an application that has little structural integrity. But perceptions of fantastic developer productivity are established and furthered. However, the rapid response times are unsustainable as an increasingly fragile application is in the making. Bugs multiply, due to unforeseen interactions between the numerous fixes, and software quality declines. Ultimately application paralysis is achieved. End users are suddenly confused and annoyed as to why their application is no longer trustworthy and by the inability to add new features easily, if at all.
A response to the above scenario is often to try to “buy time” from the end users to make the structural changes and re-designs the developers know are needed to once again achieve progress. Yet in a highly dynamic business environment, with new needs and opportunities arising daily, such time does not actually exist — it comes at the expense of addressing new issues and needs. Typically, being late with a “great” solution is of little value to the business compared with an “adequate” solution delivered on a timely basis. Thus, the importance of addressing business needs in a timely manner tends to prioritize over the need to restructure. The result is an infrastructure that grows stale relative to the business, and ironically one that leaves the business in a weaker position to quickly capitalize on new opportunities.
It is tempting to think that a given application development team can simply “do the right thing” and write code that not only meets the needs of their users, but has applicability to other projects. This is certainly possible. However, because doing so is often more difficult than writing a solution specific to the immediate needs, it typically doesn’t happen. Furthermore, when a solution is implemented for a particular need, it typically can not be leveraged to meet other business or application needs.
Second: Striving for cross-business leverage solely via application generality.
It is one thing to understand the needs of the business and to be able to write an application to address those needs. It is a very different thing to write a tool that can be leveraged across multiple businesses and that has longevity. The solution is not necessarily to make an application generic. Efforts to achieve cross-business leverage and longevity through generality tend to result in large, monolithic systems that do not serve any of their users particularly well. Also, as the applications grow, complexity and maintenance costs tend to grow even faster.
The net result of the two above scenarios is often:
- Duplication of effort across projects
- Reduced overall productivity
- Longer time to market for new solutions
- Maintenance of existing code that overwhelms the ability to do “new” development
Above all, business units suffer from sub-optimal software, delivered slowly.
Addressing the needs of the business in a timely manner and maintaining a sound infrastructure from which to address those needs must be viewed upon as being equally important. A balance must be struck between both objectives.
When the need to simultaneously solve two or more problems arises, the most effective response, where possible, is to parallelize the solution. A given developer can rarely achieve both above objectives simultaneously. It is also vital to recognize that properly addressing a set of business problems and satisfying the need for architecture and productivity often requires completely different skills sets. The solution is to formally separate the two functions, and dissolve any expectations that a single developer or team will simultaneously achieve both objectives.
Software Capital In Action
In growing a base of Software Capital, common application infrastructures and business logic are factored out into a central repository. To build an application off this base, application developers can focus on incremental development totally germane to their business. As they spend more time focusing on their business and less time on non-business specific infrastructure, their business knowledge and value to their users increases.
Software Capital — The Players
Three parties to the above, each with distinct missions, can be identified: Application Developers, Software Capital Developers, and Management.
The major differences between the Application and Software Capital Developers are summarized in the table below:
Application Developers
Application developers start with the capabilities of the Software Capital and use them to write applications. Their role is more traditional and fairly easy to define from a business unit’s perspective: solve business problems by writing software. The key is managing what they spend their time writing. Therefore, Application Developers:
- Develop close working relationships with members of the business units.
- Focus on the needs of the business and not on the infrastructure used to meet those needs.
- Have a responsibility to support the efforts of the Software Capital developers.
- Must be able to communicate specifications and requirements to the Software Capital developers.
- Keep the Software Capital developers informed as to whether their output is timely, useful, efficient, easy to use, etc.
Software Capital Developers
The roles of the Software Capital developers are somewhat more abstract and removed from the business. They build underlying platforms, tools and “core technology.” Therefore, Software Capital developers:
- Maintain constant communication with all application development teams, perhaps via a representative from each team.
- Create an information bridge between teams that may never have the business need or initiative to communicate with one another.
- Strive to identify commonality and opportunities for leverage.
- Must produce code of a sufficient quality and robustness such that no application team seeks to “do it themselves”.
- Maintain constant cost benefit analysis with regard to ongoing development.
- Actively seek to “capture” code from the applications into the Software Capital base for reapplication to other business needs.
- Promote usage of the Software Capital and look out for new development that is replicating existing solutions.
- Provide training in the usage of such tools and mentoring in areas such as application architecture.
- Must be able to extract specifications and requirements from application developers who may be unclear as to what they need. They must shift the application developer’s focus to what their applications need and away from how those needs will be met.
An interesting situation can arise when the application developers “can’t wait” for the output of the Software Capital developers. The temptation or need for the application developers to develop something that should more properly come from the Software Capital developers may indicate any of the following:
- A lack of anticipation of future needs by either or both teams.
- A productivity problem on the Software Capital side.
- A software design problem on the application side.
- Inappropriate, untrustworthy or difficult to use solutions on the Software Capital side.
- A marketing failure by the Software Capital team.
Any of the problems above can be addressed directly. However, there will be legitimate cases of the application developer’s needs not being addressable by the Software Capital team when they really ought to be. In this case, the most important action required is to try to capture the application developer’s solution back into the Software Capital base. In this case, the Software Capital team faces the following situation:
- Potentially having to repackage the solution in a way that other application developers can make use of it.
- Making sure that the application developers who spawned the new software can readily swap out their original code with the repackaged solution, so that they do not go out of sync with the Software Capital base.
As the Software Capital team seeks to achieve its goals, it must keep the following in mind: Good, albeit sometimes complex, designs will yield better software. However, caution, relevance and applicability must rule. To paraphrase a section from Design Patterns:
Techniques which make software more flexible have a disadvantage: Dynamic, highly parameterized software is harder to understand than more static software. There are also runtime inefficiencies, but the human inefficiencies are more important in the long run. These techniques are a good design choice only when they simplify more than they complicate.
The key is to use design appropriately and then leave the job to experienced people who can empower others to reap the benefits.
Management
Management’s role centers around the fact that creating Software Capital is as much an organizational and philosophical approach as it is technological. In fact, while certain technologies will provide assistance in achieving Software Capital, ultimately the presence or lack of a sound organizational strategy will play a bigger role. Similarly, without the explicit approval and support for such a strategy from the highest levels in the organization, the effort will ultimately wither and fail. The philosophy behind Software Capital acknowledges and acts upon the fact that people do have areas of expertise and that these talents should be maximized.
Therefore, Management’s roles are:
Providing resources:
- Talent
- Development tools
- Working environment
- Compensation sufficient to attract and retain quality people
Vision:
- Providing overall direction of what business opportunities lie ahead and what needs to be done to maximize profit extraction.
- Communicating to all developers the limitations of the existing software relative to where the business needs are and will be.
End-User Perception Management:
- Of time to market for applications and tools
- Of development achievement, both visible and non-visible
Supporting and marketing the value of both application developers and Software Capital developers.
Ensuring that Software Capital developers do not “temporarily” but then permanently move to the application development side:
- At times it may be tempting to “borrow” a resource from the Software Capital side to meet a short-term application development need. As developer skill sets grow and particular abilities become more apparent, it will be important to allow for people to transfer between the two development groups.
- However, one group cannot permanently cannibalize the resources of the other. If, for example, a Software Capital developer could be more valuable on the application development side, their role of the Software Capital side should be filled by another person.
Agenda items for a Software Capital team
It is helpful to examine an example of Software Capital in action. Consider the software development requirements of a typical investment bank:
A developer of a real-time pricing application should be concerned with what to request from a real-time market information service, and when, but not how to make such a request. Worse, they should not have to write the data service or the messaging service used to communicate with it. Instead, they should look to the base of Software Capital and find two (or more) relevant services to use. The services should be straightforward to use, not introduce superfluous overhead to the application and, above all, work reliably. As additional features are deemed necessary by the business unit and/or problem at hand — not by the Software Capital team — they are prioritized and communicated.
Continuing with this example environment, there are numerous opportunities for leverage from Software Capital:
- Basic software building blocks such as dates, strings, calendars, containers and generic algorithms
- Data persistence and retrieval
- Real-time data handling
- Inter- and intra-application communications
- Reporting
- User Interface components and behaviors
- Application frameworks
- On-line help facilities
- Development environment productivity tools
Combining the basic infrastructure with business logic yields an area of Software Capital one step removed from any particular business need:
- Market data information servers
- Calculation servers
- Trade and Position servers
- Risk management and analysis tools and routines
Some of the above items wind up being implemented as applications. However, they differ from a business specific application in a very important way: they provide resources to any number of business specific applications, both existing and future, through a published interface. When a new project needs to avail itself of one of the services, they can likely do so at near zero marginal cost.
The Results
The bottom line with Software Capital is better business solutions, delivered faster. As new features are developed, multiple applications stand ready to pick them up simultaneously. Communication is focused and more effective. Individual talents are leveraged and encouraged to flourish as people expand the capabilities of their particular niches — on the application side and the infrastructure side.
Better applications result from having more consistent designs, leading to reduced maintenance and easier learning curves for new personnel. In addition, applications have greater incremental emphasis on the needs of the business and less overhead in the areas of infrastructure.
Other benefits are found in the areas of staffing. Staffing becomes easier as there is no need to place undue emphasis on people who are good at both business specific application development and software design — a very rare combination that is correspondingly expensive.
Conclusion
Organizations that recognize and address the need to form a base of Software Capital will enjoy a competitive advantage. Rather than simply working harder, they work smarter. Their higher productivity and shorter time to market allow them to capture more profits from business opportunities. This capital base does not form without conscious effort. Software Capital is rooted more in an organizational than a technological philosophy. Most importantly, a key structure — the separation of application and Software Capital development — must be in place and supported by all parties.