The Growing Importance of Sustaining Software for the DoD

Architecture , Software Assurance , Software Cost Estimates , Software Product Lines , Software Sustainment Add comments

Part 1: Software Sustainment Trends and Challenges
By Douglas C. Schmidt,
Deputy Director, Research, and Chief Technology Officer

Department of Defense (DoD) programs have traditionally focused on the software acquisition phase (initial procurement, development, production, and deployment) and largely discounted the software sustainment phase (operations and support) until late in the lifecycle.  The costs of software sustainment are becoming too high to discount since they account for 60 to 90 percent of the total software lifecycle effort. Moreover, in an era where DoD new-start programs are being reduced in favor of prolonging legacy systems, significant software sustainment cost increases are themselves unsustainable. The growing expense and prolonging of legacy systems motivates the need for greater discipline and attention on defining and applying appropriate methods and technologies to improve sustainment capabilities and efficiencies.  This SEI blog posting—the first in a two part series—summarizes key software sustainment challenges faced by DoD; the subsequent post describes R&D activities conducted by the SEI to address some of these challenges.

Software sustainment involves coordinating the processes, procedures, people, information, and databases required to support, maintain, and operate software-reliant aspects of DoD systems. DoD software sustainment needs and practices are shaped by various trends, including

The confluence of these trends impacts the workload, risk, and cost of acquisition program sustainment processes and professionals. These trends also contribute to the growth in total ownership costs across program lifecycles.

Newer DoD systems rely more on software than older systems did, so the demands on software sustainment organizations are increasing as current generations of DoD systems transition from production to sustainment. For example, the percentage of avionics specification requirements involving software control has risen from approximately 8 percent of the F-4 in 1960 to 45 percent of the F-16 in 1982, 80 percent of the F-22 in 2000, and 90 percent of the F-35 in 2006. This growing reliance on software now affects most aspects of DoD systems, including mission data systems, radars/sensors, flight/engine controls, communications, mission planning/execution, weapons deployment, test infrastructure, program lifecycle management systems, and software integration labs. 

Not only are we dealing with a growing software base, but also the constantly-evolving environment in which software runs. For example, although software does not wear out, firmware and hardware become obsolete, thereby requiring software modifications. Likewise, upgraded capability must be integrated into existing systems and software defects and performance bottlenecks must continually be identified, fixed, and optimized to provide full functionality. It should therefore not be surprising that the DoD expends an increasing amount of time and effort sustaining software, often much more than was originally anticipated due to uncertainties during initial program cost estimation. While programs can take funds from later phases to cover development overruns, the sustainment phase has nothing after it to prey upon but itself!

High software sustainment costs occur for various reasons. For example, functionality originally provided by hardware may be replaced by software (e.g., fly-by-wire), thereby increasing software sustainment costs. Periodic software upgrades and enhancements throughout the lifecycle of DoD systems may also result in unanticipated increases in sustainment costs.  Moreover, costly and time-consuming effort is required by software maintainers to understand the original design and carefully make changes to avoid degrading the integrity of the design or negatively impacting key quality attributes. In addition, software scale and complexity are growing significantly to meet the expanded threat spectrum, which drives sustainment costs up.

As sustainment costs have increased, the DoD is struggling to support all its legacy systems. Economic strategies for understanding and addressing these rising costs are affected by a key difference between DoD weapon system platforms  and the software running in those platforms (example of weapons systems platforms include the physical airframes, hulls, chassis, and their associated parts like engines, weapons, sensors, and computing/communication units). The DoD has historically viewed sustainment from a weapon-system-platform perspective—where physical manufacturing and handling wear-and-tear represent a significant expense. From this perspective, sustainment costs are primarily a function of the number of weapon system platforms and parts. The DoD has traditionally handled these burgeoning costs by shrinking its inventory (for example, by retiring and/or reducing the numbers of aging aircraft, ship, and vehicle platforms).

This traditional approach worked when sustainment costs were largely a function of weapon system platform and part manufacturing costs. In contrast, software sustainment has essentially no manufacturing or wear and tear expenses. As a result, software sustainment costs are primarily a function of the number of software variants.  For example, a particular weapon system platform family (such as a class of ships, planes, or vehicles) may have scores of software variants reflecting different sensor, processing hardware, operating system, and network/bus configurations; different algorithms; and different security profiles allowed for use by customers from different countries.  Sustaining all these variants impacts the time and effort required to assure, optimize, and manage system deployments and configurations throughout the lifecycle.

As software variability grows—as it inevitably does in legacy systems unless a concerted effort is made to rein it in—it becomes increasingly hard to avoid adding unnecessary variability, reimplementing variation mechanisms more than once, selecting incompatible or awkward variation mechanisms, and missing required variations. The DoD is therefore facing “sticker shock” since software sustainment costs are unlikely to decrease by shrinking its inventory alone since roughly the same level of software sustainment is still needed, regardless of whether there are 100 or 10,000 hardware platforms.  What the DoD needs are different strategies for understanding and alleviating rising software sustainment costs—such as sustainment strategies based on managing software commonality and variability via software product lines.

In addition to the challenges above, the DoD also faces challenges with recruiting, developing, and retaining its software sustainment workforce.  For example, although the DoD needs efficient and productive software sustainers, this specialty is often not viewed as exciting or innovative as green-field developments, so key research and development challenges remain unresolved. Effective sustainment also requires engineers who have expertise in older languages, operating platforms, and tools combined with deep domain and software architect knowledge.  This combination tends to reside in more experienced members of the DoD workforce, so retaining and replenishing this cadre of software engineers is important. Fortunately, some software engineers dedicate their careers to sustaining software, though their efforts are often not as well recognized (or funded) by the DoD relative to their importance as the inventory of DoD systems continues to age.

This blog posting just scratched the surface of the technical and non-technical challenges associated with sustaining software-reliant DoD systems. Another vexing, non-technical challenge is that DoD contracts often fail to procure source code, necessary licenses and technical data rights, and technical data on design artifacts, testing facilities, and procedures during the acquisition process. The failure to procure this material complicates software sustainment activities and increases total ownership costs over program lifecycles.  The second part of this series describes R&D activities the SEI is conducting to address some of the technical challenges presented above.

Additional Resources:

More information about sustaining software-reliant DoD systems is available below.

To read about the National Research Council’s  Aging of U.S. Air Force Aircraft report, please visit www.nap.edu/catalog.php?record_id=5917.

To read about the Air Force Science Advisory Board’s Sustaining Aging Aircraft report, please visit www.airforce-magazine.com/SiteCollectionDocuments/Reports/2010/December%202010/Day29/Aging_TOR_2011.pdf

To read about the National Research Council’s Critical Code: Software Producibility for Defense report please visit www.nap.edu/openbook.php?record_id=12979&page=R1.

To read the SEI technical report, Sustaining Software-Intensive Systems, please visit
www.sei.cmu.edu/library/abstracts/reports/06tn007.cfm  or to see a presentation on Sustaining Software Intensive Systems – A Conundrum please visit www.dtic.mil/ndia/2005systems/wednesday/lapham.pdf.

Share this

Share on Facebook Send to your Twitter page  Save to del.ico.us  Save to LinkedIn  Digg this  Stumble this page.  Save this page on your Google Home Page 

11 responses to “The Growing Importance of Sustaining Software for the DoD ”

  1. Greg Houlette Says:
    An interesting review on an area of software engineering that I happen to find fascinating. How to keep a system alive and useful when its lifecycle gets extended as well as the actual contingency planning that should be occuring to allow for technology refresh and life extension.
  2. Douglas C. Schmidt Says:
    Greg, I agree this is a fascinating area of software engineering. It's ironic that despite software sustainment's growing impact on the DoD, there's not that much R&D being funded on this topic relative to its importance. There's good news for programmers with years of Jovial and Ada experience, however, since they've got an abundance of job opportunities sustaining legacy DoD software! Keep your eyes open for the second part of my blog posting that will summarize some of the work the SEI is doing on software sustainment methods, processes, and techniques.
  3. James Edmondson Says:
    Hi Doug! A couple things I don't see mentioned here are ongoing additions of security and quality-of-service to product lines or existing legacy systems. Project managers seem to be wanting more network-connected or even internet-connected subsystems that expose legacy and developing systems to security holes--holes which have resulted in some very public intrusions of DoD websites and internet-connected devices (and even software that targets hardware or low level device drivers, as in the Stuxnet worm).

    With these legacy systems becoming networked, there appears to be a need for security maintenance, more focus on quality-of-service to deal with scarce bandwidth and resources, and secure information flow between disparate systems when integrating existing systems into large scale or even ultra large systems, which SEI has outlined as a potential problem in the DoD world. Are security and quality-of-service not as big of problems as we once thought, or are they simply not as important or as big of a drain on DoD project budgets as the other challenges noted in your post? Thanks!
  4. Dave Sharp Says:
    Doug-

    This has been an important and often neglected topic for a long time. Your posting describes well why that will only accelerate. It's past time to increase our focus on it.

    It's important to decompose The DoD Sustainment Problem to effectively tackle it's distinct variants. If I applied some "sustainment problem perspectives" that I read to Microsoft Windows, it would say that all of Microsoft's work after Windows 1.0 in 1985 was "sustainment" ;-). That's always my sense when I read really high metrics like "90% is spent on sustainment".

    I normally think of "sustainment" as being "maintenance". When it is decided that "no further upgrades are going to be made", that marks the beginning of "sustainment". Maintenance is definitely a different business that development, and one that gets neglected. "How to keep a software system running" isn't typically a topic that shows up in course curricula or research agendas. We need more of that.

    "Upgrades" are often also lumped into "sustainment", but very different activities are associated with upgrades than maintenance. Depending on how much is changing in the system, upgrades can be very similar to new system development. They will have additional constraints depending on what is retained. They often benefit from having a good understanding of how the product is intended to be used (e.g., CONOPS) since there are established users. Upgrades have some unique factors, but at least they are much better served by "new development" practices than maintenance activities.

    There are obviously many other variants as well, but these two are some fundamental ones that often I see missed.
  5. Douglas C. Schmidt Says:
    Hey James, you raise some excellent points - assuring security and QoS are major challenges in legacy, new, and planned DoD software-reliant systems. These general topics are addressed (albeit somewhat obliquely) in the "trend" bullet (and associated links) that says:

    . the repurposing of systems to meet new threats and
    increasing requirements for interoperability.

    BTW, the blog has a lot of links to other material that provides additional information on the trends and challenges covered in the posting.

    My second post on this topic will focus on SEI R&D activities that address (among other things) security and QoS issues associated with software sustainment. Speaking of ultra-large-scale (ULS) systems, check out sections 4.6 and 6.6 in the ULS Systems report (www.sei.cmu.edu/uls) for a summary of research challenges associated with security and QoS, which are relevant to this discussion. Thanks, Doug
  6. Malcolm Spence Says:
    Hi Doug, I agree the DOD is not looking at how to sustain software over the long haul. Recently we tried to get access to some software that was five years old, under GPR we thought it would be easy to get and leverage it. We discovered the Prime had negotiated a 20 year ownership for all the code. It is not likely do anything with it but they have effectively precluded anybody else from doing so.

    The DOD framework initiatives such as OACE, FACE etc. help. They can spread the burden of maintenance and new development across many projects, as they all share the same infrastructure.
    regards Malcolm
  7. Douglas C. Schmidt Says:
    Hi Dave, I agree wholeheartedly that sustainment (especially software sustainment in DoD weapons systems) represents the confluence of multiple activities that are related to--but not subsumed by--classic software maintenance and upgrades. Some good sources of further insights on the "sui generis" nature of software sustainment are available at

    http://www.stsc.hill.af.mil/resources/tech_docs/gsam4/chap16.pdf

    and

    http://www.sei.cmu.edu/library/abstracts/reports/06tn007.cfm

    I think those links are included in the blog posting itself, but they may be rather hard to spot without clicking on everything ;-)
  8. James Edmondson Says:
    @Malcolm this also seems to be a trend with recent DARPA programs that involve reusable COTS open-source powered smartphones (Android), POSIX OSes, and standardized interfaces and interdependent, extensible libraries between multiple contributing vendors. So, someone up there seems to be listening! Of course, we still have integration issues but they're continuous throughout the project and encourage communication early--rather than a lot of arguing and posturing towards the final phases of development or right before major demos, which can cut into time documenting, setting up automated regression tests or even finishing up useful features. It will be interesting to see if the next blog post covers a specific approach or best practices to product lines in DoD software that will further encourage high cooperation amongst dozens of development interests to reduce overall costs. Specific, thorough examples of product line application to the DoD domain might really sell this approach better to program managers and principle investigators, which should benefit us all.
  9. Douglas C. Schmidt Says:
    Malcolm, the issues associated with restricted data rights are very thorny and contentious. There's also tension caused by DoD regulations that require a ~50/50 split between organic (government) and contractor (commercial) sustainment support. While these non-technical issues are important, I've focused this--and the subsequent--blog posting more on the technical issues since they are less vexing to address from an R&D perspective.
  10. Rick Schantz Says:
    I think its also important to note that there is a double whammy happening here. The mounting buildup and scale from yesterday's deficiencies in this area is compounded (probably many times over) by continually rising expectations and requirements looking forward for both functionality, speed and accuracy. Both are quite challenging technically by themselves. But hardly as much as recognizing both as simultaneous imperatives, else you can never break out of a non-virtuous cycle. As you point out, too often we see "borrowing" from future prospects/improvements to pay the legacy sustainment bill, or ignoring the legacy requirements to create episodic and islanding effects. Neither adequately address the real underlying need for continually getting better in a smooth trajectory from the techniques of the past to the techniques of the future as a continuum. This whole arena is a moving target, and requirements and solutions need to have both a forward and backward facing aspect, blending the two to be successful in the long haul. Solving yesterday's problem while it resurfaces again in tomorrow's capability is only treading water at best. Maybe that adds some fuel for your part 2 look at these issues.
  11. Douglas C. Schmidt Says:
    Rick, as you allude to above, the challenges the DoD face wrt software sustainment are a microcosm of more fundamental challenges caused by the general lack of understanding/appreciation of the role/value of software in weapons systems. Ironically, despite the fact that DoD weapons systems increasingly depend on software, it's been increasingly hard to motivate investments in long-term software R&D and education. There's excellent (albeit sobering) coverage of the dramatic decline in software-related R&D investments over the past decade in the NRC "Critical Code: Software Producibility for the DoD" report, which is available at http://www.nap.edu/openbook.php?record_id=12979&page=R1.

    The SEI is devoting our R&D resources to address key technology areas identified in that report to help improve software producibility and support DoD practice. We're going to need a concerted effort from the software R&D community and R&D sponsor agencies, however, to reverse the declines of the past decade. The recent round of proposed cuts to defense spending only underscores the importance of raising awareness on the need to improve the quality and rein in the costs of developing and sustaining software for the DoD.

    I'm not sure we'll cover all these topics in part 2 of the software sustainment posting, but you've certainly provided some good fodder for future blog postings!

Add Comment


Leave this field empty: