The Growing Importance of Sustaining Software for the DoD

Architecture Documentation , Common Operating Platform Environments (COPEs) , Secure Coding , Service-Oriented Architecture , Software Product Lines , Software Sustainment , System of Systems , Team Software Process (TSP) , Ultra Large Scale Systems Add comments

Part 2: SEI R&D Activities Related to Sustaining Software for the DoD
By Douglas C. Schmidt,
Deputy Director, Research, and Chief Technology Officer

Douglas C. SchmidtSoftware sustainment is growing in importance as the inventory of DoD systems continues to age and greater emphasis is placed on efficiency and productivity in defense spending. In part 1 of this series, I summarized key software sustainment challenges facing the DoD.  In this blog posting, I describe some of the R&D activities conducted by the SEI to address these challenges.

Primary Sustainment Activities
The term software sustainment is often used synonymously with software maintenance. Sustaining software for the DoD, however, requires attention to certain issues (such as operations and training) that are less essential in commercial software maintenance. There are four primary categories of software sustainment activities:

  • Corrective sustainment diagnoses and corrects software errors after release.
  • Perfective sustainment upgrades existing software to support new capabilities and functionality.
  • Adaptive sustainment modifies software to interface with changing environments.
  • Preventive sustainment modifies software to improve future maintainability or reliability.

SEI Sustainment R&D
The software engineering research community has devised various approaches to improve software sustainment.  For example, tools for detecting software modularity violations help identify eroding design structure (referred to whimsically as “bad code smells”) so the code can be refactored to enhance its sustainability. Likewise, intelligent automated regression testing frameworks help ensure that changes to legacy software work as required and that unchanged parts have not become less dependable.

SEI sustainment strategies. Over the past several decades, the SEI has created methods and guidelines for sustainingmigrating, and evolving legacy systems. For example, the SEI has devised strategies for modernizing legacy systems and reusing legacy components in service-oriented architecture (SOA)-based systems. These strategies employ risk-managed, incremental approaches that encompass changes in software technologies, engineering processes, and business practices.  In addition, the SEI has created techniques for measuring the effectiveness of software sustainment practices.  These techniques can be used to help decision-makers choose a course of continued sustainment, replacement, or selecting which redundant legacy systems to keep and which to retire.

Software product lines. Legacy DoD systems comprise a wide range of software variations, such as network, hardware, and software configurations; different algorithms; and different security profiles. This variation is a key driver of total ownership costs because it impacts the time and effort required to assure, optimize, and manage system deployments and configurations throughout the lifecycle. To manage this variation effectively, the SEI helped pioneer software product lines (SPLs), which have been applied in DoD systems to manage software variation while reusing large amounts of code that implement common features needed within a particular domain.  Software sustainment costs (particularly SPL testing) for an SPL-based family of systems can be reduced because reusable components in the SPL are maintained and validated in one place, instead of separately within each application.

Team Software Process. The Team Software Process (TSP) is another approach pioneered by the SEI that managers and engineers can use to sustain legacy software projects. TSP is a team-centric, time-boxed approach to developing software. By using TSP, organizations can better plan, measure, and improve software development productivity so they have more confidence in sustainment quality and cost estimates. The U.S. Air Force and other DoD and industry organizations have applied TSP successfully to manage software sustainment in large-scale weapons systems for the U.S. Air Force, as well as other DoD and industry organizations.

Software architecture. The SEI has also focused extensively on software architecture, which comprises the structure of the software elements in a system, the externally visible properties of those elements, and the relationships among them. SEI research has shown that a solid understanding of software architecture—and the associated methods, infrastructure, and tools—is essential to modify and improve software-reliant systems correctly, dependably, rapidly, and cost effectively throughout the lifecycle. Likewise, successful sustainment of software-reliant DoD systems requires techniques and tools for evaluating and improving software engineer and manager competence with respect to software architecture, including the following:

  • Understanding, analyzing, and engineering tradeoffs among system properties (such as performance, dependability, and security) that are critical to achieving desired levels of quality in software-reliant systems as they evolve. These properties are quality attributes that determine system viability throughout the sustainment phase.

SEI assessments, workshops, and red teaming. The SEI regularly works with DoD programs to conduct independent technology assessments, reviews, and “red teams” that apply many of the methods and approaches described above to review the planning for—and conducting of—sustainment of DoD systems.  For example, architecture practices such as the Architecture Tradeoff Analysis Method (ATAM) can help DoD programs elicit stakeholder input to identify likely long-term sources of change throughout the sustainment phase.

The SEI’s experience helping DoD programs transition from the production phase of acquisition to the sustainment phase of acquisition indicates that the DoD often focuses on how its contracts and contractors will change rather than on how its program offices will need to change.  The SEI helps acquisition programs plan for these transitions to sustainment and has collected lessons learned from these activities into software acquisition planning guidelines (including Guideline #4: Software Sustainment).  An interesting trend is that DoD programs are increasingly interdependent and interoperable, leading to sustainment interdependencies that require new coordination. To address this need, the SEI developed interoperable acquisition workshops to bring program offices together and draft plans that address sustainment.

Information assurance and software security. Increasing requirements for interdependence and interoperability also yield new challenges for information assurance and software security in legacy systems. In particular, many legacy systems were developed as isolated enclaves.  With the advent of net-centric systems-of-systems, however, these legacy enclaves are being interconnected in ways that subject them to vulnerabilities not anticipated by their original designers.

For example, legacy systems programmed in languages like C may be susceptible to buffer overflows that will not occur until they are connected to a network.  Moreover, maintainers may not resolve these types of vulnerabilities correctly. They might, for instance, simply add input validation to eliminate a particular path to a buffer overview vulnerability rather than remove the out-of-bounds write.

The CERT Secure Coding Team works with developers and maintainers to eliminate these and other types of vulnerabilities by establishing secure coding standards and processes for conformance testing against these standards. Likewise, the CERT Vulnerability Analysis Team can use an analysis of vulnerabilities based on secure coding rule violations to help handle the response. Legacy software systems can also undergo conformance testing against a secure coding standard in the CERT Source Code Analysis Laboratory (SCALe) to detect and eliminate vulnerabilities before the software is deployed. SCALe has also been used by DoD program offices to access the quality of legacy code to inform modernization versus replacement decisions.

Related SEI Blog Posts
SEI researchers have written several blog postings that are relevant to the sustainment of software-reliant DoD systems.  For example, Rick Kazman’s posting on Measuring the Impact of Explicit Architecture Documentation focused on understanding the value of documenting software architectures for complex, software-reliant systems.  Thorough software architecture documentation helps engineers who sustain DoD software understand how they can refactor, maintain, and update the software without introducing new defects or degrading existing capabilities.

Ipek Ozkaya’s posting on Enabling Agility by Strategically Managing Architectural Technical Debt examined how metrics extracted from the code and module structures of software can help repay technical debt, which is a conceptual framework for understanding how and when to defer design choices during the planning or execution of a software project.  Repaying technical debt via refactoring and re-architecting is an effective strategy to alleviate architectural dependencies that impact system-wide architectural rework and minimize software decay during sustainment. 

Steve Rosemergy’s posting on A Framework for Evaluating Common Operating Environments described a framework for exploring the interdependencies among common language, business goals, and software architecture when evaluating the sustainability of proposed software solutions.

We Want to Hear Your Thoughts
This post has just scratched the surface of the solutions that meet the challenges of sustaining software-reliant DoD systems. While the SEI has expertise in methods and tools related to software sustainment, the DoD faces deeper and broader challenges than any one organization (or blog post) can address. We welcome your feedback in the comments section below on ways to improve the technologies and ecosystems needed to sustain DoD software effectively.

Additional Resources:

More information about sustaining software-reliant DoD systems is available below.

To read about software sustainment practices for the DoD, please visit www.stsc.hill.af.mil/resources/tech_docs/gsam4.html, especially chapter 16.

To read about the SEI’s work in software architecture, please visit
www.sei.cmu.edu/architecture

To read about the SEI’s work with the Team Software Process (TSP), please visit
www.sei.cmu.edu/tsp

To read about the SEI’s work in Software Product Lines, please visit
www.sei.cmu.edu/productlines

To read about the SEI’s work in system of systems and SOA, please visit
www.sei.cmu.edu/sos 

To read about the SEI’s work on Ultra-Large-Scale Systems, please visit
www.sei.cmu.edu/uls

To read about the SEI CERT’s work in secure coding, please visit
www.cert.org/secure-coding/

Share this

Share on Facebook Send to your Twitter page  Save to del.ico.us  Save to LinkedIn  Digg this  Stumble this page.  Add to Technorati favorites  Save this page on your Google Home Page 

7 responses to “The Growing Importance of Sustaining Software for the DoD ”

  1. Alex Bell Says:
    As opposed to the DoD *suddenly* recognizing the importance of software sustainment, I see it as more of an "importance revival". Specifically, in my early projects with Hughes Aircraft Company, the consciousness of sustainment being a costly and lengthy activity seemed to be much more predominant than in today's program's. This heightened consciousness was reflected in contractor's willingness to "low ball" the development effort in exchange for the riches of sustainment, customer concerns for having to pay licensing costs of COTS products for many years, the importance of SDDs and other documentation necessary to sustain/maintain software, and the establishment of training programs to enable a customer to self-sustain his investment.

    Today, the contractor seems to be less willing to low ball the development costs because it is difficult to know how long the sustainment phase will last given the rapid pace at which technology is maturing and the possibility of a program becoming irrelevant. I also see considerably less diligence placed into software documentation as I did years ago. I am not exactly sure why this is, but, it could very well be as the result of less importance being placed on the sustainment activity.

    I would like to think that one of the forces behind the "importance revival" of sustainment is the failure of silver bullets that the DoD thought would eliminate the difficulties of developing, integrating, and sustaining software. The UML, XML, OO, SOAP, WSDL, and SOA silver bullets have certainly helped in the realms of reuse, productivity, and time to market for some systems, but, how much have these things really done to reduce the difficulty of developing, integrating, and sustaining large/complex systems? From personal experience, the XML/string based tactics which defeat strong typing at compile time in favor of flexibility at runtime have made my sustainment related efforts much more difficult over the years as opposed to easier. Could it be that the "revival" is partially in response to all of the tech hype not having lived up to expectations and recognition that a return to "old school" tactics might be the most practical for developing large-scale, mission-critical, software-reliant systems? Do I dare hope so?
  2. Douglas Schmidt Says:
    Thanks for raising these good points Alex. Here are some thoughts:

    . It's clear to me and the SEI that software sustainment is of growing importance to the DoD, particularly since legacy systems are remaining
    in service longer than expected, due in part to the risks/costs of developing new software-reliant systems. It's not as clear, however, that the *DoD* recognizes the strategic importance of software, in general, or software sustainment, in particular. More likely, software sustainment costs have simply "snuck up" on the DoD without there being an enterprise strategy for measuring or containing these costs.

    . I don't have enough data points to speculate on whether or not DoD software is documented better or worse than in the past. In the commercial world I have noticed a paucity of software documentation--particularly architecture documentation--which may stem in part from the popularity of automated API documentation tools, such as JavaDoc and Doxygen. These tools make it easy to generate lots of hyperlinked documentation from source code, thereby appearing to absolve developers from providing more detailed documentation of other key software design and architectural views.

    . As you know, I'm a fan of your ACM Queue articles (such as http://queue.acm.org/detail.cfm?id=1142053 and http://queue.acm.org/detail.cfm?id=1053347) that provide a reality check on obsessively applying the fad technologies de jour to large-scale, mission-critical, software-reliant DoD systems. I'm not sure, however, whether the current generation of legacy DoD systems being sustained (many of which were originally developed prior to the 1990s) use much of these newer technologies. Much of what I've seen is still developed in programming languages like Jovial, Ada, and C using structured design and programming methods, which may be a bit too old school for its own good ;-).

    At any rate, sustaining software for the DoD is a complex topic, so I hope we can collectively raise awareness of this important issue and help formulate more effective strategies for developing and sustaining DoD software.
  3. Don O'Neill Says:
    When it comes to large-scale software-intensive programs, it is time to move from aspirational slogans to a disruptive game changer to bring the defense industry into line with the needs of the Department of Defense. The government understands what it needs. Yet industry is not hearing the message. It is time to alter this dynamic by addressing the challenges of competition, innovation, and fixed price contracting in shifting from an expectation of compliance to one of leadership.

    The Department of Defense faces austerity challenges. The government understands what it needs. These needs were best stated by the challenges outlined by Dr. Ashton Carter, Undersecretary of Defense for Acquisition, Technology, and Logistics (AT&L). Clearly stated, these challenges are:
    1.   Target Affordability and Control Cost Growth
    2.   Incentivize Productivity & Innovation in Industry
    3.   Promote Real Competition
    4.   Improve Tradecraft in Services Acquisition
    5.   Reduce Non-Productive Processes and Bureaucracy

    Next Generation Software Engineering (NGSE) is positioned to meet these challenges. The immediate goal of practical NGSE is to drive systems and software engineering to do more with less. Well aligned with the AT&L challenges, its four practical objectives are identified to advance the goal using smart, trusted technologies:
    1.   Drive user domain awareness
    2.   Simplify and produce systems and software using a shortened development life cycle
    3.   Compose and field trustworthy applications and systems from parts [NSS2 05]
    4.   Compose and operate resilient systems of systems from systems

    Both the Department of Defense and the defense industry need to adopt a game changing vision based on competitiveness, innovation, and fixed price contracting. What is needed is a game changing vision:
    1.   To put an end to business as usual and impact the status quo.
    2.   To resurrect global software competitiveness within each enterprise of the defense industry.
    3.   To spark innovation among acquisition managers, systems engineers, and software engineers.
    4.   To adopt a new normal based on fixed price contracting and incentives.
  4. Don O'Neill Says:
    THE CHALLENGE OF COMPETITION
    One of the AT&L challenges is to promote real competition. Currently the defense industry enterprises devote extensive resources and management attention to complying with the Capability Maturity Model Integration (CMMI). The CMMI provides structures to house and control managers. This initiative fosters a culture of compliance but not one of innovation and competitiveness. Despite a two-decade history of capability maturity model improvement, software problems continue to impact defense programs. In addition, the CMMI has not kept pace with contemporary issues, such as, Cyber Security, global supply chain management, and team innovation management. Earlier the CMM and CMMI led from the front and were viewed as necessary and sufficient. Today the CMMI is a lagging indicator that is viewed as necessary but not sufficient. Competition and innovation like process improvement demand continuous improvement.

    In contrast these defense industry enterprises should strive to achieve global software competitiveness characterized by controlling the supplier, controlling the customer, controlling the competition, and controlling threat events.
    1.   Supplier control is achieved by establishing an attractive workplace culture, achieving maturity in process and skills, deepening industry relationships, and retaining personnel.
    2.   Customer control is achieved by deepening customer relationships, balancing business factors, and achieving total customer satisfaction.
    3.   Competitor control is achieved by deepening community relationships, fielding superior products, and setting the direction for the niche.
    4.   Event threat control is achieved by guarding against government intrusion, applying strategic software management, performing due diligence, and understanding reality.

    Operationally, the stages of competitiveness include make and sell, sense and respond, and anticipate and lead.
    1.   In make and sell the goals are to achieve process efficiency and deliver quality products. This is the current state to which the defense industry aspires.
    2.   In sense and respond the goals are to listen to the voice of the customer and to deliver satisfying solutions. Too often this is a failed state.
    3.   In anticipate and lead the goals are to understand the deep needs of the customer and to deliver transforming innovations. This represents the game changing state to which defense industry enterprises need to aspire and for which the Department of Defense needs to structure incentives to achieve.

    THE CHALLENGE OF INNOVATION
    Another of the AT&L challenges mentioned incentives for innovation. Dr. William Brody, former President of Johns Hopkins University, stated, "The calculus of innovation is really quite simple: knowledge drives innovation, innovation drives productivity, productivity drives our economic growth.” Innovation occurs at the intersection of invention and insight. It is not just something new; it is not just the inventiveness. To borrow criteria from the USPTO, innovation involves applying creativity and inventiveness in ways that are novel, useful, and a non-obvious extension of prior art.

    How is innovation achieved? An organization can get lucky or it can be good. In getting lucky, ideas originate from the producer, and changes are directional, that is, moving in the direction the producer is already traveling. In being good, ideas originate in the cross discipline collaboration and culture clash between producer and consumer, and changes are intersectional, that is, moving in a new direction under the combined influence of both producer and consumer. These changes are transformational.

    Since software is the carrier for innovation, an unmet need involves systematically sparking intersectional ideas between systems engineers and software engineers. However, traditional program culture, organizational structures, and supply chain management practices erect barriers and obstacles that interfere with this opportunity.

    THE CHALLENGE OF FIXED PRICE
    The Department of Defense needs to ensure that defense industry senior executives are committed to meeting the AT&L challenges and are accountable for demonstrating game changing progress towards solving these challenges.
    For example, the most significant game changer a defense industry senior executive can deliver is a commitment to accept fixed price contracts on large software-intensive programs along with a convincing capability to deliver that reflecting an understanding of the cultural changes required. Both the Department of Defense and the defense industry need to populate a tool kit of capabilities for successfully engaging in fixed price contracts and for evaluating the challenges and benefits of doing so.

    Reluctance to accept fixed price contracts within the defense industry community is based on risk and fear of failure in cost, schedule, and quality performance. This reluctance can be offset by Department of Defense incentives based on technical performance measures designed to tilt the risk calculation in favor of fixed price for those capable of delivering.
  5. Douglas C. Schmidt Says:
    Hi Don,

    Thanks for your comments. I agree that the status quo in software-reliant DoD acquisition programs is problematic. I don't see much fundamental/pervasive changes occurring, however, until the following three conditions hold:

    • There's a broad awareness/consensus within the DoD that conventional software processes, methods, and technologies are inadequate--both financially and technically--to achieve DoD mission goals given likely financial and procurement constraints. While AT&L leadership understands the overall challenges of acquisition programs, the nuances of the software-related aspects are not as well understood throughout key levels of the DoD.

    • There is a critical mass of enlightened/empowered leaders, decisionmakers, and program managers throughout the DoD who understand the main technical and programmatic challenges of software-reliant systems, are charismatic enough to motivate changes in the status quo, and forceful/commited enough to push for the necessary changes. Again, significant challenges must be overcome by DoD leadership stemming from a decade or more of neglect in strategic software R&D investments.

    • There is a critical mass of talented software engineers who have a deep understanding of the legacy and emerging technologies needed to create, secure, and sustain working software-reliant components, subsystems, and system products. The US faces a shortage of software researchers and developers with US citizenship who can help develop world class science, technology, engineering, and mathematics capabilities for the DoD and the USA.

    A good source of material that discusses these and related issues in more detail is available in the NRC "Critical Code" report available at http://www.nap.edu/openbook.php?record_id=12979&page=R1. I also gave a presentation for Congress a couple years ago when I was a professor at Vanderbilt University that summarizes the impact of increasing software scale and complexity on DoD operations. Check out http://www.dre.vanderbilt.edu/~schmidt/congress-talk.pdf for the material I covered in that talk.

    Thanks,

    Doug
  6. Mike Konrad Says:
    Hi Don,

    Doug has already responded more broadly to your postings; I’ll focus my response on clarifying how CMMI and other SEI products help address the challenges of software sustainment (in particular) and software innovation and competiveness (in general).

    Since the late 2000’s, the scope of model products offered by the SEI has expanded from product development only (CMMI-DEV) to a much larger set of products. This set of products includes models that cover not only product and service development, but also service management and delivery (CMMI-SVC) and supplier and partner management (CMMI-ACQ) as well as workforce/team management and innovation (P-CMM), and improving resilience (e.g., cyber security) (CERT-RMM).

    This larger set of products also includes the Smart Grid Maturity Model (SGMM), which helps utilities repurpose and transition their existing grid assets toward realizing a smart grid (a metaphor for the DoD software sustainment challenge of repurposing and transitioning current assets to meet increasing requirements for interoperability and security). This set of products also includes the TSP (which Doug already mentioned), the Accelerated Improvement Method (AIM), some new modeling work on Data Management, and measurement and analysis research, tools and methods, and training.

    The latest version of CMMI (Version 1.3) covers the following topics relevant to software sustainment, innovation, and competitiveness: 1) quality attributes and architecture; 2) identifying and analyzing innovations that address performance shortfalls in meeting mission/business objectives; 3) approaches to Agile and CMMI that reinforce the positive benefits of both while facilitating closer customer and contractor cooperation; 4) product line approaches; and 5) the transition of products and services to operations (e.g., in support of technology maturation or retirement).

    All CMMI models share a common core that focuses on project/work group management, process management, and support (which includes practices for decision analysis and resolution, measurement and analysis, and managing innovation), thus simplifying multi-model use, which is increasingly important because sustaining software requires taking on all the roles mentioned (product/service acquirer, developer, and provider).

    On the issue of compliance, as many users of SEI models have learned, the real benefits from using a maturity/management model lies in making the most of the added capabilities to improve mission/business performance. For example, four studies (two published by the SEI, one by DACS, and one by NDIA) summarize competitiveness-related benefits from CMMI adoption. The following are a few examples of these benefits:
    ·   The median of productivity improvement across the organizations in the study improved 62%. http://www.sei.cmu.edu/library/abstracts/reports/06tr004.cfm
    ·   Better strategic decision making (e.g., about business growth or profitability) was reported by 83% of high maturity respondents. http://www.sei.cmu.edu/library/abstracts/reports/10tr022.cfm
    ·   Warner Robins Air Logistics Center Software Engineering Division, which primarily develops software in support of test and maintenance, reported during the six months of the study: no late deliveries, no defects in fielded software, and dramatic reductions in cost and schedule variances. http://seir.sei.cmu.edu/cmmiresearch/results/pdfs/DACS-Software-Tech-News-Mar-07.pdf
    ·   In one organization, the costs for repair were reduced by 57% and there was 6.35 times less risk of cost or schedule impact late in the program. http://www.sei.cmu.edu/library/abstracts/presentations/CMMI-Benefits-to-Defense-Industry.cfm

    A recent study by Erik Brynjolfsson, Lorin M. Hitt, and Heekyung Hellen Kim titled “Strength in Numbers: How Does Data-Driven Decisionmaking Affect Firm Performance?”(April 22, 2011) draws the conclusion “firms that adopt [a data-driven decision-making approach] have output and productivity that is 5-6% higher than what would be expected given their other investments and information technology usage.” http://ssrn.com/abstract=1819486

    Although it isn’t possible to determine which of the organizations used in the study used CMMI, you might expect similar benefits to be experienced by CMMI adopters on the basis of this study because all CMMI models cover these same topics.

    Collectively, SEI maturity/management models and related products provide a comprehensive foundation for process improvement that addresses many of the challenges of software sustainment, innovation, and competitiveness. Thus, I see CMMI Version 1.3 and related SEI products as helping “lead the way.”

    Mike Konrad, CMMI Chief Architect
  7. W. G. Coberly Says:
    In the field, working on a very large-scale software-intensive program. I need an SEI point-of-contact that can address the state-of-the-practice regarding the objective/quantitative understanding of the quality of software components integrated to form a software system.

    In my context, we are integrating 80% COTS/GOTS (with no objective/quantitative understanding of the quality) and the rest is NRE. To understand scope, and understanding ESLOC, we estimate our effort to be 15 million SLOC.

    Being that in my research I find no data regarding a deterministic method for understanding the quality in an integration context, I post my question here.

    Thanks

Add Comment


Leave this field empty: