High Maturity Software Engineering Measurement and Analysis

CMMI , Measurement & Analysis , Team Software Process (TSP) Add comments

By Dave Zubrow, Manager
Software Engineering Measurement and Analysis Initiative

Dave ZubrowThe SEI has been actively engaged in defining and studying high maturity software engineering practices for several years.  Levels 4 and 5 of the CMMI (Capability Maturity Model Integration) are considered high maturity and are predominantly characterized by quantitative improvement. This blog posting briefly discusses high maturity and highlights several recent works in the area of high maturity measurement and analysis, motivated in part by a recent comment on a Jan. 30 post asking about the latest research in this area. I’ve also included links where the published research can be accessed on the SEI website.

At CMMI level 3, work is proactively managed and standard processes are used. Beyond level 3, process performance needs to be understood quantitatively. High maturity means you have the data to understand how the process is performing, how variation in the implementation and execution of the process affect performance, and what the likely costs and benefits of any change will be. A high-level description of the benefits, by process area, is shown below.

High Maturity Chart

In past years some CMMI users said they felt high maturity was not well-defined, an issue addressed by its clarification in CMMI v1.3. The CMMI community has also debated the benefits of moving up to high maturity, and asked for more examples of high maturity process implementations. Some challenges organizations face when striving for high maturity include developing an insightful set of measures, creating predictive models for process performance, project management, and product quality, and knowing which tools and methods to use for modeling and analysis. The SEI has worked to address these concerns and provide needed resources through courses, case studies, and other publications about the implementation of high maturity practices.

Publications defining and describing high maturity measurement and analysis practices include:

CMMI and TSP/PSP: Using TSP Data to Create Process Performance Models
By Shurei Tamura
This report describes the fundamental concepts of process performance models (PPMs) and describes how they can be created using data generated by projects following the Team Software Process (TSP). PPMs provide accurate predictions and identify factors that projects and organizations can control to better ensure successful outcomes, helping organizations move from a reactive mode to a proactive, anticipatory mode. PPMs are fundamental to the implementation of the high maturity process areas of CMMI and are specifically required in the Quantitative Project Management and Organizational Process Performance process areas. The three examples in this report demonstrate how data generated from projects using TSP can be combined with data from other sources to produce effective PPMs.

Approaches to Process Performance Modeling: A Summary from the SEI Series of Workshops on CMMI High Maturity Measurement and Analysis
Robert W. Stoddard & Dennis R. Goldenson
Organizations are increasingly striving for and achieving high maturity status, yet there is still an insufficient shared understanding of how best to implement measurement and analysis practices appropriate for high maturity organizations. A series of twice-yearly workshops organized by the SEI allows organizations to share lessons learned to accelerate the adoption of best measurement and analysis practices in high maturity organizations.

This report summarizes the results from the second and third high maturity measurement and analysis workshops. The participants' presentations described their experiences with process performance models; the goals and outcomes of the modeling; the x factors used; the data collection methods; and the statistical, simulation, or probabilistic modeling techniques used. Overall summaries of the experience and future plans for modeling also were provided by participants.

CMMI High Maturity Measurement and Analysis Workshop Report: March 2008
Robert W. Stoddard II, Dennis R. Goldenson, Dave Zubrow, & Erin Harper
Organizations are increasingly looking for guidance on how to implement CMMI high maturity practices effectively and how to sustain their momentum for improvement. As high maturity organizations work to improve their use of measurement and analysis, they often look to examples of successful implementations for guidance. In response to the need for clarification and guidance on implementing measurement and analysis in the context of high maturity processes, members of the SEI’s Software Engineering Measurement and Analysis (SEMA) initiative organized a workshop at the 2008 SEPG North America conference to bring leaders in the field together at a forum on the topic. Other workshops will be held as part of an ongoing series to allow high maturity organizations to share best practices and case studies.

The following reports describe results from surveys conducted related to the implementation and impacts of high maturity practices.

Performance Effects of Measurement and Analysis: Perspectives from CMMI High Maturity Organizations and Appraisers
By James McCurley & Dennis R. Goldenson
This report describes results from two recent surveys conducted by the SEI to collect information about the measurement and analysis activities of software systems development organizations. Representatives of organizations appraised at maturity levels 4 and 5 completed the survey in 2008. Using a variant of the same questionnaire in 2009, certified high maturity lead appraisers described the organizations that they had most recently coached or appraised for the achievement of similar high maturity levels. The replies to both surveys were generally consistent even though the two groups are often thought to be quite different. The results of the surveys suggest that the organizations understood and used CMMI-based process performance modeling and related aspects of measurement and analysis a great deal. Both the organizational respondents in 2008 and the appraisers in 2009 reported that process performance models were useful for the organizations.

The respondents in both surveys also judged that process performance modeling is more valuable in organizations that understood and used measurement and analysis activities more frequently and provided organizational resources and management support. In addition, results from the 2009 survey of lead appraisers indicate that organizations that achieved their appraised high maturity level goals also found measurement and analysis activities more useful than those organizations that did not achieve their targets.

Use and Organizational Effects of Measurement and Analysis in High Maturity Organizations: Results from the 2008 SEI State of Measurement and Analysis Practice Surveys
By Dennis R. Goldenson, James McCurley, and Robert W. Stoddard II
There has been a great deal of discussion about what organizations need to attain high maturity status and what they can reasonably expect to gain by doing so. Clarification is needed along with good examples of what has worked well and what has not. This clarification is particularly needed with respect to measurement and analysis. This report contains results from a survey of high maturity organizations conducted by the SEI in 2008. The questions center on the use of process performance modeling in those organizations and the value added by that use. The results show considerable understanding and use of process performance models among the organizations surveyed; however there is also wide variation in the respondents’ answers. The same is true for the survey respondents’ judgments about how useful process performance models have been for their organizations. As is true for less mature organizations, there is room for continuous improvement among high maturity organizations. Nevertheless, the respondents’ judgments about the value added by doing process performance modeling also vary predictably as a function of the understanding and use of the models in their respective organizations. More widespread adoption and improved understanding of what constitutes a suit-able process performance model holds promise to improve CMMI-based performance outcomes considerably.

We hope you find this information useful. If there are any other areas of SEI research that you would like us to highlight, please leave a comment below.

Additional Resources:

An article in the January/February 2012 issue of CrossTalk, The Journal of Defense Software Engineering, titled "High Maturity - The Payoff" discusses the value and benefits realized by organizations that adopt high maturity CMMI Level 4 and 5 software processes.

Measuring the Software Process: Statistical Process Control for Software Process Improvement
By William A. Florac & Anita D. Carleton
This book was one of the first published works that explicitly addressed how to use statistical process control methods to manage and improve software processes within an organization.  It remains a key reference for how to implement high maturity measurement and analysis.  It explains how quality characteristics of software products and processes can be quantified, plotted, and analyzed, so that the performance of software development activities can be predicted, controlled, and guided to achieve both business and technical goals.

To learn more about the SEI’s work in Measurement and Analysis, please visit


Share this

Share on Facebook Send to your Twitter page  Save to del.ico.us  Save to LinkedIn  Digg this  Stumble this page.  Save this page on your Google Home Page 

2 responses to “High Maturity Software Engineering Measurement and Analysis”

  1. Rajeev Says:
    Thank you for the insights Dave. Would you be able to additionally shed some light on the tooling requirements mandated by CMMI level 5 for the following work areas:

    - Risk Management
    - Defect Management
    - Project Management

    Looking forward. Thank you!
  2. Dave Zubrow Says:
    CMMI levels 4 & 5 include concepts related to process performance baselines and models (OPP), quantitative project management (QPM), corrective action resolution (CAR) and organizational performance management (OPM). Although the CMMI does not mandate the use of tools nor promote use of specific vendor tools, aspects of the high maturity concepts applied within many areas of a business most likely become practical with analytical and other tooling. Tools can help to generate and collect the data needed to implement the high maturity practices in the context of the processes implemented within the organization. You might think about the effectiveness of your risk management, trends in defect detection and removal, and the performance of your projects against their plans. CMMI high maturity may impart a new lens through which these topics are understood and practiced. For example, risk management may become more sensitive to the range of likelihood assessed to risks, or the range of impact that may occur if the risk materializes into an issue. Some tooling such as spreadsheets and/or Monte Carlo simulation tools may provide practical and effective solutions to account for uncertainty in the risk assessment. With respect to defect management, some tooling might prove useful to identify systemic trends using factor analysis, cluster analysis or time series analysis, such that the CMMI high maturity thinking about the resolution of whole categories of defects or system issues may occur. With respect to project management, some tooling might prove useful in addressing and modeling the uncertainty of critical resource availability, schedule duration uncertainty, or budget uncertainty may be addressed with statistical or simulation modeling tools. In summary, CMMI high maturity truly embodies a paradigm shift in thinking which includes anticipating and predicting future performance outcomes, reacting to signals of real issues rather than noise of the process, and solving entire classes of issues rather than continuing to firefight issues one at a time.

Add Comment

Leave this field empty: