Quantifying Uncertainty in Early Lifecycle Cost Estimation (QUELCE): An Update

Measurement & Analysis , Software Cost Estimates Add comments

By Dave Zubrow,
Chief Scientist
Software Engineering Process Management Program

Dave ZubrowBy law, major defense acquisition programs are now required to prepare cost estimates earlier in the acquisition lifecycle, including pre-Milestone A, well before concrete technical information is available on the program being developed. Estimates are therefore often based on a desired capability—or even on an abstract concept—rather than a concrete technical solution plan to achieve the desired capability. Hence the role and modeling of assumptions becomes more challenging.  This blog posting outlines a multi-year project on Quantifying Uncertainty in Early Lifecycle Cost Estimation (QUELCE) conducted by the SEI Software Engineering Measurement and Analysis (SEMA) team. QUELCE is a method for improving pre-Milestone A software cost estimates through research designed to improve judgment regarding uncertainty in key assumptions (which we term program change drivers), the relationships among the program change drivers, and their impact on cost.

Our Approach

According to a February 2011 presentation by Gary Bliss, director of Program Assessment and Root Cause Analysis, to the DoD Cost Analysis Symposium, unrealistic cost or schedule estimates are a frequent causal factor for programs breaching a performance criterion.  Steve Miller, director of the Advanced Systems Cost Analysis Division of OSD Cost Analysis and Program Evaluation, noted during his DoDCAS 2012 presentation that “Measuring the range of possible cost outcomes for each option is essential …Our sense is not that the cost estimates were poorly developed [but] rather key input assumptions didn’t pan out.”  For instance, an estimate might assume

  • It is possible to mature technology A from technology readiness level 4 to level 7 in three years.
  • The program will not experience any obsolescence of parts within the next five years.
  • Foreign military sales will support lower production costs.
  • An interdependent program will complete its development and deployment in time for this program to use the products.
  • We can reuse 70 percent of the code in the missile tracking system.

QUELCE addresses the challenge of getting the assumptions “right” by characterizing them as uncertain events rather than certain eventualities.  As we’ve noted previously, modeling uncertainty on the input side of the cost model is a hallmark of the QUELCE method. By better representing uncertainty, and therefore risk, in the assumptions and explicitly modeling them, DoD decision makers, such as Milestone Decision Authorities (MDAs) and Service Acquisition Executives (SAEs), can make more informed choices about funding programs and portfolio management. QUELCE is designed to ensure that DoD acquisition programs will be funded at levels consistent with the magnitude of risk to achieving program success, fewer and less severe program cost overruns will occur due to poor estimates, and there will be less rework reconciling program and OSD cost estimates.

QUELCE relies on Bayesian Belief Network (BBN) modeling to quantify uncertainties among program change drivers as inputs to cost models. QUELCE then uses Monte Carlo simulation to generate a distribution (as opposed to a single point) for the cost estimate. In addition, QUELCE includes a DoD domain-specific method for improving expert judgment regarding the nature of uncertainty in program change drivers, their interrelationships, and eventual impact on program cost drivers. QUELCE is distinguished from other approaches to cost estimation by its ability to

  • allow subjective inputs based solely on expert judgment, such as the identification of program change drivers and the probabilities of state changes for those drivers, as well as empirically grounded ones based on historical data, such as estimated system size and likely growth in that estimate
  • visually depict influential relationships, scenarios, and outputs to aid team-based development, and explicit description and documentation of assumptions underlying an estimate
  • use scenarios as a means to identify program change drivers, as well as the impacts of alternative acquisition strategies, and
  • employ dependency matrix  transformation techniques to limit the combinatorial effect of multiple interacting program change drivers for more tractable modeling and analysis

The QUELCE method consists of the following steps in order:

  1. Identify program change drivers: workshop and brainstorm by experts.
  2. Identify states of program change drivers.
  3. Identify cause-and-effect relationships between program change drivers, represented as a dependency matrix.
  4. Reduce the dependency matrix to a feasible number of inter-driver relationships for modeling, using matrix transformation techniques.
  5. Construct a BBN using the reduced dependency matrix.
  6. Populate BBN nodes with conditional probabilities.
  7. Define scenarios representing nominal and alternative program execution futures by altering one or more program change driver probabilities.
  8. Select a cost estimation tool and/or cost estimating relationships (CERs) for generating the cost estimate.
  9. Obtain program estimates of size and/or other cost inputs that will not be computed by the BBN.
  10. For each selected scenario, map BBN outputs to the input parameters for the cost estimation model and run a Monte Carlo simulation.

Improving the Reliability of the Expert Opinion

Early cost estimates rely heavily on subject matter expert (SME) judgment, and improving the reliability of these judgments represents another focus of our research. Expert judgment can be idiosyncratic, and our aim is to try to make it more reliable. QUELCE draws upon the work of Dr. Douglas Hubbard, whose book How to Measure Anything describes a technique known as “calibrating your judgment” that we are adapting for our DoD cost estimation analysis.

For example, if you state you are 90 percent confident, you should be correct in your answers 90 percent of the time.  If you state you are 80 percent confident, you would be correct 8 times out of 10.  Performing in agreement with your statement of confidence is termed “being calibrated.” 

Hubbard’s technique operates by giving participants a series of questionnaires.  The participants are asked to provide an upper and lower bound for the answer to each question such that they believe they will be correct 90 percent of the time. Hence, a participant should get 9 out of 10 answers right.  If they answer all 10 correctly, they are being too conservative in their answers; they provided too wide of a range.  If they get fewer than 9 correct, they are over confident and providing too narrow of a range for their answers. Hubbard’s approach provides feedback so that participants are consistently correct 90 percent of the time.  Through this method of testing and feedback, they learn to calibrate their judgment.

Applying that same approach to DoD cost estimation analysis would ideally mean that if two calibrated judgments are being applied to the same cost estimate, there is now a more precise idea of what those judgments mean. Hubbard, who taught a class at the SEI, demonstrated that most people start off being highly over confident in terms of their knowledge and judgment.

We plan to test Hubbard’s approach of calibrating judgment with questions specific to software estimating at several universities, including Carnegie Mellon University and the University of Arizona. To develop the materials for these experiments, we are mining information from open-source repositories, such as Ohloh.net.  Our objective is to increase the consistency and repeatability of expert judgment as it is used in software cost estimation.

Addressing Challenges

A key challenge that our team faces in conducting our research is validating the QUELCE method.  It can literally take years for a program to reach a milestone against which we can compare its actual costs to the estimate produced by QUELCE.  We are addressing this challenge by validating pieces of the method through experiments, workshops, and retrospectives.  We are currently conducting a retrospective on an active program that provided us access to its historical records.  Key to this latter activity is the participation of team members from the SEI Acquisition Support Program (ASP).   The ASP members are playing the role of program experts as we work our way through the retrospective.

Another challenge that our work on QUELCE addresses is insufficient access to DoD information and data repositories may significantly jeopardize our ability to conduct sufficient empirical analysis for the program change driver repository. To address this, we have been working with our sponsor and others in the Office of the Secretary of Defense to gain access to historical program data stored in a variety of repositories housed throughout the DOD.  We plan to use this data to develop reference points and other information that will be used by QUELCE implementers as a decision aid when developing the BBN for their program.  This data would also be included in the program change driver repository.

Developing a Repository

We are creating a program change driver repository that will be used as a support tool when applying the QUELCE method. The repository is envisioned as a source of program change drivers—what events occurred during the life of a program that directly or indirectly impacted its cost—along with their probability of occurrence.  The repository will also include information that will be used as part of the method for improving the reliability of expert judgment such as reference points based on the history of Mandatory Procedures for Major Defense Acquisition Programs

Developing the repository is a major task planned for FY13.  We also plan to conduct additional pilots of the method including use of the repository and support tools. From those pilots, we will develop guidance for the use of the repository and make it available on a trial basis within the DoD.  After the repository is adequately populated and developed, we intend it to become an operational resource for DoD cost estimating.

Transitioning to the Public

During the coming year, our SEMA team will work to

  • create guidance and procedures on how to mine program change relationships and related cost information from DoD acquisition artifacts for growth of the program change driver repository
  • collaborate with Air Force Cost Analysis Agency to include results from analyzing Software Resources Data Report data in the program change driver repository
  • assemble a catalog of calibrated mapping of BBN outputs to cost estimation models and make it available to the DoD cost community
  • continue discussions with Defense Acquisition University (DAU), Service Cost Centers, and the DoD cost community about research and collaboration opportunities (for example, discussions at the DoD Cost Analysis symposium)

Additional Resources

To read the SEI technical report Quantifying Uncertainty in Early Lifecycle Cost Estimation (QUELCE), please visit www.sei.cmu.edu/library/abstracts/reports/11tr026.cfm.

For more information about Milestone A, please see the Integrated Defense Life Cycle Chart for a picture and references in the “Article Library.”


Share this

Share on Facebook Send to your Twitter page  Save to del.ico.us  Save to LinkedIn  Digg this  Stumble this page.  Save this page on your Google Home Page 

6 responses to “Quantifying Uncertainty in Early Lifecycle Cost Estimation (QUELCE): An Update”

  1. Kathy Smith Says:
    While this method is specifically directed towards Acquisition, it could be applied to any large proposal or bid made early in the lifecycle. Good information on how to improve predictability of models that use judgement as an input.
  2. Bob Ferguson Says:
    Ms. Smith,
    Thank you for the comment.
    You are correct. The method is applicable to any large proposal, even an internal development proposal. Not only does the method provide different ranges based on scenario selection so that we can improve the decisions. It also makes many assumptions more visible.

    In our experiments (both live and retrospective) organizations have seen immediate payback. In one instance the scenario analysis exposed some assumptions that caused the company to make an immediate change to the proces in order to reduce the risk (mitigation). In many cases the mitigation can be performed prior to the bid.

    Calibration tests of expert judgment and the repository concept have been interesting as well. We should have some articles about interim results by this fall.
  3. James Passaro Says:
    Hello, i work for the Space and Missile Systems Center (SMC) Developmental Planning Directorate (SMC/XR), and our office is intrigued by your paper and the QUELCE tool/process. SMC/XR is a unique organization responsible for incubating and transitioning new space system acquisitions from the technology phase to the formal acquisition program start-up phase. As such there may be a synergy in collaboration. Please contact me if you are interested in further discussion. Thank you! JP
  4. Dave Zubrow Says:
    James, Thank you for the comment. I will be in touch.
  5. Donald Beckett Says:
    While I commend your desire to improve early lifecycle estimation of large government systems, I believe your approach will only yield modest improvement at best. It is fine to train estimators by the Hubbard Technique to be more accurate in determining their uncertainly ranges. However, early in the lifecycle one does not even know what all the factors are that require uncertainty ranges, nor much less how these interact. Rather than trying to identify all of the program change drivers, which is impossible, why not use the history of government projects to help determine a confidence range. For example, if the history of large programs indicates that they are typically underestimated by 25 to 50%, come up with your best estimate and add 25 to 50% to it. This may sound too simple; but it has a sounder empirical basis than trying to identify all of the critical factors and their interactions of a program that is still in the early planning stage.
    Thank you, Don
  6. Dave Zubrow Says:

    Thanks for the response. I will address a couple of points in your comment. First, we need not identify all of the program execution change factors. One of our goals, however, is to develop a catalog of these factors based on subject matter expert input and historical experience. We believe this information can be fruitfully utilized and modeled early in a program’s life. Second, and perhaps more directly to your comment and question, we want to move the recognition of uncertainty to the input side of the cost estimation equation rather than simply tacking on a range after the estimate has been produced. This has several benefits. It makes the assumptions about the magnitude and sources of uncertainty explicit for the current estimate rather than relying solely on analogies. This, in turn, allows them to be discussed and debated among the estimation team. Further, the model can then be adjusted to represent different acquisition and execution scenarios and strategies. We have observed that the modeling exercise itself provides value and insight. One of the virtues of the method is that it taps both subjective expertise and historical data. We do not agree that this makes the method less empirically valid. Rather, it is maximizing the use of relevant information regardless of where it is found. Finally, we advocate producing the estimate as a distribution so decision makers get a clear depiction of the risk associated with any specific dollar amount. Our method, as do some others, does this. In the end, we are both trying to recognize and convey the uncertainty and risk associated with an estimate. The difference is how we get there.

Add Comment

Leave this field empty: