Estimate Bias / Estimate Validation

Of all the systemic risks one must rate in the ValidRisk parametric tool, the bias of the base estimate and schedule are the most challenging to rate objectively. Users need to understand that cost growth and schedule slip (i.e., contingency usage) depend in large part on how much “allowance” for risk and uncertainty have already been built into (or buried in) the base cost and duration estimates by the team or contractor. The allowance may be visible (e.g., allowance for waste) or hidden (e.g., padded unit costs or activity durations). Best practice is to review the visible allowances and to “validate” the base estimate and schedule against internal and/or external benchmark metrics for the net result of the visible and hidden biases. If the company has a project historical database, validation capability is usually built into that tool. Validation provides an objective indicator of bias and arguably should be the first step in every quantitative risk analysis. AACE provides RP 110R-20 for performing cost estimate validation.

Note that every base estimate and schedule is biased; it is only a question of which way and by how much. AACE recommends in RP 34R-05 that the Basis of Estimate document define the “cost estimate strategy” which is in essence a planned bias. For example, if the company wants to achieve continuous performance improvement, a target should be set; for example, one might target a base estimate that represents the p25 of historical performance. Using “average” historical metrics as a base embeds past mediocre performance into one’s control baseline. That may result in predictable outcomes (i.e., set the budget high enough and one will always underrun) but it is uncompetitive and uneconomical. One should be able to see the importance of having a project historical database; it is part of a broader risk quantification process.

In addition to validating each estimate, one can perform a study of a sample of company projects to see if there are persistent biases in the company’s general estimating and schedule process. For example, if the company’s mean cost growth is always several percent less than predicted by the model, this is likely an artifact of not rating the base estimate bias correctly (i.e., not an issue with the model, but with one’s ratings). The ValidRisk partners can support calibration studies and perform estimate review and validation services.