A Decade of Project Management
Martin Dean Martin
John O. Lenz
William L. Glover
A major problem facing program managers responsible for Department of Defense
(DOD) weapon systems development programs today is how to effectively predict
and ultimately control and manage program cost growths. During the past ten to
fifteen years, cost growths have plagued major development programs. There are
many tools available for managers to use in estimating program costs, but most
of the methods used do not consider the uncertainty associated with the
successful completion of' a program in any rigorous or formal manner.
The specific inclusion of uncertainty in estimating program costs formed the
basis of a recent research effort by the authors while assigned to the Air
Force Institute of Technology (AFIT), Wright-Patterson Air Force Base, Ohio.'
The purpose of the research effort was to validate an entropic cost model for
use in predicting and controlling the final cost of weapon system development
programs in the DOD. The model was originally formulated under Air Force
sponsorship at the University of Oklahoma. A brief background will facilitate
an understanding of the model.
To understand the entropic cost model, a few points must be introduced
relative to the acquisition environment and the characteristics of
information.
There are two phases of a development program which relate to the potential
of a cost growth: pre-award and post-award (See Figure 1). For the entropic
cost model, the critical point in time is the contract award for a
development program.
During the pre-award phase (before actual award of the contract) the program
manager is primarily concerned with influencing future cost growth of his
program. To accomplish this task the program manager (PM) has at his disposal
certain information which should permit him to structure his decisions in a
rigorous manner at the time of contract award. This information includes
technical data, cost estimates, and results of risk analysis. Technical data
consists of engineering estimates and feasibility studies conducted by either
the government or the contractor. Cost estimates are available in four
principal forms: Cost Analysis Improvement Group (CAIG) estimates;
Independent Cost Estimates (ICE); estimates made by personnel organic to the
Systems Program Office (SPO); and finally, estimates in contractor proposals.
Uncertainty analysis is relatively new as far as being a formal and integral
part of the PM's information base; that is, only recently have serious
efforts been made to formalize and structure the process of uncertainty
analysis during the development program pre- award phase.
During the post-award phase (See Figure 1), the PM must monitor control
systems and act to preclude a program cost growth based on his own expertise
and that of his subordinates in the SPO. The information available to the PM
has certain characteristics that are central to the fundamental concepts of
the entropic cost model.
The universe of program information relative to the development of a weapon
system is comprised of two subsets: ordered information and information that
lacks order. Ordered information relates to factors of the program which
appear relatively certain as to their ultimate outcome (See Figure 2). These
factors generally form the basis for the target (the theoretically "most
likely") cost of' the program at contract award. The information in a program
which lacks order relates to aspects of a program with uncertain outcomes and
form the basis for cost growth during development and possibly production. A
conceptual visualization of the SPO information base is illustrated in Figure
2.
The lack of order, or uncertainty, in program information forms the
foundation for the entropic cost model. In the terminology of set theory, it
is the complement of the ordered portion of the information base. For the
entropic model, the uncertainty of information is conceptualized as
approximately equal to entropy, or the disorder, in the program system of the
PM and his information. The concept of entropy has its roots in
thermodynamics. Entropy in a physical system is the amount of disorder
present in the system due to molecular state changes when heat is applied.
This property of disorder was extended to information systems in the
development of information and communication theory by Shannon and Weaver.
The concept was used in an attempt to explain noise in communication
systems.2 An extension of the concept forms the basis for the entropic cost
model.
The entropic cost model is formulated as follows:
Validation of the model using the SRAM required a re-creation of the program
information environment in existence immediately subsequent to the award of
the development contract. A review of program documentation disclosed that
the data necessary to reconstruct the environment was contained in the
source-selection documentation file. This file was not readily available as a
result of the document security required by the source-selection process. An
alternative method was required to reconstruct the information base for the
test of the entropic cost model.
The limited access to documentation caused an additional secondary objective
to be added to the research. That was, in essence, to develop a method by
which the uncertainty of information could he quantified in a structured
manner. The method needed to not only satisfy accepted research tech- niques,
but be practically useful, if possible, in the realm of the PM and his
decisions.
The work of C. Jackson Grayson in his organization of expert judgment and its
application to oil well drilling decisions was known to the researchers. The
technique rested largely on the use of probability statements as responses to
questions of well-drilling experts relative to the potential success of the
drilling operation in terms of oil production. Specifically, Grayson
requested geologist faced with making decisions relative to the location of
oil-bearing formations to develop probability distributions as to the
probable success of drilling at a specific location. The geologist could,
then, subjectively formulate a risk function based on his experience or could
select a "Classical" distribution, such as Normal, Gamma, or Poisson, which
seemed to fit his mental pattern. The individual's subjective risk function
(distribution) would be derived from a verbal lottery with successive
questions to specify quantitative point estimates, which when plotted would
give a probability distribu- tion. Later, Grayson extended this technique to
derive a group risk preference function. A similar type of question/answer
format was perceived to be applicable to the program management environment
in DOD.
A somewhat exhaustive review of techniques used to structure the opinions of
experts revealed the DELPHI method, developed at the RAND Corporation, as a
candidate for application of the Grayson method to the instant research.
DELPHI is a method for predicting the probability of future events by
polling experts, concerning their subjective evaluation as to event
occurrence. Each participant is interrogated individually by means of an
interview or questionnaire. The process involves four rounds of
interrogation. The results of each round are fed back to the participants in
an anonymous manner to eliminate the influence of strong personalities. The
goal is to refine and revise the subjective probabilities which are being
formulated as a measure of the uncertainty of occurrence for each future,
alternative outcome. Generally, the DELPHI technique is normally used in
forecasting from the present to the future, with responses in the form of
what might happen. As applied to the recollection of SRAM program
source-selection panel members, the responses not only expressed what was to
happen, but assigned probability statements to the outcome measures of
unacceptable, acceptable, and exceptional.
By the controlled interview/feedback/interview cycle central to the DELPHI
method, the researchers were able to identify some 19,683 possible SRAM program
factor-outcome combinations, and assign a probability to each. The calculation
base was the nine (9) factors identified during the DELPHI interrogation; each
having three categories of outcomes assigned. Thus, there were 39 or 19,683
possible outcomes or states for the program. By means of the computer, entropy
was then calculated using the following formula:
The DELPHI method was used to poll the original participants in the SRAM
source selection to determine the subjective probabilities associated with
the uncertain outcomes expected in the development program for the SRAM.
These probabilities were used to calculate the entropy in the program at the
time of source selection on a retrospective basis. As related to the
validation goal for the model, the findings for the effort are significant.
The actual total cost for the SRAM development program was $439 million. The
estimate for this cost obtained by applying the entropic cost model was $456
million. This estimate was based on encountering the worst possible cost
conditions during development. Adjustments based on approved changes which were
not contemplated at the time of source selection were made to the final cost
data. As a consequence, results of the study indicate that the entropic cost
model is a valid predictor of development program cost. The power of the model
rests in its ability to readily explain uncertainty in a single measure,
entropy. Admittedly,, the results of' one research effort do not validate the
model for general applicability. However. the model does have potential as a
cost estimating tool for program managers. Further research applying the model
to other developmental programs to determine the extent of the usefulness of
the entropy concept is planned.
Another significant finding that was a by-product of the research endeavor was
the use of DELPHI in uncertainty analysis. Application of the DELPHI
methodology to determine uncertain aspects of' a development program provides a
structured process by which a PM can use his experts to develop rigorous inputs
to assist in making key and significant program decisions as related to cost,
time, and performance.
Both the DELPHI technique and the entropic cost model merit consideration for
application in various areas external to the defense environment. Planning and
control for a large, high-dollar value project which entails a moderate to high
degree of uncertainty as related to cost could be managed by application of'the
methodology at selected decision points over time.
For example, large-scale projects in construction, marketing new products or
services, advancing technology and exploration for new deposits of natural
resources are a few possible areas for application. The amount of effort
expended naturally depends on the magnitude of the project and how much time
and money management is willing to invest for informa- tion, a measure of the
entropy in the information base, and the estimated cost outcome for a specific
program. Certainly, the benefits as related to cost control should exceed the
cost of administration.
Air Force Institute of Technology
Air Force Systems Command
Wright-Patterson AFB
BIBLIOGRAPHY
Growth Model For Weapon System Development
Programs, Unpublished Master's Thesis (Wright-
Patterson Air Force Base, Ohio: Air Force Institute
of Technology, 1974), p. 124.
The Mathematical Theory of
Communication (Urbana, Illinois:
The University of Illinois Press, 1949). p. 23 .
Decisions Under Uncertainty
(Boston, Massachusetts: Harvard Business
School, Division of Research, 196O, p. 70.
A Cost Growth Model for Weapon System Development Programs.
Published Master's Thesis. Wright-Patterson Air Force Base.
Ohio: Air Force Institute of Technology, 1974.
The Mathematical Theory of Communication,
Urbana. Illinois: The University of Illinois Press, 1949.
Decisions under Uncertainty,
Boston, Massachusetts: Harvard Business School.
Division of Research, 196[?]