|
from Configuration Management Plans: The Beginning to your CM Solution
Three well established standards were evaluated by the authors using six
different criteria. For each criterion used, the criterion was defined, the
rationale for rating described, and how the criterion applied to the standards
in general, and each standard individually, was provided. The standards
compared were: the IEEE Standard for Software Configuration Management Plans
(IEEE Std 828-1990), the NASA Software Configuration Management Plan Data Item
Description (NASA-Sfw-DID-04), and the DoD Software Development Plan Data Item
Description associated with DoD-STD-2167A (DI-MCCS-80030A).
Ease of use, in general, is measured by the time and effort required to learn
how to use a standard, and the recurring effort required to further use the
standard.
Ease of use is described by many different attributes, but perhaps the most
significant attribute of this criterion is the ability of a first time CM user
and an expert in the field to be able to easily use the standard to the degree
that each needs it. A person who is writing a CM plan for the first time, and
is not familiar with CM concepts, needs a great deal of information and help in
developing the CM plan. A person who has written many CM plans and just needs
some simple guidance only needs a few reminders about key elements in a CM
plan. A standard that is easy to use would be adaptable to both these
situations. It would allow the expert user to glance through the standard
quickly, locating the information needed while still providing a great deal of
information for the first time user.
Ease of use also implies that the standard be well organized and written, and
that specific information can be quickly located. The specific attributes for
this criterion that must be met for each of the ratings are shown in the next
subsection.
Ease of use is determined by the attributes listed below. These attributes are
placed in four categories. The first category contains those attributes that
must be met for a standard to receive a rating of 1. The second category
contains additional attributes that must be met for a standard to receive a
rating of 2. The third category contains additional attributes that must be met
for a standard to receive a rating of 3. Finally, the fourth category contains
attributes that would be desirable in a standard but that don't have to be met
for a standard to be rated a 3. These are attributes that a standard can do
without, but shouldn't. A rating of 0 implies that the standard did not meet
the minimum attributes.
The minimum attributes a standard must have to be "easy to use" are:
- A user should gain an increased understanding of how to write a CM plan.
- The standard is structured and formatted (laid out) to make information
easily accessible.
- Key words are used prominently and sections and paragraphs are built around
these key words.
- A user does not have to read a passage of text multiple times to understand
it.
The additional attributes that must be met for a standard to be rated "good"
are:
- The standard is easy to read--it is clear and concise, and is well written
grammatically.
- The standard assumes little or no prior user knowledge in the development
of CM plans; it is appropriate for more than one audience.
- Definitions of terms used in the standard are grouped together in one
logical place.
- The material is well organized so that a user can easily locate a section
of information. It takes a user no longer than three minutes to locate specific
information.
- All information related to one topic is located in one location--the user
does not have to look in more than one location in the standard for information
on a topic.
- Themes throughout the standard are clear to the user.
- Sections of the standard are short so that a user looking for a specific
piece of information does not have to look through pages of material to find a
specific reference in a section.
- The standard satisfies its requirements and fulfills the user's needs.
The additional attributes that must be met for a standard to be rated
"excellent" are:
- It takes a user no longer than one minute to locate specific information.
Important concepts are explained well and the text is structured to emphasize
the important information. There are two desirable, though not rated, attributes
of this criterion. They are:
- An indexing scheme for quick access to subject matter is included; this
index allows the user to look up any keyword and quickly locate all references
to it throughout the standard; this index must include, at a minimum, all
headings, acronyms, and defined terms.
- The standard does not require a high level of CM knowledge nor does it
assume a low level of knowledge--instead, it provides a quick reference for the
expert user and a tutorial for the first time user.
The three standards are well structured and formatted. Each uses section
headings well to control the flow of logic throughout the standard. They are
all easy to read, clear, concise, and well organized. Additionally, they all
keep information related to one topic in one location within the standard, thus
making it easier to read and use.
The major drawback in this area relates to the need to reference more general
standards for descriptions of many of the plan components. It appears, for
example, that both the NASA and DoD standards were written with the assumption
that the user would be using the more general standards for supplemental
information. Thus, these standards are not easily used as stand-alone
documents, especially for a novice user. The IEEE standard, however, does
function well as a stand-alone document. This is because it provides a greater
amount of detail and depth on each of the CM plan components. This point is
further demonstrated by the fact that the IEEE standard is 15 pages in length
while the NASA standard is nine pages and the DoD standard (that portion
dealing with CM) is four pages.
The IEEE standard was extremely easy to use. Not only did the standard meet the
attributes required to be rated excellent, but in a several areas it provided
added value. First, the standard made it easy for the user to recognize those
items in a CM plan that were required versus optional by using the words
"shall" and "required" to indicate required items. Another attribute of the
standard is that it is very good at providing examples of types of items when
an item is required (e.g., types of items to be controlled for each identified
software library, a list of possible interfaces with the minimum information
that must be defined for each). Finally, the standard very effectively maps one
CM activity to another. For example, the standard maps change controls,
identified in the configuration control section of the standard, to the
software libraries, identified in the configuration identification section of
the standard.
However, while the standard met all of the attributes required to be rated
excellent, there were several areas where it could have been better. The first
area is in terms of its definitions. While it had a section for definitions,
and did provide some definitions, it also listed quite a few terms that needed
to be defined but then referenced another IEEE standard for these definitions.
All terms should have been defined in this document rather than being
externally referenced. The second area deals with the level at which the
standard was written. Although this standard could easily be used by both a
novice and an expert, it could have been even more effective if it would have
provided a quick reference section for the expert to use. The effect of this
would be that the expert would not have to glance through all the information
to find the few key words that are used as content reminders for the plan. It
also could have added yet more content to some of the sections to provide a
better basis for the novice user (e.g., a better description of the
identification tracking scheme, a description of how to track and use change
control forms). Finally, it should have provided examples of CM plans or parts
thereof. This would have made using the standard that much better.
The NASA standard was easy to use and fairly complete. This standard would have
been rated good except for two omissions. The first is that it is somewhat
difficult to use this standard as a stand-alone document. Thus, it failed the
attribute of satisfying its requirements and fulfilling the user's needs. If a
user had good knowledge in the CM area, the NASA standard would be a very good
reference. However, if the user is a novice, it would be difficult, and
probably impossible, to write a complete CM plan based only on the information
provided in this standard. This also made it difficult for the standard to
meet the attribute of being appropriate for more than one audience. The second
omission is that the standard had no definitions of terms, so once again, it
assumed a fair amount of knowledge on the user's behalf.
Positive aspects of this standard are that it is well organized, with all the
information related to one topic placed in one location; it is easy to read and
relatively short, thus a user would not have to spend much time reviewing it;
and it is formatted in a manner that makes locating a key component very easy.
The DoD DID was fairly easy to use but not very complete. It only provided a
very high level of information on each of the four key CM components
(configuration identification, control, status accounting, and auditing). This
DID is an excellent reference for the CM expert but is not usable as a
stand-alone document by anyone but an expert because it lacks too much
pertinent information. Thus it fails the attributes of assuming little or no
prior user knowledge in the development of CM plans and being appropriate for
more than one audience. Another drawback to this DID is that there is no table
of contents or index. Thus, a user that wanted to locate a piece of information
would have to leaf through the entire DID until that information was located.
This could take several minutes. The end result is that the DID is not well
organized, thus harder to use. To add to this complexity in use, there is no
definition of terms in the DID. Consequently, an uninformed user would have to
reference other material to support this DID. Finally, this DID is not easy to
use because all of the information related to the CM function is not located
together within the DID. While the majority of the CM information is located in
one area of the DID, two key components are addressed in the section relating
to software development management. These are the software development library
and the corrective (change) action process, including the change report. These
two components should have been addressed in the section related to
configuration management.
Positive aspects of this DID are that it is structured well, using sections as
keys to important material, and sections are kept short so that they cannot
become confusing.
In general, a standard is complete if the user does not have to look elsewhere
for information or ask for other assistance in developing the configuration
management plan. However, this is a somewhat vague definition of completeness.
So that completeness can be objectively measured for each of the three
standards, specific attributes have been established. These attributes are
shown in the next subsection.
Completeness is determined by the attributes listed below. These attributes are
placed in the same four categories as the first criterion. Again, a rating of 0
implies that the standard did not meet the minimum attributes.
The minimum attributes a standard must have to be "complete" are:
- The standard provides a description of the following components of a CM
plan:
- Document introduction and purpose
- CM organization and responsibilities
- Configuration Identification
- Naming and numbering/freshening scheme for documents and files
- Identification and descriptions of baselines
- Procedures for handling changes to baselines
- Information on all CCBs
- Configuration status accounting records and reports
- Description of configuration audits that will involve CM
- Description of CM scheduling/milestones
- The purpose and audience of the standard is clearly defined.
- The standard is clear, concise, unambiguous, meaningful, and simple.
The additional attributes that must be met for a standard to be rated good are:
- The standard provides a description of the following components of a CM
plan:
- Interfaces between CM and other organizations
- Configuration Identification
- Description of identification tracking scheme
- How the identification scheme addresses versions and releases
- Backup, recovery, and retention policies
- Change request processing to include change classification scheme
- Change reporting
- Document revisions
- Reviews to be supported by CM
- Description of the version release process
- Subcontractor/vendor support
- The standard does not contain information that does not apply to the
creation of a CM plan (e.g., it should not be a tutorial on general CM
concepts, it should not discuss the entire software development life cycle).
- All information in the standard is necessary and is also sufficient.
- There is no missing information.
- All external references are defined.
- Each component described in the standard is covered in adequate depth
(i.e., the user should not have to reference any other document for information
on any of these components).
- The standard is well thought out, not thrown together.
- The standard is comprehensive in that it addresses the handling of items
such as documents and vendor software as well as application source code files.
The additional attributes that must be met for a standard to be rated excellent
are:
- The standard provides a description of the following components of a CM
plan:
- Relationship of CM to the software process life cycle
- How the identification scheme addresses hardware, system software, COTS,
etc.
- Library disaster procedures
- Non-CM organizations assigned responsibilities for change control
- Interfaces between multiple CCBs and the overall hierarchy
- How change control can change throughout the life cycle
- Use of automated tools to perform CM
- Relationship between CM milestones and the life cycle/development process
- CM training
- The standard provides examples of CM plans, or major portions thereof, to
assist the user.
There are four desirable, though not rated, attributes of this criterion. These
are:
- The standard contains a complete index that includes references to all
headings, acronyms, and terms that were defined in the standard.
- The standard provides a template of a basic CM plan.
- The standard addresses the handling of more than one client (e.g., how
releases are done for multiple clients, how variants of the system are managed).
- The standard provides new ideas, understanding, or improvements to the
software engineering practice of CM plan development.
In the area of completeness, one of the standards was rated good, another was
rated average, and the third was rated fair. The primary reason for these
ratings is because information that was deemed critical for each category was
missing in one or more of the standards. For example, two of the standards did
not address backup, recovery and retention. If these necessary concepts are not
addressed in the standard, they cannot be expected to be addressed in the CM
plan produced from the standard. A second key reason for these ratings is
because two of the standards contained incomplete information on the topics
they did address.
The IEEE standard was rated good for this criterion and was very close to
receiving an excellent rating. There are many positive points about this
standard in the area of completeness. First, the standard provides a good level
of depth on each of the CM plan components it discussed. Second, the standard
addresses all CM components that would be necessary in a CM plan and would
receive a rating of excellent in this one attribute. Important to this coverage
was that the standard very adequately addressed topics not covered well in
other standards or many CM plans, such as baselines and changes to them,
interface control, subcontractor/vendor control, and multiple levels of change
control boards (CCBs). Additionally it provided detailed coverage of topics not
required of even an excellent standard. These topics were tailoring and CM plan
maintenance. A final positive aspect of this standard is that it provided a
template for a CM plan upon which a user could immediately begin building their
own plan.
Aspects of this criterion that were not addressed well by this standard are
lack of examples, lack of added emphasis on several components, and minimal
emphasis on automated tools. The first point, lack of examples, is the only
reason this standard did not receive a rating of excellent. This standard would
have been rated excellent if it had included a sample change control diagram or
a sample completed CM plan. Ironically, there were three sample CM plans
included in the general standard for CM concepts (IEEE Std-1042-1987). The
authors would suggest moving one of the samples to this standard. We do not,
however, recommend including all three samples as we believe that this would
add unnecessary volume to the standard and would provide minimal additional
benefit.
The second point, lack of added emphasis on several components, can best be
described by reference. While the standard does address identification
tracking, it does not elaborate well on the entire tracking scheme. The user
would know, by inference, that a tracking scheme needs to be described but the
standard does not explicitly state this. The standard also does not place
enough emphasis on change processing classification. It discussed
classification generally in terms of priority and urgency but did not express
the need for a classification scheme in terms of high, medium, low or 1, 2, 3,
for example, with associated definitions. Also, the standard only addresses
documentation at the overall level as part of CM. It would have been more
effective to re-address or remind the user of documentation when the standard
discussed identification and change control procedures.
Finally, the standard would have been even more outstanding if it would have
discussed the use of automated tools at a more detailed level. Though it did
discuss automated tools in one section, an expanded discussion of automated
tools describing how they can be applied in the various functional areas (e.g.,
change control, status accounting) would have been useful.
The NASA standard was only rated average for completeness because it did not
address many of the components required for a CM plan to be rated good, nor did
it provide all the information required to make it a complete document.
Examples of CM plan components that it did not address, but should have, are
software libraries, a version release process, backup, recovery and retention,
change classification, subcontractor/vendor support, and the use of automated
tools. This standard is not considered complete because the detail of
information provided on the CM components addressed is insufficient. The
standard is written, for the most part, in an outline form providing no
explanation of the items listed in the outline. It appears as though the
authors of this standard never intended for it be used without the accompanying
guidebook on the general CM concepts (NASA D-GL-11 VO.2). Because there was
very little detail in this standard it can only be effectively used by an
expert in CM as a quick reference guide.
Although the standard was only rated average, there were three positive aspects
about it that should be noted. First, this standard included a section on
resources beyond what is typical. That is, it addressed resources such as
machine and data storage resources as well as the commonly addressed human
resources. Second, it contained a very complete section on configuration
control forms, addressing this component at a level of detail much greater than
the other two standards. This level of detail provided added benefit to the
user versus being excessive information on a topic. Finally, the standard
handled the identification, storage and release of documentation separately
from the CSCIs. This adds benefit to the standard in that it reminds the user
not to forget to address documentation.
The DoD DID was rated as not satisfying the minimum requirements for
completeness because it did not address one of the components required for a CM
plan to be rated average. This component is the identification and description
of baselines. A CM plan cannot be considered adequate without defining and
describing baselines, as baselines provide the fundamental structure upon which
all changes occur. This is especially true in the DoD environment. Thus, it is
critical that the standard address baselines. In addition to the problem of not
addressing all required components, this DID is considered incomplete because
the level of detail provided for the addressed CM components is insufficient.
As with the NASA standard, it appears as though the authors of this DID
intended that it be used in conjunction with DoD-STD-2167A and MIL-STD-973 or
MIL-STD-483A. Because there was very little detail in this DID it can only be
effectively used by an expert in CM as a quick reference guide.
One positive aspect of this DID is that it provided a good example of a
configuration control flow chart. This chart would be of significant benefit to
a user of a standard, especially a non-expert user. This was the only standard
that provided a sample chart.
A standard is considered to be tailorable if it can be customized to the needs
of any project. The attributes for tailorability are shown in the next
subsection.
Tailorability is determined by the attributes listed below. These attributes
are placed in the same four categories as the first criterion. Again, a rating
of 0 implies that the standard did not meet the minimum attributes.
The minimum attributes a standard must have to be "tailorable" are:
- A CM component that does not apply to a particular project can be easily
removed from the plan (i.e., the standard addresses each component in only one
section versus scattering the discussion of that component throughout the
entire plan/standard).
- CM components can be reorganized within the plan to accommodate the project.
The additional attributes that must be met for a standard to be rated "good"
are:
- The standard can be adapted, though perhaps not easily, to any application
domain.
- The standard can be adapted, though perhaps not easily, to any process
model or life cycle phase.
- The standard can be adapted, though perhaps not easily, for use with
hardware, software and firmware.
- The standard addresses subcontractors and vendors, but does so in such a
manner that a plan can be tailored not to address them if they do not apply to
the project.
The additional attributes that must be met for a standard to be rated
"excellent" are:
- The standard can be easily adapted to any application domain without
resorting to extensive expandability (i.e., it can be tailored by removing
rather than adding information).
- The standard can be easily adapted to any process model or life cycle phase
without resorting to extensive expandability.
- The standard can be easily adapted for use with hardware, software, and
firmware without resorting to extensive expandability.
- The standard can be tailored easily, completely, and consistently--this
requires a coherent and easy-to-use organization that has no redundancy.
Each of the standards met the minimum qualifications to be considered
tailorable, and one standard met the qualifications to be rated excellent. The
other two standards met all of the qualifications to be rated good, with the
exception of one attribute. Thus, overall the authors would state that all the
standards lent themselves well to tailorability.
The IEEE standard was rated excellent for tailorability. This standard was
intentionally written to be tailorable and explicitly addresses tailorability
as part of the standard. The standard contains a complete section on tailoring
that addresses both upward and downward tailoring. Although the standard
handles upward tailoring, it is the authors opinion that there will be very
little need to tailor up from this standard for most projects, because the
standard is so complete. Thus, the attributes dealing with resorting to minimum
expandability are still met by this standard. This standard also addresses the
tailoring of a document to handle hardware, firmware, and subcontractors and
vendors quite clearly. Additionally, this standard contains specific
instructions that forbid a user from removing any of the required information
without stating specifically why it was removed. This standard has no negative
aspects related to this criterion.
The NASA standard was rated average, and would have been rated good except that
it does not address subcontractors and vendors, thus it failed this attribute.
Because the standard is well organized, it would be easy to remove a CM
component that did not apply to a project, or to reorganize the CM components
to accommodate a project. This standard can also be adapted, though perhaps not
easily, to any application domain, process model, or life cycle, and can be
adapted for use with hardware and firmware. The reason that the tailoring may
not be easy is because the standard is incomplete. Thus, any adaptation will,
in all likelihood, require a considerable amount of upward tailoring.
The DoD standard was rated average, and would have been rated good except that
it also did not address subcontractors and vendors. The same comments that
apply to the NASA standard apply to the DoD standard.
A standard is considered to be consistent if it has a structure and there is
uniformity throughout the document. The attributes for consistency are shown in
the next subsection.
Consistency is determined by the attributes listed below. These attributes are
placed in the same four categories as the first criterion. That is, a rating of
0 implies that the standard did not meet the minimum attributes; a rating of 1
means it is average, 2 mean it is good, and 3 means it is excellent.
The minimum attributes a standard must have to be "consistent" are:
- A standardized representation format is used for the production of the plan.
- Naming and use of terms is standardized.
- The same CM component or term is not referenced by more than one unique
name, and is not defined by more than one set of characteristics.
The additional attributes that must be met for a standard to be rated "good"
are:
- A structured format is used for the production of the plan.
- Unique names given to CM components or terms are commonly known and used in
industry.
- Uniformity is achieved throughout the standard.
- The standard is consistent with other standards for the project life cycle.
There is only one additional attribute that must be met for a standard to be
rated "excellent". It is:
- The standard is consistent with other known CM standards in industry.
Each of the standards received a rating of excellent for this criterion. All
standards were very consistent within themselves. More importantly, they were
all consistent with other known standards in the industry. This consistency to
other known standards exists primarily because all three standards addressed
the same key components. These were CM organization, configuration
identification, configuration control, configuration status accounting,
configuration auditing, and configuration management milestones.
The IEEE standard, like the other standards, met all of the attributes of
consistency. Perhaps the attribute that lent the standard towards being the
most consistent was its uniform structuring. One unique characteristic about
the IEEE standard is that it contained a section on consistency criteria for
the CM plan. This section contains a list of three criteria. This was the only
standard that addressed consistency explicitly for the plan, as well as being
consistent itself. This standard had no negative aspects related to this
criterion.
The NASA standard was also rated excellent in this category. Again, as with the
IEEE standard, the attribute that lent this standard most towards being
consistent was its uniform structuring. This standard also had no negative
aspects related to this criterion.
The DoD standard was also rated excellent. The same comments that apply to the
NASA standard apply to the DoD standard.
A standard is considered to be correct if it contains only valid information.
The attributes for correctness are shown in the next subsection.
Correctness is determined by the attributes listed below. These attributes are
placed in the same four categories as the first criterion. Again, a rating of 0
implies that the standard did not meet the minimum attributes.
The minimum attributes a standard must have to be "correct" are:
- The standard does not provide any incorrect information.
- The standard does not contain any contradictory information.
The additional attributes that must be met for a standard to be rated good are:
- The standard contains only that information that is necessary to clearly
explain a topic. Any less than is needed forces the user to look elsewhere
making the standard ineffective. Discussion of non-related information, or
excessive supporting material on a topic is unnecessary and makes it difficult
for the user to concentrate on the key items in the standard.
- The approach taken in the standard makes sense both technically and
practically.
The additional attributes that must be met for a standard to be rated excellent
are:
- There is evidence of correctness to back up the standard (i.e., it should
have been successfully applied to case studies).
There is one desirable, though not rated, attribute of this criterion. It is:
- The standard represents the state-of-the-art in CM and does not contain
outdated guidance.
One standard was rated excellent in this category and the other two were rated
average. The only reason for this difference in ratings was that the two lower
rated standards did not contain a sufficient amount of information. Other than
failing this attribute, all three standards met all the remaining attributes
for both a good and excellent rating.
The IEEE standard was rated excellent for this criterion. Not only was the
information contained in the standard correct, but the correct level of
information was also provided. There were a couple of topics where a little
more information would have increased the value to the user, however, the
information provided was still complete enough to clearly and correctly explain
each topic.
The one thing the standard could have done better is to attempt to represent
the state-of-the-practice a bit more. For example, the standard could have
placed more emphasis on how automated tools could be used for each of the CM
activities, and could have more clearly addressed the relationship of CM to the
project life cycle.
The NASA standard was rated average for this criterion. As stated above, the
only reason for this low rating is that the standard does not contain all the
information needed by any user to develop a CM plan. Had the standard passed
this attribute, it would have been rated excellent.
The DoD standard was also rated average for this criterion for the same reason
as the NASA standard. The comments that apply to the NASA standard also apply
to the DoD standard.
A standard is considered to be connected to the life cycle if it refers to how
CM connects to the life cycle. This criterion only has two attributes. These
attributes only apply to a "good" or "excellent" standard. There are no
attributes associated with an "average" standard for this criterion.
Life cycle connection is determined by the attributes listed below. These
attributes are placed in the categories for a good and excellent standard. A
rating of 1 implies that the standard did not meet the attributes associated
with being a good standard. There is no rating of 0 for this category.
The attribute that must be met for a standard to be rated good is:
- The standard addresses how the CM plan fits into and evolves throughout the
overall project life cycle.
The additional attribute that must be met for a standard to be rated excellent
is:
- The standard addresses how the individual CM plan activities fit into the
overall project life cycle.
All three standards were rated average in this category. This means that the
attributes for life cycle connection do not apply to any of the standards. The
standards do not meet the first attribute, of addressing how the CM plan fits
into and evolves throughout the overall project life cycle, because none of the
standards address the evolution of the CM plan throughout the life cycle.
Additionally, the NASA and DoD standards do not address the life cycle at all,
while the IEEE standard addresses life cycle, but not the evolution of the CM
plan through the life cycle. The standards do not meet the second attribute, of
addressing how the individual CM plan activities fit into the overall project
life cycle, either.
Because none of the standards meet the attributes of either the good or
excellent rating, as stated above, and there were no attributes associated with
the average rating, there is no need to discuss each standard individually with
regard to this criterion.
![[Table of Contents]](../../icons/toc.button.gif)
The Software
Engineering Institute (SEI) is a federally funded research and
development center sponsored by the U.S. Department of Defense
and operated by Carnegie Mellon University.
Copyright
2007
by Carnegie Mellon University
Terms of Use
URL: http://www.sei.cmu.edu/legacy/scm/papers/CM_Plans/CMPlans.AppendixC.html
Last Modified: 11 January 2007
|
