Case
1: Allow sufficient
time for testing, and plan to test COTS products early. The time needed
to conduct test and integration activities is often grossly underestimated,
especially for critical products undergoing changes. This program estimated
1600 hours and easily spent 10,000 hours. Therefore, test and integration
of COTS products should start as soon as the product is available in-house.
It is sometimes necessary to bring in the vendor representative (as a consultant)
to assist in this discipline.
Case
2 : Including end users in testing can be used to set expectations.
This program used some end user staff to perform the testing. This introduced
the people who would eventually have to use the system in a gradual way.
|
|
Recommendation
1: Plan your system testing carefully.
Make a realistic determination of what kind/levels of testing are required
for the COTS and non-COTS parts of a system. Prototype your testing strategy,
process, and tools early in the life cycle.
Recommendation
2: Product maturity drives the
amount of necessary testing. Design your individual test scenarios to
match the maturity level of each COTS product. There is some evidence
that for mature products, there is little or no unit testing required.
Functional tests subsequent to installation are sufficient. For less mature
products, you should institutionalize a process of physically testing
product features in a closed environment to verify the product's release
claims, not unlike testing in a traditional development environment. The
test scenarios can be as simple as functional checkout and as complex
as formal simulation tests. While some unit testing for any custom software
components, and for software written to adapt a given COTS product into
your system, is useful, unit testing for the COTS products themselves
makes little sense. Thus, the focus of test for COTS-based systems changes
to system-level testing.
Recommendation
3: Plan to benchmark COTS products
early, as soon they come in-house to establish the validity of their capabilities.
Plan to test immature and fast-changing COTS products often.
Recommendation
4: If the system is making use
of standards, expect the vendor to bring standards conformance certification.
This will assist (although not replace) your own testing
Recommendation
5: DII COE can have an impact
on testing. There is a perception that use of DII COE segments has simplified
the integration and test process. This comes about because of the single
product, single segment nature of the COE. But caution must be exercised
when interfacing different application segments.
|
