Control Channel Toolkit: A Software Product Line Case Study
7 Lessons and Issues
With asset development complete, CCT offers its users a set of tested assets to support development of new ground-based spacecraft command and control systems. Many lessons were learned during the CCT effort, and several major challenges still face the CCT program as it evolves from an effort primarily focused on asset development to one dedicated to product development by external organizations. The use of CCT for development of the system test architecture and the Spacecraft C2 System brought many nagging issues to the surface. Future development and collaboration with external users will shed light on others. We conclude the CCT case study with a brief tour through some of these lessons and outstanding issues.
Tool Support Is Inadequate
Tools for adequate support of software product lines lag behind the adoption of product line approaches, and so it’s no surprise that the tools for support of CCT processes were not ideal.
In common with many domain analysis efforts, the CCT domain analysis used existing object-oriented analysis and design tools to capture and represent domain-level concepts. The outputs of the domain analysis process were spread over a number of documents that incorporate text, outputs from the Rational Rose tool, and outputs from the Slate tool. Collectively, these documents comprise the requirements database for CCT; there is no single, easily maintainable model. Maintenance is painful, which may pose a risk for long-term use of the assets. The same is true for the Architecture Model, which contains the layer model diagrams, use-case maps, and overall execution and planning architecture diagrams. Owing to a lack of tool support, these diagrams exist separately. The distribution makes maintenance awkward and time-consuming and increases the risk that changes will not be incorporated consistently.
Lack of consistent tool support for traceability among the CCT assets continues to hamper ongoing CCT maintenance. The CCT team documented a complete set of generic requirement specifications based on the initial analysis and subsequent understanding achieved through design and implementation. A formal, tool-supported mechanism should record issues and decisions, and feed them back into the domain model and other analysis results. Unfortunately, the tool support doesn’t yet exist, and the resultant manual efforts are cumbersome.
The CCT domain analysis products did not contain a record of issues and decisions regarding the presence or absence of particular domain features, or a record of tradeoffs that might influence subsequent design decisions. Such information would have been useful to CCT developers. It would also be useful now to targeted CCT users as they try to decide whether or not to use CCT for their products and to determine how CCT could be used.
An Early Architecture Focus Is Best
A software product line approach relies on a strong product line architecture that is used to structure all of the products in the product line. The architects need to take great care to ensure that the product line architecture is accessible to and understandable by all future product builders. Because of the acquisition nature of this product line effort, even greater care was needed; the product builders will not be members of the CCT development team and probably will not belong to the Raytheon Company, nor will the CCT Program Office commission them. In order for the NRO to benefit fully from the product line that CCT is intended to support, the CCT architecture documentation and its attached process must be of exceptionally high quality. Moreover, the NRO would like some means to ensure that product architectures (called target architectures by the CCT effort) actually conform to the CCT architecture. They want to ensure that the CCT assets are used in the intended way and not simply to feed some ad hoc reuse of components that would undermine the true potential of CCT.
Unfortunately, the CCT effort was not architecture-centric from the start. Remember that the original vision called for a toolkit. The architecture and architectural knowledge were in the heads of the CCT architects, who did a fine job and who remained involved in every aspect of the CCT development and its initial use. Although this involvement was a plus, it didn’t really permit the architecture documentation to be exercised to the fullest by the development team. Team members could always go ask the architects, and for that matter, so could the first product builders.
Early on, the SEI was concerned about the architecture and its documentation and advocated two architecture evaluations and the development of the Reuse Guide. The CCT Program adopted these suggestions but had to insert these efforts into the schedule after the project was well under way. It is to the Raytheon team’s credit that they embraced both activities even though architecture evaluations weren’t in the original contract.
During the first software architecture evaluation, it was apparent that there was a lack of architecture documentation that could stand on its own. The CCT Architecture Group was established to address this need. This group advocated creation of the Architecture Model and was a strong proponent of the Reuse Guide. Both documents attempt to communicate CCT architectural concepts and decisions to CCT users who would build future products. These documents, however, were not in the original plans and suffer a bit from their late start. They are not as user-friendly as they should be, and in their current format it is more difficult to determine architecture conformance.
The CCT effort focused on the creation of a set of assets for spacecraft command and control and much less on the integration of those assets. There were two outcomes that have proven to be disadvantageous to future product builders:
The Reuse Guide is the CCT production plan, and its very existence is positive. It provides extensive documentation on the use of individual component categories, their variation points, mechanisms, and tailoring¾
what we might call the attached process for components and what they call component categories. However, other information is less well organized and in some cases missing. Critical architectural information regarding the range of overall qualities (for example, runtime performance and reliability) of the systems built from CCT assets is either difficult to locate or missing. There really is no attached process for creating an instance of the architecture and no attached process for system integration and testing. A decision was made to leave the process up to the builder, which might be fine, but a product builder wants to get a system perspective first, and the Reuse Guide offers no direct support for such a perspective. The product builder has to glean this in a bottom-up fashion. The Reuse Guide is also missing any sort of CCT primer that explains basic concepts and assumptions and helps a potential user mount the CCT learning curve.
Production capability was tested in two ways: by having an internal application engineering team (the system test architecture developers) act as early users of CCT, and by having the first real user¾
the Spacecraft C2 System¾
cooperate in the CCT development effort. Both were
positives for CCT. However, the end-system perspective wasn’t pervasive enough during the creation of the test system, despite the involvement of domain experts and architects: the test cases were in some instances too simple to address the kinds of issues involved in integrating CCT assets into the development of an operational system. Integration and other system-wide issues, such as meeting a range of real-time performance requirements, were not adequately tested. The CCT component level requirements were met, but it was not entirely clear from integration-level testing that the needs of real, full-up command and control applications would be met. The first user was successful, but was intimately involved in the entire CCT development process and so is not a typical future user, who will need more support. These key reuse aspects were not addressed early in the program to the same extent as were the technical requirements
[Ohlinger 00].
There are two types of CCT users: those who would commission systems to be built from CCT (acquirers) and the contractors who would be doing the building. The acquirers are interested in knowing that the systems built by their contractors do in fact use the CCT architecture and do in fact take advantage of CCT as a product line asset base. They want to measure architecture conformance. The CCT Architecture Group developed conformance guidelines to support this user need. The conformance guidelines define six conformance levels. The Architecture Group applied these guidelines to the system test architecture to determine its level of conformance. The SEI conducted a conformance analysis of the system test architecture and the CCT architecture, at the component category level. Both results provided useful metrics, and also led to some recommendations on architecture enhancement.
There was no metrics program that identified specific measures for analyzing the reusability of the CCT assets or the benefits that would accrue from their reuse. Full tracking of the extent and remediation of defects provided some measures, but these measures were not preplanned. Since CCT completion, there has been an effort to collect data that speak to the reuse potential and the inherent benefit of using the CCT assets. These benefits were reported earlier and are now included in a business case being developed by the NRO. Such metrics are essential for attracting new users (acquirers) and convincing their contractors of CCT’s merits; they both need to know the benefits and the costs in quantitative terms. Data should have been proactively collected from the beginning. There were potential users who rejected CCT because of the lack of such quantitative information.
True to its name, CCT was conceived as a "toolkit" development effort. The increments were created on the basis of staged delivery of component categories. The incoming assumption was that a rich set of highly flexible class categories could support the requirements of future spacecraft command and control systems. Significant effort was applied to understanding and documenting the variation points for support of variability and to the mechanisms for implementing the variation points, but the focus was on the functionality and availability of components.
The shift to an architecture-based approach came about as a result of the first architecture evaluation and a subsequent decision made by John Ohlinger and other key CCT players to form the CCT Architecture Group. The Architecture Group had representatives from the entire spectrum of major players in CCT development, including the contractor, technical support consultants to the CCT Program Office, the SEI, the Aerospace Corporation, and end-user organizations. Its chartered purpose was to guide the evolution of the CCT product line architecture, and its functions were to:
The Architecture Group was the major influence in moving CCT from a component-based effort to an architecture-based effort. The lesson that was learned is really twofold: (1) structuring the organization is an ongoing process that requires organizational flexibility, and (2) a permanent crosscutting architecture group has great benefits.
The Spacecraft C2 System was really the first to test the reusability of CCT. This program consisted of two major subsystems:
The program integrated CCT assets into its spacecraft command and control application, and it also exercised the variability of the components by adding mission-unique extensions. The Spacecraft C2 System supported CCT by providing essential feedback in two areas:
Spacecraft C2 was a major influence on CCT and a factor in CCT evolution since the initial increment. This real product demonstrated the strategic intent of CCT¾
to show that satellite programs could use CCT as the basis for their spacecraft command and control solutions¾
and that demonstration was critical. Here is the lesson for organizations that commission a product line asset base: commission or develop a product in concert with the asset base development. The schedules and the demands of the product team have to be managed carefully, but the advantage of having a real product to provide valuable feedback and prove the usability of the assets outweighs any extra management required.
[Title Page]
[Abstract]
[Figures]