SEI Documents List

[2007] [2006] [2005] [2004] [2003] [2002] [2001] [2000] [1999] [1998] [1997] [1996] [1995] [1994] [1993] [1992] [1991] [1990] [1989] [1988] [1987] [1986] [PDF]

2006 Reports

Handbooks

CMU/SEI-2006-HB-003, ADA456877
Defense-in-Depth: Foundations for Secure and Resilient Enterprises
May, C.; Hammerstein, J.; Mattson, J.; & Rush, K.

The Defense-in-Depth Foundational Curriculum is designed for students, ranging from system administrators to CIOs, who have some technical understanding of information systems and want to delve into how technical assurance issues affect their entire organizations. The course material takes a big-picture view while also reinforcing concepts presented with some details about implementation. Therefore, this course can be a useful pursuit for system administrators and IT security personnel who would like to step up to the management level. It also can provide a refresher for IT managers and executives who want to stay up to date on the latest technological threats facing their enterprises.

The curriculum consists of eight main modules: (1) Compliance Management, (2) Risk Management, (3) Identity Management, (4) Authorization Management, (5) Accountability Management, (6) Availability Management, (7) Configuration Management, and (8) Incident Management. The document also contains an introduction, "Foundations of Information Assurance," which focuses on how the overarching concepts of confidentiality, integrity, and availability can lead to a comprehensive security strategy.

http://www.sei.cmu.edu/publications/documents/06.reports/06hb003.html

CMU/SEI-2006-HB-001, ADA455116
QUASAR: A Method for the Quality Assessment of Software-Intensive System Architectures
Firesmith, D.

This handbook documents the QUASAR (QUality Assessment of System ARchitectures) method for assessing the quality of the architecture of a software-intensive system. It begins by discussing the challenges that are faced when assessing a system's architecture and outlines the development history of the method. The next section of the handbook documents the concept of quality cases and the claims, arguments, and evidence that compose them. This is followed by a description of the teams that collaborate to perform QUASAR tasks. Next, individual tasks and associated steps performed as part of the QUASAR method are documented. Next, the work products produced by these teams when performing these tasks are described. Finally, lessons learned during the development and use of the method when assessing the quality of major subsystems during the development of a very large, software-intensive system of systems are presented. Also provided are appendices that define common quality factors and subfactors, offer reusable checklists, and give examples of quality cases. The example quality cases illustrate valid quality goals and requirements that compose claims, example architecture decisions and associated rationales that compose arguments, and the types of evidence that architects might provide.

http://www.sei.cmu.edu/publications/documents/06.reports/06hb001.html

CMU/SEI-2006-HB-002, ADA454685
Standard CMMI Appraisal Method for Process Improvement (SCAMPI) A, Version 1.2: Method Definition Document
SCAMPI Upgrade Team

The Standard CMMI Appraisal Method for Process Improvement (SCAMPI) is designed to provide benchmark quality ratings relative to Capability Maturity Model Integration (CMMI) models. It is applicable to a wide range of appraisal usage modes, including both internal process improvement and external capability determinations. SCAMPI satisfies all of the Appraisal Requirements for CMMI (ARC) requirements for a Class A appraisal method.

The SCAMPI Method Definition Document describes the requirements, activities, and practices associated with each of the processes that compose the SCAMPI method. It is intended to be one of the elements of the infrastructure within which SCAMPI Lead Appraisers conduct a SCAMPI appraisal. Precise listings of required practices, parameters, and variation limits, as well as optional practices and guidance for enacting the method, are covered. An overview of the method's context, concepts, and architecture is also provided.

http://www.sei.cmu.edu/publications/documents/06.reports/06hb002.html

Special Reports

CMU/SEI-2006-SR-005, ADA453524
Adapting CMMI for Acquisition Organizations: A Preliminary Report
Dodson, K.; Hofmann, H.; Ramani, G.; & Yedlin, D.

CMMI (Capability Maturity Model Integration) is a collection of best practices that helps organizations improve their processes. It was initially developed by a product team from Industry, U.S. government and the Software Engineering Institute for application to process improvement in the development of products and services covering the entire product life cycle from conceptualization through maintenance and disposal. Following the success of CMMI models for development organizations, the need was identified for a CMMI model addressing the acquisition environment. This need was reinforced and gained further attention due to similar needs expressed by General Motors (GM), which acquires information technology (IT) solutions. Aligned with GMs strategy, GM projects or programs develop requirements and design constraints and oversee multiple suppliers that develop IT solutions and then deploy the resulting products and services into one or more of GMs business units. This approach parallels the acquisition processes used in many government organizations.

General Motors, in collaboration with the SEI and with approval of the CMMI Sponsors and Steering Group, sponsored the development of an initial draft CMMI for Acquisition (CMMI-ACQ) constellation, which will lead to a CMMI- based acquisition model formally accepted by both government and industry after piloting of the initial draft CMMI- ACQ has been completed. This draft is based on the CMMI Version 1.2 architecture and framework which incorporates the concept of constellations, which are groupings of components to support a specific model application such as Development (DEV) or Acquisition (ACQ).

http://www.sei.cmu.edu/publications/documents/06.reports/06sr005.html

CMU/SEI-2006-SR-011, ADA460413
Army ASSIP System-of-Systems Test Metrics Task
Sledge, C.

The Army Strategic Software Improvement Program goal is to dramatically improve the acquisition of software- intensive systems by focusing on acquisition programs, people, and production/sustainment and by institutionalizing continuous improvement.

This special report contains a briefing (slides and accompanying notes) on the results of one subtask of this effort conducted during FY06. The subtask called for three actions: (1) explore the (then) current processes and test results/ metrics used to address system-of-systems integration and testing, (2) develop findings and recommendations for improvement based on this initial exploration, and (3) recommend future work to further improve the Army's system- of-systems integration and test practices.

The Army is in the lead in addressing the many challenges associated with system-of-systems integration and testing, paving the way for the rest of the U.S. Department of Defense (DoD). As a result, the information contained in this report is useful to other organizations facing similar challenges.

http://www.sei.cmu.edu/publications/documents/06.reports/06sr011.html

CMU/SEI-2006-SR-017
Examination of a Structural Modeling Risk Probe Technique, An
Anderson, W.; Boxer, P. & Brownsword, L.

The integration of demand dynamics into a structural model is a key conceptual shift for software engineering. This report examines the utility and transition characteristics of a structural dynamic analysis modeling technique called Projective ANalysis (PAN) that was used on an interoperability technical probe of a North Atlantic Treaty Organization (NATO) modernization program. The report focuses on the process, rather than the findings, of the probe. Organizational entities are referred to generically and, in some instances, aggregated.

The probe involved workshops and interviews, conducted over a two-week period with more than 25 people, followed by analysis of the data gathered. PAN was used to model the NATO program as a system of systems. The model is a rapid assessment based on the subjective understanding of the interviewed subject matter experts. It is a snapshot in time; while dynamic stocks and feedback loops are represented, their temporal characteristics are not. From the model, five perspectives were analyzed for different forms of interoperability risk. These analyses produced three- dimensional projections that depict clusters of shared interfaces. The separation between these clusters identifies the interoperability risks.

The report notes that the PAN technique starts from a client-driven context and builds visual representations that are easily understood by, and bring immediate value to, the client. Further, the report observes that the modeler is critical to this technique and must possess expert skills in the Microsoft Visio application as well as an ability to quickly grasp and characterize the constructs and objects revealed through dialog-based inquiry.

The report concludes that PAN appears to offer a fresh approach, new insights, and appropriate mechanisms to study complexity in systems of systems. The potential for applying and amplifying this technique appears to be significant. The report also determines that an experienced process modeler would have little difficulty adapting to this modeling paradigm.

http://www.sei.cmu.edu/publications/documents/06.reports/06sr017.html

CMU/SEI-2006-SR-008, ADA465913
Global Information Grid Survivability: Four Studies
Ciampa, R.; Day, D.; Franks, J. & Tsuboi, C.

The four studies in this document are student contributions to the SEI Global Information Grid (GIG) Survivability Study. Each study explores an issue relevant to the survivability of networks which are systems of systems. Since the GIG is inherently a system of systems, the survivability of operational concepts such as Joint Battle Management Command and Control (JBMC2) will largely depend on the extent to which GIG architecture is approached from this perspective. Systems of systems differ from large, monolithic systems because of the simultaneous independence and interdependence of their constituent parts, and therefore traditional survivability methods are not sufficient. To deal with the operational complexity resulting from qualities peculiar to systems of systems, planners and builders of the GIG will need to formulate broad strategic approaches taking these qualities into account.

These four studies have attempted to identify characteristics of systems of systems which may be useful in this endeavor. The specific areas explored in this document include the following: the applicability of autonomous agents in a system of systems; the suitability of conventional software testing in a system-of-systems environment; emergent properties and unanticipated consequences in a system of systems; the role of ontologies in systems-of-systems interoperability; the architectural properties and operational survivability effects of internet protocol version 6 (IPv6) technology.

http://www.sei.cmu.edu/publications/documents/06.reports/06sr008.html

CMU/SEI-2006-SR-007, ADA452451
Information Assurance: Building Educational Capacity
Sledge, C.

This report is the fourth in a series describing the efforts by the Software Engineering Institute (SEI), and in particular those of its CERT Program to increase the capacity of institutions of higher education to offer information assurance (IA) and information security (IS) courses. Other goals are to expand existing IA and IS offerings and to include IA and IS topics and perspectives in other courses. For each participating institution, these efforts are aligned with the focus of its involved academic department, current curriculum, and accreditation requirements. The report describes SEI activities for accomplishing its goals: participating in faculty capacity building programs funded by the National Science Foundation; creating and transitioning courseware, materials, and a newly created survivability and information assurance curriculum; and collaborating with key regional educational institutions. This report also presents four approaches the SEI has developed for its educational outreach in IA. The SEI applies these approaches as it works with all institutions of higher education, with a particular focus on minority-serving institutions and community colleges in the United States.

http://www.sei.cmu.edu/publications/documents/06.reports/06sr007.html

CMU/SEI-2006-SR-001, ADA443799
Proceedings of the First International Research Workshop for Process Improvement in Small Settings, 2005
Garcia, S.; Graettinger, C.; & Kost, K.

The first International Research Workshop for Process Improvement in Small Settings was held October 19-20, 2005 at the Software Engineering Institute in Pittsburgh, Pennsylvania. Attendees from Australia, Canada, Chile, China, Germany, Ireland, India, Japan, Malaysia, Mexico, Spain, and the United States discussed the challenges of process improvement in small and medium size enterprises, small organizations within large companies, and small projects. The presentations addressed starting and sustaining process improvement, qualitative and quantitative studies, and using Capability Maturity Model Integration (CMMI), Agile, Modelo de Procesos para la Industria de Software (MoProSoft), International Organization for Standardization (ISO), Quality Function Deployment (QFD), and Team Software Process (TSP) in small settings. The workshop also had working groups that discussed issues unique to small settings, such as regional support centers and process improvement "on a shoestring."

This report includes the papers from this workshop and presents conclusions and next steps for process improvement in small settings. This report also contains the workshop breakout session results.

http://www.sei.cmu.edu/publications/documents/06.reports/06sr001.html

CMU/SEI-2006-SR-002, ADA448167
R2PL 2005-Proceedings of the First International Workshop on Reengineering Towards Product Lines
Graaf, B.; O'Brien, L.; & Capilla, R.

This report contains the proceedings from the First International Workshop on Reengineering Towards Product Lines (R2PL) 2005, which was held on November 10th, 2005 in Pittsburgh, Pennsylvania, USA and colocated with the Working Conference on Reverse Engineering (WCRE) 2005 and WICSA 2005-the Working Institute of Electrical and Electronics Engineers/International Federation for Information Processing (IEEE/IFIP) Conference on Software Architecture. This report consists of an overview of an invited presentation, a set of position papers, and details of the workshop's outcomes.

http://www.sei.cmu.edu/publications/documents/06.reports/06sr002.html

CMU/SEI-2006-SR-003, ADA452453
Security Quality Requirements Engineering (SQUARE): Case Study Phase III
Chung, L.; Hung, F.; Hough, E.; & Ojoko-Adams, D.

This special report is the third in a series by the Software Engineering Institute focusing on the practical application of the Security Quality Requirements Engineering (SQUARE) process. In this report, a student team presents their results of working with three clients over the course of a semester. Each client was developing a large-scale software application and worked with the students to generate security requirements. The students main contribution to the SQUARE process was to determine how existing software requirements-elicitation techniques could be applied to software security requirements (as opposed to end-user requirements).

With each client, the students implemented a different structured requirements-elicitation technique: Issue-Based Information Systems with an information technology firm, Joint Application Development (JAD) with the Delta client, and the Accelerated Requirements Method (ARM) with the Beta client. The ARM technique, which is a variant of JAD, held the most promise for inclusion in future applications of SQUARE. In addition to an analysis of the three elicitation techniques, the student team also generated feedback and recommendations on different steps of the SQUARE process, such as requirements prioritization and inspection. They found the Analytic Hierarchy Process to be highly useful for prioritizing requirements quickly; however, they did not find a requirements inspection technique that was well suited for any of the clients.

http://www.sei.cmu.edu/publications/documents/06.reports/06sr003.html

Technical Notes

CMU/SEI-2006-TN-001, ADA443683
Acquiring Evolving Technologies: Web Services Standards
Levinson, H. & O'Brien, L.

Software development projects rarely are started or proceed without risks involving the technologies used. Typically, many facets of a project such as system functionality and tool support depend on the availability of a specific technology. This dependency poses risks: the required technology can disappear within the project's life cycle or a promised technology may not be available when it's required.

A popular software technology today, Web services standards, is a widely supported approach to implementing a service-oriented architecture. Because Web services standards promise system interoperability and flexibility to large projects, commercial and government organizations are including it as the cornerstone of future computer-based systems. In fact, many systems currently being architected and designed assume the availability of products built upon a stable and effective set of Web services standards. This assumption presents project stakeholders with a large technology availability risk.

This technical note discusses some of the challenges of using Web services standards and presents the results generated by an assessment tool used to track the appropriateness of using this technology. The appendix includes an example built using the authors' opinions about the current level of appropriateness of using Web services standards in a typical, large software-intensive project.

http://www.sei.cmu.edu/publications/documents/06.reports/06tn001.html

CMU/SEI-2006-TN-010, ADA448425
Applying OCTAVE: Practitioners Report
Woody, C.

The CERT Operationally Critical Threat, Asset, and Vulnerability Evaluation (OCTAVE) method, an approach for managing information security risks, was designed to be sufficiently flexible for organizations to address unique and highly contextual analysis needs through tailoring capabilities. This document describes how OCTAVE has been used and tailored to fit a wide range of organizational risk assessment needs. Guidelines for successful tailoring, built on the reporting practitioners' successes, are provided to help an organization fit the OCTAVE approach to their specific domain and organizational needs. The range of applications demonstrates the flexibility of the OCTAVE approach and its value in addressing security risk management.

Readers should already be familiar with the general concepts of the OCTAVE approach.

http://www.sei.cmu.edu/publications/documents/06.reports/06tn010.html

CMU/SEI-2006-TN-011, ADA455842
The Architecture Analysis & Design Language (AADL): An Introduction
Feiler, P.; Gluch, D.; & Hudak, J.

In November 2004, the Society of Automotive Engineers (SAE) released the aerospace standard AS5506, named the Architecture Analysis & Design Language (AADL). The AADL is a modeling language that supports early and repeated analyses of a system's architecture with respect to performance-critical properties through an extendable notation, a tool framework, and precisely defined semantics.

The language employs formal modeling concepts for the description and analysis of application system architectures in terms of distinct components and their interactions. It includes abstractions of software, computational hardware, and system components for (a) specifying and analyzing real-time embedded and high dependability systems, complex systems of systems, and specialized performance capability systems and (b) mapping of software onto computational hardware elements.

The AADL is especially effective for model-based analysis and specification of complex real-time embedded systems. This technical note is an introduction to the concepts, language structure, and application of the AADL.

http://www.sei.cmu.edu/publications/documents/06.reports/06tn011.html

CMU/SEI-2006-TN-028, ADA460424
Assume-Guarantee Reasoning for Deadlock
Chaki, S. & Sinha, N.

The use of learning to automate assume-guarantee style reasoning has received a lot of attention in recent years. This paradigm has already been used successfully for checking trace containment, as well as simulation between concurrent systems and their specifications. In this report, the learning-based automated assume-guarantee paradigm is extended to perform compositional deadlock detection. Failure automata is defined as a generalization of finite automata that accept regular failure sets. A learning algorithm L^F is developed that constructs the minimal deterministic failure automata accepting any unknown regular failure set using a minimally adequate teacher. This report shows how L^F can be used for compositional regular failure language containment and deadlock detection, using non-circular and circular assume-guarantee rules. Finally, an implementation of techniques and encouraging experimental results on several nontrivial benchmarks are presented.

http://www.sei.cmu.edu/publications/documents/06.reports/06tn028.html

CMU/SEI-2006-TN-006, ADA448227
Autonomic Computing
Muller, H.; O'Brien, L.; Klein, M.; & Wood, B.

This report examines selected aspects of autonomic computing and explores some of the strengths and weaknesses of that technology. It also makes connections between autonomic computing and current work in several initiatives at the Software Engineering Institute. Furthermore, it describes the potential and impact of autonomic computing for Department of Defense (DoD) systems and outlines some of the challenges for the DoD as it moves to exploit autonomic computing technology.

http://www.sei.cmu.edu/publications/documents/06.reports/06tn006.html

CMU/SEI-2006-TN-044, ADA461460
Case Study of the NENE Code Project
Kendall, R.; Post, D.; & Mark, A.

The Defense Advanced Research Projects Agency (DARPA) High Productivity Computing Systems (HPCS) Program is sponsoring a series of case studies to identify the life cycles, workflows, and technical challenges of computational science and engineering code development that are representative of the program's participants. A secondary goal is to characterize how software development tools are used and what enhancements would increase the productivity of scientific-application programmers. These studies also seek to identify "lessons learned" that can be transferred to the general computational science and engineering community to improve the code development process.

The NENE code is the fifth science-based code project to be analyzed by the Existing Codes subteam of the DARPA HPCS Productivity Team. The NENE code is an application code for analyzing scientific phenomena and predicting the complex behavior and interaction of individual physical systems and individual particles in the systems. The core NENE development team is expert, agile, and of moderate size, consisting of a professor and another permanent staff member, five post docs, and 11 graduate students. NENE is an example of a distributed development project; the core team is anchored at a university, but as many as 250 individual researchers have made contributions from other locations.

http://www.sei.cmu.edu/publications/documents/06.reports/06tn044.html

CMU/SEI-2006-TN-030, ADA456867
Certifying the Absence of Buffer Overflows
Chaki, S. & Hissam, S.

Despite increased awareness and efforts to reduce buffer overflows, they continue to be the cause of most software vulnerabilities. In large part, these problems are due to the widespread use of unsafe library routines among programmers. For reasons like efficiency, such routines will continue to be used, even during the development of mission-critical and safety-critical software systems. Effective certification techniques are needed to ascertain whether unsafe routines are used in a safe manner.

This report presents a technique for certifying the safety of buffer manipulations in C programs. The approach is based on two key ideas: (1) using a certifying model checker to automatically verify that a buffer manipulation is safe and (2) validating the resulting invariant and proving it with a decision procedure based on Boolean satisfiability. This report also discusses the advantages and limitations of the approach with respect to today's existing solutions for buffer- overflow detection. Experimental results are presented that position the technique favorably against other static overflow-detection tools and indicate that the procedure can complement and augment these tools from a purely verification perspective.

http://www.sei.cmu.edu/publications/documents/06.reports/06tn030.html

CMU/SEI-2006-TN-014, ADA446836
Common Elements of Risk
Alberts, C.

Traditionally, responsibility for completing a mission and the resources needed to pursue it aligned with organizational boundaries. However, key drivers in the business environment, such as the globalization of business and the fast pace of technological change, have resulted in increased outsourcing and partnering among organizations. It is now common for multiple organizations to work collaboratively in pursuit of a single mission, which creates a degree of programmatic and process complexity that can be difficult to manage effectively. In today's business environment, management and staff must be able to deal with intricate and unclear interrelationships and dependencies among technologies, data, tasks, activities, processes, and people. Mission success in these complex environments requires people to sort through the inherent complexity when making important decisions. Effective risk management that is based on a solid conceptual foundation is an essential part of this decision-making process. This technical note begins to define this foundation by identifying the basic elements of risk and exploring how these elements can affect the potential for mission success.

http://www.sei.cmu.edu/publications/documents/06.reports/06tn014.html

CMU/SEI-2006-TN-027, ADA456882
Evolutionary Systems Design: Recognizing Changes in Security and Survivability Risks
Lipson, H.

A fundamental truth of system design is that, in the absence of countermeasures, a system's security and survivability will degrade over time. Changes in the environment or usage of a system, or changes to the elements that compose the system, often introduce new or elevated threats that the system was not designed to handle and is ill-prepared to defend itself against. The first step in evolving to meet new threats to your system's security and survivability is to recognize the need to modify your system-that is, to recognize changes in security and survivability risks that trigger the need to enter the evolution phase of the system development life cycle.

It is essential that significant risk management resources be devoted to the ongoing evolution of any mission-critical system. The successful evolutionary design of a secure and survivable system is dependent on the continual monitoring of the system and its environment to detect changes that may affect the risk management assumptions on which the system's security and survivability are founded.

http://www.sei.cmu.edu/publications/documents/06.reports/06tn027.html

CMU/SEI-2006-TN-034, ADA461385
Interoperable Acquisition for Systems of Systems: The Challenges
Smith, J. & Phillips, M.

Large, complex systems development has always been challenging, even when the "only" things a program manager had to worry about were cost, schedule, and performance within a single program. The emergence of operational concepts such as network-centric operations, greatly expanded use of joint and combined operations, and rampant growth in system complexity has led to the prevalence of interoperable systems of systems as the preferred solution to providing operational capability. This report explores how systems-of-systems realities necessitate changes in the processes used to acquire, develop, field, and sustain operational capability. Interoperable acquisition is defined, and key concepts are explored through an analysis of some of the ways in which traditional (i.e., system-centric) acquisition approaches can result in problems when applied to a system-of-systems context.

http://www.sei.cmu.edu/publications/documents/06.reports/06tn034.html

CMU/SEI-2006-TN-029, ADA454691
Joint Capabilities and System-of-Systems Solutions: A Case for Crossing Solution Domains
Anderson, W.; Brown, M.; & Flowe, R.

Recognizing the need to succeed in a new multilateral, asymmetric threat environment, the U. S. Department of Defense has initiated a radical transformation in operations to promote agility and enhance responsiveness. The transformation process, as well as the resulting new order of operations, relies heavily on system-of-systems solutions to bridge existing gaps in operations. To date, a pervasive, and possibly detrimental, assumption has dominated the program management arena: management tools and methods that work for single systems apply equally well to the acquisition of system-of-systems solutions. This technical note questions the general assumption that single-system methods are effective in a system-of-systems arena. Taking the position that the field, as a whole, lacks an adequate understanding of the unique challenges that influence system-of-systems initiatives, this report presents a case for the investigation and adaptation of structural and dynamic modeling techniques to the engineering of systems of systems. The report also includes results from a survey of subject matter experts providing evidence that resource expenditures in areas important to a system-of-systems environment are becoming high priorities.

http://www.sei.cmu.edu/publications/documents/06.reports/06tn029.html

CMU/SEI-2006-TN-041
Management and Education of the Risk of Insider Threat (MERIT): Mitigating the Risk of Sabotage to Employers Information, Systems, or Networks
Cappelli, D.; Desai, A; Moore, A.; Shimeall, T.; Weaver, E.; & Willke, B.

The Insider Threat Study, conducted by the U.S. Secret Service and Carnegie Mellon University's Software Engineering Institute CERT Program, analyzed insider cyber crimes across U.S. critical infrastructure sectors. The study indicates that management decisions related to organizational and employee performance sometimes yield unintended consequences that increase risk of insider attack. The problem is exacerbated by a lack of tools for understanding insider threat, analyzing risk mitigation alternatives, and communicating results. To develop such tools is the goal of Carnegie Mellon University's Management and Education of the Risk of Insider Threat (MERIT) project. MERIT uses system dynamics to model and analyze insider threats and produce interactive learning environments. These tools can be used by policy makers, security officers, information technology and human resource personnel, and management. The tools help these users to understand the problem and assess risk from insiders based on simulations of policies, and on cultural, technical, and procedural factors. This technical note describes the MERIT insider threat model and simulation results.

http://www.sei.cmu.edu/publications/documents/06.reports/06tn041.html

CMU/SEI-2006-TN-040
Modeling and Analysis of Information Technology Change and Access Controls in the Business Context
Moore, A. & Antao, R.

Ongoing field work centered at the Information Technology Process Institute (ITPI) makes clear that processes that control change and access within information technology (IT) management and operations simultaneously reduce security risk and increase efficiency and effectiveness. The CERT's Coordination Center is building on this work. This technical note describes a system dynamics model that embodies CERT's current hypothesis of why and how these controls reduce the problematic behavior of the low-performing IT operation. CERT has also started to extend the model in ways that reflect the improved performance seen by high performers. In the longer term, the hope is that this model will help to specify, explain, and justify a prescriptive process for integrating change and access controls into organizations' business processes in a way that most effectively reduces security risk and increases IT operational effectiveness and efficiency.

http://www.sei.cmu.edu/publications/documents/06.reports/06tn040.html

CMU/SEI-2006-TN-018, ADA454363
Model Problems in Technologies for Interoperability: OWL Web Ontology Language for Services (OWL-S)
Metcalf C. & Lewis, G.

Application developers often do not have control over the services they utilize. What would happen if a service required by an application were removed from the environment or had its interface changed? What if a new and better service were introduced that an application might be able to utilize? Existing services-oriented frameworks do not protect application developers against these contingencies.

The OWL Web Ontology Language for Services (OWL-S) is a language to describe the properties and capabilities of Web Services in such a way that the descriptions can be interpreted by a computer system in an automated manner. This technical note presents the results of applying the model problem approach to examine the feasibility of using OWL-S to allow applications to automatically discover, compose, and invoke services in a dynamic services-oriented environment.

http://www.sei.cmu.edu/publications/documents/06.reports/06tn018.html

CMU/SEI-2006-TN-021
Model Problems in Technologies for Interoperability: Web Services
Lewis, G. & Wrage, L.

Web service technologies (or Web services) are experiencing a growing popularity in U.S. Department of Defense, industry, and non-defense government organizations due to their potential to enable interoperability between applications implemented on different platforms. This potential stems from Web services being based on standards that have been widely accepted and implemented, such as the Simple Object Access Protocol and the Web Services Description Language. The large number of products and tools created to facilitate the development of Web services has also contributed to their popularity. This technical note presents the results of applying the model problem approach in an initial investigation of the potential of Web services to enable interoperability.

http://www.sei.cmu.edu/publications/documents/06.reports/06tn021.html

CMU/SEI-2006-TN-012, ADA457003
On System Scalability
Weinstock, C. & Goodenough, J.

A significant number of systems fail in initial use, or even during integration, because factors that have a negligible effect when systems are lightly used have a harmful effect as the level of use increases. This scalability problem (i.e., the inability of a system to accommodate an increased workload) is not new. However, the increasing size (more lines of code, greater number of users, widened scope of demands, and the like) of U.S. Department of Defense systems makes the problem more critical today than in the past.

This technical note presents an analysis of what is meant by scalability and a description of factors to be considered when assessing the potential for system scalability. The factors to be considered are captured in a scalability audit, a process intended to expose issues that, if overlooked, can lead to scalability problems.

http://www.sei.cmu.edu/publications/documents/06.reports/06tn012.html

CMU/SEI-2006-TN-020, ADA447911
Product Line Acquisition in a DoD Organization-Guidance for Decision Makers
Bergey, J. & Cohen, S.

In the Department of Defense (DoD) acquisition environment, many organizations have not seriously considered adopting a product line approach or are reluctant to because it is not a well-understood acquisition paradigm. Nonetheless, a compelling case can be made for adopting a product line approach because it addresses a problem facing many program managers today-how to cost-effectively acquire, develop, and maintain a set of related software- intensive systems and how to respond to the needs of greater product agility in the face of the current DoD transformation.

This technical note chronicles the decisions a program manager might face in considering the adoption of a product line approach. This report uses a hypothetical acquisition to focus on why an acquisition organization should consider adopting a product line approach-instead of the traditional stovepipe approach-when acquiring a number of software- intensive systems that have a lot in common. The technical note provides a program manager with insight into the many benefits of adopting a product line approach and examines alternative acquisition approaches for acquiring a product line capability.

http://www.sei.cmu.edu/publications/documents/06.reports/06tn020.html

CMU/SEI-2006-TN-039
Proposed Taxonomy for Software Development Risks for High-Performance

Computing (HPC) Scientific/Engineering Applications, A Kendall, R.; Post, D.; Carver, J.; Henderson, D.; & Fisher, D. Because the development of large-scale scientific/engineering application codes is an often difficult, complicated, and sometimes uncertain process, success depends on identifying and managing risk. One of the drivers of the evolution of software engineering, as a discipline, has been the desire to identify reliable, quantifiable ways to manage software development risks. The taxonomy that follows represents an attempt to organize the sources of software development risk for scientific/engineering applications around three principal aspects of the software development activity: the software development cycle, the development environment, and the programmatic environment. These taxonomic classes are divided into elements and each element is further characterized by its attributes.

http://www.sei.cmu.edu/publications/documents/06.reports/06tn039.html

CMU/SEI-2006-TN-017, ADA452977
PROxy Based Estimation (PROBE) for Structured Query Language (SQL)
Schoedel, R.

This paper presents a method for applying the PROxy Based Estimation (PROBE) technique to Structured Query Language (SQL). Estimating program size is a critical component of successful software project effort estimation and cost estimation. The PROBE technique is a simple estimation method that can be used for estimating program size and effort. To date, PROBE has been used more often to estimate programs written in third-generation programming languages (3GL) such as C, C++, and Java. Its application to IT development has been inhibited by the lack of demonstrated applicability to database work. For data storage, most IT departments have transitioned from file-oriented storage (accessed by traditional 3GL languages) to relational database server software, which uses an implementation of 4GL languages such as SQL to manipulate data. SQL's logic encapsulation properties differ dramatically from those of traditional 3GL languages, so it is not clear to most developers how to effectively apply the PROBE techniques to SQL. The method presented here enables a level of estimation detail similar to the application of PROBE to traditional 3GL languages.

http://www.sei.cmu.edu/publications/documents/06.reports/06tn017.html

CMU/SEI-2006-TN-026, ADA460422
Quantitative Methods for Software Selection and Evaluation
Bandor, M.

When performing a "buy" analysis and selecting a product as part of a software acquisition strategy, most organizations will consider primarily the requirements (the ability of the product to meet the need) and the cost. The method used for the analysis and selection activities can range from the use of basic intuition to counting the number of requirements fulfilled, or something in between. The selection and evaluation of the product must be done in a consistent, quantifiable manner to be effective. By using a formal method, it is possible to mix very different criteria into a cohesive decision; the justification for the selection decision is not just based on technical, intuitive, or political factors. This report describes various methods for selecting candidate commercial off-the-shelf packages for further evaluation, possible methods for evaluation, and other factors besides requirements to be considered. It also describes the use of a decision analysis spreadsheet as one possible tool for use in the evaluation process.

http://www.sei.cmu.edu/publications/documents/06.reports/06tn026.html

CMU/SEI-2006-TN-015, ADA449727
Requirements Management in a System-of-Systems Context: A Workshop
Meyers, B.; Smith, J.; Capell, P.; & Place, P.

This report summarizes the results of a workshop focused on requirements management in a system of systems. The workshop attendees were affiliated with the Army Program Executive Office (PEO) Aviation and Training and Doctrine Command (TRADOC) Combat Developers. During the workshop, issues were identified in a number of areas, including requirements management, system-of-systems management, and system construction. Many of the issues raised address some form of the conflict that exists between a top-down, policy driven approach to the acquisition of a system of systems and a bottom-up, program-centric approach to the acquisition of an individual system.

http://www.sei.cmu.edu/publications/documents/06.reports/06tn015.html

CMU/SEI-2006-TN-032
Risk Management Considerations for Interoperable Acquisition
Meyers, C.

This report addresses interoperable risk management: the interoperability of organizations that engage in risk management in the context of a system of systems. The state of risk practice management--the specification of standards and the methodologies to them implement--is addressed and examined with respect to the needs of system-of-systems interoperability. The current practice is found to be insufficient to achieve interoperability with regard to risk management. A number of research questions are raised to associate this topic with the needs of the larger context of interoperable acquisition.

http://www.sei.cmu.edu/publications/documents/06.reports/06tn032.html

CMU/SEI-2006-TN-004, ADA44645
SAT-Based Software Certification
Chaki, S.

This report formalizes a notion of witnesses as the basis of certifying the correctness of software. The first part of the report is concerned with witnesses for the satisfaction of linear temporal logic specifications by infinite state programs and shows how such witnesses may be constructed via predicate abstraction and validated by generating and proving verification conditions. In addition, the first part of this report proposes the use of theorem provers based on Boolean propositional satisfiability (SAT) and resolution proofs in validating these verification conditions. In addition to yielding extremely compact proofs, a SAT-based approach overcomes several limitations of conventional theorem provers when applied to the verification of programs written in real-life programming languages.

The second part of this report formalizes a notion of witnesses of simulation conformance between infinite state programs and finite state machine specifications. The report also proves that computing a minimal simulation relation between two finite state machines is an NP-hard problem. Finally, the report presents algorithms to construct simulation witnesses of minimal size by solving pseudo-Boolean constraints. The author's experiments on several nontrivial benchmarks suggest that a SAT-based approach can yield extremely compact proofs-in some cases by a factor of over 10⁵-when compared to existing non-SAT-based theorem provers.

http://www.sei.cmu.edu/publications/documents/06.reports/06tn004.html

CMU/SEI-2006-TN-035
Schedule Considerations for Interoperable Acquisition
Meyers, B. & Sledge, C.

The role of schedule is fundamental to the acquisition of a particular system. This topic is of even more importance to acquisition in a system-of-systems environment. This report examines the issue of schedule considerations for interoperable acquisition. First, a Gedanken red team project is used to explore concerns about schedule in interoperable acquisition. Then, those concerns are examined in light of current requirements regarding schedule. From that examination, several research questions are proposed.

http://www.sei.cmu.edu/publications/documents/06.reports/06tn035.html

CMU/SEI-2006-TR-006, ADA449432
Specifications for Managed Strings
Burch, H.; Long, F.; & Seacord, R.

This report describes a managed string library for the C programming language. Many software vulnerabilities in C programs result from the misuse of standard C string manipulation functions. Programming errors common to string manipulation logic include buffer overflow, truncation errors, string termination errors, and improper data sanitation. The managed string library provides mechanisms to eliminate or mitigate these problems and improve system security. A proof-of-concept implementation of the managed string library is available from the Secure Coding area of the CERT Web site.

http://www.sei.cmu.edu/publications/documents/06.reports/06tr006.html

CMU/SEI-2006-TN-023, ADA453489
Specifying Initial Design Review (IDR) and Final Design Review (FDR) Criteria
Lapham, M.

Many Department of Defense (DoD) development programs, such as aircraft development programs, are typically complex and long-lived. Often, these programs are structured to demonstrate significant capability in the form of prototypes, which may be additionally intended to provide lingering operational capability. As such, technology development activities frequently include design reviews known as the Initial Design Review (IDR) and the Final Design Review (FDR) that are not present in most other systems acquisitions.

IDR and FDR content is not explicitly defined in regulations or policies; rather, it is defined by the program office. However, since IDR and FDR are the Technology Development phase's equivalent to Preliminary Design Review and Critical Design Review, this technical note proposes that they should have similar criteria, scaled for Technology Development work.

This technical note presents definitions of IDR and FDR, their context in the acquisition life cycle, a comparison of engineering emphasis during IDR and FDR, IDR and FDR pre- and postconditions, and IDR and FDR criteria and how to apply it. The audiences for this technical note are managers and developers of medium to large DoD systems that employ technology that is not mature enough to transition directly to systems development.

http://www.sei.cmu.edu/publications/documents/06.reports/06tn023.html

CMU/SEI-2006-TN-009, ADA446757
Sustaining Operational Resiliency: A Process Improvement Approach to Security Management
Caralli, R.

Organizations face an ever-changing risk environment. The risk that emanates from the day-to-day activities of the organization, operational risk, is the subject of increasing attention, particularly in the banking and finance industry, because of the potential to significantly disrupt an organizations pursuit of its mission. Security, business continuity, and IT operations management are activities that traditionally support operational risk management. But collectively, they also converge to improve the operational resiliency of the organization-the ability to adapt to a changing operational risk environment as necessary. Coordinating these efforts to sustain operational resiliency requires a process- oriented approach that can be defined, measured, and actively managed. This report describes the fundamental elements and benefits of a process approach to security and operational resiliency and provides a notional view of a framework for process improvement.

http://www.sei.cmu.edu/publications/documents/06.reports/06tn009.html

CMU/SEI-2006-TN-007, ADA453287
Sustaining Software-Intensive Systems
Lapham, M.

As today's systems become increasingly reliant on software, the issues surrounding sustainment become increasingly complex. The risks of ignoring these issues can potentially undermine the stability, enhancement, and longevity of fielded systems. Questions about sustaining new and legacy systems include

What does it mean to perform sustainment from a software perspective?

What types of development and acquisition activities are required to sustain software-intensive systems?

Although the Department of Defense (DoD) has a technical definition of sustainment, does the DoD typically consider sustainment as maintenance?

How does the increased use of commercial-off-the-shelf software complicate sustainment?

This technical note discusses these questions and presents definitions, related issues, future considerations, and recommendations for sustaining software-intensive systems. Sustainment done well leads to well-supported software- intensive systems and reduced total ownership costs and should help organizations meet current and new mission area and capabilities requirements.

The information contained in this technical note is based on information that the Software Engineering Institute gathered during work with Air Force software-intensive systems. While the information is pertinent and can be applied to systems in the commercial sector, keep in mind minimal effort was made to convert "DoDspeak" into commercial sector language.

http://www.sei.cmu.edu/publications/documents/06.reports/06tn007.html

CMU/SEI-2006-TN-036
System-of-Systems Governance: New Patterns of Thought
Morris, E.; Place, P.; & Smith, D.

Systems of systems introduce complications for information technology (IT) governance because their individual system components exhibit considerable autonomy. This technical note examines the ways in which six key characteristics of good IT governance are affected by the autonomy of individual systems in a system of systems. The characteristics discussed are (1) collaboration and authority, (2) motivation and accountability, (3) multiple models, (4) expectation of evolution, (5) highly fluid processes, and (6) minimal centrality. This report examines each characteristic in detail and, where possible, provides guidance for the practitioner.

http://www.sei.cmu.edu/publications/documents/06.reports/06tn036.html

CMU/SEI-2006-TN-019, ADA449276
System-of-Systems Navigator: An Approach for Managing System-of-Systems Interoperability
Brownsword, L.; Fisher, D.; Morris, E.; Smith, J.; & Kirwan, P.

We have crossed a threshold where most of our large software systems can no longer be constructed as monoliths specified by a single, focused, and unified team; implemented as a unit; and tested to be within known performance limits. They are now constructed as groups of interoperating systems (as systems of systems) developed by different but sometimes related teams and made to interoperate through various forms of interfaces. Unfortunately, while we can easily conceive these large systems of systems, we have trouble building them. Software engineering practices have not kept pace, and the problem will only get worse as the community begins to build Internet-scale systems of systems like the Global Information Grid.

This technical note introduces the System-of-Systems Navigator (SoS Navigator), the collection and codification of essential practices for building large-scale systems of systems. These practices have been identified through the work of the Integration of Software-Intensive Systems Initiative at the Carnegie Mellon Software Engineering Institute. SoS Navigator provides tools and techniques to characterize organizational, technical, and operational enablers and barriers to success in a system of systems; identify improvement strategies; and pilot and institutionalize these strategies.

http://www.sei.cmu.edu/publications/documents/06.reports/06tn019.html

CMU/SEI-2006-TN-037
Topics in Interoperability: Structural Programmatics in a System of Systems
Smith, J.

This technical note presents a case study on how choices of structural programmatics (e.g., hierarchical or peer-to- peer organization, centralized or decentralized execution) affect the ability to achieve programmatic interoperability in the context of large, complex systems of systems. Key systems-of-systems concepts and definitions are introduced and explored through the case study. In addition, this report illustrates the pitfalls of focusing on only one aspect of a problem and discusses the need to balance management's desires for control with the realities of systems-of-systems programmatics. This report also introduces an alternative to conventional program management practice that addresses the pitfalls previously identified.

http://www.sei.cmu.edu/publications/documents/06.reports/06tn037.html

CMU/SEI-2006-TN-013, ADA446559
Toward Measures for Software Architectures
Chastek, G. & Ferguson, R.

This technical note describes the results of a preliminary investigation into measures for software architecture. It focuses on measures that directly indicate the health of or detect a problem with the software architecture of an up-and-running software system.

Defining these architectural measures is very difficult. The software architecture deeply affects the subsequent development and project management decisions, such as the breakdown of the coding tasks and the definition of the development increments. Most existing measures for up-and-running software systems capture the cumulative results of architectural, developmental, and managerial decisions and do not directly address the health of the software architecture.

The investigation into measures requires the joint participation of the software architecture and measurement communities. Since the software architecture community has made such rapid progress over the past ten years, this report first describes what the measurement community needs to know about software architecture to understand the difficulty of defining architectural measures. The current relevant literature is then described in terms of its potential contribution to this research. Finally, the report identifies areas for future research into the application of measurement technology to software architectures.

The ultimate goal of this body of work is to provide measurement guidance and quantitative decision support to software practitioners, including software architects and project managers.

http://www.sei.cmu.edu/publications/documents/06.reports/06tn013.html

CMU/SEI-2006-TN-031
Workshop on Model-Driven Architecture and Program Generation
Lewis, G.; Meyers, C.; & Wallnau, K.

This technical note summarizes the results of a workshop held on June 2, 2006, at the Software Engineering Institute in Pittsburgh, Pennsylvania (USA). The workshop explored business and technical aspects of program generation in the context of the Object Management Groups model-driven architecture development approach. The workshop was structured around consideration of the perspectives of five different communities: standards body, vendor, acquisition, development, and research. This note recapitulates these individual perspectives and highlights important themes.

http://www.sei.cmu.edu/publications/documents/06.reports/06tn031.html

Technical Reports

CMU/SEI-2006-TR-011
Appraisal Requirements for CMMI, Version 1.2 (ARC, V1.2)
SCAMPI Upgrade Team

The Appraisal Requirements for CMMI (ARC) V1.2 defines the requirements considered essential to appraisal methods intended for use with Capability Maturity Model Integration (CMMI) models. In addition, a set of appraisal classes is defined, based on typical applications of appraisal methods. These classes are intended primarily for developers of appraisal methods to use with CMMI capability models in the context of the CMMI Product Suite. Appraisal methods, as used in this document, may be applied for different purposes, including assessments for internal process improvement and capability evaluations for supplier selection and process monitoring. This document defines the requirements for such methods, but not necessarily the conditions or constraints under which they might be applied.

The approach employed to provide guidance to appraisal method developers is to define a class of typical applications of appraisal methods (which are based on years of experience in the process improvement community) called appraisal method classes. Requirements are then allocated to each class as appropriate based on the attributes associated with that class. Thus, a particular appraisal method may be declared to be an ARC Class A, B, or C appraisal method. This designation implies the sets of ARC requirements that the method developer has addressed when designing the method.

http://www.sei.cmu.edu/publications/documents/06.reports/06tr011.html

CMU/SEI-2006-TR-008, ADA455858
CMMI for Development, Version 1.2
CMMI Product Team

CMMI for Development (CMMI-DEV), Version 1.2 is an upgrade of CMMI-SE/SW/IPPD/SS, Version 1.1. The focus of the CMMI Version 1.2 effort is on improving the quality of CMMI products and the consistency of how they are applied. This report represents the model portion of the CMMI Product Suite. Other portions of the CMMI Product Suite include the SCAMPI A appraisal method and the Introduction to CMMI training course.

CMMI now includes the concept of CMMI "constellations." A constellation is a set of CMMI components designed to meet the needs of a specific area of interest. A constellation can produce one or more related CMMI models and related appraisal and training materials. CMMI for Development is the first of these constellations.

This report contains the two models that comprise the CMMI for Development constellation: the CMMI for Development and CMMI for Development +IPPD models. The report consists of three parts. Part one is the overview, which describes CMMI concepts, model components, and guidance on using the CMMI Product Suite. Part two contains the generic goals and practices and process areas, which are used by organizations to improve their development processes. Part three contains references, acronyms, project participants, and a glossary.

http://www.sei.cmu.edu/publications/documents/06.reports/06tr008.html

CMU/SEI-2006-TR-023, ADA460414
Attribute-Driven Design (ADD), Version 2.0
Wojcik, R.; Bachmann, F.; Bass, L.; Clements, P.; Merson, P. Nord, R. & Wood, B.

This report revises the Attribute-Driven Design (ADD) method that was developed by the Carnegie Mellon Software Engineering Institute. The motivation for revising ADD came from practitioners who use the method and want ADD to be easier to learn, understand, and apply.

The ADD method is an approach to defining a software architecture in which the design process is based on the software quality attribute requirements. ADD follows a recursive process that decomposes a system or system element by applying architectural tactics and patterns that satisfy its driving quality attribute requirements.

This technical report revises the steps of ADD and offers practical guidelines for carrying out each step. In addition, important design decisions that should be considered at each step are provided.

http://www.sei.cmu.edu/publications/documents/06.reports/06tr023.html

http://www.sei.cmu.edu/publications/documents/06.reports/06tr026.html

http://www.sei.cmu.edu/publications/documents/06.reports/06tr013.html

CMU/SEI-2006-TR-005, ADA448156
Detecting Scans at the ISP Level
Gates, C.; McNutt, J.; Kadane, J.; Kellner, M.

Scans are often used by adversaries to determine the potential weaknesses in a target network or system prior to an intrusion attempt. In other cases, exploits are packaged with the scans themselves. This report presents a novel approach to detecting scans (including very stealthy scans) against, or passing through, very large networks. It meets operational requirements that are particular to detecting scans in ISP level networks.

This scan-detection approach performs an ongoing, incremental analysis of flow-level data regarding traffic inbound to a network. It is multi-dimensional and flexible, based on up to 21 characteristics describing traffic collected from any single source.

The report describes in detail a method developed to provide a probability that a particular traffic sample contains a scan. In validation testing using a manual analysis of traffic collected from a high-volume network, this method correctly classified 99.3% of TCP traffic samples.

http://www.sei.cmu.edu/publications/documents/06.reports/06tr005.html

CMU/SEI-2006-TR-003, ADA449020
Emergent Perspective on Interoperation in Systems of Systems, An
Fisher, D.

This technical report characterizes systems of systems from several perspectives; shows the role of emergent behavior in systems of systems; and introduces interoperability as the domain of development, use, sustainment, and evolution for systems of systems. It argues that the increasing importance of systems of systems was inevitable, emergent behavior is inherent in systems of systems, traditional software and systems engineering methods are inadequate for interoperation of systems of systems, and emergent methods offer a potential for cost-effective and predictable solutions. This report aims to facilitate discussion and reasoning about interoperation within systems of systems by showing some of the interdependencies among systems, emergence, and interoperation. It establishes a sizable but incomplete repertoire of topics, characteristics, and principles that are fundamental to the intersection of systems of systems, emergent behavior, and interoperation.

http://www.sei.cmu.edu/publications/documents/06.reports/06tr003.html

CMU/SEI-2006-TR-004, ADA454687
Performance Results of CMMI-Based Process Improvement
Gibson, D.; Goldenson, D.; & Kost, K.

There is a widespread demand for factual information about the impact and benefits of process improvement based on Capability Maturity Model Integration (CMMI) models. Much has been learned since the Software Engineering Institute (SEI) published a special report on this topic over two years ago. There now is evidence that process improvement using the CMMI Product Suite can result in improvements in schedule and cost performance, product quality, return on investment, and other measures of performance outcome.

This technical report summarizes much of the publicly available empirical evidence about the performance results that can occur as a consequence of CMMI-based process improvement. In addition, the report contains a series of brief case descriptions that were created with collaboration from representatives from 10 organizations that have achieved notable quantitative performance results through their CMMI-based improvement efforts.

The report is meant for members of engineering process groups, middle and first-line management, and other potential process improvement participants who wish to learn more about how CMMI can contribute to measurable improvements It also may be useful for executives and senior managers who are faced with decisions about the allocation of scarce resources for improvement efforts.

http://www.sei.cmu.edu/publications/documents/06.reports/06tr004.html

CMU/SEI-2006-TR-010, ADA457165
Proceedings of the Second Software Architecture Technology User Network (SATURN) Workshop
Nord, R.;

The second Carnegie Mellon Software Engineering Institute (SEI) Software Architecture Technology User Network (SATURN) Workshop was held April 25-26, 2006 in Pittsburgh, Pennsylvania. A total of 61 software systems engineers, architects, technical managers, product managers, and researchers exchanged best practices and lessons learned in applying SEI software architecture technology in an architecture-driven development or acquisition project. In the closing session, workshop participants noted these highlights: presentations showing the methods in action, a comparison of multiple SEI Architecture Tradeoff Analysis Method (ATAM) evaluations and cross-wise analysis, the workshop format using interactive presentations, a good mix of academic and industry perspectives, and a sharing of workshop results.

This report describes the workshop format, discussion, and results, as well as plans for future SATURN workshops. Key topics covered in the workshop and noted by the participants were the future plans of the SEI's Software Architecture Technology Initiative, the overall integration of software architecture methods and techniques, and the experiences others shared in applying the methods and transitioning them for use. Slides for the presentations and recordings of the keynote talks are available at the SATURN workshop Web site.

http://www.sei.cmu.edu/publications/documents/06.reports/06tr010.html

CMU/SEI-2006-TR-012, ADA456884
Risk Themes Discovered Through Architecture Evaluations
Bass, L.; Nord, R.; Wood, W.; & Zubrow, D.

This technical report analyzes the output of 18 evaluations conducted using the Architecture Tradeoff Analysis Method (ATAM) developed by the Carnegie Mellon Software Engineering Institute. The goal of this analysis was to find patterns in the risk themes identified during those evaluations. The major results are

• a categorization of risk themes
• the observation that twice as many risk themes are risks of "omission" as are risks of "commission"
• a failure to find a relationship between the business/mission goals of a system and the risk themes revealed during an ATAM evaluation of that system
• a failure to find a relationship between the domain of a system being evaluated and the risk themes associated with the development of that system

http://www.sei.cmu.edu/publications/documents/06.reports/06tr012.html

CMU/SEI-2006-TR-009, ADA463962
State of Software Measurement Practice: Results of 2006 Survey, The
Kasunic, M.

In February 2006, the Software Engineering Measurement and Analysis Initiative at the Carnegie Mellon Software Engineering Institute (SEI) conducted the first in a series of yearly studies to gauge the state of the practice in software measurement. To conduct this study, a structured, self-administered survey consisting of 17 questions was distributed to a random sample of software practitioners who had contacted the SEI during 2004 and 2005. The results of this study, which are revealed in this technical report, offer these benefits: they can be used to indicate (1) what measurement definition and implementation approaches are being adopted and used by the community, (2) the most prevalent types of measures being used by organizations that develop or acquire software, and (3) what behaviors are preventing the effective use of measurement (so that these barriers can be addressed). In addition, when the studies are conducted on a periodic basis, the results can indicate trends over time.

http://www.sei.cmu.edu/publications/documents/06.reports/06tr009.html

CMU/SEI-2006-TR-017, ADA454679
Systems of Systems: Scaling Up the Development Process
Humphrey, W.

Some systems have some but not all properties of systems of systems (SoS). We refer to these as SoS-like systems. This report reviews the fundamental process and project-management problems of large-scale SoS-like programs and outlines steps to address these problems. The report has eight sections. Section 1 summarizes current thinking on the nature of future complex systems, and Section 2 discusses the systems-design problems of the future, particularly the partitioning of massive systems into system-of-systems structures. Section 3 points out how large-scale systems development efforts have typically failed because of project-management and not technical problems, and that the solutions to these problems are known and highly effective, but not widely practiced. It explains why, if the project- management problems of the past are not promptly and effectively addressed, large-scale systems development programs will likely be unmanageable. Section 4 discusses the requirements for a scalable process, and Section 5 both reviews and explains the quality-management principles upon which any scalable process must rest. Section 6 reviews the nature of the project-management problems currently faced by large-scale software-intensive system development efforts and explains why attempts to scale up current methods to very large-scale systems work will almost certainly fail. Section 7 describes process strategies for supporting development of a network-like system of systems and it outlines the process and project-management topics needing further research and development. Finally, Section 8 reviews the process considerations for supporting the very large-scale integrated development programs of the future. The report concludes that, unless steps like those outlined in this report are taken in conjunction with continuing technical research and development, the large-scale systems development efforts of the future will almost certainly fail, and often catastrophically.

http://www.sei.cmu.edu/publications/documents/06.reports/06tr017.html

CMU/SEI-2006-TR-002, ADA454461
Techniques for Developing an Acquisition Strategy by Profiling Software Risks
Ward, M; Elm, J.; & Kushner, S.

The goal of acquisition planning is to create a roadmap that a program can follow to maximize its chances of successfully fielding a system that meets users' needs within cost and on schedule. Developing an acquisition strategy is a key component of acquisition planning that provides a means of addressing risks through the program structure. Programs need structured ways to reason about software risks, formulate acquisition strategies to mitigate software risk, and evaluate their current acquisition strategy in an ongoing, systematic manner.

This report introduces a taxonomy of strategy drivers and strategy elements and provides a method for performing a comparative analysis of the strategy drivers and the resulting strategic choices for the elements. The primary audience for this technical report and the accompanying Excel-based tool is program managers of government acquisition programs. The main prerequisite for successfully using this information is working knowledge of government acquisition practices.

http://www.sei.cmu.edu/publications/documents/06.reports/06tr002.html

CMU/SEI-2006-TR-021, ADA460415
Technology Foundations for Computational Evaluation of Software Security Attributes

Walton, G.; Longstaff, T.; & Linger, R.

In the current state of practice, analysis of the security attributes of software systems is typically carried out through subjective evaluations by security experts who accumulate system knowledge in bits and pieces from architectures, specifications, designs, code, and tests. In contrast, this report describes foundations for a new computational security attributes (CSA) technology. This innovative approach provides precise computational methods for defining and analyzing security attributes based solely on the data and transformations of data found within programs. CSA permits security attributes to be evaluated through automatable analysis of the functional behavior of programs. The technology can support specification of security attributes of systems before they are built; specification and evaluation of security attributes of acquired software; verification of the as-built security attributes of systems; and real-time evaluation of security attributes during system operation.

http://www.sei.cmu.edu/publications/documents/06.reports/06tr021.html

[2007] [2006] [2005] [2004] [2003] [2002] [2001] [2000] [1999] [1998] [1997] [1996] [1995] [1994] [1993] [1992] [1991] [1990] [1989] [1988] [1987] [1986] [PDF]

© 2008 Carnegie Mellon University