Software Engineering Institute | Carnegie Mellon University
Software Engineering Institute | Carnegie Mellon University

Publications by Topic: Evaluation and Analysis

SEI Reports

Adaptive Flow Control for Enabling Quality of Service in Tactical Ad Hoc Wireless Networks
(December 2010) This report details the results from 18 experiments to investigate Adaptive Quality of Service, an approach to enable applications to fulfill their missions despite network infrastructure limitations. (CMU/SEI-2010-TR-030)

A Workshop on Analysis and Evaluation of Enterprise Architectures
(November 2010) This report summarizes a workshop on the analysis and evaluation of enterprise architectures that was held at the SEI in April of 2010. (CMU/SEI-2010-TN-023)

COVERT: A Framework for Finding Buffer Overflows in C Programs via Software Verification
(August 2010) This report presents COVERT, an automated framework aimed at finding buffer overflows in C programs using state-of-the-art software verification tools and techniques. (CMU/SEI-2010-TR-029)

Case Study: Model-Based Analysis of the Mission Data System Reference Architecture
(May 2010) This report describes how AADL support an instantiation of a reference architecture, address architectural themes, and provide a foundation for the analysis of performance elements and system assurance concerns. (CMU/SEI-2010-TR-003)

System Architecture Virtual Integration: An Industrial Case Study
(December 2009) This report introduces key concepts of the SAVI paradigm, describe the POC scope, and discusses the series of development scenarios used in a POC demonstration to illustrate the feasibility of improving the quality of software-intensive aircraft systems. (CMU/SEI-2009-TR-017)

Evaluating a Service-Oriented Architecture
(September 2007) This report contains technical information about SOA design considerations and tradeoffs that can help the architecture evaluator to identify and mitigate risks in a timely and effective manner. (CMU/SEI-2007-TR-015)

Quality-Attribute-Based Economic Valuation of Architectural Patterns
(May 2007) This 2007 report shows how an analysis of the options embodied within architectural patterns allows a software and system architect or manager to make reasoned choices about the future value of design decisions, considering this value along multiple quality attribute dimensions. (CMU/SEI-2007-TR-003)

Risk Themes Discovered Through Architecture Evaluations
(September 2006) This 2006 report analyzes the output of 18 evaluations conducted using the Architecture Tradeoff Analysis (ATAM). The goal of the analysis was to find patterns in the risk themes identified during those evaluations. (CMU/SEI-2006-TR-012)

Toward Measures for Software Architectures
(March 2006) This report describes the results of a preliminary investigation into measures for software architecture. (CMU/SEI-2006-TN-013)

Acquiring Evolving Technologies: Web Services Standards
(March 2006) This technical note discusses some of the challenges of using Web services standards and presents the results generated by an assessment tool used to track the appropriateness of using this technology. (CMU/SEI-2006-TN-001)

Quality Attributes and Service-Oriented Architectures
(September 2005) This report examines the relationship between service-oriented architectures (SOAs) and quality attributes. (CMU/SEI-2005-TN-014)

Reasoning Frameworks
(July 2005) This report describes a vehicle for encapsulating the quality attribute knowledge needed to understand a system's quality behavior as a reasoning framework that can be used by nonexperts. (CMU/SEI-2005-TR-007)

Discovering Architectures from Running Systems: Lessons Learned
(December 2004) This report describes a technique that uses automatically generated runtime observations of an executing system to construct an architectural view of the system. (CMU/SEI-2004-TR-016)

Integrating Software-Architecture-Centric Methods into Extreme Programming (XP)
(September 2004) The report presents a summary of XP (Extreme Programming) and examines the potential uses of the SEI's architecture-centric methods. (CMU/SEI-2004-TN-036)

SACAM: The Software Architecture Comparison Analysis Method
(December 2003) The report outlines the first version of the Software Architecture Comparison Analysis Method (SACAM). This method was created to provide rationale for an architecture selection process by comparing the fitness of architecture candidates for required systems. (CMU/SEI-2003-TR-006)

Integrating the Architecture Tradeoff Analysis Method (ATAM) with the Cost Benefit Analysis Method (CBAM)
(December 2003) This technical note reports on a proposal to integrate the SEI ATAM (Architecture Tradeoff Analysis Method) and the CBAM (Cost Benefit Analysis Method). (CMU/SEI-2003-TN-038)

Quality Attribute Workshops (QAWs), Third Edition
(October 2003) This report describes the newly revised QAW (Quality Attribute Workshop) and describes potential uses of the refined scenarios generated during it. (CMU/SEI-2003-TR-016)

Interactions Among Techniques Addressing Quality Attributes
(June 2003) This report provides software architects a chart for determining the relationships among techniques that promote different architectural qualities. (CMU/SEI-2003-TR-003)

SEI Architecture Analysis Techniques and When to Use Them
(October 2002) When analyzing system and software architectures, the Quality Attribute Workshop (QAW) and the Architecture Tradeoff Analysis Method (ATAM) can be used in combination to obtain early and continuous benefits. (CMU/SEI-2002-TN-005)

Illuminating the Fundamental Contributors to Software Architecture Quality
(August 2002) This 2002 report presents the basic concepts of analysis models for two quality attributes-modifiability and performance, identifies a collection of tactics that can be used to control responses within those models, and discusses how to analyze the models in terms of these tactics. (CMU/SEI-2002-TR-025)

Use of Quality Attribute Workshops (QAWs) in Source Selection for a DoD System Acquisition: A Case Study
(June 2002) This case study outlines how a DoD organization used architecture analysis and evaluation in a major system acquisition to reduce program risk. (CMU/SEI-2002-TN-013)

Using Economic Considerations to Choose Among Architecture Design Alternatives
(December 2001) The SEI developed the CBAM (Cost Benefit Analysis Method), which incorporates the costs and benefits of architectural design decisions and provides an effective means of making such decisions. This paper reports on the application of this method to a real world case study. (CMU/SEI-2001-TR-035)

Active Reviews for Intermediate Designs
(August 2000) This paper describes ARID, a piloted software design review technique. (CMU/SEI-2000-TN-009)

Using Quality Attribute Workshops to Evaluate Architectural Design Approaches in a Major System Acquisition: A Case Study
(July 2000) This report describes a series of Quality Attribute Workshops (QAWs) that were conducted on behalf of a government agency during its competitive acquisition of a complex, tactical, integrated command and control system. (CMU/SEI-2000-TN-010)

Architectural Evaluation of Collaborative Agent-Based Systems
(October 1999) This report identifies features in agent-based systems that could be used to classify agent-system architectures and to guide the generation of scenarios applicable to these architectures. (CMU/SEI-1999-TR-025)

Principles for Evaluating the Quality Attributes of a Software Architecture
(May 1997) This report describes a few principles for analyzing a software architecture to determine if it exhibits certain quality attributes. (CMU/SEI-1996-TR-036)

Recommended Best Industrial Practice for Software Architecture Evaluation
(January 1997) This report details the results of two workshops on software architecture evaluation, held at the SEI in 1996. (CMU/SEI-1996-TR-025)

Quality Attributes
(December 1995) This report describes efforts to develop a unifying approach for reasoning about multiple software quality attributes. (CMU/SEI-1995-TR-021)

SEI White Papers

SAAM: A Method for Analyzing the Properties of Software Architectures

Rick Kazman, Len Bass, Gregory Abowd, and Mike Webb

May 2007

This paper describes three perspectives by which we can understand the description of a software architecture and proposes a five-step method for analyzing software architectures called SAAM (Software Architecture Analysis Method).

An Approach to Software Architecture Analysis for Evolution and Reusability

Chung-Horng Lung, Sonia Bot, Kalai Kalaichelvan, Rick Kazman

November 1997

This paper presents an approach to capturing and assessing software architectures for evolution and reuse. The approach consists of a framework for modeling various types of relevant information and a set of architectural views for reengineering, analyzing, and comparing software architectures.

Scenario-Based Analysis of Software Architecture

Rick Kazman, Gregory Abowd, Len Bass, Paul Clements

November 1996

Software architecture is one of the most important tools for designing and understanding a system, whether that system is in preliminary design, active deployment, or maintenance. Scenarios are important tools for exercising an architecture in order to gain information about a system’s fitness with respect to a set of desired quality attributes. This paper presents an experiential case study illustrating the methodological use of scenarios to gain architecture-level understanding and predictive insight into large, real-world systems in various domains.

Assessing the Quality of Large, Software-Intensive Systems: A Case Study

Alan Brown, David Carney, Paul Clements, Craig Meyers, Dennis Smith, Nelson Weiderman, Bill Wood

September 1995

This paper presents a case study in carrying out an audit of a large, software-intensive system.

An Architectural Analysis Case Study: Internet Information Systems

Rick Kazman, Len Bass, Gregory Abowd, Paul Clements

April 1995

This paper presents a method for analyzing systems for nonfunctional qualities from the perspective of their software architecture and applies this method to the field of Internet information systems (IISs).

SEI Books

Evaluating Software Architectures: Methods and Case Studies

Paul Clements, Rick Kazman, Mark Klein

November 2001

This book is a comprehensive, step-by-step guide to software architecture evaluation, describing specific methods that can quickly and inexpensively mitigate enormous risk in software projects.

SEI Presentations

A Case Study in Assessing the Maintainability of Large, Software-Intensive Systems

Alan W. Brown, David J. Carney, Paul C. Clements

March 1995

This paper presents a case study in assessing the maintainability of a large, software intensive system. The techniques we used are described, and their strengths and weaknesses discussed.

Non-SEI Publications

Assessing Design Quality From a Software Architectural Perspective
Jeromy Carrière & Rick Kazman. OOPSLA'97 Workshop on Object-Oriented Design Quality. October 5, 1997.

Making Architecture Reviews Work in the Real World
Rick Kazman & Len Bass

Predicting Software Quality by Architecture-Level Evaluation
Paul Clements, Len Bass, Rick Kazman, & Gregory Abowd. Proceedings, Fifth International Conference on Software Quality. Austin, Tx, October 1995.

Quality-Attribute-Based Economic Valuation of Architectural Patterns
Ipek Ozkaya, Rick Kazman, & Mark Klein

Using SAAM: An Experience Report
M. DeSimone, Rick Kazman. Proceedings of CASCOM '95, Toronto, ON, November 1995, pp.251-261.