NEWS AT SEI
This article was originally published in News at SEI on: December 1, 2002
For several years, the SEI has been conducting architecture evaluations with a variety of companies and DoD organizations, helping them to identify system and software architecture concerns and risks. We have noticed particular problems from an activity referred to as “enterprise integration.” The goal of enterprise integration (EI) is to provide timely and accurate exchange of consistent information between business functions to support strategic and tactical business goals in a manner that appears to be seamless. However, problems often emerge from overly ambitious or imprecise requirements resulting from inadequate plans for integrating different systems (legacy or otherwise). In this column, we describe the challenges associated with EI and provide a road map toward a solution.
Reasons for Enterprise Integration
Depending on the organization, there are different legal and business reasons for integration. For example, U.S. federal law, under the Clinger-Cohen Act, mandates that each federal agency develop a plan for EI (for example, the Veterans Administration, the Internal Revenue Service, and the DoD Office of the Secretary of Defense have all instituted EI efforts). U.S. and international corporations regularly engage in major integration efforts (the telecommunications industry consortium, KLM Airlines, EURONEXT, and the Australian Stock Exchange, for example). Effective integration is often a prerequisite for e-business success (for example, successful enterprise-wide software integration was a critical factor for the rapid success of Dell online sales).
Enterprise Integration Challenges
Integration of information systems is expensive and time consuming. Between 20% and 40% of labor costs can be traced to the storage and reconciliation of data. In addition, 70% of code in corporate software systems is dedicated to moving data from system to system.
Efforts at EI should transcend a single system or a single department, since optimization at a broader level helps to resolve the problems of redundancy and inconsistency. However, the term “enterprise” can be ambiguous; it has been referred to variously at the level of a division, a strategic business unit, a profit center, a public sector agency, and a corporation. While the scope of the term “enterprise” can legitimately vary depending on the needs of an organization, it is important to define the scope for the purpose of an integration project and to stick with that definition.
Despite the large investments that are made, many integration efforts fail for a number of technical, organizational and management, and migration planning reasons. The technical issues include the following:
- Legacy systems originated from unplanned, stovepipe development or were developed as batch, single tier.
- Data is specific to the applications and was not designed for sharing.
- The legacy systems were not initially designed for new quality-attribute requirements and are being affected by needs for interoperability, performance, security, and usability.
- The scope for the integration effort is not adequately defined.
- Decisions are made without performing adequate analyses (sometimes referred to as “management by magazine”)
The organizational and management issues include the need to obtain and maintain sponsorship and financial commitment at all levels of the organization for the duration of a potentially lengthy project; the need to break a project up into subprojects so that consistent, intermittent milestones can be measured; and the need to communicate effectively.
Finally, issues involved in migration planning are often poorly understood. These include a consideration of migrating databases and data, understanding user issues, providing training, and developing both temporary and long-term bridges between legacy systems and the newer applications.
Critical Success Factors for Enterprise Integration
EI requires developing an overall plan to determine the scope of a set of projects and then implementing this plan with a sound technical approach. Both of these activities are crucial for the success of EI. We next briefly describe the overall planning activity and the activities associated with implementing the plan.
Planning for Enterprise Integration
One way to plan for enterprise integration is to develop an enterprise architecture that represents how major information systems across the enterprise fit together. The architecture identifies the scope of individual systems and the boundaries between them. Thus, an enterprise architecture is essentially a planning activity, rather than a development activity.
In practice, however, this distinction between planning and development is often ignored. Organizations focus on the wrong sets of issues in developing an enterprise architecture, and as a result get little value from it. Here are two basic problems that often occur in practice:
- overscoping the enterprise architecture, resulting in an open-ended effort that is too ambitious to ever successfully implement
- driving the enterprise architecture down to low levels of detail
- This results in losing the focus of the enterprise architecture as well as modeling low levels of detail that will only need to be repeated when actual systems are developed.
- It also results in focusing on the functionality of individual systems rather than the interconnections between them.
Examples of enterprise architectures are shown in sidebar 1 and sidebar 2.
An enterprise architecture has a very different scope from a software architecture. The software architecture is the structure or structures of a system, which comprise software elements, the externally visible properties of those elements, and the relationships among them [Bass 2003]. Thus, a software architecture should begin where the enterprise architecture ends.
Implementing the Plan
Implementing the plan for enterprise integration requires a set of projects that rely on a systematic approach for representing business, information, and technology perspectives; a systematic method to plan for specific tactical solutions, including budgets and project plans; and an understanding of infrastructure issues, such as processors and storage, networking, data management, and management and staffing. The management and organizational issues are critical for enterprise integration. In this column, we will focus primarily on technical issues.
To understand the relationship between the technical issues, we developed an integration model, which is shown in Figure 1.
Figure 1: Integration Model
The Technical Issues
Requirements and principles deal with determining the business drivers and guiding principles that help in the development of the enterprise architecture. Each functional and non-functional requirement should be traceable to one or more business drivers. Organizations are beginning to become more aware of the need for capturing and managing requirements. Use-case modeling is one of the techniques that is used for doing this.
Business integration concerns the design and modeling of business processes. Processes represent how the user sees the system. They are often highlighted when companies merge. Business process reengineering (BPR) decisions are often made in the context of enterprise integration.
Presentation integration involves with integration of corporate knowledge and business processes to enable an enterprise to share applications, services, and core competencies among the enterprise’s main constituents: employees, customers, partners, and suppliers.
Data integration deals with aspects of integrating data within an enterprise. It deals with how data is modeled and the meaning of the data. It deals with normalization, validation, and integrity of data and what translations need to be applied to the data for exchange between applications within the enterprise or between the enterprise and outside systems.
Control integration deals with different types of messaging between applications and how these messages are handled based on different modes of communication and protocols.
Connectivity relates to workflow, data, and service connectivity and is handled by application bridges and gateways, message-handling services, and basic communication protocols.
Quality attributes involve architectural decisions that influence quality characteristics such as performance, dependability, and security. Quality attributes span several aspects of the integration model. When the software architecture is specified, designers need to determine the extent to which architecture decisions influence quality attributes, the extent to which techniques used for one quality attribute support or conflict with those of another attribute, and the extent to which multiple quality-attribute requirements can be satisfied simultaneously.
Quality-attribute issues do not get enough attention in enterprise integration. Quality-attribute requirements are usually defined too late in the development process. One problem is to determine how business requirements and drivers translate into quality-attribute requirements.
Application integration deals with getting different applications to work together using mechanisms such as automatic event notification, flow control, and content routing. Transactional integrity and application-context transformation across applications are important aspects of application integration. Application integration spans several layers of the integration model. Clear techniques are not available for making decisions about which applications to integrate and determining the nature of the problem. Applications on the same server may be easier to integrate, as cross-network integration can be complicated by different operating system versions, network delays, and different protocols.
Example Requirements and Solutions
The following cases illustrate optimal examples of successful control, data, and presentation integration:
- Delivering information to the point of care: A physician accesses information about a patient, notifies appropriate specialists, and secures resources. (On average, hospital physicians need access to information they don’t have three to five times a day.)
- Business-to-business transactions: Corporations implement their transactions and enact their business-process models using intelligent agents to access back-end corporate services.
- Corporations build customized services by fusing data content and services originally available as HTML web pages, programmatic scripts, and back-end applications so they can be accessed, fused, processed, and presented in various platforms.
Figure 2 shows a publish/subscribe configuration that can be used as a solution for integrating diverse applications. Figure 3 illustrates the component interactions. In the figures, publisher is the provider of the information, subscriber is the consumer of the information, and broker is the service that controls and routes all the messages from the publishers to the subscribers. Topic is the subject of the information that is contained in the message. A publisher provides information on a specific topic, while a subscriber receives information for only the topics that it is interested in.
Figure 2: Publish/Subscribe Configuration
Figure 3: Publish/Subscribe Component Interactions
The solution depends on integration of heterogeneous services (operating systems, programming language, network protocol, data representation) and Internet accessibility of services (transactional services, security, messaging, naming services). Markup languages provide the ability to achieve these requirements more easily. One common approach to enterprise integration problems is the use of Enterprise Resource Planning (ERP). See sidebar 3 for details.
A Roadmap to Future Progress
As we have seen, the demand for enterprise integration is broad. This affects all organizations, and addressing enterprise integration can often be a key to meeting the strategic goals of the organization.
A number of recent trends provide a foundation that can lead to future success. Middleware technologies have advanced rapidly over the past 10 years, and these enable more options for integration than had been previously available. The maturing of markup languages such as XML enable more effective integration, particularly between structured data, such as data from databases, and non-structured data, such as email. The Web can serve as a common front end for integrating a variety of applications, and it can enable effective presentation integration. The emergence of enterprise portals over the past several years demonstrates the strong interest and need for effective presentation integration. In addition, ERP applications, while still displaying significant problems, have become more mature, and their interfaces are better able to share data with other applications.
Despite this progress, the overall status of the field is immature. The issues of planning and scoping continue to hamper many efforts. There is significant confusion between the roles of different types of architectures. As a result, organizations sometimes go into too low a level of detail in developing an enterprise architecture, leading to a failure to adequately specify the scope, as well as a duplication of the effort that will later be applied to defining the software architecture. Although middleware technologies have matured significantly, clear guidelines are not yet available about when to use different types of technologies. As a result, some organizations have made decisions in a haphazard manner and have sometimes focused on vendor-driven approaches without adequate analysis. Some ERP products are heavily promoted as a total solution to enterprise integration. However, substantial modifications are often required to the ERP product for it to work in a specific organization. In addition, interfaces with legacy systems and other ERP products can be problematic.
The organizational issues that we mentioned earlier can be significant. An enterprise-integration effort affects an entire organization, and it is necessary to have long-term management support and financial commitment, realistic plans, and systematic migration planning.
To address these issues, a community-wide enterprise integration agenda needs to be developed. Such an agenda should focus, as a starting point, on the following issues:
- effective techniques for developing an appropriate scope for enterprise integration, including understanding business goals and how these goals are affected by current systems
- effective mapping between enterprise architectures and software architectures
- understanding the role and influence of organizational issues on enterprise integration
- understanding the role (and limitations) of middleware technologies and ERP solutions: when and how to select and use them
- understanding the role of migration planning
Bass, L.; Clements, P.; & Kazman, R. Software Architecture in Practice (Second Edition). Reading, MA: Addison-Wesley Longman, 2003 (to be published).
Information Technology Management Reform Act (ITMRA)
SEI Enterprise Integration White Papers
Examples of Enterprise Architectures and Frameworks
The Zachman framework is a classification system for design artifacts. It enables concentration on aspects of an object as well as the overall context of the object.
© John A. Zachman
(Note: A PDF version of the Zachman framework can be downloaded from http://www.zifa.com/framework.html and enlarged for easier reading.)
Zachman distinguished between roles and product abstractions. Roles include planner, owner, designer, builder, and subcontractor. Product abstractions include data, function, network, people, time, and motivation.
The Zachman framework has been used extensively as the starting point for enterprise architectures. The framework was originally proposed in 1978, using terms appropriate for structured analysis techniques. The conceptual approach needs to be updated to account for object-oriented language and client-server and Web-centric approaches.
While most analysts recognize the top two levels of the framework as being the appropriate role of an enterprise architecture, in reality they tend to delve into a much lower level of detail.
The U.S. Department of Defense requires an architectural approach called C4ISR for systems within the domain of command, control, and intelligence. C4ISR has three components:
- operational architecture, to describe tasks, operational elements, and information flows
- technical architecture, to define a minimal set of rules (or set of standards) governing the arrangement, interaction, and interdependence of components
- systems architecture, to describe systems and interconnections necessary to carry out tasks defined in the operational architecture
In practice, the C4ISR goes into substantial levels of detail. There is not a clean separation between the planning and development stages and, as a result, there tends to be substantial duplication of effort between development and planning. Many practitioners don’t have a clear understanding of the purpose and scope of the C4ISR.
Enterprise Resource Planning Solutions
Enterprise resource planning (ERPs) solutions are integrated, technology-based solutions that provide support for standard enterprise needs in such areas as finance, human resources, and logistics. A number of vendors provide ERP solutions, including PeopleSoft, SAP, Baan, and Oracle. ERPs are popular because they offer the promise of enabling an organization to leverage the research and development efforts of the ERP vendors. Functional areas such as taxes, purchasing, and human resource management have a significant amount of commonality between organizations, and there are strong arguments for purchasing a ready solution rather than developing an application from scratch.
ERPs can make good sense for an organization. However, the benefits are far from automatic. In fact, a number of disasters in ERP implementations have occurred. A decision to implement an ERP requires careful analysis of the following factors:
an understanding of the gap between the underlying object, data, or functional models of the ERP solution and those currently supported by an organization’s legacy systems (often substantial effort is required to customize the ERP or change the organizational processes to match those of the ERP)
an understanding of specific ways in which the ERP will interface with legacy systems, other ERPs, and future development efforts
an understanding of migration issues, such as user training, data migration, phasing in of the ERPs, and phasing out of the legacy systems
development of realistic cost and schedule estimates reflecting realistic expectations