Selecting Advanced Software Technology in Two Small Manufacturing Enterprises
Bill Anderson
Len Estrin
Charles Buhman
CMU/SEI-2003-TN-020
May 2004
Technology Insertion Demonstration and Evaluation Program
Unlimited distribution subject to the copyright.
[Acknowledgements] [Executive Summary] [Abstract] [1 Introduction] [2 Case Study]
[3 Lessons Learned] [4 Summary] [Appendix A: Analytic Hierarchy Process]
[Appendix B: AHP Requirements Matrix] [Appendix C: Product Dossier Guidelines]
[Appendix D: Cost Comparison Spreadsheet]
[Appendix E: Manufacturing Execution Systems Product Survey]
[References] PDF File
Acknowledgements
The authors wish to gratefully acknowledge the contributions of
- U.S. Representative Mike Doyle (18TH Congressional District) whose vision of the Technology Insertion, Demonstration, and Execution (TIDE) Program has and will help many small manufacturing enterprises;
- Joseph Magdic and Todd Sterlitz for their patience and determination to make this a successful selection effort;
- Grace Lewis for her expert assistance with the PECA1 methodology;
- Ed Morris for the Commercial off-the-shelf (COTS) Product Dossier Guidelines;
- Simon Frechette, Al Jones, Michelle Steves, and Craig Schlenoff for their team support and background research; and last but not least,
- David Allinder for bringing the opportunity to us and helping to make the execution a success.
[Acknowledgements] [Executive Summary] [Abstract] [1 Introduction] [2 Case Study]
[3 Lessons Learned] [4 Summary] [Appendix A: Analytic Hierarchy Process]
[Appendix B: AHP Requirements Matrix] [Appendix C: Product Dossier Guidelines]
[Appendix D: Cost Comparison Spreadsheet]
[Appendix E: Manufacturing Execution Systems Product Survey]
[References] PDF File
Executive Summary
This paper documents two small manufacturing enterprises' (SMEs') efforts to select advanced software technologies for their business operations. While the two companies' market spaces are completely different, each faced business and operational issues that are common to the broad SME community. Conducting both companies' technology selection efforts concurrently allowed the Technology Insertion, Demonstration, and Execution Program to address a wide range of issues and better leverage the selection expense.
The generic selection methodology used was a downsizing of the PECA methodology augmented by Analytic Hierarchy Process (AHP) decision support (see Appendix A). PECA was developed by the National Research Council of Canada and the Carnegie Mellon Software Engineering Institute. The body of this report describes the companies, the process, the issues, and the lessons learned during the software selection. The lessons taught us how important it is for SMEs to
- understand their business and how the proposed software will support their firm's growth strategy
- develop or use a process to assign tasks and involve stakeholders
- if necessary, involve specialists in decision support and technology adoption to help clarify issues and identify potential pitfalls
- investigate vendors and their software offerings from a variety of perspectives
[Acknowledgements] [Executive Summary] [Abstract] [1 Introduction] [2 Case Study]
[3 Lessons Learned] [4 Summary] [Appendix A: Analytic Hierarchy Process]
[Appendix B: AHP Requirements Matrix] [Appendix C: Product Dossier Guidelines]
[Appendix D: Cost Comparison Spreadsheet]
[Appendix E: Manufacturing Execution Systems Product Survey]
[References] PDF File [0.60MB]
Introduction
1.1 Background
Small manufacturing enterprises (SMEs) today are faced with many challenges: among them are global competition, volatile markets, and rapidly evolving technology. These challenges require SMEs to raise their performance to new levels. SMEs must operate with increased efficiency to meet the demands of global competition. They must reengineer their processes to reduce the time to market for new products. They must also continually improve their products and services to meet ever-increasing customer demands. Advanced Information Technology (IT) tools such as computer-aided engineering (CAE) and integrated manufacturing execution systems (MESs) can help SMEs achieve these goals.The indications are, however, that many SMEs have yet to adopt advanced, commercial off-the-shelf (COTS) software tools. For example, the Air Force white paper, Initiative for Small and Medium Enterprises, quoted a study of 1,002 companies. According to the study, 35% of companies with 50 or fewer employees had a computer-aided design and engineering capability. For companies with 500 employees or more, the figure was 85% [Boden 99]. Similarly, a survey of 200 SMEs in southwestern Pennsylvania found that
- 28% have solid modeling capabilities.
- 23% have simulation capabilities.
- 16% use Finite Element Analysis.
- Fewer than 30% communicate directly to suppliers and customers over the Internet [Catalyst 02].
SME reluctance to adopt advanced software technologies may be attributed to various factors, barriers, and constraints. These factors include the perception that advanced software represents a cost, not an asset; the lack of knowledge of the technologies; the lack of financial resources; and the lack of expertise in technology adoption. The Technology Insertion, Demonstration, and Evaluation (TIDE) Program investigated these issues by collaborating with two SMEs to specify and select advanced software technologies. In this effort the TIDE program and the Software Engineering Institute also collaborated with the Duquesne University Institute for Economic Transformation (IET) and the National Institute of Standards and Technology (NIST).
1.2 Magdic Precision Tooling, Incorporated
Magdic Precision Tooling, Incorporated designs and manufactures sophisticated compaction tooling for the powder metal industry. Over several years, Magdic has worked with the Duquesne University IET to implement strategic business planning, cross-training, process flow optimization, and other continuous improvement activities. To maintain its growth and profitability, the company began looking at technology improvement. Specifically, the firm identified the following areas of concern regarding its design and manufacturing process:
- The total cycle time was greater than desired.
- An opportunity existed for manufacturing process improvement.
- Processes were paper based.
- Retrieval of legacy information was difficult.
- Revision control of drawing sets was manual.
- Shop scheduling was not optimized.
- Shop capacity was difficult to monitor.
These issues are typical of the "sneaker net" communication scenario found in many SMEs. Processes and tasks are described verbally. Job orders are delivered by hand throughout the shop floor. If paper documentation exists, it is often incomplete or out of date.
Magdic personnel felt that electronic data display tools could enhance data management, improve documentation, lead to parallel job processing, and ultimately help to reduce product cycle time. At the same time, these improvements would help Magdic to enhance its position as a market leader for rapid product design and delivery. IET consultants subsequently matched Magdic's improvement need with the technology adoption research being performed by the TIDE Program.
1.3 Gentile Manufacturing Company
Gentile Manufacturing Company, Incorporated designs and manufactures sophisticated parts and assemblies. Gentile had all of Magdic's challenges plus the following unique issues:
- Cost tracking was manual and not progressive.
- The quotation process was lengthy and difficult.
- A key client required real-time visibility of work status.
- The raw material inventory was not tracked.
Most importantly, Gentile's largest customers were demanding that the company conduct business in a "pure" electronic format. The ability to electronically manage and integrate business and operational data would enable Gentile to respond to this demand.
1.4 Project Motivation
From the perspective of the TIDE Program, the ability to help two different companies adopt a common technology solution had a number of advantages. It would allow the TIDE Program to increase the amount of technology adoption data gathered while leveraging program resources. It would enable TIDE personnel to acquire information about two different types of manufacturing businesses. (Magdic specializes in low-volume, custom products. Gentile was a higher volume job shop.) Finally, it would enable the TIDE Program to measure the impact of the software on both businesses going forward.
1.5 Project Scope
As initially conceived, the effort to document the process and benefits of adopting a manufacturing execution system (MES) would be conducted in three phases: Discovery and Planning; System Implementation; and System Analysis and Publication. However, shortly after completing the Discovery and Planning Phase, Gentile lost a large customer. That loss along with continued weakness in the metalworking market forced the company to reorganize its business. As a result, Gentile opted to not continue with the implementation phase of the project. Magdic Precision Tooling implemented the software technology that was selected. Magdic's implementation effort will be documented in a future technical note.
The remainder of this paper describes the specification and selection effort, and presents lessons learned to help other SMEs streamline their technology selection efforts.
[Acknowledgements] [Executive Summary] [Abstract] [1 Introduction] [2 Case Study]
[3 Lessons Learned] [4 Summary] [Appendix A: Analytic Hierarchy Process]
[Appendix B: AHP Requirements Matrix] [Appendix C: Product Dossier Guidelines]
[Appendix D: Cost Comparison Spreadsheet]
[Appendix E: Manufacturing Execution Systems Product Survey]
[References] PDF File
2 Case Study
During the Discovery and Planning Phase, TIDE, Magdic, and Gentile personnel
- performed a needs assessment and business case analysis
- established a selection team and selection process
2.1 Needs Assessment and Business Case Development
To participate in the TIDE Program, Magdic submitted a technology adoption proposal. TIDE personnel reviewed the proposal, compared the proposal to Magdic's growth strategies, and evaluated Magdic's ability to implement the proposal. Gentile was introduced to the TIDE Program through the TIDE workshop "Introduction to Ecommerce for SMEs" conducted in December of 2001.
In the case of Magdic Precision Tooling, consultants from the Duquesne University IET had previously helped Magdic to implement a series of business process improvement activities. These changes resulted in a 20% increase in capacity without an increase in overhead. Magdic's strategy was to continue improving workflow to further reduce delivery times, enhance customer service, and obtain a competitive advantage. The company wanted help implementing a computer-based system for controlling job information. That system would allow Magdic personnel to
- scan and store drawings electronically
- enter and save job information with electronic order files
- display drawings along with the latest changes at each machine
- retrieve archived job files
- integrate engineering data with electronic order information
The consultant from the Duquesne University IET helped Magdic develop a business case. Based on an investment of $70K, company officials predicted a 10% increase in capacity and a 30% reduction in cycle time on new tool sets, resulting in a capacity to add $200K in new sales annually. Magdic presented this business case in its proposal.
After reviewing Magdic's proposal, TIDE personnel recommended that Magdic expand it to cover an integrated MES that would provide the desired capabilities while linking accounting, billing, and other front-office functions. It would also enable customers to review the status of their orders in real time. When fully implemented, the MES could help Magdic to establish a virtually paperless manufacturing environment.
In the case of Gentile manufacturing, three customers had asked the company to set up and maintain electronic Web portals. TIDE staff members invited experts from NIST to explore the possibility of a one-to-many portal translator or some level of automation to assist in the maintenance of these portals. While some one-to-many portal automation was possible, NIST specialists concluded that Gentile lacked the basic manufacturing execution software needed to automate a portal translation. An MES could provide that capability. In addition, Gentile had an opportunity to take over the renewal parts manufacturing business for another company. This activity also would require an MES to manage the volume of business. While no formal business plan was developed, the new business-forcing function justified Gentile's interest.
2.2 Establish Selection Team and Selection Process
In the next step, TIDE staff members worked with Magdic and Gentile personnel to identify roles and responsibilities for participants and to develop a process for software selection. Based on their experience in analyzing and specifying COTS systems, TIDE specialists suggested using the PECA methodology2. PECA was jointly developed by the National Research Council Canada and the Carnegie Mellon Software Engineering Institute (SEI) to evaluate COTS software, document the factors involved, and record the decision-making process.
Initially, the TIDE specialists proposed that Magdic and Gentile employees form a joint selection team. TIDE staff members would train team members on the PECA process and document their efforts. However, the lack of time and available SME personnel forced the TIDE team to change the strategy. Instead of training a software evaluation team, TIDE members ended up serving on it. Selection team members included Charles Buhman, Bill Anderson, and Grace Lewis from the SEI;3 Todd Sterlitz, Vice-President, Gentile Manufacturing Co.; Joe Magdic, President, Magdic Precision Tooling; and Simon Frechette, NIST.
In its first activity, the team tried to define the goals and scope of evaluation. The team struggled with "scope" verses "goal" semantics, wasting time in the process. Eventually, the team agreed on the following:
Scope: Evaluate a small set of software packages, hardware, and infrastructure to support shop floor control, visualization, and a paperless, Internet-enabled environment.
Goal: Select software that satisfies the needs of two small manufacturers and considers the shop floor environment, stakeholders' needs, ecommerce, shop floor visualization, and collaboration capability.
These statements are not significantly different. The teams suggests that the scope and goals must be precisely defined and clearly differentiated to avoid wasting time on semantics.4
Next, the team identified a series of tasks and a schedule for completion. This too required discussion and negotiation. For example, SME managers Joe Magdic and Todd Sterlitz simply did not have the time to take advantage of external resources such as the SEI software demonstration laboratory. They also did not have time to travel across town to the SEI facility. And they needed to limit the time that they and their employees could commit to this effort. The time factor remained an issue throughout the demonstration project, and a number of activities were modified to expedite matters.
2.2.1 Software Requirements Specification
This task involved identifying the fundamental specifications of any MES. The team agreed on a number of fixed requirements. These included
- budgets for purchase and implementation
- limits on the training time and effort that the MES would require
- an ability to implement the system within time deadlines imposed by customers.
Additional fixed requirements stated that the selected system would have to
- be PC based
- use Windows NT/XP/2000
- be compatible with Peachtree software
- meet key customer criteria (e.g., electronic collaboration and ecommerce capabilities)
- support AutoCAD and Unigraphics packages
Early on, the team dropped the requirement for compatibility with Peachtree software because each candidate MES featured an internal accounting package.
Having identified an initial set of selection criteria, the team discussed the need to interview stakeholders from accounting, purchasing, engineering, quality, production, technology, shipping, and receiving. The goal was to solicit lower level requirements. In the end, however, only the accounting stakeholder was interviewed. Joe Magdic and Todd Sterlitz provided the additional input to save the selection team valuable time. Joe and Todd were familiar with both manufacturing and business operations, and had previous experience implementing computer-based systems.
In general, the criteria covered four areas:
- functionality
- ability to integrate with legacy systems
- adoptability
- strength of the vendor
Under each area, the team listed specific requirements. For example, "adoptability" included having Windows conventions, being very intuitive, and having a short learning curve.
The team used a decision support tool to weigh and prioritize the requirements. The tool helped facilitate communication among team members. At the same time, it provided a yardstick to measure candidate MES packages, and also allowed the team to analyze the impact of decisions on candidate software packages. This "sensitivity analysis" feature became important later in the evaluation.
2.2.2 Comparing Alternative Software Packages
TIDE members from NIST investigated MES packages (see Appendix D) and CAD viewer packages [Steves 03]. In addition, the selection team researched the Internet, reviewed trade publications, and informally polled SME employees and customers. Based on that input, the team developed a short list of four MES packages. Figure 1 illustrates the process they used to compare the packages.
The selection team interviewed candidate vendors for first-pass fit against the requirements. Each package appeared to meet the requirements. It became a matter of judgment as to how well, how easily, how quickly, and so forth, each package would perform. Furthermore, each vendor featured local support, a large base of installed systems, and an active user community.
Next, the team asked for product demonstrations. All the product vendors could provide interactive live demonstrations of their systems via Internet remote-session-viewing technology (WebEx5 in this case). The vendors ran their software on their local systems while the evaluation team watched live via the Internet. This eliminated one candidate MES; the team felt that the package did not fit the underlying make-to-order business process.
The selection team asked the remaining three candidate vendors to bid to a representative system. The three bids were compiled into a spreadsheet that grouped costs into equivalent categories. (See Table 2) The spreadsheet included one aspect of life-cycle cost (annual maintenance fees) to indicate operational cost.
Figure 1: Comparing Alternate Software Packages
On the surface, the price between the high and low bids differed by less than seven percent. However, each vendor took a different path to reach the net price. One package carried a high list price that was deeply discounted and that served as the base against which a multiplier was applied to derive annual maintenance fees. In this first case, a high list price resulted in high annual maintenance fees ($5,400 per annum). The next package had a low list price, offered no discount, but used a high multiplier to derive the highest annual maintenance fees ($5,900 per annum). The third package had a list price in the middle that, when extended by the multiplier, produced the lowest annual maintenance fees ($2,600 per annum). When these recurring fees are considered over a few years they became a significant cost differentiator.
Another large variable was the cost and recommended levels of training and start-up assistance. The recommended packages varied from eight days of consulting plus two distance-learning classes for $11,300, six days of consulting at $7,200, and three days of consulting plus unlimited factory and Web-based training for $2,900.
The team cut the short list to two vendors and then checked those vendors' references. Using the Analytical Hierarchy Process (AHP) tool, a method for prioritizing decisions by incorporating relevant decision criteria, the team evaluated and compared the two vendors' software packages, and conducted a sensitivity analysis. (See Appendix A.) That analysis raised questions about whether either vendor's package could provide paperless, shop floor control. This prompted the selection team to ask for a second round of demonstrations. One vendor was unable to demonstrate the requested functionality, despite earlier claims that the product's current version could provide it. As a result, that vendor-which had been the leading candidate-lost the selection team's confidence and the sale. The second vendor readily admitted that this functionality was new, and although the vendor was confident that the software could provide paperless shop floor control, the vendor could not provide a reference that could vouch for that functionality; Magdic would be the first company to use it. The selection team appreciated that vendor's frankness and awarded it the contract.
The second set of demonstrations put the project behind schedule. However, it validated the benefit of the decision-support software and the structured COTS selection process (and verified the importance of demos to validate vendor claims).
2.2.3 Hardware Considerations
The implementation of an MES at Magdic required a system server, a large format scanner, and 10 terminals, one at each shop floor station. TIDE team members qualified hardware with the software vendor's and Magdic's concurrence. The TIDE-supplied hardware was purchased following SEI equipment guidelines, leveraging the university's purchasing agreements with an existing supplier. A mix of PCs and less expensive thin client workstations for the shop floor was purchased. Magdic purchased the Uninterruptible Power Supply (UPS), firewall, and network upgrades. A third-party network installer conducted a site survey and added several network drops and a cabinet to house the server and UPS.
2.2.4 Procurement and Licensing Issues
Procurement was much more complicated than originally anticipated. Despite the advice of vendors, certified Microsoft service representatives, and several experienced IT personnel, TIDE members were unable to discern the correct licensing requirements for the MES. Some products (the thin client terminals) required Certified Application Licenses. Others (the PC thin client emulators) came pre-licensed and were compatible with the terminal services environment. This was counterintuitive as the PCs (which have a stand-alone utility) came with an embedded license for the relatively less popular terminal services mode of operation, while the thin client terminals that only have utility in that mode required a separate license expenditure. This situation made projecting costs difficult. For example, the price of backup software tripled when its license incompatibility was finally resolved.
Furthermore, the maintenance contract provided by the MES vendor did not cover the integrated third-party packages. When one of these packages publicly announced feature updates, the selection team learned that it did not have the right to them. This raises a number of concerns.
- If a company purchases additional software licenses, which versions of the third-party software are included? Are these versions compatible?
- When the third party drops support of a given version, will the vendor take over?
- Who is responsible for purchasing updates? One would assume that the vendor would provide them under the maintenance agreement, but this was not the case.
The vendors' recommendation to purchase separate upgrades for the embedded third-party products generated a set of issues as well.
- If the SME buys upgrades separately, what level of coordination through the base vendor is available?
- Does the SME have the legal right (without harming its ability to get continued support from the base vendor) to integrate a revision of third-party software into the suite?
- Will the base vendor test and notify compatibility with future third-party revisions?
- Does the vendor supply a clear interface specification and instructions for installing the third-party software? Does the third party sanction the practice and procedure?
These are not just theoretical considerations. When the MES vendor had difficulty correcting a fault in a Web viewer portal, Magdic was willing to hire a third-party expert to remedy this fault. However the license restricted reverse engineering, derivative work, and remedial software repairs.
The solution is to establish a sound vendor relationship and to make sure company needs align with the vendor's target market, so that the vendor will want to address (or at least not ignore) customer requests.
2.2.5 Tools and Artifacts
As part of the software selection process, TIDE staff members applied a number of artifacts:
- the specific hierarchical requirements tree that guided our product evaluation (see Appendix B).
-
a product dossier document that originated from the SEI Evolutionary Process for Integrating COTS-based systems (EPIC) [Albert 02]. A product dossier highlights a broad range of product evaluation issues (see Appendix C).
For information on EPIC see http://www.sei.cmu.edu/cbs. - a cost comparison spreadsheet (see Appendix D).
- a Manufacturing Execution Systems Product Survey, a product comparison matrix to aid in the software selection process (see Appendix E).
[Acknowledgements] [Executive Summary] [Abstract] [1 Introduction] [2 Case Study]
[3 Lessons Learned] [4 Summary] [Appendix A: Analytic Hierarchy Process]
[Appendix B: AHP Requirements Matrix] [Appendix C: Product Dossier Guidelines]
[Appendix D: Cost Comparison Spreadsheet]
[Appendix E: Manufacturing Execution Systems Product Survey]
[References] PDF File
3 Lessons Learned
Budget and time limitations, a bewildering array of products, and lack of expertise can pose serious challenges to SMEs interested in adopting advanced software. The TIDE demonstration responded to these issues by emphasizing both preparation and process. The TIDE Program offers the following guidance for selecting advanced software technologies:
- The size of the company will determine the type and amount of process required. With fewer than 30 employees each, both Magdic and Gentile lacked the "mass" needed for the formal PECA methodology, forcing the process to adapt to the circumstances. However, the participants confirm that some structure was necessary to move the software selection process forward.
- Team composition affects team tasking. If the top decision makers are on the team, tasks that are motivated by upward communication and authority enablement are less important, if not unnecessary. Top-level management in a small enterprise is also well founded in comprehensive business process knowledge, reducing the discovery value of non-team stakeholder involvement. Stakeholder involvement becomes more motivated toward buy-in, training, and user acceptance.
- Beware the PowerPoint6 demonstration. When a vendor switched from a live WebEx demonstration to a canned PowerPoint slide show, vaporware7 was soon uncovered.
- Decision-support software can be very helpful for software selection and other issues. Properly implemented, decision-support software can help rank, compare, and clarify subjective issues, improve communications among different stakeholders, and facilitate the "what if" thinking that can lead to better decisions. However, the key to efficient use of this software is in limiting the scope of the investigation. Joe Magdic felt that an SME user, before using the software with confidence, would need someone to guide the process several times.
- Stay in the vendor's sweet spot. Finding a vendor that knows and is committed to the SME's business is critical. If the vendor is committed to the SME's market, the SME's issues will be market issues, creating more incentive for the vendor to resolve them. In addition, there may be other users who have already addressed common questions and issues.
- SMEs must do their homework. Often, vendors and prospective customers focus on the "bells and whistles" of the software, rather than the "nuts and bolts." Once that software has been installed, however, "nuts and bolts" features become very important. With so much on the line, SMEs need to learn as much as they can about the software's capabilities, compatibilities, and processes. This requires the SME to do more than check references. Ideally, the SME should visit and interview customers who have similar operations, if possible. The SME should require the vendor to demonstrate typical or critical tasks.
- Trainers should have domain expertise. Pay attention to the trainer's background and domain expertise before you engage. The accounting functions generally demand a deep background and understanding of accounting and how the software operates. This is not the same knowledge that it takes to understand the operations on the shop floor.
- Vendors will sell flexibility. The marketplace forces the vendor to be all things to all people (or at least a broad enough set of people to generate a market). But in reality the software will have "optimal use scenarios"-those ways of using the system that are tried and true. These are the scenarios that will have the lowest implementation risk; SMEs should find them and change their practices to take advantage of them.
- The SME must be prepared to change. COTS software is designed around a general business model. In most cases, the SME will have to modify its business and operational processes to use the software. To minimize the changes, the SME should select a package that fits its needs and follows the way it does business. Still, the SME should expect that changes will be necessary and desirable, especially if the software embodies improved or "industry best" practices. This also will keep the SME closer to the vendor's sweet spot.
- Ask questions and more questions. Such questions include the following:
- What 3rd party packages are bundled in the software suite and will the vendor support them?
- What is involved in converting legacy data to work with the new system?
- Does the vendor have a mechanism to educate employees about the optimal process scenarios to leverage his software?
- Do you need editable versions of (and rights to use) the training materials, perhaps to generate your own process procedures?
[Acknowledgements] [Executive Summary] [Abstract] [1 Introduction] [2 Case Study]
[3 Lessons Learned] [4 Summary] [Appendix A: Analytic Hierarchy Process]
[Appendix B: AHP Requirements Matrix] [Appendix C: Product Dossier Guidelines]
[Appendix D: Cost Comparison Spreadsheet]
[Appendix E: Manufacturing Execution Systems Product Survey]
[References] PDF File [0.60MB]
4 Summary
Advanced software technologies can increase productivity and reduce costly errors. However, selecting the best software requires understanding a number of factors. The TIDE Program investigated these factors during the course of selecting MES software for two SMEs. The effort underscored the need for SMEs to
- understand their business and how the proposed software will support their firms' growth strategy
- use a process to understand requirements and correlate to capabilities
- scale the selection process to fit the organization
- involve experienced personnel (including outside decision support and technology adoption specialists if necessary) to clarify issues and identify pitfalls
- investigate vendors and their products from a variety of perspectives
[Acknowledgements] [Executive Summary] [Abstract] [1 Introduction] [2 Case Study]
[3 Lessons Learned] [4 Summary] [Appendix A: Analytic Hierarchy Process]
[Appendix B: AHP Requirements Matrix] [Appendix C: Product Dossier Guidelines]
[Appendix D: Cost Comparison Spreadsheet]
[Appendix E: Manufacturing Execution Systems Product Survey]
[References] PDF File
Appendix A: Analytic Hierarchy Process
The Analytic Hierarchy Process (AHP) was proposed by Saaty over 20 years ago and is a widely used technique for multi-attribute decision making [Saaty 80]. It is a method of prioritizing decisions by incorporating relevant decision criteria.8 This prioritization achieved through pair-wise comparisons of competing objectives and through making subjective judgments. This results in a ratio scale of relative values. The AHP is carried out in two phases. In the Design Phase, a criteria hierarchy is set up. In the Evaluation Phase, pair-wise comparisons are used to evaluate alternatives. Figure 2 illustrates the major steps involved in an AHP facilitated evaluation.
Structuring the Evaluation
The initial step in using the AHP tool is structuring the decision to be made. In this case, the method was used to evaluate and eventually recommend a COTS product.
Criteria Development
Criteria are statements or conditions that serve to validate that a requirement has been met. They help to translate the subjective to a more objective perspective. Criteria development can be a layered process that repeatedly asks "Why?" or "What does that mean?" This recursive decomposition must be used with caution however; it is very easy to quickly build a model that becomes cumbersome in future steps.
User Requirements Definition
The first step is to gather key stakeholders to brainstorm user requirements. Figure 3 shows the beginning steps of establishing user requirements for a COTS software product. Three different requirements have been identified:
- All functionality is provided.
- Product integrates with legacy systems.
- Product is "adoptable" by the organization.
Figure 2: AHP Methodology of Evaluation
Figure 3: AHP Tool Capturing Initial User Requirements
User requirements definition requires more than brainstorming a wish list of features and functions. Users' needs and wants must be identified and structured to facilitate evaluating alternatives. AHP-based tools provide a consistent and repeatable process for translating requirements into evaluation criteria.
Clustering
One of the more time- and labor-intensive aspects of the evaluation process is establishing a list of criteria in such a manner that all requirements can be clearly understood and communicated to stakeholders and decision makers. This task is aided by "clustering"-grouping requirements into "theme categories" that will become the evaluation criteria hierarchy. Figure 4 shows the three previously mentioned requirements (functionality, integrability and adoptability) as the theme categories containing nine different evaluation criteria.
Figure 4: Using Clustering to Identify Criteria
Identifying Alternatives
Once high-level user requirements and related evaluation criteria have been established, viable alternative COTS products can be identified. At this stage, the evaluation team will frequently incorporate a user requirement involving the strength of the company. Figure 5 shows how an AHP tool handles the list of alternatives (COTS1, COTS2, COTS3, and COTS4). It also shows the user requirement "company strength" along with four associated evaluation criteria.
Figure 5: Adding Alternatives and Developer-Related Requirements
Establishing the Evaluation Hierarchy
Once the alternatives have been identified and a sufficient number of criteria established, the AHP tool can automatically create the evaluation hierarchy. At this stage all criteria are equal; no attempt has been made to establish weights or priorities.
Figure 6: The Initial Hierarchy for Evaluation
Some criteria (e.g., "Has short learning curve") may require additional definition or another level of refinement. The following screen shows the addition of another "branch" to the "adoptable" portion of the hierarchy.
Figure 7: Addition of "Branch" to Improve Criteria Definition
Deriving Requirement and Criteria Weights
Once requirements and criteria have been identified, the team can establish priorities for each. The process is established through the mechanism of pair-wise comparisons in which each requirement and criterion is compared against its "siblings" within the evaluation hierarchy. In the example, the high-level requirements that will be compared are
Similarly, within the theme category (requirement) "adoptable," several criteria will be compared:
In this example, a "verbal" approach has been used to compare the top-level requirements. The following screen capture shows the comparison matrix between the four user requirement categories. Initial comparisons have been made, and normalized weights have been assigned by the AHP tool. Note that the tool provides a histogram to show the relative importance of each requirement as the process unfolds.
Figure 8: Pair-Wise Comparison of High-Level Requirements
This pair-wise approach allows evaluators to compare tangibles or intangibles on a reliable scale. Each evaluator expresses an opinion and all individual judgments are collected and aggregated into a group judgment.
Figure 9: Evaluation at the Completion of All Comparisons
At this stage the evaluation team has established weights for each of the criteria and expressed its opinion of the four COTS software packages and their ability to satisfy the different criteria.
The high-level requirements have been weighted and sorted. Their relative importance is illustrated using bar graphs. In this example, the evaluation team deemed "functionality" as by far the most important requirement. It has a normalized score of .480, nearly half the total assessment of utility. In other words, the team felt that functionality was nearly as important as all other requirements combined.
Figure 10: Relative Importance of High-Level Requirements
At this stage, each team member will understand how other team members feel about the requirements and criteria and how their different perspectives influence the evaluation. Further, the team members will see where they agree. Effort can therefore be focused on areas of disagreement or where there are points of uncertainty or misunderstanding.
Sensitivity Analysis
Because the evaluation process is inherently uncertain, it must accommodate sensitivity analysis, which determines what influence each assumption has on the recommendation. At each level of the hierarchy, the evaluation team can see the relative importance of its criteria (left-hand pane below). The team can also dynamically change these relative weights and view the outcome (right-hand pane).
Figure 11: Dynamic Sensitivity Analysis
"What If?" Scenarios
An important use of dynamic sensitivity analysis is in "what if" scenarios, which test the robustness of the recommendation under a variety of different assumptions. Since a number of the criteria in a typical evaluation may be quantifiable, a sensitivity analysis can show the extent to which the recommendation might change if an assumption were altered.
In the example, products COTS1 and COTS4 have virtually the same ratings (.265 vs. .263). Increasing the importance of "functionality" does not cause the relative scores of COTS1 and COTS4 to change. The recommendation is very insensitive along the dimension of functionality. Similarly, the importance score of "integration" must substantially increase before it changes the rank of the products. However, increasing the importance of adoptability changes the recommendation from COTS1 to COTS4. Increasing the importance of "company strength" only serves to increase the score of the leader, COTS1.
By conducting this type of sensitivity analysis, the evaluation team can focus on issues that can potentially change the recommendation: the criteria in the "adoptability" requirement. Examining the "adoptability" branch in detail provides insight into the recommendation. As the following screen capture illustrates, the team felt that COTS4 was significantly more intuitive than the other products. This rating was sufficient to award COTS4 the highest rank in this dimension. By highlighting these criteria, the team can double-check the reasons for the initial ratings to make sure that all team members are comfortable with the conclusions.
Figure 12: Closer Examination of the "Adoptability" Criteria
Building Consensus
As with criteria comparisons, the evaluation team sees where it agrees and disagrees on a product evaluation. Where there is no disagreement on where the decision is insensitive to changes in assumptions, there is no benefit in protracted discussion. Where there is disagreement or where the decision can change with a modest change in assumptions, it is worth the team's time to scrutinize. Building a consensus within the evaluation team and communicating such to the stakeholders is arguably the most valuable aspect of this methodology.
[Acknowledgements] [Executive Summary] [Abstract] [1 Introduction] [2 Case Study]
[3 Lessons Learned] [4 Summary] [Appendix A: Analytic Hierarchy Process]
[Appendix B: AHP Requirements Matrix] [Appendix C: Product Dossier Guidelines]
[Appendix D: Cost Comparison Spreadsheet]
[Appendix E: Manufacturing Execution Systems Product Survey]
[References] PDF File [0.60MB]
Appendix B: AHP Requirements Matrix
The following pages contain the specific AHP requirements matrix that was generated on this project. This matrix was formatted for use as a checklist and note template for use during product demonstrations, to aid in Product Management (PM) tool selection.
Table 1: User Requirements Review of Shop Control Software
[Acknowledgements] [Executive Summary] [Abstract] [1 Introduction] [2 Case Study]
[3 Lessons Learned] [4 Summary] [Appendix A: Analytic Hierarchy Process]
[Appendix B: AHP Requirements Matrix] [Appendix C: Product Dossier Guideliness]
[Appendix D: Cost Comparison Spreadsheet]
[Appendix E: Manufacturing Execution Systems Product Survey]
[References] PDF File [0.60MB]
Appendix C: Product Dossier Guidelines
By Edwin Morris9
Overview
The product dossier artifact captures all the information regarding a single COTS product, including characteristics of the vendor; product architecture and functional capabilities; standards supported; required hardware and software configurations; nonfunctional characteristics such as usability, supportability, reliability, interoperability, portability, and scalability; quality of documentation; costs; and licenses. A Product Dossier is created when the product is introduced and updated as appropriate.
Purpose of Captured Information
- The product dossier artifact accumulates and organizes information sufficient to record
- the history of contacts with the vendor regarding the product
- the history of consideration and use of the product
- raw (unfiltered) information about a product and product vendor gathered directly from the vendor (documentation, claims, price lists, demonstration versions, etc.), and from third parties (such correspondence and reviews by other users, trade journal articles, business/financial analysis, etc.)
- processed (filtered) data obtained during consideration of a product including the results of investigations into the product and vendor, information describing the exact configuration of the product evaluated, and data gathered during evaluation activities and benchmarking
- the analysis of the product and vendor, including product/vendor strengths, weaknesses, related products and ensembles, and architecture or usage constraints identified during evaluation
- the history, rationale, and specific activities for customization and tailoring of the product
- the history, rationale, and specific activities for integration of the product
- the history of version releases
- the history and rationale for upgrade decisions and certification activities
Information Needed
The goal for populating the Product Dossier is to capture information sufficient to select (or rule out) a specific product version, to maintain data about the architectural, design, implementation and testing ramifications of using the product, to transition necessary skills to stakeholders (such as maintainers and end users), and to support the maintenance/evolution process of the product in the system.
The categories of information maintained within the Product Dossier are extensive. Some of this information is developed to support the selection of the product. Other information is developed as the product is incorporated and maintained in the system. Thus, a Product Dossier is a living document that represents the state of knowledge about a product during the time it is considered, used in, and maintained for the system. Examples of the categories of information that are maintained in a Product Dossier are identified below. The type and degree of information maintained for each category will depend on a number of factors, including the characteristics of the product, the stage in product selection and use, and how the product is or will be used in the system. In addition to example categories, sample questions that illuminate the intent of the categories are provided.
Vendor Characteristics
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
