Salion, Inc.: A Software Product Line Case Study
3 How Salion Builds Its Software Product Line
A product line approach to software involves three essential activities: (1) development of core assets, (2) development of products, and (3) management of the product line operation at both technical and enterprise levels [Clements & Northrop 02]. The order in which core asset development and product development occurs is not fixed.1
Many organizations begin a software
product line by developing the core assets. These organizations take a
proactive approach [Krueger
01]. They define their product line scope to define the set (more often,
a space) of systems that will constitute their product line. This
scope definition provides a kind of mission statement for designing the
product line architecture, components, and other assets with the right
built-in variation points to cover the scope. Producing any product within
that scope becomes a matter of exercising the variation points of the
components and architecture--that is, configuring--and then assembling and
testing the system. Other organizations begin with one or a few existing
products and use them to generate the product line core assets and future
products. Such organizations take a reactive approach.
The proactive approach has obvious
advantages--products come to market extremely quickly with a minimum of code
writing. But that approach also has disadvantages. It requires a significant
up-front investment to produce the architecture and components that are
generic (and reliable) across the entire product space. And it also requires
copious up-front predictive knowledge--something that is not always available.
In organizations that have long been developing products in a particular
application domain, this is not a tremendous disadvantage. For a "green
field" effort like Salion's, where there is neither experience nor existing
products, this is an enormous risk.
The reactive approach has the advantage of a much lower cost of entry to software product lines, because the asset base is not built up front. However, the quality of the architecture and components of the product(s) used as the basis for the asset base must be robust, extensible, and appropriate to future product line needs. This is not a trivial prerequisite.
Krueger describes a form of the
reactive model that works by building the first system (or a number of small
systems) and, when a new product is defined for a new customer, "reacting" to
it by refactoring its design to define what is now common across all members
of the emerging product line and what varies [Krueger 01]. Variability among systems
is handled, if practical, by making the system configurable with respect to
that variability. Otherwise, the special need is treated as a
customization that the organization may or may not be willing to
implement for that customer, depending on business criteria. According to
Krueger's model, the difference between a configuration and a customization is
that the customization requires new code.
Salion adopted Krueger's reactive model. It first produced a "standard"
product as its entry into the market. That product formed the basis for Salion's software product line and the basis for each new customer-specific product it fielded. The standard product was more than an engineering model from which "real" systems were produced; it gave Salion a very enviable entrée capability into customer organizations (see Section 2.7).
The rest of this section describes in greater detail Salion's software product
line approach in terms of the product line practice areas [Clements & Northrop 02] that are especially germane to it. Each practice area is categorized as software engineering, technical management, or organizational management. Rather than organizing the practice areas by category, we organize them as follows to better show the evolution of Salion's product line:
We begin by describing one of the most key aspects of the Salion story: how Salion's reactive model gave it a way to handle unforeseen variations through partially reactive product line scoping.
3.1
Scoping
A product line's scope is a statement about which systems are in and out of the line's declared area of business. For organizations that take a proactive approach, their initial scope consists of the description of a set (more often, a space) of systems that will constitute their product line.
Salion was unwilling to a priori rule any system out of
its product line's scope; a small company finding its legs
cannot afford to turn down business. Moreover, in any "green
field" approach, a firm scope definition would be very risky.
Salion instead began with an informal scope definition based
on the variabilities outlined in Section 2.4--a list
that itself has grown over time. As customer products were
added to Salion's repertoire, a de facto scope
definition has emerged. As more products have been delivered,
Salion's confidence in its original product space has grown.
Now the scope revolves around the following variations that
will cause the company to engage in customization work: These variations represent the three
ways in which Salion expects to customize products. On the other hand,
workflows, for instance, also remain a point of variation, but do not require
customization. Rather, Salion's software is configurable with respect
to workflow. Salion does not regard workflow as a dimension of its product
line scope, because practically no work is required to achieve that variation.
And now, Salion's reactive scope
definition has come to play the same primary role as a proactive scope
would--it defines the company's business area. For example, it provides the
marketing department with cues about how to engage the customer (see Section 3.7). The marketers now ask customers what
their requirements are in the three areas listed above, whereas before, the
marketers were much more likely to engage in an open-ended discussion about
requirements. The scope also gives strong hints about what's "out." For
example, when an early customer wanted to have a new kind of domain object
modeled, Salion was quick to accommodate the request. Now, however, Salion is
more likely to push back and try to frame such a requirement in terms of the
standard (OAG) business object model that it has adopted.
3.2
Understanding Relevant Domains
In Salion's case, there was no long history of application domain expertise from which to precisely define anticipated products. In many ways, its products chart new territories. However, despite the newness, it did not proceed without domain investigation. Its founding organization, McKinsey & Company, turned over to Salion its extensive research into the manual management processes of custom and semi-custom manufacturing organizations that rely on bids and proposals to win business. This research provided not only the necessary market analysis for Salion's business proposition but also a domain knowledge base in acquisition management. Salion also drew on its own experience in related domains such as workflow and process enactment. In addition, several key people at McKinsey & Company had prior experience in ASP companies--a model that would come in handy at Salion.
Given this foundation, there is considerable evidence of the high degree of domain understanding that Salion accrued and modeled. In particular, it
The "Understanding Relevant Domains" practice area is likely to be a source of risk for new organizations, especially if they are venturing into new application areas. Salion seems to have successfully mitigated that risk.
3.3
Process Definition
A strong software process played a role from the beginning. Salion knew that the complexity of an unknown number of customers requiring an unknown number of variations would threaten to spin out of control of the company's small staff. So it put some strong process safeguards in place.
First, Salion decided to adopt the Rational Unified
Process® (RUP®) to guide its
development activities [Kruchten 00]. It
overlaid that with the VRAPS (vision, rhythm, anticipation,
partnering, and simplification) approach [Dikel et al. 01]. VRAPS
represents an approach to software development that transcends
tactical issues (such as what objects to create) and instead
emphasizes strategic concerns such as keeping everyone in the
organization aligned, dealing with change events, making sure
that the right people are involved, keeping the architecture
simple, and establishing a regular pulse-like release schedule
in concert with market conditions.
Into that mix Salion added some aspects of Extreme Programming
[Beck 00]--namely, it carries out twice-daily builds, performs constant unit testing, and uses contracts as the interface among units. Also, every month it refactors its system's design to identify new commonality.
Keeping everyone aligned with the company's vision is a central theme of Salion's story. Salion applies RUP to write use cases not just for the company's software, but also for its business.
Figure 2 illustrates the Salion software solutions that realize the business use cases defined in the business use case model. Each product is represented as a package of functionality that provides extensions or modules that can be turned on or off, depending on the subscription level of each customer. This way of thinking permeates the business.
The business use case model sets the strategic direction of the company. Salion uses it to gauge the attractiveness of prospective clients. It keeps everyone on the same page in terms of knowing in which businesses the company is--and is not--engaged.
This capacity for keeping a group of people working synchronously toward a common goal is the essence of strong process definition and discipline.
Every successful software product line depends on the right architecture. For Salion, this means an architecture that blends general concerns about runtime efficiency and protection of confidential data with production concerns about taking advantage of product line commonalities.
Salion's system
architecture, shown in Figure 3,
provides three tiers: (1) a front-end tier on top, (2) a core
platform in the middle, and (3) a persistence layer (database)
on the bottom. The front-end tier consists of a pair of
JavaServer pagesTM (JSPTM) servers, to
provide redundancy for increased reliability. Hosted systems
(e.g., Salion hardware running at Salion's site) share the
core platform and persistence layer, but each customer gets a
dedicated pair of front-end servers with a unique uniform
resource locator (URL) for access. This is done to provide
high availability and prevent data confidentiality breaches
across customer systems.
Customized software runs only on the front-end servers; the
other platforms run common software. The front-end servers are
currently simple and inexpensive Solaris boxes. The other
hardware consists of about 10 "really beefy" (in the words of
the architect) application servers that are responsible for
carrying out the systems' business logic and core (common)
applications, as well as hosting Web servers, the network
infrastructure, and the security authentication subsystem.
This system architecture, and the way that the software is allocated to it, makes upgrades and customizations straightforward. If customers want a small upgrade to their customized software, Salion can provide it easily. If customers want an upgrade that will tax the capabilities of the front end, Salion can scale up that area without affecting any other part of the architecture or any other customer. A major upgrade that has widespread functionality or performance requirements can be done in concert with an upgrade to the application servers.
The software
architecture is decomposed to be compatible with the
deployment scheme just described. Also, the availability and
use of commercial off-the-shelf (COTS) components dictate
additional decomposition criteria (see Section 3.5).
Variation points are implemented using the Façade design
pattern [Gamma et al.
95].
The architecture for the product line is a firm by-product of the VRAPS approach's simplification principle. The chief architect has adopted a set of strong, sensible principles to ensure that the architecture is no more complex than is necessary.
As mentioned previously, security and availability lead the list of required attributes. Seamless 24/7 operation is a requirement, as is the unconditional preservation of customers' data confidentiality. These mandates precipitated the following foundational architectural decisions:
Optimizations such as the use of careful SQL provide the key
to performance and are chosen carefully. The architect takes
responsibility for targeting the optimizations that will bring
about real improvement and avoiding those that do not address
a demonstrated need. For example, early on, the architect
gathered data about where cycles were being spent in the
architecture, namely, in Java Naming and Directory InterfaceTM
(JNDI) lookups, object invocations, and static hypertext
markup language (HTML) traffic. Armed with this data, the
architect implemented four minor optimizations that quadrupled
the system's throughput and provided a performance cushion
that still holds. This is in contrast to an "optimize
everything" approach that consumes developer resources and may
not provide a return in runtime benefits.
The result is a software architecture that consists of four logical layers:
In addition to those four layers, a unifying security framework exists alongside them. It protects the applications and data at each level by providing authentication and security services.
A typical product consists of about 40 independently built submodules totaling some 150 thousand lines of code (KLOC).
The goal of a product line architect is to provide a development and runtime environment in which developers don't have to worry about debugging cache, SQL queries, or transaction protocols--in short, to let the developers concentrate on the application's business logic. The architect has tried to instill an architectural culture within Salion, based on the following principles that ground the nimble, flexible architecture:
This attempt to instill an architectural philosophy throughout the company is another example of how Salion wants to make sure that the entire staff shares a unified vision. If architectural stakeholders hold the same vision as the architect, they will come to understand which kinds of changes are feasible, which are likely to be met with resistance, and (most importantly) why the decisions to make them support the organization's business model.
3.5
COTS Utilization
Salion's philosophy about using COTS components reflects its deliberate commitment to simplicity, but not at the expense of quality. Salion's chief architect and vice president of engineering are both former employees of a major Web search engine site/content provider. In four years, they watched the site/provider's usage go from 45,000 to 45,000,000 page views per day. With millions of people using the system, they learned very quickly to do what it takes to avoid being awakened in the middle of the night with a business-threatening problem.
Salion's major concern about COTS products centers on its ability to provide the necessary quality attributes (i.e., performance, security, and reliability) and the product's ability to scale. When the product's ability is in doubt and no practical workarounds are available, Salion will not hesitate to build the necessary components in-house.
Salion's rule of thumb is that if the function is unimportant, COTS will do. If there's an actual or de facto standard for some aspect of the system, COTS will do (as there is likely to be a choice of more than one vendor that meets the standard). Otherwise, that component is a strong candidate for in-house implementation.2
An example is the commercial workflow system that Salion adopted initially. After discovering that the system didn't have the necessary flexibility, scalability, or reliability, and that it didn't obey any applicable standard, Salion jettisoned it in favor of a homegrown version.
3.6
Tool Support
Tool support is an
area where Salion has shown a willingness to be innovative.
In addition to the usual array of software development tools,
Salion employs two emerging product-line-specific tools: Gears
(http://www.biglever.com)
and CloneDR ( http://www.semanticdesigns.com).
Gears is designed to permit organizations to manage the commonality and variability present in its product line assets. Gears works by identifying artifacts (e.g., source code files) that are used in every member of the product line and artifacts that are used in some, but not all, members. Then it facilitates the build process for a particular product by retrieving the assets that belong to that member and making them available to the normal build tool.
The advantages of
Gears are four-fold. First, it obviates the need for
complicated layers of #ifdef statements in the code that
quickly become impossible to read and understand in all but
the simplest cases. Second, it allows a vastly simplified
build process that can now be identical for every member of
the product line. Third, because it is file based, it can work
for non-code artifacts such as design documentation, user
documentation, test cases, data files, work schedules, and so
forth. And fourth, it reinforces a fundamental concept in
product line development--that the entity being built is not a
set of products but a single product line.
The last point is subtle but fundamental. Product line organizations with the most success view their endeavor not as building an ad hoc set of products that might have things in common, but rather as developing and nourishing a strong singular product line. Gears reinforces this by providing a single venue for building every product, by explicitly identifying common artifacts and product variants, and by mapping artifact variations to products.
Operationally, Gears lets Salion run two variants at the same time without having to worry about "breaking" Salion's standard product.
Currently, Gears helps Salion manage a total of 3,333 files, 88 (2.4%) of which represent variations. Put another way, 97.6% of the files are reused (i.e., do not change) across products.
CloneDR is a tool that provides clone detection--places where the code is almost similar and can be refactored to extract the commonality. CloneDR uses sophisticated algorithms that go far beyond simple syntactic comparison. Salion found that some 13% of its stable code was essentially duplicate and could be removed and combined after some simple reengineering. In one case, the removal of three clones eliminated some 27 KLOC.
Together, Gears and CloneDR provide a naturally synergistic capability for generating a product line from two or more already-fielded systems that were developed from each other. Identical clones become common components; almost-alike clones and product-specific software define dimensions of variation to Gears, which can then quickly cast the products as variations of a single product line.
3.7
Marketing/Customer Interface Management
In the beginning,
Salion employed a "solution selling" strategy to connect with
its customers. This strategy involved performing a needs
assessment for a customer and then selling to those needs. The
needs turn into business use case models, following RUP, at a
two-hour meeting held every Wednesday. This so-called "joyous
vision" meeting is an executive-level discussion among the
chief financial officer (CFO), the vice president of
marketing, the vice president of engineering, the chief
architect, and others at this level to choose and prioritize
among business opportunities, all under the auspices of
Salion's own business use case model that drives the company.
Salion still employs this approach, but as mentioned in Section 3.1, is much more inclined
these days to channel the customer's statement of needs into
the three main dimensions of variation inherent in its product
line.
Thus, like most product line organizations, Salion's marketers do not have carte blanche to sell whatever the customer wants. Rather, they must first receive approval or confirmation that a proposed system is in the product line's scope. Of course, they try to sell each customer a basic system first (one that can satisfy the customer's needs through preplanned configuration) or, failing that, a variant already fielded.
The first litmus test applied to a new customer need is whether it's in Salion's business use case model. For instance, a customer who wanted to buy Salion's workflow engine would likely be turned down, because there is no use case that says Salion is a component provider.
If the customer's needs can be met only by a new system,
Salion first predicts the cost based on past performance (see
Section 3.9). At that point, whether to achieve the requirements by customization (new code) or configuration (expanding variabilities already in place) is a technical question that must be answered. But its answer is linked to business questions dealing with costs, the likelihood of future use, and which strategy is likely to make Salion more money. Thus, Salion's marketers must walk a fine line between keeping customers happy by selling features and limiting promised customizations.
But as a start-up
company, Salion has been loathe to turn down business, and
being able to say "yes" to a customer right out of the gate is
an enormous advantage. Because of its flexible architecture
and robust development process (see Section 3.8), Salion can say "yes" most
of the time.
Also contributing to this ability to say "yes" is an innovative marketing arrangement made possible solely by the product line approach. All implementation work for a customer occurs in two phases. During the first phase, Salion installs its standard system for a customer within 60 days. During the second phase, Salion builds and installs a customized product. While the standard system is running, Salion works with the customer to define customizations and then, based on this elaboration step, commits to a delivery date and (fixed) contract price. This two-phase approach has many benefits:
The customer's primary interface with Salion is through its Professional Services group, that configures delivered systems, initiates customizations, and (in the case of one-time-use-only components) might actually build custom parts. Salion understands that its customers just want a system and don't want to be bothered with configuring it. So that's what Salion delivers, and whether customers get their systems through customization or configuration is a detail Salion can keep hidden and over which it can maintain flexibility.
As mentioned in Section 3.1, the marketing group gently
steers the customer to three areas of customization: (1) forms
and screens, (2) export and reporting, and (3) bulk load
(i.e., legacy data migration) requirements. Other variations
are possible, but they are done via the configuration of
built-in preplanned variation mechanisms, and so do not
represent an area of intense concern to the development group.
3.8
Operations
The "Operations" product line practice area is concerned with the day-to-day running of the product line and how various stakeholders come together to cooperatively make sure that new products in the product line are launched, built, and deployed smoothly.
3.8.1
Normal Operation
Salion's operational strategy manages three different product cycles at any one time. While one product is in the inception stage, another is in requirements design and elaboration, while a third is in development.
The business use case model is the driving impetus for all three stages. This leads to other use case models for a given product, subsystem, or sub-subsystem. Those models are standard RUP practices, but as mentioned previously, Salion has also adopted some of the tenets of Extreme Programming. These tenets include Salion's twice-daily builds, emphasis on unit testing, use of contracts as the interface among units, and monthly refactoring to identify new commonality (which the Gears tool helps manage).
Salion's "rhythm" part of the VRAPS process comes from a planned release every 30 days. At that interval, a packet is given to the user-interface developers working on the front end and to the developers working on back-end components. There is one packet per component, and typically 5 (out of the 40) are being worked on at a time. A packet contains any of the following, as needed:
A new release might contain enhanced generic properties or features, or represent a new variation contained in a new member of the product family.
After coding is complete, the system goes into quality assurance (QA) and testing. Most of the process after this point is scripted, handling checkout from the CM system, the build step, and the application of a suite of some 350 white-box tests. All told, this process takes about two hours.
To keep problems from derailing the clockwork rhythmic process of development and release, a so-called "joyous love" meeting is held every Tuesday and Thursday to address implementation-level concerns and keep projects on track. The objectives of these meetings include
At end of each release iteration, the chief architect and some of the key developers have a "quiet week." They run scripts that show coupling/cohesion among modules and inspect code. They post a list of recommendations, which are addressed in the following release, after being prioritized in the next "joyous love" meeting.
The QA team, by contrast, is never idle and always working to maintain and improve the quality of the artifacts.
3.8.2
Introducing New Variants
If a new product is given the go-ahead, the architect must decide how to handle the new requirements. Depending on several factors, the requirements might
At that point, developers are assigned to make the necessary changes to the various artifacts. In a pilot demonstration of the product line concept, developers were tasked with building a new variant that produced a special kind of report detailing the history of the kinds of bids the supplier lost. This change resulted in adding new business logic to that layer of the architecture and required about two days of effort. Out of almost 3,000 classes, only about 20-30 required modification.
3.8.3
Phasing Out Old Products
To simplify its life-cycle support costs, Salion prefers not to maintain old versions of products. Where possible, this provision is built into Salion's contracts. When a customer product undergoes maintenance, the first inclination is to upgrade that product to the current version of all software, if possible.
3.9
Data Collection, Metrics, and Tracking
Salion uses metrics as a key planning tool. At the time of our first visit, it had accumulated 12 months worth of data and heuristics that help it predict with confidence what it can do by when.
Use cases are used to drive productivity measures. Numbers that are lower than expected help Salion spot potential problems, such as wheel spinning due to inadequately specified contracts.
For each release, Salion measures the
These measures help Salion determine its capacity for new work and prevent it from promising more to a customer than it can likely deliver. They also help to avoid "yes you can, no we can't" battles by simply establishing an historical baseline and allow Salion to answer questions such as how much work it could do if it doubled its staff.
3.10
Structuring the Organization
Every software product line
organization must adopt an organizational structure that promotes the most
advantageous production process. Although other variations are possible [Bosch 00], fundamentally there are two
organizational structures for product lines that differ in who produces the
core assets that are reused across all the products in the family. One model
establishes a separate core asset group (sometimes called the "platform"
group) that is separate from the groups that turn out products. The second
model assigns responsibility for building and maintaining the core assets
directly to the product groups.
Salion is small enough--with only 10 people involved in product development-that crafting a formal organizational structure is not paramount. However, it is still instructive to examine how Salion addressed the question that every product line organization must face: who builds the core assets?
Early on, before
Salion sold its first product, it worked on building its
so-called "standard system." This was the system that would
form the basis of Salion's reactive software product line
model. Future systems would be built from this one, and it is
still the system that is installed initially for a new
customer. Since a reactive model proceeds largely without
knowing in advance what is going to be reused, all the assets
were regarded as at least potentially core. During this
start-up period, Salion's three-cycle operational approach
(see Section 3.8) drove its
organizational role assignments. There was no separate core
asset team; everyone worked on producing successively more
capable versions of the product.
This model has persisted even with Salion's third customer product under way. When Salion needs to make a major customization, a product-based team is assigned to make it happen.
Figure 2: Salion's Business Use Case Model
3.4
Architecture Definition
Figure 3: Salion's Product Line System Architecture
1
Regardless of the order, there is perpetual iteration between these two
activities throughout the life of the product line.
2
This defined decision process also qualifies as a practice in
the "Make/Buy/Mine/Commission" practice area.
TM
Rational Unified Process and RUP are registered trademarks of
Rational Software Corporation.
TM
Java and all Java-based marks are trademarks or registered
trademarks of Sun Microsystems, Inc. in the United States and
other countries.
®
CORBA is a registered trademark of Object Management Group, Inc.