Testing in SOA Environments

Our research into testing service-oriented architecture (SOA) implementations focuses on the elements for testing, the aspects to be evaluated, and the challenges arising from SOA characteristics.

Elements

SOA testing strategy must encompass infrastructure components, individual and composite services, and mission threads spanning multiple services and applications.

The SOA infrastructure typically consists of capabilities and services to register and discover services, manage metadata, provide security, and deliver messages. The infrastructure may be composed of custom developed software, commercial products, or some combination of custom and commercial capability.
An individual web service typically provides coarse-grained, business-level capability. The elements of a web service from a testing perspective are the service interface, service implementation, message format, message payload, and service level agreement (SLA). Composites involve multiple web services that interact to provide a capability.
End-to-end threads are composites of humans, applications, services, back-end applications, and databases that utilize the SOA and network infrastructure to perform a mission task. End-to-end threads include services along with other interacting components (human, functional, infrastructural), along with the operating environment.

Aspects

Viewed a a high level, the aspects to be tested for in a service-oriented architecture (SOA) implementation are like that are tested for in any system. These aspects include

functionality: The functional capability provided by each SOA element must be tested to determine whether the element meets requirements and other expectations. For example, functional testing for SOA infrastructure verifies that individual services are made available correctly (i.e., they are published), that the services can be located, and that connections can be established (called find and bind).
non-functional attributes: The non-functional characteristics that are essential for determining the fitness of the SOA element for operational use, also called its quality attributes, must be tested.
conformance: For SOA implementations, there are many standards and guidelines. Information about standards and guidelines developed by consortiums interested in SOA can be found from the World Wide Web Consortium (http://www.w3.org/), the Web Services Interoperability Organization (http://www.ws-i.org/), and the Organization for the Advancement of Structured Information Standards (OASIS, http://www.oasis-open.org/home/index.php). In addition, many organizations develop standards and guidelines specific to their lines of business that must also be tested.

Challenges

Testing SOA implementations is challenging for various reasons, such as

Services are often outside the control of the organization using them, leading to potential mismatches and misunderstandings between parties.
SOA implementations are often highly dynamic, with frequently changing services and service consumers and varying load on services, SOA infrastructure, and the underlying network. Consequently, it is normally not possible to replicate all possible configurations and loads during the testing process.

Research direction

We are investigating answers to the challenges of testing SOA implementations by (1) surveying the current state-of-the-art and state-of-research for testing in SOA environments and (2) applying techniques in a U. S. Department of Defense (DoD) context.