Despite increased adoption and advances in machine learning (ML), studies show that often ML projects do not reach the production stage because they do not fit into the overall business workflow and context, or the use case is not well understood. As such, testing ML components is still largely limited to model properties, such as accuracy, with limited consideration of requirements derived from the system or workflow it will be a part of, such as throughput, resource consumption, or robustness. This limited view of testing can lead to failures in model integration, deployment, and operations.

In traditional software development, quality models, such as ISO 25010, provide a widely used structured framework to assess software quality, define quality requirements, and provide a common language for communication with stakeholders. A newer standard, ISO 25059, defines a more specific quality model for AI systems. However, a problem with this standard is that it combines system attributes with ML component attributes, which is not helpful for a model developer or evaluator, as many system attributes cannot be assessed at the component level. In this blog post, adapted from a recently published paper, we present a quality model for ML components that serves as a guide for requirements elicitation and negotiation.

By developing a quality model specifically for ML components, developers, evaluators, acquirers, and stakeholders can draw upon the model’s quality attributes (QAs) and definitions to establish a common vocabulary for defining system- and mission-derived requirements and focus testing efforts accordingly.

• Behavior Analysis: QAs related to ease of observing and analyzing ML component behavior at development time and run time.
  • Analyzability. Capability of an ML component to expose attributes that enable it to be effectively and efficiently assessed regarding its behavior or the impact of an intended change, to diagnose it for deficiencies or causes of failure, or to identify sub-components to be modified.
  • Monitorability. Capability of an ML component to produce data that can be used to measure runtime attributes (e.g., performance degradation, latency) that can effectively be observed and used by the ML-enabled system or operator to take action aligned with requirements.
  • Testability. Capability of an ML component to support testing against expected behaviors by offering information relevant to test results or ensuring the visibility of failures.
  • Understandability. Ease with which the code, implementation choices, and design choices of an ML component can be understood.

• Confidence: QAs related to producing information that increases insight into the model and its outputs.
  • Explainability. Capability of an ML component to provide information that can explain model outputs (decisions) in human terms.
  • Functional Correctness. Capability of an ML component to meet system-derived correctness or model performance requirements.
  • Interpretability. Capability of an ML component to be transparent in its execution and produce information concerning the relationship between inputs and outputs.

• Consistency: QAs related to producing consistent results over time.
  • Repeatability. Capability of an ML component to produce equivalent results when run on equivalent inputs during inference time.
  • Reproducibility. Capability of the training algorithm for the ML model to produce functionally-equivalent ML models when run on similar datasets during training time.

• Continued Operation: QAs related to providing continued operation within specified boundaries and constraints.
  • Deployability. Capability of an ML component to be deployed into production when needed, without any unanticipated side effects and within specified resource and time constraints.
  • Resilience Capability of an ML component to provide and maintain an acceptable level of service in the face of technical challenges to normal operation.
  • Reliability. Capability of an ML component to perform specified functions without failure under normal operation.
  • Resource Utilization. Capability of an ML component to use no more than the specified amount of resources to perform its function.
  • Robustness. Capability of an ML component to preserve its level of functional correctness under specified fault modes or conditions.
  • Time Behavior. Capability of an ML component to produce outputs within required response time and throughput rates.

• Maintenance and Evolution: QAs related to ease of maintaining and evolving ML components over time.
  • Domain Adaptability. Capability of an ML component trained on a source domain to be used in a different, but related domain.
  • Maintainability. Ease with which the ML component can be modified to correct faults, to improve performance or other attributes, or to adapt to a changed environment.
  • Modifiability. Capability of an ML component to be changed without introducing defects, unintended side effects, or degrading existing component quality, and within specified resource and time constraints.
  • Portability. Capability of an ML component to be adapted for or transferred to different hardware, software, or compute platform, without any unanticipated side effects and within specified resource and time constraints.
  • Replaceability. Capability of an ML component to replace (or be replaced by) another ML component for the same purpose in the same environment.
  • Retrainability. Ability to generate a new version of the trained model using the same process on a new set of training data, either manually or automatically depending on system-derived requirements.
  • Reusability. Capability of an ML component (or its sub-components) to be used in other ML-enabled systems with minimal change.

• Responsible AI: QAs related to the ethical and societal impact of ML component usage.
  • Compliance. Capability of an ML component to satisfy requirements, as required by rules, regulations, laws, standards, or security policies.
  • Economic Risk Consideration. Capability of an ML component to mitigate the potential risk of its results to financial status, efficient operation, commercial property, reputation, employment, educational status, or other resources.
  • Fairness. Capability of an ML component to produce results that are not biased towards certain populations, including, but not limited to protected attributes (e.g., race, gender, income).
  • Health and Safety Risk Consideration. Capability of an ML component to mitigate the potential risk of its results to the health and safety of system users.
  • Inclusivity. Capability of an ML component to be used by systems serving people of various backgrounds, characteristics, or capabilities.
  • Societal and Ethical Risk Consideration. Capability of an ML component to mitigate the potential risk of its results to society.

• Security and Privacy: QAs related to protection and handling of sensitive information.
  • Privacy. Capability of an ML component to produce outputs that cannot be mapped to personally-identifiable information (PII) in inputs.
  • Resistance. Capability of an ML component to safeguard against malicious purposes.

Details of how the quality model was generated and validated with practitioners are available in the paper Quality Model for Machine Learning Components.

Quality Model in Practice

The quality model is integrated in the MLTE (ML Test and Evaluation) tool, co-developed by SEI and the Army AI Integration Center (AI2C) for systematic and repeatable test and evaluation of machine learning components. MLTE can be used for test and evaluation of both traditional ML components and generative AI components.

The quality model guides the elicitation and specification of system-derived requirements for the ML component, ensuring a broad coverage of quality attributes beyond simply functional correctness. While it is not expected that all quality attributes will be relevant for the model under test, using the quality model as a guide places the requirements discussion in the context of the system that the model will be a part of.

MLTE enforces the definition of each model requirement as a quality attribute scenario, which provides a template for ensuring that each requirement is clearly defined and testable. Model testers and evaluators use MLTE to define a test case for each requirement, which becomes part of a test suite that can be executed as needed, repeatedly for a single model, or across different versions or implementations of a model to compare results. A MLTE Report summarizes test results and clearly illustrated tests that passed or failed.

Activities in which organizations have used MLTE include model testing, model acquisition, off-the-shelf LLM evaluation, and analysis of alternatives between vendor implementations of a model.