Contract driven testing is a technique in software testing that aims to ensure the correctness of software systems by defining and testing the contracts that describe their expected behavior.
With the rise and ubiquity of APIs and microservices, agreements between not only interfaces but data and system behaviors, has become a critical element to ensuring that complex distributed systems will and are working to provide excellent end-to-end experiences.
But many of these interfaces are often shipped without machine-to-machine readable schema, human-focused documentation, proper testing across functional, performance, security, and other aspects of quality. Constant changes to algorithmic underpinnings, code, data sources and shape, infrastructure, component patches…how do we know that what was expected yesterday and still needed works adequately, now and in the future?
One key approach is to have clear technical ‘contracts’ between components implemented…and tested. These contracts don’t just include schema of programmatic interfaces, but also include data model constraints, expectations over failure modes and metadata contents, fallback semantics, and other ancillary (but still mission critical) system behaviors.
Contract driven testing is a technique in software testing that aims to ensure the correctness of software systems by defining and testing the contracts that describe their expected behavior. These contracts specify the requirements and guarantees of the software components, such as their inputs, outputs, and error conditions. By verifying the adherence of the software to these contracts, contract-driven testing can identify defects early in the development cycle and prevent regression issues during future changes.
In contract-driven testing, contracts are formalized as code, which enables them to be easily maintained, versioned, and integrated with automated testing tools. The most common type of contract used in contract-driven testing is the Design by Contract (DbC) technique, which was first introduced by Bertrand Meyer in the Eiffel programming language. DbC consists of preconditions, postconditions, and invariants that define the assumptions and guarantees of a software component.
A modern example of contracts in distributed systems is often found as OpenAPI specs over cloud-based platform APIs and microservices, which are not simply documentation, but rather are machine-to-machine readable specifications over interface functionality and expected behaviors.
Contract-driven testing focuses on testing the contracts that describe the expected behavior of software components, such as their inputs, outputs, and error conditions. These contracts are formalized as code and specify the assumptions and guarantees of the components. Contract-driven testing verifies that the software adheres to these contracts and can identify defects early in the development cycle. The goal of contract-driven testing is to ensure the correctness and maintainability of the software.
Integration testing, on the other hand, focuses on testing the interactions between different software components, such as modules, subsystems, or services. Integration testing verifies that the components can work together correctly and produce the expected results. Integration testing can identify defects that arise from the integration of different components, such as communication errors, data inconsistencies, or timing issues. The goal of integration testing is to ensure the interoperability and reliability of the software.
In summary, contract-driven testing and integration testing differ in their scope and focus. Contract-driven testing tests the contracts of individual components, while integration testing tests the interactions between different components. Contract-driven testing ensures the correctness and maintainability of the software, while integration testing ensures the interoperability and reliability of the software.
API testing focuses on testing the application programming interfaces (APIs) of the software. API testing verifies that the APIs are functional, reliable, and secure. API testing can identify defects that arise from the misuse, misconfiguration, or vulnerabilities of the APIs. The goal of API testing is to ensure the interoperability and security of the software.
Contract-driven testing can be a subset of API testing when APIs are designed with contract specifications such as OpenAPI or Swagger. However, contract-driven testing can also be applied to non-API software components, such as libraries, frameworks, or modules. In contrast, API testing focuses specifically on the APIs of the software.
One of the critical benefits of contract-driven testing is that it promotes collaboration between developers and testers by providing a shared understanding of the software requirements and expectations. It also helps developers to write better code that adheres to the defined contracts, which leads to fewer defects and faster debugging. Contract-driven testing also enables testers to write more comprehensive and precise test cases that cover all possible scenarios and edge cases, which increases the confidence in the software’s quality.
Another advantage of contract-driven testing is that it can improve the maintainability and scalability of the software. Contracts serve as a form of documentation that facilitates the comprehension and modification of the codebase. They also provide a way to ensure that new features and changes do not violate the existing contracts, which prevents regression issues and preserves the correctness of the software.
Design by Contract (DbC): DbC is a programming methodology that uses contracts to define the preconditions, postconditions, and invariants of software components. DbC can be implemented in different programming languages, such as Eiffel, Java, or Python.
OpenAPI (formerly known as Swagger): OpenAPI is a specification for defining RESTful APIs. OpenAPI allows developers to describe the inputs, outputs, and error conditions of the API endpoints in a standardized format. OpenAPI can be used to generate documentation, client libraries, and automated tests for the API.
GraphQL: GraphQL is a query language and runtime for APIs. GraphQL allows developers to define a schema that describes the data types and operations of the API. GraphQL can be used to generate client code and automated tests for the API.
Pact: Pact is a consumer-driven contract testing tool. Pact allows developers to define the expected interactions between the consumer and provider of an API. Pact can generate contract tests that verify the compliance of the provider with the consumer’s expectations.
REST-Assured: REST-Assured is a Java library for testing RESTful APIs. REST-Assured allows developers to write automated tests that validate the responses of the API endpoints against the expected results.
Postman: Postman is a popular API testing tool that allows developers to create and execute API requests and test cases. Postman supports different types of APIs, such as RESTful, SOAP, and GraphQL, and provides features for automating tests, generating documentation, and collaborating with team members.
Contract-driven testing often depends on a significant amount of effort to design, implement, and maintain the contracts of the software components. If the value of contracts is ONLY understood as benefitting testing, then it may be hard to make the case for adopting this approach. However, if the full value of contract-driven design, implementation, documentation, testing, versioning, and maintenance are part of your organization’s approach to delivering distributed systems, then contract-driven testing is a no-brainer.
Contract testing is not itself, alone, a sufficient approach for complete coverage over all aspects of quality. It is a specialization to optimize the process of verifying certain aspects of systems faster and easier than with other types of testing. However, contract testing should be complemented with other testing techniques, such as integration testing, system testing, or exploratory testing, to achieve sufficient coverage.
Additionally, contract-driven testing may not be compatible with all types of software components, such as legacy code, third-party libraries, or proprietary systems. This incompatibility may limit the applicability and usefulness of contract-driven testing in certain contexts, but should not be considered a blocker to adoption wherever and with teams that can externalize or sufficiently abstract away legacy componentry in order to accelerate their overall practices.
In conclusion, contract-driven testing is a critical technique in quality engineering that can help software development teams to build more reliable, maintainable, and scalable systems. By defining and testing contracts that describe the expected behavior of software components, contract-driven testing can improve collaboration between developers and testers, reduce defects, increase the confidence in the software’s quality, and prevent regression issues.