If you asked ten software developers what a contract test is, you’d get twelve different answers. One might say that it’s something to do with schema validation. Another might say that it’s something to do with API specifications. Yet another might tell you that it’s got something to do with the legal profession and kindly ask you to stop bothering them with irrelevant questions.

A few years ago, this confusion would have been endearing. But in 2025, countless new frameworks and tools are competing for the attention of software teams everywhere.

It’s time we get to the bottom of this. What are we actually talking about when we’re talking about contract testing?

Defining the Key Players: Schemas vs. Contracts

A schema uses a generalized notation to define the data types and set of inputs and outputs, like an OpenAPI definition (OAD), that a single system supports at a point in time. In HTTP, for example, these are the syntactical rules that requests and responses must follow. Put another way, a schema is simply data that describes other data. It doesn’t contain any actual code that relates to the API being described. In other words, and to quote the great philosopher Alfred Korzybski, “The map is not the territory.”

A contract, however, defines how two systems can communicate by agreeing on what interactions can be sent between them and providing concrete examples to test the agreed behavior. If I were going to single out one part of this definition, it’d be “two systems.” If I were going to single out a second part of this definition, it’d be “concrete examples.”

The Limitations of Schemas

Schemas are abstract — contracts are concrete. They’re concrete because they include real examples of interactions between two systems instead of just abstract information about one system. Both have uses, but using schemas alone can result in some challenges:

1. Ambiguity: Schemas are abstract, which can lead to misinterpretations. For instance, in an OpenAPI definition, you may define that an API can return a 400, a 403, or a 200, but you cannot say for sure which specific set of inputs will result in those status codes by looking at the specification alone. This is particularly problematic for API schemas with optional fields and polymorphic types.
2. API drift: Unexpected or unintentional changes to an API can cause issues if they aren’t tracked correctly and consistently synchronized with the schema. If a schema is also serving as API documentation, this can result in API documentation that doesn’t accurately describe the actual behavior of the API.
3. Test coverage: Following from the points above, it’s easy to check if a system is compatible with a schema, but it’s very difficult to be sure it fully implements the API definition.

The Levels of Sophistication

Hopefully, you’re starting to understand that there is a difference between simple schema validation and a fully-fledged contract test. Let’s think about this difference in terms of levels of sophistication:

• Level 1 — Schema test: Asserts that a single system is compatible with a schema (such as an OAD) at a point in time.
• Level 2 — Schema-based contract test (also known as a bidirectional contract test): A method of testing that checks that a consumer communicates messages that match a (subset of a) given schema, and that a provider produces output that matches this schema. This may be produced by static analysis, via code generation, or other means.
• Level 3 — Code-based contract test: A method of generating contracts and testing that contracts are valid, using code-based automation tests (i.e., the tests must execute real application code on both sides of the interaction).

Choosing the Right Approach

I’ve already talked about the pros and cons of the kind of schema testing described at Level 1, so let’s look at some of the trade-offs to be aware of when it comes to Levels 2 and 3.

Code-based contract testing (like Pact) executes real application code to prevent implementation drift, provides clear specification-by-example documentation that removes ambiguity, and supports service evolution (with Pact, this is facilitated by the Pact Broker). The downside is that it requires more learning time, test maintenance, and can involve complex test data management through provider states.

Schema-based contract testing has its advantages, including a simpler developer experience, faster setup, and broader tooling options. However, it also comes with significant drawbacks. These include:

• Schemas can’t reliably capture application-level semantics.
• They introduce ambiguity about which inputs produce specific responses.
• They require additional processes for evolution, collaboration, and reliable sharing across teams.

These challenges are similar to those faced when performing Level 1 schema tests. By understanding both the benefits and limitations, you can make informed decisions about your contract testing approach.

Are We Clear Now?

Next time a colleague tells you that they’ve written a contract test, ask them how many systems are involved in the test and whether there are any examples of concrete data involved. If the answer to the first question is “one” and the answer to the second question is “no,” you’ve got yourself a schema test. Send this article to them and invite them to check out contract testing tools, so they can learn more about these techniques.