This guide will explore how to create tests for web APIs from an OpenAPI specification using property-based testing as the technique for generating tests. We will also show a real-world example of how these techniques found bugs in production code from one of the most famous DevOps platforms, GitLab.

OpenAPI

OpenAPI specifications are a programming language-agnostic interface description for REST APIs. Comparable to interface descriptions in different programming languages, OpenAPI allows you to specify what developers need to integrate an API. It supports authentication, endpoints, HTTP verbs, parameters, request body, and response data.

Reading an OpenAPI specification lets both humans and machines quickly understand the underlying web service’s capabilities.

Property-Based Testing

Property-based testing revolves around generating input data and checking if specific properties are valid when executing tests. Property-based testing helps verify that the output data is in the correct format and that the values are valid. There are libraries available to make it easier to use property-based testing. These libraries often consist of two parts:

• A generator that creates randomized test data that fit the needs of the function to test
• Helper methods that make it easy to use the generator

Property-based testing is all about failure and is hard to implement in your code, but it becomes doable with helper libraries and tools available. We often use tests written with examples, and we try to find useful input and check if the output is correct. Based on a specification, a property-based testing approach will generate random data as input and verify if the output is correct.

Example-based tests verify what we think about the function to test, whereas property-based testing tries to prove that it works as expected. For example, an example-based test might check that a function add(3, 3) = 6 works. However, that is only one test, and we can´t say for sure that we implemented the add-function correctly.

A property-based testing approach could test add to check that:

• order of parameters doesn´t matter
• add(x, y) = add(y, x)
• adding 1 two times is equal to adding 2
• add(add(x, 1), 1) = add(x, 2)

Generating data for and using the properties above would give us more confidence that the add-function works as expected compared to writing a few example-based tests.

Benefits of Property-Based Testing

Besides allowing us to generate thousands of tests, shrinking input data is an important benefit of property-based testing. When discovering a problem, the algorithm will determine the input data’s properties that affect the result and repeatedly generate different values for those properties. Hence, it will try to figure out the smallest repeatable test case.

Another benefit is that it will not put any constraints on generated input data if not told otherwise. Also, executing the same test scenario again using a seed value makes it easy to repeat the failed tests.

Finding a Property When Testing Web APIs

Services exposing RESTful Web APIs should handle different inputs correctly and return a result with a status code that a client application can understand.

There are five categories of HTTP response status codes, where the first digit defines the response class.

• 1XX — for information, e.g., backend received a request
• 2XX — request successfully handled
• 3XX — a redirection occurred
• 4XX — cannot fulfill client request, e.g., wrong syntax for the data
• 5XX — server failed internally and couldn´t satisfy the request

For example, service endpoints returning a status: 400 Bad Request tells the client politely that the service could not understand the request and that the problem is on the client-side.

Creating a property that fails on the 5XX status code is a good starting point since this indicates that the web service couldn’t handle the request without an internal failure.

Finding Bugs Using Existing Tooling

Humlix is a REST API client similar to Postman but can generate tests using the Property-based testing approach explained above.

Let’s look at a simple example of an endpoint from the GitLab Groups API OpenAPI specification. If we were to create tests for this endpoint, we would look at what it expects as input. In this example, we see the page query parameter of type integer. We would probably create tests that call it using a negative number and a positive number, and even zero.

Seems simple enough, right?

paths:
  /v4/groups:
    get:
      tags:
        - groups
      summary: Get a groups list by page number
      description: Get a groups list by page number
      operationId: getV3Groups
      parameters:
        - name: page
          in: query
          description: Current page number
          schema:
            type: integer
            format: int32
      responses:
        "200":
          description: Get a groups list
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/Group"

What if we were to apply Property-based testing on the endpoint and verify that we don’t get any 500 Internal Error status in the response? We can do that with Humlix. After importing the OpenAPI specification, we can generate tests.

The screenshot below shows that we received a 500 Internal Error status code generating tests. It took Humlix 956 tests to figure out that calling <https://localhost/api/v4/groups?page=461168601842738792> caused an error inside the GitLab backend.

The property-based testing functionality in Humlix generated many different values for the page query parameter. Finally, it concluded that 461168601842738792 is the smallest value needed to repeat the failing test.

Humlix screenshot failing test

When going into the GitLab production.log file, we can see that calling the endpoint with /api/v4/groups?page=461168601842738792 caused a NumbericValueOutOfRange:Error:bigint out of range exception in an SQL statement.

GitLab production log

To find this issue using example-based testing could have been hard. If we didn’t think of using a big enough value, we wouldn’t discover this issue until our code runs in production.

Conclusion

That was a quick guide showing a small example of combining property-based testing and an OpenAPI specification to generate RESTful API tests. With the real-world example above, we found a bug in production code from GitLab and can safely state that releasing bug-free software is hard.

I’m sure GitLab has a lot of example-based unit-tests in their codebase. A combination of property-based testing and example-based testing would have helped them a lot in producing high-quality software.