Software development these days is about iterating fast, and releasing often. In the 10 years since Martin Fowler wrote his original paper, continuous integration has become a cornerstone of the software development process. It is a fundamental part of Agile development (being one of the core tenants of Extreme Programming) and has helped to spawn organizational change with the creation of the DevOps approach to application development.

Within the API community, continuous integration (CI) tools generally fall into two categories. The first wave of tooling saw the role of APIs in CI as being one of a building block, with the CI servers available tending to reflect this. Solutions such as Jenkins, TeamCity, Travis CI and Bamboo provide or consume APIs to help drive the build process, allowing automated builds to invoke actions during the course of execution: this role persists and is still expanding.

The second category of tools help organizations implement CI for the APIs they are creating. These tools can perform test execution and metric collection associated with running a CI process, allowing organizations to introduce a greater degree of automation in their approach to testing APIs.

In this post we’ll cover the latter of the two categories and look at some examples of the tools available that help spur agile API development. While it is possible to create tests for your API using no more than shell script and cURL, this takes time and effort that may not be available, so using tools built for the purpose of testing APIs can be a great way of accelerating your efforts. At the time of this writing there is a burgeoning number of both open source and commercial solutions available that offer a mixture of SaaS-based and on-premise solutions. Ideally these solutions implement some of the following characteristics:

• Easy integration with CI servers using either plugins or webhooks;
• Automatic comprehension of APIs through ingestion of description specifications such as Swagger, RAML and API Blueprint;
• Polyglot support for test creation, giving users choice of the language they use to construct their integration tests;

Abao

Abao is a Node.js module that allows developers to test their RAML API description against a backend instance of their API. To execute tests, a developer simply executes the abao command line tool, passing the RAML file and API endpoint as parameters. Abao also does not provide CI server integration of the box, but scripting its execution from a build job definition would not be difficult to achieve.

DHC

DHC is a tool offered by Restlet that allows developers to create a suite of unit and functional tests for their own or any third-party API. It includes support for security, authentication and hypermedia. It also includes the ability to perform scenario-based testing, with each scenarios allowing multiple features of an API to be tested.

In terms of CI server integration, Restlet offers a plugin for Jenkins and also a Maven plugin that can be incorporated into other solutions, allow developers to incorporate their tests into the build pipeline.

Dredd, by Apiary

In 2015 Apiary introduced continuous integration features into their development platform, enabling developers to continuously and automatically test their APIs with test definitions generated from their API Blueprint description. This functionality is implemented with Dredd, a “language agnostic command-line tool for validating API documentation written in API Blueprint.” Developers use Dredd to execute their tests, with the results posted in the Apiary development console.

Obviously, using Apiary and/or Dredd for continuous integration necessitates the use of API Blueprint to describe the API. Apiary also does not provide CI server integration out of the box, but again, scripting the execution of Dredd from a build job definition would not be difficult to achieve.

APIMATIC

APIMATIC is a solution that allows developers to automatically generate SDKs from API descriptions, which can be imported from existing API Blueprint, Swagger and RAML API descriptions. APIMATIC provides the functionality to initiate the code generation process from CI, by providing an API that can be called from a build job definition, allowing the compiled SDK to keep step with the state of the API automatically.

Chakram

Chakram is an API testing framework built on Node.js and Mocha. It implements a behavior-driven development style of testing (although it does not use Gherkin-style test definitions). Chakram does not provide CI server integration of the box, but like the other examples scripting it’s execution from a build job definition would not be difficult.

Frisby.js

Frisby.js is a Node.js module for testing APIs, allowing developers to create tests in JavaScript. Whilst Frisby does not have built-in integration with CI servers, again it can be called from a script-based build job definition using its command line interface. A notable feature that Frisby implements is the ability to generate JUnit format test reports, compatible with several CI servers including Jenkins and Bamboo, making it easier for test results to be displayed and understood.

Postman

Postman is a well-known and popular API testing client, with support for importing API descriptions in RAML, Swagger and WADL formats. Developers can create tests using JavaScript that can be bundled into “collections” for executing automated test suites.

Postman is accompanied by its Newman command line client that can be incorporated into a script-based build job definition on most CI servers. Postman provides a walkthrough of implementing this with Jenkins on their blog, where they use the built-in scripting capability to call the client, passing the required Postman collection as a parameter.

Runscope

Runscope markets itself as an API monitoring and testing tool capable of being used throughout an organization’s development lifecycle, with capabilities such automatic test generation from a Swagger description or Postman collection.

Runscope appears to be one of the most mature tools in this space with several features for CI. It can be easily plugged into a build via a webhook, triggered at any point to initiate API testing, the scope of which includes validation of the data returned in an API response. Runscope also provides detailed information on how to configure the webhook, what they call the “Trigger URL” for a series of CI servers and platforms. They also provide functionality that helps users to analyze results by providing integration to analytics platforms, allowing detailed digestion of test results. Finally, Runscope has developed a plugin for Jenkins that is available in the Jenkins plugin repository, and has provided a walkthrough of the implementation.

SmartBear Ready! API Platform

SmartBear Software has a long history in both SOAP and REST API testing and are probably most well known as the providers of the commercial version of SOAP UI. SmartBear has entered the CI market with their Ready! API platform, which provides developers with the means to create API tests with their TestRunner and TestServer products. Using SmartBear products, tests can also be integrated with Jenkins using a plugin, which allows the tests to be included in the build pipeline for a given API. For CI servers other than Jenkins, SOAP UI comes with a command line tool that could be called using script from a build job definition in the manner we’ve already mentioned above.

Swagger Diff

Swagger Diff is a command line tool created by Civis Analytics to test backward compatibility between two versions of a Swagger description for a given API, the idea being to plug this into a CI server using a script-based build job definition as described above. On finding a breaking change the tool will return an error, providing details of the diff that highlights the source of the incompatibility.

Conclusion

Continuous Integration – Develop, Test, Deploy

This collection highlights the fact that continuous integration tools related to APIs are a growing presence in the API economy. With a mixture of both open source and commercial solutions available, companies can spur their API development through effective, automated integration processes.

Another area we have not covered that is likely to be increasingly important is from the perspective of the API consumer; publicly available API descriptions, whether hosted at the API provider or in an API directory will become triggers for continuous integration, allowing build jobs to run based on published changes to the format, generating new client libraries for the API in question.

Real benefits for organizations lie in the practice of continuous integration. All API providers should consider leveraging the practice to ensure their efforts in exposing APIs have the best chance of success.