It’s a universal truth that testing for errors, and debugging any that show up, isn’t a fun process.

Almost all API providers are familiar with the process of monitoring and testing. Before you commit code, API behaviors must be thoroughly vetted, as with any software release.

So why is it that when it comes to API documentation, many people adopt the stance of “set it and forget it”? Too often, API owners write up some static documentation, post it in their developer center, and never touch it again.

According to Rouven Weßling of Contentful, who spoke about automating API documentation testing at our 2016 Platform Summit, API owners need to put more emphasis on keeping their documentation up to date and consistent with their API behavior, as developer users “would rather have no documentation, or incomplete documentation, than wrong documentation.”

In other words, any API owner that provides broken documentation runs the risk of infuriating users to the extent that they may abandon the API altogether provided they can find a viable alternative. In order to prevent that, we need to make sure that API documentation testing becomes the norm.

Thankfully, to avoid any headaches, there are tools out there that can automate a good amount of this process. So within this article, we’ll give a high level overview of using API Blueprint, Dredd, and Travis CI to perform continuous API documentation testing.

Testing API Documentation 101

We’ve written extensively on why providing comprehensive API documentation is so important, and the reasons should be pretty obvious:

• Can provide a quick and easy guide to getting started with your API
• Demonstrates useful functions
• Is the front face of your developer program, acting as a point of sale for acquiring new developers
• Helps reduce user frustration

But API documentation can only perform these functions if it’s succinct, useful, and accurate. While regular testing can’t help with the succinct aspect – testing will help ensure that documentation is technically accurate and up to date. Though it’s not a magic recipe to turn it into great, stunning documentation – it can definitely help improve accuracy.

The first way to do so is to adopt standardization. Crafting human-readable documentation involves a certain personal touch, however, if you’re manually annotating documentation, it can be difficult to automate tests if there are elements of subjectivity.

Weßling’s answer to this is to use an API specification:

“If you want to test your documentation, you really need to base it off a spec. There’s no way of just testing prose you write down; you need to have some well-defined format to test against.”

Using API Blueprint could certainly work, but other popular specification formats work too: Swagger/Open API Specification and RAML. As in so many cases, it’s less about the tool you use but rather the measure you take to facilitate the documentation testing process.

Set up Hooks to Test Every API Transaction

As Weßling points out, testing read-only functions using GET requests doesn’t give you the whole story. To test the complex interactions of your entire API, he recommends to start experimenting with hooks.

This is where Dredd comes in. Dredd acts as a CLI for validating that your API documentation and actual implementation are synced up. Used in this context, hooks allow you to execute code around each test step and override certain things in the lifecycle of an individual test. Here are a few examples of what they look like:

• beforeAll – called at beginning of the whole test run
• beforeEach – called before each HTTP transaction
• before – called before some specific HTTP transaction
• beforeEachValidation – called before each HTTP transaction is validated
• and so on…

Or, if you want to hook in after transactions, swap in the word ‘after’ for ‘before’ in the functions above. Using hooks will help you avoid errors that result from security issues, help clean up after test step(s), debug, and change/set expectations.

The bad news is that creating all of the hooks you need isn’t an especially fun experience, particularly if you need to create them for many different transactions in a test. “It’s just like writing integration tests… just even less fun. But it doesn’t replace integration tests unfortunately,” says Weßling. Still, it’s one that’s necessary to reap the full benefits.

Rouven remarks that “if you just test your GET requests, you’re doing more than most people are” but, if you’ve ever been infuriated by inaccurate documentation, then you’ll surely see the value in going the extra mile to test the rest of your API calls as well. And, with the help of some of the tools out there, this whole process doesn’t have to take as long as you might think.

How Often Should You Be Testing?

In Weßling’s opinion, this isn’t something you should be doing every few weeks but rather every time you make an adjustment to your API or the documentation associated with it. That probably sounds like a lot to take on, particularly if you’re continuously integrating new code or are part of a team that’s already spreading itself too thin, which is why it’s a great idea to automate this process.

There’s a range of tools available to help API developers looking to test their documentation, like Swagger Test Templates or Hippie Swagger. Because Dredd is used and recommended by Weßling, we’ll stick with that here.

Dredd takes an API Blueprint or Swagger specification and validates whether or not the production API functions is as described in its documentation. It’s language agnostic, compatible with PHP, Go, Python, Ruby, and other languages. If you use it with Travis CI, CircleCI, Jenkins or another *nix based CI, you can use it in tandem with continuous integration.

Using something like Travis CI for testing is helpful because it will notify you of errors in real-time as opposed to if you bundle them with your nightly integration tests. And pairing it with Apiary is even better, because it allows you to review test results in a more user-friendly format rather than having to dig through CI logs. Once you’ve successfully set up this process, you really can set and forget your API documentation because any changes you make to it will be tested automatically.

Common API Documentation Testing Issues

Unfortunately, because there’s not much in the way of standardization when it comes to API documentation testing, there isn’t yet an easy step-by-step process to transition from manual testing to automating the experience. At least, not without a few hiccups along the way.

Some of the problems that Weßling encountered, which may well apply to your API as well, are as follows:

1. API tests are going to run into issues of rate limiting, so you need to set a timeout.
2. You may need to use hooks to change request data before a request is sent if, say, a particular example in your documentation requires an authorization key.
3. Non-public API keys can result in security problems. In this case you’ll need to override things like authorization headers, again using hooks, before it gets sent to the log. Otherwise, as Weßling jokes, “you’re going to have a bad time.”
4. Mutating or deleting unneeded data created by testing CRUD (Create, Read, Update, Delete) operations during the API testing process.

Setting up the automation of API documentation testing is something that requires a significant initial time investment because, let’s face it, everything won’t necessarily go right the first time around. But, of course, it’s worth it for the time you’ll save in the long run.

API Documentation Testing Eventually Will Be Standard Practice

You might be looking at this post and wondering whether or not you have to add automatic API documentation testing to the growing heap of design practices and supplemental API tooling. Right now, perhaps it’s not that critical. But we think Weßling is onto something when he puts up a slide showing that the search “documentation error” has more than 225,000,000 results on Google.

Customer-centric companies are already testing their documentation by the thousands, and stable, predictable API documentation will continue to be a hallmark of quality developer experience. Especially as APIs become more dominant, comprehensive, up-to-date documentation with interactive sandboxes will become even more important than they already are.

API developers who neglect their documentation/specification might get away without implementing automatic update validation for a little while longer, but they run the risk of being left behind in years to come.