With API breaches emerging by the week, testing is more crucial than ever

There’s a tendency to believe that if something is working, it’s fine. As the saying goes, “if it ain’t broke, don’t fix it.” The problem with this is that, particularly when it comes to APIs, it isn’t always easy to spot whether or not something is broken.

At our 2017 Platform Summit Patrick Poulin of API Fortress asked the following question: “Do you put the same level of effort into testing your API that you do testing websites and apps?” If we’re going purely by the numbers, the answer is that most people don’t.

Patrick cites a Smartbear study that found that less than 10% of API issues are resolved within 24 hours, and 25% of issues remain unresolved for 7 days or more.

Below we’ll be looking at why that’s the case — it’s worth pointing out right off the bat that, as you’ll see from the three real world examples of “leaky APIs” provided by Patrick below, it usually comes down to ignorance of potential API breaking issues and not negligence. Let’s discover what is typically left out, and zone in on three ways such issues can damage your business and its reputation.

Time and Use are Factors

The easiest answer to the question of why API errors persist is that, unless we’re talking about a company that deals exclusively with APIs, developers are typically juggling API maintenance with other projects.

If ownership of an API, and issues relating to it, is poorly defined then developers may be inclined to think that someone else will be the one to fix the problem. And that’s if they look at the API often enough to spot the problem in the first place.

Then there’s that fact that many APIs are used infrequently by a small number of people, sometimes by just a handful from within the organization itself. In other words, there simply may not be enough active users to identify and draw attention to the problem.

But there’s a problem with this explanation. We know from the study linked above that even dedicated API companies with products that have tens of thousands of users aren’t exempt from responding sluggishly, so there must be other issues at work here…

Flaws ≠ Broken (but they do hurt trust)

Patrick says in his talk that “we notice when an API is down. We don’t notice when it’s a little bit leaky.” That’s true not just of APIs but modern tech in general; we get angry when a device won’t connect to the internet at all but we’re usually happy to write off slow performance to our router, the weather or even that “everyone in the building must be online right now”…at least for a little while.

Although most API developers take performance and uptime into account when assessing the quality of their APIs, Patrick makes a case for considering accuracy alongside these factors. In other words, that all the functions of the API are doing exactly what they’re supposed to do in a reasonable timeframe.

The first example he gives in his talk is a major book publisher that provided a self-reported uptime of 99.87% on their affiliate API. It turned out that, on closer inspection, their actual uptime was 97.3% because there were hundreds of 404 errors coming back.

The reason for this was that their API manager was caching product endpoints and, every time the database was updated, that endpoint would be incorrect for a few hours per week. The fix here was to test on deployment, which brought their uptime closer to what they had originally been reporting.

We know that downtime is a significant factor when it comes to building trust with consumers, with 18% of companies saying that IT outages had a “very damaging” impact on their reputation. If users can’t trust your API to work reliably, they’ll look elsewhere.

However, as we’ll see in the examples below, undetected API issues can have a more immediate financial impact as well…and not just on your business.

Flawed APIs Harm Bottom Lines

Patrick also describes his company’s work with Etsy, and how they were missing out on up to $75,000 worth of sales every day. Testing conducted based on their own API documentation found that 3,000 products lacked proper categorisation because a field was marked optional instead of required.

As a result, these products couldn’t be found using the top level navigation, let alone via external applications using their API. No appearance in search results lesser sales, and less of a cut for Etsy.

The final example Patrick provides is a print on demand company that was using a stock image company’s API and reported zero revenue on one particular Wednesday. Black Wednesday, if you will. A small change in structural development by the stock image company meant that the print on demand company’s particularly strict integration could no longer get the information it needed.

The really scary thing? It was an accountant who caught the issue, rather than a developer from one of the teams involved. Again, this goes to show how easy it is for issues to go undetected when no-one knows to (or bothers to) look for them. Incidentally, the outcome of the situation was that the stock image company started testing their API on deployment much more strictly.

Clearly, missing out on revenue is a bad thing. But for an API company, where facilitating processes with other companies is key, recommendations often come from positive reviews or via word of mouth. The repercussions of such errors, then, could cause a loss of potential business too. Have enough days like Black Wednesday, and you’ll soon find your API has no users at all.

Final Thoughts

Patrick points out early in his talk that just because something is up doesn’t mean that it’s functioning properly. We’ve seen above just how true that can be. Later on, he makes two interrelated statements that summarise the ideal approach to testing and APIs:

1. “We’ve reached that point of maturity now that testing has to be taken seriously.”
2. “We all think you can’t test an API very easily, and that’s just not true anymore.”

Providing examples of API monitoring tools and products such as SmartBear, API Fortress, Runscope and Postman, he highlights the fact that even non-technical employees can pitch in with API testing.

Based on what we’ve seen above, the following steps are vital to preventing a leaky API:

• Build and run integration tests, or have someone else do it
• Test on deployment more thoroughly, i.e. tests need to be detailed, rather than loose
• Encourage your users to get in touch about any issues they encounter
• Test the entire lifecycle (see slide below), including live environment

Patrick stresses how important it is to test for API issues throughout the entire development lifecycle.

Let’s return to the example mentioned earlier in this post: users of a certain website or app will only blame sluggish performance on internal issues (e.g. signal noise messing with their router, someone else in the house downloading files etc.) for so long before they start trying to pin the blame elsewhere.

Typically, they’ll start with the site owner and/or Internet Service Provider (or, in our case, API provider). If they can’t fix the issue in a reasonable timeframe then the odds are good that they’ll ditch the site/ISP forever and navigate somewhere else.

Most APIs are designed to reduce friction and make life easier for their users. If an API isn’t doing that due to a fault, however minor it might seem to you, then you risk users looking for an alternative way to get the job done.