Nowadays, you can find web APIs everywhere you look, and developers create more of them each day. On the one hand, that’s a good thing: it shows we’re living in an age of interoperability. On the other hand, the shift towards heavily API-based platforms brings with it severe technical overhead. Developers now face the high complexities of distributed computing systems. The solution to this, says Tina Huang, Co-Founder, and CTO of Transposit, is API composition. By packing the complex tedium of API and service integration into a separate layer, we can allow developers to focus on functionality while minimizing developmental friction.

The Problem: Growing Complexity in API Ecosystems

Driven in part by the boom of the SaaS industry, organizations around the world are looking to adhere to API-first strategies. Indeed, the reasons for doing so are numerous: decoupling tech systems, allowing teams to operate independently, and catering to a growing number of client devices, among others. The result of this is that all web applications are effectively becoming platforms. Not only must apps share data with others, but they also must consume data from a variety of external sources. Thus, fewer and fewer developers build applications purely from scratch. Instead, the work of developers is increasingly dominated by the need to connect and combine existing APIs into new applications (and new APIs). Unfortunately, connecting APIs is rarely as easy “snapping together LEGO blocks,” says Tina. Every API is a little different — a snowflake, you could say. Developers have to spend additional time reading nitty-gritty documentation on how one particular API works before they can start using it.   In addition to documentation, several technical nuances make every API unique:

• How data is separated across endpoints (e.g., the N+1 problem)
• How pagination is managed; for example, the Slack API uses a whopping five different mechanisms for pagination
• How authentication takes place
• Rate limits, retries, and so on

These nuances cause friction and increase onboarding time. Ultimately, they lead to a lot of complexity — not within an application’s core logic — but in the interactions of dozens of services and APIs. If you pause and think about it, this means that all application developers in the API ecosystem need to be experts on distributed computing. They need to be able to code against unreliable networks, manage long latencies, and avoid invoking rate limits. Our industry already faces a shortage of developers. Tina says these new standards only raise the bar for what it takes to be an effective developer in the API world.

The Solution: API Composition

Tina defines API composition as all of the technical work needed to connect and combine data sources. If you’re familiar with developer libraries, think of API composition as the same principle — cutting out the technical overhead of API adoption — on a bigger scale. Here’s how Tina defines API composition:

“An API composition platform abstracts away those mechanical details of working with different APIs and lets developers focus on the core application logic. It lets developers express their intent at a higher level, for example, the developer can specify the quantity and criteria for results, and the API composition platform paginates and filters data automatically.”

Since some compositional logic is often shared within and even across organizations, you can begin to consolidate this excess code as time goes on. Tina says doing so can help to amortize the development costs of building against APIs across multiple developers. An approach of shared compositional logic has several benefits. By using a single, shared cache, compositional logic enables applications to perform better. It also improves reliability by ensuring you don’t push the technical constraints of individual APIs. Tina provides a real-life example of how this kind of solution might be useful in the context of rate-limiting. She once spoke to a company whose main product depended on the Salesforce API. In another part of the organization, a contractor had been building a Marketo-SalesForce integration. Unfortunately, the contractor blew through his rate limits, thus taking down the organization’s core production service. So, how do you get started with API composition?

Your Options for API Composition

Tina says you have a few choices when it comes to your API composition solution.

Libraries and Modules

The most lightweight approach to API composition is using libraries and modules. However, it won’t solve all of your problems. One issue is the lack of governance, which in time means you’ll end up with many versions of the same library, with no easy way to deprecate old versions. If there’s no easy method for upgrading libraries throughout your different products, this could become a pretty serious issue when there’s a security bug to patch. Tina mentions a few horror stories associated with libraries — like when the author of a simple NPM package called left-pad unpublished his work, breaking thousands of projects across the web. Or, when a hacker briefly hijacked the Eventstream library. Subtle changes to a library on one side can have massive consequences for the applications using them. Of course, these are extreme examples but demonstrate potential issues associated with libraries.

Microservices

Another solution for API composition is a microservices architecture. Using a management layer for microservices can provide a centralized place for caching, rate-limiting, and deploying security fixes. Tina believes this is overkill for a lot of problems. Taking this microservices management route requires developers to create even more technical overhead — allocating new infrastructure, configuring deployment, and setting up monitoring for each new service. If you only have a handful of microservices, you’re probably doing it right: using microservices to encapsulate internal databases and represent your core infrastructure. However, if you have hundreds or thousands of microservices, that’s a sign you might be using them to house compositional logic, too.

Serverless

A lighter-weight alternative is a serverless approach, also known as the Functions as a Service architecture. In this case, you use servers that you don’t run or manage yourself. Serverless allows developers to focus on their code with minimal “housekeeping.” This does have its own complexities — having to route requests to the right code for a gateway, or, if you use a solution like AWS Lambda, accounting for the latency of spinning up a new service. There’s also the problem of not being able to store state within functions. Ultimately, this is the solution that Tina recommends. As long as the platform allows developers to focus on their code while it takes care of the compositional “heavy lifting,” it’s doing its job.

If You’re Not Ready Yet…

If you’re not yet ready to implement or build an API composition solution, Tina has some pointers for you. The problem of more APIs and more technical overhead is inevitable, and the first step is accepting that. Then, Tina’s advice is as follows:

• Adopt internal standards about how you create APIs (authentication, pagination)
• Adopt standards and specifications like OpenAPI
• Check for Functions as a Service solutions in your existing cloud provider
• Manage your APIs like products, always thinking about the user experience

Final Thoughts

There’s a rapidly-growing number of APIs out in the wild, and connecting with them is becoming seriously complicated. Tina says we can eliminate a large amount of this complexity with API composition solutions, which absorb the tedious integration logic of API ecosystems and allow developers to focus on the core functionality of their application. Between libraries, microservices architectures, and the serverless/Functions as a Service approach, Tina believes the latter is the best solution for many of us.