Designing Evolvable APIs for the Web: Identification

The current business and technical relevance of Web APIs is visible and documented in an increasing number of articles, products and conferences. However, these efforts have been mostly centered on the API (Application Programming Interface) side and less on the Web.

This article is the first in a series that aims to bring the focus back to the Web, on its underlying architecture, and on what it means to build evolvable APIs for it. We address not only the technologies that build the Web but also the fundamental architectural principles that guided the development of these technologies.

As we will see, the Web architecture is perfectly adequate for exposing APIs. However, it has some non obvious characteristics that should be considered if we want to take full advantage of what the Web has to offer.

Introducing the 3 Pillars of the Web

The “Architecture of the World Wide Web, Volume One” document helps define resources and describes the three architectural pillars of the Web:

Identification: All resources are globally identified identified via URIs (Uniform Resource Identifiers).
Interaction: Web agents interact via the exchange of messages, namely HTTP messages.
Formats: The exchanged messages convey information based on pre-defined formats.

In this series we expand on each pillar of this core architectural foundation, and their relevance for Web APIs. In this specific article we’ll focus on defining Identification, and venture into Interaction and Formats in the following posts.

Defining Resource Types

Our journey starts on a rather old (and perhaps not very well known) document, entitled “Architecture of the World Wide Web, Volume One”, authored by the World Wide Web Consortium in 2004. This document starts by stating:

The World Wide Web (WWW, or simply Web) is an information space in which the items of interest, referred to as resources, are identified by global identifiers called Uniform Resource Identifiers (URI)

This sentence introduces two important characteristics of the Web. First, the Web is an information space and not just a set of pages, making it very applicable for APIs that interact with information. Second, the elements that compose this space are called resources. The resource concept is central to Web architecture, however its characterization may not be obvious.

Types of resources, or “items of interest”, are varied. For an API such as the one provided by GitHub, these items will be repositories, commits, branches, issues, users, organizations, and so on. In an IoT (Internet of Things) system items will probably be sensors and actuators.

Read: Microservice Showdown: REST vs SOAP vs Apache Thrift

Identification: URI vs URL

Each Web resource is identified by at least one global and uniform identifier, called URI. On the Web, a URI unambiguously identifies a single resource, without needing additional context.

The global property of this identification system comes from the fact that no further information, other that the URI, is required to identify a resource. This contrasts with local identifiers that are only meaningful given additional context. As an example, the identifier 1234567 is probably not enough to identify and locate an entry on a relational database. However, given the additional information that it is an invoice identifier we are probably now able to access the invoice via a select * from invoices where id = '1234567' or via a GetInvoiceById operation.

A special type of URI called Universal Resource Locator (URL) provides even more information: URLs are enough to interact with the resource they identify. For instance, an “http” URL is enough to obtain a representation of the identified resource state via an HTTP GET request.

Instead of resource local identifiers (e.g. 1234567) and operations (e.g. GetInvoiceById), the Web provides us with global identifiers (e.g. https://api.example.com/invoices/1234567) and generic interaction operations (e.g. HTTP GET method).

Defining URI/URL Syntax and Structure

The URI syntax is generally defined by RFC 3986 as:

URI         = scheme ":" hier-part [ "?" query ] [ "#" fragment ]

For the particular case of the “http” (and “https”) scheme, defined on RFC 7230, the syntax becomes:

http-URI = "http:" "//" authority path-abempty [ "?" query ] [ "#" fragment ]

Defining the URL structure is one of the required tasks when designing a Web API, and is guided by multiple requisites. First, we need to make sure that a URL encodes all the information required to fully identify the target resource, including any local identifiers such as the invoice number or the invoice line number. If the resource represents a collection, then additional information such as pagination as well and filtering and sorting criteria must be also included.

Note that the same collection filtered or sorted by a different criteria are different resources. For instance, the URLs https://api.example.com/customer/123/invoices?sort=date and https://api.example.com/customer/123/invoices?sort=amount identify two distinct resources. It is not adequate to interpret these two URLs as the same URL with different parameters, namely because the query string is an integral component of the URL and not just a decorator over it. Since other components such as caching intermediaries rely on these specifics, the distinction isn’t just superfluous pedantry.

A typical question that arises when designing the URL structure is what to include on the path versus what to include on the query string. Keeping in mind that both are URL components, the answer comes from the distinction stated on RFC 3986, which outlines URI syntax:

The path component contains data, usually organized in hierarchical
form, that, along with data in the non-hierarchical query component
(…), serves to identify a resource (…)

This means that the path should be used to represent information that has some hierarchical relation. For instance, the GitHub API uses the path to encode the repository owner, the repository name, and the issue number when identifying a single issue comment.

https://api.github.com/repos/webapibook/issuetracker/issues/1

On the other hand, non-hierarchical data such as sorting and filtering criteria are typically included in the query string, using name-value pairs. As an example, the GitHub API uses the query string to encode the search keywords, the sort field, and the sorting order on its search API.

https://api.github.com/search/repositories?q=user:webapibook&sort=issues&order=desc

Notice however that this name-value structure used on the query string URI part is not mandated by the RFC 3986. It is a very common usage, made popular by the HTML specification which uses this format, named application/x-www-form-urlencoded to represent the form data set. The URL structure must also be designed to be easy to handle on the server side. Unfortunately this depends on the concrete HTTP server framework, namely its routing engine, so it’s hard to provide general rules here. However, special care must be taken to enable API evolvability by allowing new resource types to be identified without creating ambiguities.

On URL Readability

Especially in the realm of APIs, URL readability should be considered as it makes URL processing easier for humans. For example, the meaning of https://api.example.com/customer/123 is much easier to understand than https://api.example.com/2335bbb6-8e33-4f88-8088-03d996f52752, namely when looking at log traces.

All this discussion regarding the URL structure design is mostly relevant for the server side only. For the clients, URLs are mostly opaque as stated by the best practices in “Architecture of the World Wide Web, Volume One”

Good practice: URI opacity

Agents making use of URIs SHOULD NOT attempt to infer properties of the referenced resource.

Namely, clients should not be dependent on a specific URL structure, rather, they should just use the URLs as they are provided by the server using hypermedia controls. (We will address hypermedia on a forthcoming article in this series).

In the scenario where part of the information to encode a URL must be provided by the client, then the Web API should use URI templates, such as https://api.example.com/customer/123/invoices{?sort}. This template syntax is defined by RFC 6570 (URI Template) and in this example informs the client that the sort variable must be expanded on in the query string. The client must provide the sort variable in order to expand the URI template into a URL. This means that most probably the client, or the user on whose behalf it is acting, know that a sort variable exists, and also understand the semantics and possible values that can be entered. However, since the URL structure is not hard-coded on the client side, it can evolve without breaking the clients. As an example, GitHub uses URI templates on its root resource — https://api.github.com — namely to describe the structure and method required to perform searches.

"code_search_url": "https://api.github.com/search/code?q={query}{&page,per_page,sort,order}",

The {&page,per_page,sort,order} means that the query string accepts four optional keys: page, per_page, sort and order.

Conclusion: Use URI Templates for Evolvability

As we will see multiple times throughout this series, the interaction between the client and servers always require some form of contract. However, the way these contracts are defined determine the evolvability of the interaction. Using URI templates doesn’t eliminate the a priori knowledge required on the client-side, however, it allows the server to change the overall URI structure.

This ends our first article in the series, in which we aim to put the focus of Web APIs back on the Web, on its underlying architecture, and on what it means to build evolvable APIs for it. In the following article we will be looking into the second pillar of the Web – interaction. Sign up to our newsletter to make sure you don’t miss the next posts in this series.