Every so often, a new image file format comes along that does a better job of compressing images than JPEG or PNG. We’ve seen Google pushing its own Webp format, (purportedly 26% smaller in size compared to PNGs), and new formats BPG and FLIF are contenders, packaging images in less and less space and time then their predecessors.

Aside from compression, user experience is being improved with SVG, a vector format suited for retina screens. Animated GIFs have soared in popularity, now present in too many memes, and available in most messaging apps. Imgur now supports GIFV, a video alternative. It’s apparent that new image formats will likely continue to emerge and shift in usage in the coming years. The same high volatility toward change is true for all media types.

With that in mind, how can we design long-lasting APIs that adapt to an ever changing world of file formats? According to Erik Michaels-Ober of Soundcloud, the million dollar answer is content negotiation — a technique that has been around for decades, but surprisingly has not been adopted by many REST APIs that serve structured data or media.

What is Content Negotiation?

”HTTP has provisions for several mechanisms for “content negotiation” – the process of selecting the best representation for a given response when there are multiple representations available.”
– RFC 2616 Fielding

Content negotiation allows a user to determine which media types they prefer to receive from the server. It’s a mechanism defined in the HTTP protocol (RFC 7231), which is great, because REST APIs work alongside HTTP.

The most common example of content negotiation is browser-server behavior. Let’s say a browser has been programmed to privilege Swedish over English, and if neither are available, to serve up Danish. The Accept header will look something like this:

Accept-Language: sv; q=1.0, 
en; q=0.7, 
da; q=0.5

Accept: text/html; q=1.0, 
text/*; q=0.8, 
image/gif; q=0.6, 
image/jpeg; q=0.4, 
image/*; q=0.3, 
*/*; q=0.1

Browsers can send information as part of each request about the representations they prefer, with q-factors to denote the usage preference relative to other languages, text formats, and image types. Then, the server responds to best fit these needs. The user agent can also request a specific data format to be served from a web service — like application/json or application/xml.

State of API Design

Many tend to see RESTful APIs as a better design choice than SOAP. Though there are common cases where using SOAP makes sense, REST as an architectural style holds much credibility, designed by Roy Fielding, the creator of HTTP, which is the most successful protocol ever invented.

In the spirit of designing long-lasting APIs, REST is arguably better than SOAP for content distribution. Take the varying ways to retrieve a single resource.

With SOAP, all data is stored in the XML body, so a POST request to /api.xml would access the same endpoint for every request. This design throws out the URI (Unique Resource Identifier) and lumps all data into a single resource. There are then additional remote procedures necessary to access a single image file. This isn’t exactly how the web naturally functions.

REST is specifically designed work with HTTP. Since web resources are each identified by URIs, GET requests can be made to unique endpoints. A call to /users/avatar/nordicapis.png can easily retrieve an image file. This design is simpler, removing duplication, complex remote procedures, and working with the HTTP protocol the way it was intended.

The Case to Erase Filename Extensions

.txt, .doc., .rtf, .pdf, .jpeg, .gif, .png, .mp3, .wav— nearly every operating system uses these filename extensions, but are they completely necessary? In a way no – within a UNIX environment, whether a program is executable or not is determined by the permissions on the file, not the extension. This added layer of complexity was, out of habit, evolved into the web as a matter of style. According to Michaels-Ober, we should not be hard coding these file extensions into URIs. Below we’ll explore some cases why.

Accept Header HTTP Documentation

The Accept request-header field can be used to specify certain media types which are acceptable for the response. Accept headers can be used to indicate that the request is specifically limited to a small set of desired types, as in the case of a request for an in-line image.
–Accept Header RFC

Take a scenario in which we want to retrieve a user’s avatar photo. As the RFC says you may specify multiple formats that you can accept, requesting a PNG becomes redundant. If we are using an accept header, we can drop the PNG from the resource name, and replace it with a list of acceptable image types, with a q-factor to determine how strongly we prefer a certain format relative to others.

GET /user/avatar/nordicapis.png

Accept: image/png,
	image/jpeg; q=0.8,
image/gif; q=0.8,
	image/*; q=0,5
	application/json; q=0.1
…

What if there is an error response? We want the ability to return JSON, and handle the error on the client side. Since this request may return JPEG, GIF, and or even JSON, having a .png in the URI doesn’t really make much sense.

Let’s say the social network releases new features, and now also supports video avatars. The great thing is that the URI remains constant, and now all the client has to do is say that they accept videos in the HTTP accept header.

GET /user/avatar/nordicapis

Accept: video/avi; q=0.8,
	video/mov; 8=0.5,
video/*,
	image/gif; q=0.8,
	image/*; q=0.5,
	application/json; q=0.1,
…

This is content negotiation. The client says what is supports, what it accepts, and the server responds with the relevant return formats, if available.

Dynamically Serving XML/JSON Formats

If your API returns an XML representation of a user, you may want to reference the same user in a completely different format — it may change over time, or depending on the client or device accessing it. Two representations of the same resource — JSON/XML representations — should have the same URI for the sake of longevity, but if clients have varying format preferences, how do we know which format to serve?

If we get rid of filename extensions, and use an accept header, we can remove redundancies, replacing these:

GET /users/nordicapis.json
GET /users/nordicapis.xml

With this:

GET /users/nordicapis

Using information in the accept header, the most appropriate representation can be automatically chosen, depending on the client’s preferences.

Accept-Language Header

Here is a more practical example. Also, as defined in the RFC:

The Accept-Language request-header field is similar to Accept, but restricts the set of natural languages that are preferred as a response to the request.

We’re used to requesting an index, usually specified like this: GET /index.se.html. However, most browsers already send along an Accept-Language header.

GET /index.se.html

Accept-Language: se

Again, specifying this in the URL is unnecessary. Using content negotiation, language preferences can be stipulated in the Accept-Language header:

Accept-Language: se,  
    en-us; q0.8,  
    en;q=0.7  

Using this approach, we’re not encoding the language or the format of the resources into the URI, which makes a lot of sense. Using a single /index/ endpoint erases duplication, and produces nice, clean, readable URLs. Letting the clients specify their preferences in this way can also help improve marketing with cleaner documentation.

Real World Reasons to Adopt Content Negotiation

File formats change over time, affecting the way content is distributed throughout the web. Since APIs released today may be in client systems for years, we must implement content negotiation design now to increase longevity.

Travis CI

What if you eventually want to drop support for a certain file format? When Travis CI updated their badge status API to SVG, they stopped serving a PNG file. But, for backwards compatibility, the SVG version will be returned even if you request PNG or SVG. Here is the folly of coding file formats into the URI — when format adoption changes, you may end up returning a different file format from what is technically requested.

Google Formats

WebM and Webp are alternative file formats that have been pushed by Google. Though Google supports the formats within their own apps, they haven’t really taken off with too much steam in third party environments. However, it’s hard to underplay the influence a behemoth like Google has in the web economy. If any new format does catches on, you will need a way to evolve your content API.

Twitter

In June 2014 Twitter began supporting animated GIFs. In 2015 GIFs were adopted into JSON payloads as well. The way they support it is by taking a GIF, and converting it into a video (Mp4) for compression benefits. Just another example of format evolution on a trusted API-first platform.

Software Design on the Scale of Decades

It wasn’t too long ago that PNG was the hot new format, praised for it’s lossless compression. In the coming years, we may say goodbye to the file type in favor of crystal clear vector SVGs. A similar change will inevitably occur in text as well — JSON has usurped XML in popularity, and in the future, JSON could be replaced in favor of another format, like EDN, an extensible alternative to JSON with more base types.

With forecasts set for further file format fluctuation (alliteration!), we must design our APIs as evolvable from the onset if we want them to stand the test of time. This means exposing resources, not representations, and not encoding file format into the URL. According to Rob Zazueta of Mashery, a content negotiation approach could even replace traditional API versioning.

REST is software design on the scale of decades: every detail is intended to promote software longevity and independent evolution. Many of the constraints are directly opposed to short-term efficiency. Unfortunately, people are fairly good at short-term design, and usually awful at long-term design. Most don’t think they need to design past the current release.
–Roy Fielding

What is your team doing to design web APIs “on the scale of decades” ?