Learn how Target scaled its extensive corporate API spec catalog

Many developers are filled with dread at the thought of sitting down to write or maintain a specification or documentation for their API. That’s not because they lack confidence in their writing skills (although that may be a factor), but because they’re so close to the product that they find it difficult to isolate what readers will know by instinct and what needs explaining.

Now, imagine that feeling of dread multiplied by 1,000x. That’s what Jay Dreyer, lead engineer at Target, was faced with when trying to get the company’s… shall we say, extensive collection of APIs in good shape.

At the ASC 2020 event, hosted by Linux Foundation, Dreyer spoke via Zoom about some of the challenges he faced (and still faces today), as well as his use of things like OpenAPI Specification and Swagger tooling. Whether you have a thousand API specs to manage or just the one, Dreyer has some great advice.

API Specification vs. API Documentation

We’ve previously written about the advantages offered by extensive documentation and designing APIs to spec But, before we get too deep into talking about managing large numbers of specs, it might be worth touching on the differences between API documentation and API specifications.

In the simplest terms, a specification might explain what an API does and how it links to other APIs. Documentation gets into the nitty-gritty of how developers can actually use different functions, often including use cases, and how to call those functions.

While that doesn’t necessarily mean that managing huge numbers of specs is easy, it’s worth noting that specifications are slightly more straightforward in that they rely on the same key questions:

• How is the API designed?
• What does it do?
• What functions does it facilitate?

Out of the Box API Management Vs. Custom Solutions

Dreyer describes how, back in 2012, Target was using Apigee to manage their APIs via an API gateway. “As we kept adding specifications, we didn’t have a very good process in place… We had specs everywhere and it was kind of a nightmare.”

Specs arrived in different formats — e.g., PDF, Microsoft Word, Google Docs — and weren’t kept up to date effectively after uploaded to the Apigee developer portal using Drupal.

“In 2016,” Dreyer said, “we created our own platform (ending the Apigee relationship), and the company became more of a developer-centric company.” That meant building a custom gateway with a homegrown developer portal.

Their team also decided to document in a Swagger spec. Part of that process meant documenting things like:

• Who the owner of the API is
• The lifecycle status of the API
• Who to talk to if something breaks
• How long the API has been out there

Beyond simplifying the procedures around dealing with problems, the end goal was to improve consistency around how individual APIs were cataloged and managed.

API Specifications And Consistency

“Initially, we let teams do whatever they wanted to do,” Dreyer said. But that caused problems.

Although taking a more hands-off approach is often seen as an effective management strategy, especially when you’re trying to let skilled development teams work in the way that best suits them, it can complicate things down the line.

Dreyer said that “it became almost impossible for us to help teams because the way they were doing things was so different from API to API.”
The solution? The implementation of standards. Standard response codes and standard error codes, for example, and using JSON for responses.

The use of snake_case was mandated, verbs in endpoint names were outlawed, and plural nouns were used for API names and endpoints. Generally speaking, standard RESTful best practices are encouraged.

All of this information was outlined in comprehensive internal documentation and stored alongside all the other information you’d expect from a good developer portal.

Automating API Specification Processes

At Target, every API goes through an onboarding process that includes a formal review of the specification. Dreyer uses a custom tool called SpecVet, not currently open-source, on any pull request to identify errors (categorized either as critical or warnings) in specifications.

Bringing an element of automation to the evaluation process is almost essential when trying to manage upwards of 1,000 APIs. Yet, it probably isn’t necessary if you’re only dealing with a handful of APIs or fewer.

Besides, Dreyer suggests that their attempts to create consistency have been successful enough that most APIs pass this aspect of the evaluation: “once teams have been through the process once or twice, they know what we’re looking for.”

It’s also worth pointing out that you can’t automate every aspect of the approval process; there are no guarantees on accuracy, no way to force teams to update their specifications when changes are made. Dreyer acknowledges how, at times, he still has to chase down specifications that are already in production.

The Future of API Specifications

The OpenAPI Specification, now at version 3.0.3, is almost certainly the best-known spec out there. And, while it’s certainly not compulsory, it’s something that more and more API developers are adopting when putting out specifications.

As for automating the production of API specifications, we’re not quite there yet. That’s actually one of the key issues that Dreyer highlights in his talk, i.e., that creating and checking specifications is still mostly a manual process.

That’s not to say, however, that this can’t change. Checklists, samples, and templates for REST API documentation are already pretty standard, and Swagger already offers a Codegen tool that can generate server stubs and client SDKs for an API defined with the OpenAPI specification.

Tools like OpenAPI Generator and others generate documentation from OpenAPI documents too. Perhaps a similar product that’s capable of generating specifications just by examining an API isn’t too far away after all.

For now, the future of API specifications probably depends heavily on how widely the OpenAPI spec and other similar standards are adopted. We’ve been singing its praises for a while, but read this post if you’re not sure where to start.