A Software Developer Kit, or SDK is perhaps one of the most important things a web API can have. SDKs are a helpful system of libraries and frameworks to support the API, often including documentation and other guidelines.

What can’t be argued are their significance to developers. According to Tristan Sokol of Square, “If you think about the end user experience, SDKs are the biggest touchpoints for people.” To this end, creating proper SDK material is highly vital to the success of an API.

One problem is handcrafted SDKs require much time and effort to maintain. Square takes an alternative approach; they automatically generate their SDKs using an API specification as a single source of truth.

At our Austin Summit in June 2018, Tristan described Square’s automatic SDK generation process in detail. So, in this article, we revisit that process to see how it works. As we explore what they’re doing at Square to automate SDK generation, we’ll see if a similar process could be applied to create and maintain developer-facing kits in other organizations.

Creating the OpenAPI Specification

“With SDK generation, one of the most important parts is your specification.”

Square uses OpenAPI Specification as a single source of truth for SDK generation.

The entire system Square uses for SDK generation rests on a superb API specification. At each stage, the specification is used as a foundational source of truth. With so much weight resting solely on a single point, Square knew that the specification solution must be something that is standardized, efficient, and understandable.

To this end, Square uses the OpenAPI Specification standard. In their case, the specification is a JSON file that defines the URL, what kind of HTTP requests can be made, what info must be provided for such requests, what info should be expected in return, and other general information. The specification is broadly separated into three parts — general info/metadata, paths, and models.

General info includes content such as descriptive licensing terms and constraints, contacts to reach out to in the case of issues, and more. A rough equivalent would be the beginning credits of a script ot other program, wherein developers typically provided commented out fields detailing the author, license information, etc.

Next, the specification delves into paths. Paths are essentially the endpoints provided for the API, and details as to what kind of HTTP request to make and what can be expected. During this stage, information concerning the process of authorization, authentication, and more are also detailed.

Finally, the specification delves into models. Models describe the objects that the API interacts with, and are mostly used to serialize the JSON response from the API into native objects for each language that is supported. This stage provides much of the later co-interpretative functionality, allowing for SDKs to be generated in a variety of languages efficiently and effectively.

Again, the value of this specification should be stressed — what is essentially being created at this stage is a single, universal source of truth, and because of this, the specification must at all times be correct and standardized.

Filling Templates Using Swagger Codegen

The SDKs are created with Swagger Codegen

With a strong specification in place, Square begins its process of automating the SDK generation itself. It does this using Swagger Codegen, and specifically leverages the templates it offers.

The specification is fed into a series of templates that take each part of the specification and separate the values into the appropriate places in the template according to the language in question and the functions that are being covered. The template uses a language called Mustache to do this, defining each part of the template and tying it to equivalent parts in the specification file.

During this stage, code comments are also generated, allowing for built-in documentation and comments from the specification codebase itself. This is hugely important, as it allows for in-line documentation and easy to follow resources for developers utilizing the SDK outside the native environment, such as those utilizing a different language than the core implementation.

Swagger Codegen also makes substantive use of configuration files that describe aspects of the SDK that are part of the specification layer. This adds another huge benefit in that package managers can very easily serialize or transpose the SDK formats and languages with additional templates.

Using Repos and Travis CI

Travis CI is used to automatically update SDK repos

This entire process happens as part of an automated process, so it helps to look at it within the context of the underlying repo system. When the codebase is altered, a pull request is made to update the remote repo in question, which triggers the Travis CI instance to begin running.

Travis CI then automatically installs Swagger Codegen, and calls for a decryption token from the repo core user. The token is used to decrypt the uploaded changes, exposing them to Travis CI. From here, a script generates the SDKs, looping through languages, looking through the specification, sticking the relevant data points into the templates, and ultimately, pumping out the SDK in question. From here, Travis CI takes that SDK, and pushes it out to the secondary repo, which functions as an SDK repo apart from the main branch that hosts the main codebase.

A big benefit for this automated process is that Travis CI also conducts automatic tests to ensure that the SDK doesn’t break major functionality or provide quantitatively bad user experience. By automating the entire process and coupling a sort of quasi-quality control, the entire code push can be handled quickly and with minimal pain to the affected end customer.

Protocol Buffers Feed the Specification

While this is all well and good, keeping the specification itself updated can be tricky when there are so many systems floating around, constantly iterating and changing. Square handles this through the extensive use of protocol buffers. In essence, a protocol buffer is a layer between internal services and the specification. This buffer helps construct the specification.

When any changes are made, such as a new field or modification to an existing field, this change is pushed to the protocol buffer, which then notes the change and automatically records it as part of the specification. What this fundamentally means is that the protocol buffer is actively “feeding” the specification.

By doing this, the specification, in theory, will never get out of sync. As long as the specification is up to date, so too will the SDK be current and updated.

Final Thoughts

Square is as much a case study in automation of SDK generation as it is in the value of automation in general. With proper planning, active understanding of the underlying systems, and sensible licensing of the systems that drive it, automation can serve not only as a workhorse, carrying out rote tasks, but as a system for enabling greater functions.

Square is a great example automation done right. Generating SDKs at this scale would not be possible without automation, and it’s this freedom that facilitates the easy spinning up of additional services, resources, forks, and more.

With that in mind, it should be noted that this all depends on having a single verifiable source of truth. Accordingly, keeping everything as lean, correct, and updated as possible is key to making something like this work. If you can properly balance it, however, it is a very powerful approach and a great tool for active, consistent development and external integration.