Have you noticed that your API sometimes feels a bit misunderstood, seen as a sideline or a purely technical solution?

This might happen because you put your API to do menial work with no real business strategy. It may also be because the API documentation is not pretty, correct, or understandable. Or because error messages and endpoints are “cryptic” to understand for the developer.

When developing with someone else’s API, many usability issues may arise. The goal for any API provider should be to listen to feedback, and eliminate these sort of issues.

Liberate our APIs

Stop keeping your API tucked away sweeping dust from corners and washing pots and pans.

What if the oppression from your API’s evil step sisters “Flat Files” and “Custom Code” was put to an end? What if we could give it a pumpkin carriage, glass slippers, and an awesome party dress and make it presentable for wider audiences?

When PlanMill introduced their API in 2009, it was one of the first business applications in Finland to have any kind of API. API-fying an ERP has been an interesting process, but now 7 years later, they have at least a promising beauty on their hands. They just need to get it ready for the party.

Put your Flat Files to a Pumpkin Carriage

Consider one of your customers has an integration challenge and asks for a very specific flat file. How can you offer this without lots of customized code or a heavy integration solution converting database to files? One way to get around this hurdle is by anticipating this need and building a dedicated dog-food solution.

For PlanMill, the first version of a pumpkin carriage consists of a flat file consuming /integrations endpoint, an import user interface built with React.js, and a Single-page-application-architecture that consumes it. The CSV files consumed by the API are transformed with a simple and configurable transformation layer to JSON files filled with normal resource-specific API calls.

As the Integration-endpoint re-uses the other API-endpoints, it also applies all checks and business rules to the imported data — as if the data was inserted through the user interface or API calls directly. This helps to avoid duplicate data in the system and helps users to use a flat file in their system and get the data updated or inserted with one upload, while data from the file is compared to the data already in system.

Anticipating a more custom solution for clients to consume via API was critical for PlanMill’s success in streamlining partner integrations. Here are a few pre-API and post-API cases to explain the impact of engineering a similar solution can have on your business model, resources, and customers:

Customer wants a simple synchronization between a 3rd party contract management and ERP:

“How much does it cost to make an SFTP + CSV integration, where we might want to transfer customer’s basic information to our new contract management system?”

Just as you have given a quote or possibly implemented it, the customer needs something more. Then another customer needs something a bit similar, but of course the data formats are different and columns need to be in a bit different order. Oh, and you have to include 15 custom fields in there, too.

Pre-API: 100–2000 lines of custom non-reusable code and settings, done by you to extract the file. In addition, setting up, testing and monitoring the SFTP.

Post-API: Everything including custom fields and metadata of field values comes out of the API. A generic transformation layer of data and you can pick out the fields you need. Depending on tooling and generic solution, 100–500 lines of fully re-usable code, either at your end, in an IaaS, or in the 3rd party system.

A bit more complex example with an on-premise ITSM (IT Service Management) system:

“How much does it cost to integrate our new ITSM system to your system, it’s an on-premise solution with limited access from outside, we are a couple of countries to the South from you and actually we need it yesterday. Couldn’t you just cook up a nice CSV we could use to synchronize pretty much all core data umpteen times a day to both directions?”

Pre-API: 5 days of development, probably half of it on-site

Post-API: “Give this API documentation to your integration party, contact us if you have questions”

And the nice and “simple” import cases:

“We just need our leads uploaded to your system. It is simple; we get them from the 3rd party booking agencies every week. Oh, what does the file look like? Well it can be anything, they all send us different looking files from their own systems. And the most important thing is to avoid duplicate leads or updating existing companies already in the system. Oh you need a unique id to handle that? Well, our data may have business id, account name or at least a domain name in it.”

Or

“A part of our business is SaaS but we also sell professional services and such. Can we somehow import the SaaS-related transaction costs from our other systems to your system monthly so we can invoice them from our customers? We would really like everything to be handled through one system.”

Or

“We re-sell search engine optimization, social media and other campaigns to our customers. We need to get the media campaign costs to your system, so we can invoice them from customers and monitor our own performance and profitability. All providers have different looking files and formats, but everyone has a campaign account number, which needs to be used for matching which project each campaign belongs to.”

Pre-API: At least 3–5 days of specifications, comparing data models, testing etc.

Post-API: “Here is our use case-specific, but still generic import documentation. Check that and decide if you want to do the import by manually uploading the files, or if you want to integrate with our API”

Does IaaS Fit Your API’s Feet like a Glass Slipper?

The new princes in town courting your API are the Integration-as-a-Service providers (IaaS), also called integration platforms (IPaaS), or even more high-level business-area specific operators like Service Integration and Management (SIAM) co-ordinators.

You may find the roles reversed: Traditionally, it was the prince trying the glass slipper to the feet of hundreds of girls. In API-land it will be you trying on the slippers provided by tens of integration providers to your APIs endpoints.

During this fitting process, you may find that some suit your API and your client’s purposes better than others do. Providers you may consider include Zapier, IFTTT, Mulesoft, BizTalk or business-area specific solutions in Finland like Sofigate, Verco, and Ambientia’s ITSM related integrations. Important things you want to consider when choosing among these providers are:

Sharing data must be done with care

Consider where your data is routed through or possibly even stored depending on the integration platform. Be extra careful if handling any of these:

• Company Secrets: Inside company & immediate network;
• Personal Data: Payroll integrations, social security numbers, sick leaves, bank account and credit card numbers, passwords;
• Real Money: E-invoicing, accounting, invoices, expense reports, salaries, etc.

Personal Data Act and/or Company Policies control some of the data that can be shared. For example, the Finnish social security number must not be stored to systems accessible outside the European Union. Some data is also government and trade treaty controlled.

This means that if you integrate your API with something like Zapier, and you provide access to /users, you may have to omit, encrypt or otherwise specifically protect the sensitive data even though the user may have access to use that data.

Pushing or polling

When providing an API, you must consider how your platform will be affected when thousands (or millions) of users are polling for new data every 1–15 minutes. It usually means they are pulling all records from you and then comparing with themselves if there is any new or changed data they should handle.

High usage of large data calls is resource-consuming for both ends. This is why you should seriously consider implementing RESTful hooks to your API. Instead of everyone tugging your API’s skirts with requests like “Do you have something new for me? Do you? Do you?” your platform would be in charge and only receive subscription requests to the hooks it provides.

Your platform then just gracefully pops a message to the subscribers saying, “You have a new project!”, “Your account 123 was just deleted” or “Your sales order 456 was just updated”.

Be extra careful with hook design, for both security and usability reasons. If you have reasonable-sized resources, you can deliver the whole resource as the payload of the request when the hook triggers, so the recipient does not have to make an actual request to fetch the data. The recipient might want to filter out some cases — if they are only interested in accounts with type customer, for example, they should be able to subscribe only to the triggers they want or be able to filter out the data based on the content received. However, if you serve the full content of the resource when the hook triggers, you must also implement authentication to that post, which makes things a bit more complicated than they have to be.

As an example, for the PlanMill API v1.5 it was decided to offer minimal but useful information, mostly the resource id and some filtering information the client has given when subscribing to the hook. This way a client can ask (using authentication) more information about the resource if they need to. This also solved a specific issue of handling asynchronous processes, which would change the data milliseconds after the initial save operation. In an ERP system, these are often needed for pricing and consolidation logics, because heavy calculation operations are usually involved when updating data for invoicing, project planning, time reporting, and others.

Example of a hook subscription from PlanMill API:

 /hooks
   {
       id: 2836851,
       hook: "timereport.delete",
       url: "https://requestb.in/1cebrdl1",
       eventUser: 123,
       eventProject: 456
    }

This works really well with services like Zapier, Google Cloud Messaging, and Apple and Microsoft’s messaging. Hooks are subscribed by the client and hooks trigger when a suitable event happens. When designing your REST hooks, a good resource is the “Stop That Polling Madness” manifest by Zapier

Authentication and Documentation Dress Up our API for the Party

Most APIs developed in the early days of APIs were using a simple API key for authentication, and XML and a version of JSON as a return data format.

As API tools and libraries have developed and security has become a bigger concern, OAuth2 and OpenID Connect have gained more support. Most tools like Postman, SOAPui, Zapier, and others offer a plug-and-play experience.

Finally, you should consider how to document your API. Consider both the developer user of your API as well as how easily your own team can maintain the documentation. For example, PlanMill API is documented using RAML and generated to HTML using Raml2Html. There are other methods, such as hosting the documentation on GitHub, which then acts as the developer hub for issues and questions surrounding the API.

Is your API ready to inherit the Kingdom?

When your users start to adopt your API or when you decide to plug it into an integration platform, consider carefully what your most interesting endpoints and functionalities are. Should you instantly offer everything through your API? It’s more likely best to start lean with a couple of endpoints, see how they are used in the wild, and implement additional features or changes requested by your customers, partners, or in-house UI and integration developers. An iterative development mindset helps decrease large revisions, minimizing wasted time and resources. You also receive results and feedback faster.

Be prepared though — the initial cost of “API-fying” an existing system could be quite large. You may need to redesign key parts of your system, like authentication, the access to your business objects, and the data formats you offer. At least you must put up a layer in front of your existing core, to offer these services in an API-friendly way.

At the risk of sounding too feminine for the average API-engineer out there, consider the process of API-fying your existing system as putting on a coat of make-up (authentication, JSON, data formats, resources) and a beautiful crown on your system: this is what you will need to make your API a crown princess ready to inherit the kingdom.

Remember, Cinderella didn’t do it all alone. She had a fairy godmother, and magical forest friends by her side… attend the API Stack Conference on April 12th to meet and learn from others to improve your overall API strategy!