Large organizations often talk about “beginning their API journey” as if they don’t yet have APIs. In a narrow sense of the word this may be true if we look at APIs as something that is (a) designed for reuse, and (b) that is designed as a networked interface (two fundamental aspects of APIs that I point out in a video about what an API is).

But we can also view APIs in a more pragmatic sense: What are they used for? Speaking broadly, APIs are used for interconnecting components to accomplish tasks or implement processes that are spanning more than one individual component. Looking at APIs form this angle, you can identify many “proto-APIs” that may not use the technology and standards we have today but were created in the same spirit of interconnecting components. These proto-APIs accomplish similar business goals and span more than one component. Typical examples for these proto-APIs include:

• File Transfer: File transfer is the oldest existing way of making systems exchange data. It is still widely used, even though nowadays it is often wrapped in sophisticated managed file transfer solutions that provide security and management capabilities that go far beyond manually moving files between components.
• Data Integration: One step up from file transfer are integration activities where two components are enabled to communicate without the file detour, but with the help of an intermediary. The enterprise version often is referred to as Extract, Transform, Load (ETL). ETL accomplishes interconnections by having one component export the data, massaging it so that another component can read it, and sending it to another component to understand. This is an asynchronous process, and it is a pattern that is still widely used to interconnect components.
• Application Integration: In this case, components communicate synchronously, but still with the help of an intermediary that provides the connectivity on both ends. There are many names for these intermediaries; middleware and Enterprise Service Bus (ESB) are two popular ones. In the end, these intermediaries are not all that different from ETL, but instead of doing it in batch mode, they support synchronous interactions.

APIs are a bit different from these approaches because instead of focusing on 1:1 connections (asynchronous in the file transfer scenario and synchronous in the application integration scenario), they focus on 1:n connections. APIs also assume that components are built to communicate with other components, eliminating the need for traditional methods.

But this article is not about explaining APIs or their advantages. It is about showing you a good path for your API journey. The important observation here is that you already have a landscape of communicating components — it may just be the case that those components are not using the most modern way to communicate. We can call these existing communication paths “accidental APIs,” because somebody did create a communication channel, but just not in the best possible way (given what we know today about the advantages of APIs).

You might think that discovering accidental APIs is an interesting thought experiment, but not all that consequential. However, analyzing your communication systems in this way brings many benefits. It shows how the value is currently flowing in your organization. It reveals bottlenecks and potential ways to remove them. By engaging co-conspirators to streamline infrastructure, your organization becomes more proficient as a whole. In the end, these actions will gradually unlock more and more value that is currently trapped within silos in your organization.

Starting Your API Journey

Before we go into the details: This article encourages you to finally get started with your API journey if you have not started yet. It encourages you to look at all the things that are already in place and keep your organization running. But it also encourages you to take a critical look at what may be holding you back and how you can address them. Digital doesn’t have to be disruptive, but it does require the willingness to change and to understand the biggest obstacles that are holding you back. You must start small but think big. A future API investment will likely be driven by looking at your current practices and the opportunities that you identify.

It is also important to stress that APIs are an essential component of your digital journey, but they are not the only important thing. APIs will not solve all your problems, even though they are a necessary aspect of solving many of them. You still must take a step back to view business and organizational issues, and without fixing those, you cannot expect your API journey to deliver the value that you expect from it. With that said, let’s dive into API archaeology!

API Archaeology: Archaeology is the practice of unearthing artifacts and understanding them in the context of their origins in time and location. That exact concept can be applied to APIs as well. API archaeology thus is the practice of finding integrations, understanding why and how they were created, and documenting them as a way to better understand the history and structure of complex IT systems. Practicing API archaeology in organizations can be extremely valuable in terms of finding out about existing ways in which IT components interact (in some way or form).

Analyze the Flow of Value

The main idea behind API archaeology (as introduced in Continuous API Management) is that by analyzing existing interconnections, you get an excellent picture of where value is flowing in the organization. Each existing proto-API is an indication that the interconnection was deemed valuable enough to invest in 1:1 integration efforts. Of course, it may not be clear whether that perceived value was ever realized. Nonetheless, understanding its history is a powerful starting point to understand the current status, and to decide on future investments for the modernization of the organizational landscape.

Before you start putting values on existing interconnections, it is usually a good idea to create some form of unbiased catalog that serves as a starting point. There is no need to be overly obsessed with this catalog being complete: It could potentially take a very long time to find all of the existing interconnections. On the other hand, many organizations have some form of this already, so maybe you can use this as a starting point instead of having to do a lot of digging yourself.

Identify Bottlenecks and Potential

Once you have an understanding of value flows throughout the organization (and with its partners), it is now time to start assessing how well things are working. There are two main things to look out for when it comes to identifying potential bottlenecks:

• • Integration Calcification: Many “old school integrations” were one-off projects that connected components. These integrations then became calcified. Due to their custom construction, it is tough to make any changes on either side, meaning that both the provider and the consumer have a hard time making any changes because they fear breaking things. In system design, this is called tight coupling because components are coupled through many different facets of their interconnection. APIs have a strong culture of loose coupling, making it easier to change components without breaking existing interconnections.

• Potential for Reuse: Some existing interconnections may have an obvious potential for reuse. For example, if you have data about products, customers, and purchase histories, this may currently feed into product catalogs and accounting systems. If that information was accessible more openly, you might be able to feed it into analysis and machine learning components to understand your customers and their purchases better. The more easily this kind of information can be consumed, the more easily you can build these new interconnections. Reuse makes it easier to improve value chains and rid costly one-off integration projects.

The purpose of this step is to allow you to identify the potential of API investments. In many large and old organizations, it is not realistic to move all existing interconnections to API-based approaches. It is thus vital to understand which ones have the most potential to unlock new value when you progress them to APIs. However, like any modernization effort, API journeys are an art. Since, by definition, all interconnections involve more than one component, the next consideration is how to get teams on board.

Engage Co-Conspirators

The value proposition of APIs is fundamentally different from traditional integration approaches. In conventional integration approaches, organizations create value by interconnecting components, and both the interconnection and the realized value are known because it is a closed system.

For APIs, the value is an open-ended proposition. Though APIs may be designed for a small initial consumer base, ideally, they are useful and available to other consumers as well. This makes the value proposition and calculation a bit more complicated. Thus, it is always better to build an MVP for a subset of initial consumers so that you don’t have to “design into the void.”

This general observation of how to identify and build valuable APIs is true for our API archaeology scenario as well. In these scenarios, it is important to locate consumers willing and able to move their interconnections to APIs. When identifying these consumers and including them in your design process, also make sure to have an eye on your API guidelines. After all, you want these new APIs to be in line with the API landscape that you are managing, meaning that coherence at the landscape level should become part of the design process.

Each pre-existing integration that you can retire because of an API is a victory. It means that you have realized additional value along three axes:

• Your ability to change: By moving to an API, integration is now based on a well-defined contract and not on a specific implementation. This means that you can make changes to the API provider side, as long as you still adhere to the contract. You can even completely switch out the implementation if that’s what you want to do.
• Your consumer’s ability to change: By switching to loose coupling, your consumer has made the same change as you: interconnection is now based on a contract, not on implementation. They can make changes if they need to, and you might not even know about this, and you don’t have to be involved.
• Enabling other consumers: Since there now is a contract for the interconnection, others can use it as well. Of course, there have to be security mechanisms and access controls at the technical level of securing the API. But if there is a business reason to allow others access to the API, nothing on the provider side needs to be changed and then can start consuming the API.

If you look at these advantages carefully, you will notice they revolve around the contract that the API represents. This also is a strong indication that this contract needs to be well-designed. Do not fall into the trap of merely building an API that directly represents what was exchanged in the old interconnection method. Instead, go through a proper design process that allows you to identify what the value is, and how best to represent that value in a well-designed and extensible API.

Release, Realize, Respond, Repeat

As mentioned in the last section, it’s a good idea to identify some initial candidates with a good value proposition and willing co-conspirators. This approach allows you to get started quickly, to learn along the way, and to improve your effectiveness.

• Release the API as soon as possible so that you can test operations and learn how well it works for the consumer. Consider ways to improve the developer experience, and how API tooling and an API platform may support.
• Realize the value of the API by making sure it is managed effectively and discoverable by those who want to use it.
• Respond to developer feedback about specific API behavior, as well as functional gaps. This feedback will help improve the APIs or re-prioritize the APIs you build. As APIs should support value chains, attentive listening is critical to improving the allocation of resources.
• Repeat this process by gradually covering more interconnections and moving them to APIs. Like any modernization effort, this will take a while, but in this case, it is crucial to get started as soon as possible so that you can learn from your own experiences and the feedback of others.

Using this iterative approach, you can make sure that your modernization starts more quickly, self-adjusts as it is moving along, and is driven by feedback that allows you to make the most valuable investments. It is unlikely that APIs will ultimately replace all the interconnections that you have, but that should not be the goal, to begin with anyway. The purpose of this approach is to unlock value that is currently not realized in your organizational landscape — that really should be the primary measurement of success.

Conclusions

API archaeology is not something fundamentally new, but it follows patterns and experiences we have seen in our work with many large organizations. With enough digging, every organization today turns into a complex landscape of proto-APIs. Understanding this landscape is essential for directing investments when it comes to getting started with the API journey.

One of the mistakes we see is that organizations often approach their API journey in a “waterfall,” making far-reaching plans and investments for their API landscape. Though the result is a fundamental IT shift, the API journey itself should be treated as an evolutionary process that slowly but surely changes the organizational approach and mindset when it comes to creating and managing value chains.

With all of this being said, please keep in mind that API archaeology is just one wrench in the toolbox of the digital transformation practitioner. As mentioned above, business and organizational challenges must be addressed, as well. And like anything starting from the past, while it is helpful to develop a sense of how to improve the current set of interconnecting components, do not forget that there may be equally large potential in enabling entirely new value chains that aren’t reflected in the current organizational landscape.