One of the main balancing games in the API space is the balance between efficiency and privacy. The most efficient system in the world exposes everything, minimizing the level of exchange and conversion that is necessary in a system. The most private system in the world has everything behind lock and key, which introduces substantial inefficiency at scale, slowing the system and ultimately resulting in a costly effort for minimal work done.

Balancing these two attributes is a complex question. Below, we’ll discuss one such solution in the world of authorization: the Phantom Token Approach. This powerful pattern unlocks great efficiency — but how does it work, exactly, and where did it come from?

What is the Phantom Token Approach?

Phantom Tokens are a relatively new approach to tokens offered within the OAuth security flow. They’re designed to be more privacy-preserving. In essence, Phantom Tokens are meant to combine the benefits of opaque tokens and JWTs, two different token types in OAuth. Therefore, let’s first define the pros and cons of these other token types to understand why Phantom Tokens are so necessary.

Opaque Tokens

Opaque tokens are a type of token represented as a random string. Because this string is random, no meaning can be extrapolated by any other system or user — this lack of context means that its purpose is entirely opaque. While a certain amount of metadata is connected to the token, this information is not contained within the token itself. It is thus meta-contextual information that can only be gleaned through authorization server introspection. During the process of introspection, information can be returned through an internal endpoint, which provides more context about the token and its data.

While this creates a level of privacy, it does introduce cost. This kind of token is often referred to as passing a token by reference, and this referencing carries its own cost to the authorization server and, by means of the back and forth in such a flow, the infrastructure itself.

• Privacy: Private; context is obfuscated and only discoverable via contextual introspection.
• Efficiency: Inefficient; context is only gleaned through contextual conversation with authorization servers.

JWT Tokens

JSON Web Tokens (JWTs) are designed to get around this additional load and limitation of context. A JWT (often said aloud as “jot”) is compact and highly efficient. A JWT contains resources and contextual data that can be used by the system to make an authorization decision. Notably, this typically includes a lot more contextual data, meaning that the cross-talk with authorization servers is significantly lower than in the opaque token type.

This introduces some issues, however. While JWTs are much more efficient, this efficiency comes through bundled data. While JWTs can be encrypted, it still contains the data in question, resulting in a lower level of privacy in theory. JWT tokens are often referred to as passing a token by value, with the value being the meta-contextual information that otherwise was obfuscated in the opaque system. While there are mechanisms that protect the integrity of the data — notably the signing of content that prevents tampering —these only protect the integrity of the data rather than the data itself.

• Privacy: Not private; context is included within the token itself, resulting in surfacing that undermines security.
• Efficiency: Efficient; context is included, meaning communication can be limited by including context in the token itself.

The Phantom Token Paradigm

The Phantom Token is thus a hybrid approach that attempts to provide the efficiency of a JWT with the privacy of an opaque token. The Phantom Token makes some assumptions based on the typical structure of an OAuth flow within an API. Principally, it assumes that there is a gateway or reverse proxy as a middleware within the authorization flow, a design that is all but required within most OAuth instances. By using this intermediary to do some conversion, we can make for a more private pattern.

This works by pairing opaque and JWT tokens. The opaque token is the token issued to the end-user or public application — this is the external-facing token and is the one carried by the application, whatever form that application may take. When making a request, the application passes this opaque token into the system. A middleware solution, say an API Gateway, then takes this token and utilizes an internal introspection endpoint to convert it to a JWT token. This JWT token is then used in its place internally and is passed upstream.

This is where the term Phantom Token comes from. To the end user, the only known token is the Opaque one, and without the contextual information it provides, it might as well be any random number. Internally, however, the quick exchange into a JWT token makes for a kind of Phantom Token that the end user is unaware of but nonetheless carries the context and data necessary to make more efficient requests internally.

Benefits of Using a Phantom Token

The Phantom Token may make some assumptions that are somewhat restrictive to the end user. Notably, it requires some sort of middleware. But in exchange, it unlocks significant benefits. Phantom Tokens combine the best of Opaque Tokens and JWTs by the nature of the fact that they use both. By combining these systems, the ultimate exchange flow results in a system that grants the privacy of the Opaque token with the efficiency to internal services of the JWT token. This efficiency is incredibly important to the overall flow requirements and can fundamentally change the nature of the API authorization flow by orders of magnitude at scale.

Notably, the adoption of Phantom Tokens also results in improved consistency across services. Since JWTs expose data and context that can be used to formalize the structure of data across services within a larger system, the use internally of JWTs is significant, allowing for efficiency and scalable interactions that are predictable. Additional data can always be added to the JWT internally without exposing this data to the external user, promoting extensibility without necessarily introducing issues to the end user.

Ultimately, Phantom Tokens are the best of both worlds, and if your system is designed in a way that works with the pattern, it can be an incredibly powerful solution that boosts efficiency, scalability, and extensibility.

Conclusion

What do you think of the Phantom Token approach? Are there other patterns that should be discussed within this context? Let us know below!