Everyone’s excited about microservices, but actual implementation is sparse. Perhaps the reason is that people are unclear on how these services talk to one another; especially tricky is properly maintaining identity and access management throughout a sea of independent services.

Unlike a traditional monolithic structure that may have a single security portal, microservices pose many problems. Should each service have it’s own independent security firewall? How should identity be distributed between microservices and throughout my entire system? What is the most efficient method for the exchange of user data?

There are smart techniques that leverage common technologies to not only authorize but perform delegation across your entire system. In this article we’ll identify how to implement OAuth and OpenID Connect flows using JSON Web Tokens to achieve the end goal of creating a distributed authentication mechanism for microservices — a process of managing identity where everything is self-contained, standardized, secure, and best of all — easy to replicate.

What Are Microservices, Again?


Microservice Architecture

For those readers not well-versed in the web discussion trends of late, the microservice design approach is a way to architect web service suites into independent specialized components. These components are made to satisfy a very targeted function, and are fully independent, deployed as separate environments. The ability to recompile individual units means that development and scaling can be vastly easier within a system using microservices.



Monolithic design

This architecture is opposed to the traditional monolithic approach that consolidates all web components into a single system. The downside of a monolithic design is that version control cycles are arduous, and scalability is slow. The entire system must be continuously deployed since it’s packaged together.

The move toward microservices could have dramatic repercussions across the industry, allowing SaaS organizations to deploy many small services no longer dependent on large system overhauls, easing development, and on the user-facing side allowing easy pick-and-choose portals for users to personalize services to their individual needs.

The Good, Bad, and What You Should Be Doing Better: Read Our Post on Microservice Best Practices

Great, So What’s The Problem?


Simplified monolithic flow

The problem we’re faced with is that microservices don’t lend themselves to the traditional mode of identity control. In a monolithic system security works simply as follows:

  1. Figure out who the caller is
  2. Pass on credentials to other components when called
  3. Store user information in a data repository

Since components are conjoined within this structure, they may share a single security firewall. They share the state of the user as they receive it, and may also share access to the same user data repository.


The problem with microservice security

If the same technique were to be applied to individual microservices, it would be grossly inefficient. Having an independent security barrier — or request handler — for each service to authenticate identity is unnecessary. This would involve calling an Authentication Service to populate the object to handle the request and respond in every single instance.



The Solution: OAuth As A Delegation Protocol

There is a method that allows one to combine the benefits of isolated deployment with the ease of a federated identity. Jacob Ideskog of Twobo Technologies believes that to accomplish this OAuth should be interpreted not as Authentication, and not as Authorization, but as Delegation.

In the real world, delegation is where you delegate someone to do something for you. In the web realm, the underlying message is there, yet it also means having the ability to offer, accept, or deny the exchange of data. Considering OAuth as a Delegation protocol can assist in the creation of scalable microservices or APIs.

To understand this process we’ll first lay out a standard OAuth flow for a simple use case. Assume we need to access a user’s email account for a simple app that organizes a user’s email — perhaps to send SMS messages as notifications. OAuth has the following four main actors:

  • Resource Owner (RO): the user
  • Client: the web or mobile app
  • Authorization Service (AS): OAuth 2.0 server
  • Resource Server (RS): where the actual service is stored

A Simplified Example of an OAuth 2 Flow

In our situation, the app (the Client), needs to access the email account (the Resource Server) to collect emails before it can organize them to create the notification system. In a simplified OAuth flow, an approval process would be as follows:

  1. The Client requests access to the Resource Server by calling the Authorization Server.
  2. The Authorization Server redirects to allow the user to authenticate, which is usually performed within a browser. This is essentially signing into an authorization server, not the app.
  3. The Authorization Server then validates the user credentials and provides an Access Token to client, which can be use to call the Resource Server
  4. The Client then sends the Token to the Resource Server
  5. The Resource Server asks the Authorization Server if the token is valid.
  6. The Authorization Server validates the Token, returning relevant information to the Resource Server i.e. time till token expiration, who the token belongs too.
  7. The Resource Server then provides data to the Client. In our case, the requested emails are unbarred and delivered to the client.

An important factor to note within this flow is that the Client — our email notification app — knows nothing about the user at this stage. The token that was sent to the client was completely opaque — only a string of random characters. Though this is a secure exchange, the token data is itself useless to the client. The exchange thus supplies access for the client, but not user information. What if our app needed to customize the User Experience (UX) based on which membership level the user belonged to, a group they were a member of, where they were located, their preferred language, etc.? Many apps provide this type of experience and for that they require additional user information.

Learn How APIs Are Disrupting The Way We Think

The OpenID Connect Flow

Let’s assume that we’re enhancing the email service client so that it not only organizes your emails, but also stores them and translates them into another language. In this case, the client will want to retrieve additional user data and store it in it’s own user sessions.

To give the client something other than the opaque token provided in the OAuth flow, use an alternative flow defined in OpenID Connect. In this process, the Authorization Server, which is also called an OpenID Connect Provider (OP), returns an ID Token along with the Access Token to the client. The flow is as follows:

  1. The Client requests access to the Resource Server by calling the Open ID Connect enabled Authorization Server.
  2. The Authorization Server redirects to allow the user to authenticate.
  3. The Authorization Server then validates the user credentials and provides an Access Token AND an ID Token to the client.
  4. The Client uses this ID Token to enhance the UX and typically stores the user data in it’s own session.
  5. The Client then sends the Access Token to the Resource Server
  6. The Resource Server responds, delivering the data (the emails) to the Client.

SSO_with_OpenIdConnect_microservicesThe ID token contains information about the user, such as how they authenticated, the name, email, and any number of custom data points on a user. This ID token takes the form of a JSON Web Token (JWT), which is a coded and signed compilation of JSON documents. The document includes a header, body, and a signature appended to the message. Data + Signature = JWT.

Using a JWT, you can access the public part of a certificate, validate the signature, and understand that this authentication session was issued — verifying that the user has been authenticated. An important facet of this approach is that ID tokens establish trust between the Authorization Server/Open ID Connect Provider and the Client.

Using JWT For OAuth Access Tokens

Even if we don’t use OpenID Connect, JWTs can be used for many things. A system can standardize by using JWTs to pass user data among individual services. Let’s review the types of OAuth access tokens to see how to smartly implement secure identity control within microservice architecture.

By Reference: Standard Access Token

user_valueThis type of token contains no information outside of the network, simply pointing to a space where information is located. This opaque string means nothing to user, and as it is randomized cannot easily be decrypted. This is the standard form of an access token — without extraneous content, simply used for a client to gain access to data.


By Value: JSON Web Token

user_referenceThis type may contain necessary user information that the client requires. The data is compiled, and inserted into the message as an access token. This is an efficient method because it erases the need to call again for additional information. If exposed over the web, a downside is that this public user information can be read easily read, exposing the data to an unnecessary risk of decryption attempts to crack codes.

The Workaround : External vs. Internal

To limit this risk of exposure, Iskedog recommends splitting the way the tokens are used. What is usually done is as follows:

  1. The Reference Token is issued by the Authorization Server. The client sends back when it’s time to call the API.
  2. In the middle: The Authorization Server validates the token and responds with a JWT.
  3. The JWT is then passed further along in the network.

In the middle we essentially create a firewall, an Authorization Server that acts as a token translation point for the API. The Authorization server will translate the token, either for a simple Reverse Proxy, or a full scale API Firewall. The Authorization Server shouldn’t be in the “traffic path” however — the reverse proxy finds the token and calls the Authorization server to translate it.

Let All Microservices Consume JWT

So, to refresh, with microservice security we have two problems:

  • We need to identify the user multiple times: We’ve shown how to leave authentication to OAuth and the OpenID Connect server, so that microservices successfully provide access given someone has the right to use the data.
  • We have to create and store user sessions: JWTs contain the necessary information to help in storing user sessions. If each service can understand a JSON web token, then you have distributed your identity mechanism, allowing you to transport identity throughout your system.

In microservice architecture, an access token should not be treated as a request object, but rather as an identity object. As the process outlined above requires translation, JWTs should be translated by a front-facing stateless proxy, used to take a reference token and convert it into a value token to then be distributed throughout the network.


Why Do This?

By using OAuth with OpenID Connect, and by creating a standards based architecture that universally accepts JWTs, the end result is a distributed identity mechanism that is self contained and easily to replicate. Constructing a library that understands JWT is a very simple task. In this environment, access as well as user data is secured. Creating microservices that communicate well and securely access user information can greatly increase agility of the whole system, as well as increase the quality of the end user experience.

Check Out This Talk For More:

Watch Jacob Ideskog of Twobo Technologies talk at a Nordic APIs event. Slides here.

This graphics used in this post are owned by Twobo Technologies.

Bill Doerrfeld

About Bill Doerrfeld

Bill Conrad Doerrfeld is an API specialist, focusing on API economy research and marketing strategy for developer programs. He is the Editor in Chief for Nordic APIs, and formerly Directory Manager & Associate Editor at ProgrammableWeb. Follow him on Twitter, visit his personal website, or reach out via email.

  • Martin Streicher

    In your OpenID flow, item #5 uses the word “this”, which is a little ambiguous. I assume “this” refers to the access token since you are talking to the email server? Thanks for clarifying.

    • travisspencer

      Clarified in the text to reflect the basic rule: the ID Token is for the client and the Access Token is for the Resource Server. Let us know if the text still isn’t clear, and thanks for pointing out the ambiguity.

  • Martin Streicher

    Many OAuth servers return user data after authentication, depending on the scope parameter and what the user approves during authentication. Is your ID token a special version of that?

    • travisspencer

      Many OAuth servers do many funky things, so it’s hard to say exactly. The ID Token we’re talking about here is a standardization of what many OAuth implementations were doing before OpenID Connect was ratified. So, it could be similar to things you’ve come across when integrating with various services. This kind of need for one-off integrations was one of the primary drivers that led the OpenID Foundation to standardize an identity layer atop OAuth.

  • Alon Lavi

    Great article and talk, thanks! I have a small concert, though – Isn’t it a security issue that we share the same secret (for validating the JWT token) among the all micro services?

    • travisspencer

      Validation of the JWTs is done using the public key of the issuer. The micro-services don’t have a shared secret or a symmetric key. They only have a copy of the OAuth server’s public/asymmetric key. They can even obtain this dynamically as they start up using standards like JWKS. This is how trust is created in the ecosystem, a topic that was touched on a bit at the end of talk IIRC.


      • Alon Lavi

        Thanks for the answer!

  • Steven Webb

    Thanks for the great article. Could you please provide some more information about how to set up the firewall/reverse proxy? Would you use something like this https://github.com/pingidentity/lua-resty-openidc or do I need to build my own?

    • travisspencer

      For simple deployments or ones where NGINX and/or PingFederate are already being used, that is a good one. (We’ve used it in one customer deployment @2botech. We had it up in a few minutes and it uses the standard introspect endpoint of an OAuth server.) When things start to get more complicated, you’ll probably need something else though. At that point, I’d suggest using a full-blow API security gateway, like Layer 7 (who’s a partner BTW of @2botech’s so you know). If you are gonna code it yourself though, LUA in NGIX is probably gonna be very fast (or C in Apache if that’s your high-perf lang or Go if LUA and C aren’t your thing). I would cation you from heading down the DIY path though as these integrations can get hairy.

      No matter what you use — DIY, NGINX, Layer 7, etc. — the things you need to do are:

      1. Check if the incoming request has an Authorize header with a type equal to Bearer (e.g., Authorize: Bearer XYZ)
      2. Check if a by-val token is cached using the by-ref token XYZ as they cache key. If so, forward the by-val token from the cache to the back-end API (i.e., the actual API)
      3. If the by-ref token XYZ isn’t yet cached, call the OAuth server’s introspect endpoint (or equivalent if your OAuth server doesn’t support RFC 7662). The response should include an equivalent by-val token that should be used on the internal network that’s behind the API gateway (call this JWT)
      4. Store JWT in the gateway’s cache using XYZ as the cache key to avoid subsequent calls to the OAuth server for the lifetime of the by-ref XYZ token
      5. Call the back-end API with the by-val JWT

      Then, the microservices behind the gateway can validate the JWT using asymmetric crypto and have all of the identity data they need without ever contacting the OAuth server, as Jacob discussed in the video.


      • Steven Webb

        Thanks for the thorough answer!

      • Sam Hass

        Great Article. I’m planning to do a college project about the “firewall/reverse proxy” that you mentioned here. Could you please give me some more good points that I should consider?

        • travisspencer

          There are three classes of proxies that come into play with APIs:

          1. Network firewall — This type of intermediary does port forwarding/mapping, layer 3 defenses (like how many requests has this client IP made in the last X seconds), etc. These often do network load balancing as well (i.e., measure the size of TCP packets to determine which back-end to forward the call to). Examples I see a lot are F5, Bluecoat, and Cisco.

          2. Reverse proxy — An intermediary that’s intention is to obscure or hide services that are running in a protected network. A reverse proxy is typically placed in a DMZ and used to pass restricted (usually authenticated) calls to the “green zone” (i.e., a corporate network). Common examples are NGINX and Apache.

          3. API Security Service — An API security service usually is a reverse proxy in that it brokers from a DMZ into a protected network where APIs are deployed. Using OAuth like @disqus_ky9OCATU3q:disqus and I were discussing above, the API Security Service can ensure that only authenticated calls can pass into the green zone where microservices/APIs are deployed.

          In addition to token translation discussed above, an API Security Service should provide the following capabilities:

          * DoS protection (at layer 7 and maybe even layer 3 if the network firewall isn’t doing this, which it should)

          * SQL, XML, JSON, LDAP, etc. injection protection

          * Require and control secure HTTPS communication with clients. Ideally, it should be possible to control which trust anchors are allowed and if mutual TLS is required.

          * Routing and exposing of many different virtual endpoints/APIs. You should be able to split apart various back-ends and expose them in different ways.

          * Mapping of content (e.g., from REST to SOAP). This mapping can be *very* complicated in some cases. In these situations, it’s better to do this in code that’s source controlled, unit/integration testable, and not in an API gateway’s policy editor.

          * Caching.. The API Security Service should provide the ability to cache tokens and other content in a local or cluster-wide store.

          Typical examples that I see in the field for API Security Services are Layer 7, Apigee, SOA Software, etc.

          BTW, notice how I didn’t say anything about a dev portal. All the API gateway vendors have one, but the gateway needs to verify tokens not manage clients. Client management is needed so that tokens can be issued by an OAuth server. So, the dev portal is more tightly coupled to the OAuth server and there actually shouldn’t be any direct communication between the dev portal and the API security gateway.


          • Sam Hass

            Thanks for the answer, I extended my project to “identity distribution and session handling in microservices”. Your answer will be very helpful.

      • Satish K.

        Thanks for the article and the incredibly useful discussion. Any thoughts (or links to other resources) for 1) client-side storage of the by-ref token for SPAs and 2) protecting against replay and CSRF attacks when using the pattern described above? I was considering implementing the synchronizer token pattern but I’m unclear whether the gateway should be issuing and verifying the CSRF token or whether that’s best left to the individual services.

        • travisspencer

          I would only recommend storing the token in a cookie. I would consider using the code flow with the the SPA’s back-end. That way you can authenticate the client and that can store the token in an encrypted cookie or hash it and store the result in plaintext. This will give you help protecting it because of the browser’s single origin policy. If you don’t have a back-end at all for your SPA, I don’t have a good answer for you.

          Ref tokens don’t tend to be nonces, so replay isn’t an issue with them. To get a ref from an Access Code, which is a nonce (e.g., if you use the code flow with the SPA’s back-end), the OAuth server will take care of replay protection. It’ll do this by “remembering” which codes have been redeemed and which have not. So, the app shouldn’t need to be concerned with replay.

          CSRF protection will be very good to have if you end up storing the Access Token in a cookie. Ideally, you would tie the cookie’s life to the SPA. That way, if the user surfed away from the page, the cookie would become unusable. You want to ensure that if requests made by an attacker’s site would cause the cookie to flow to the SPA’s back-end, then that back-end has a way to determine that the request is invalid (this is essentially the synchronizer pattern you mentioned). To this end, I would suggest that:

          1. You make sure the cookie is a session cookie (i.e., it is one that’s deleted when the user closes their browser)

          2. On the SPA’s back-end that gets the Access Code (effectively this is the real OAuth client), you hash the resulting token. Store the actual token, the unhashed value, in a session DB. Sign the hash and set the hash and signature in the cookie in the response to the SPA. Also, in this reply, include this hash outside of the cookie. Save this hash in a global, page-wide JavaScript variable. When you make API requests, include the cookie and the hash. The hash could be in the header as a bearer token (e.g., Authorization: Bearer myGoodHash). On the server, it’ll see that the cookie is present, that the signature of the value matches the one in the cookie, and that the hash matches the one in the HTTP request header. Given all of this, it looks up the *real* Access Token associated with that hash in the session DB and the API call is made.

          So, where does all this logic of token validation end up? I would *never* put this into an API proxy/gateway unless I can write it in code and run that code in the gateway. This is complicated stuff, and I wouldn’t want to maintain it in some graphical, drag-and-drop UI. (I’m a programmer though, so I’m a little biased. Even without that bias though, it just makes sense to code this that can be tested easily, version controlled, commented, backed-up, etc.)

          So where? The SPA’s back-end. That does this validation and forwarding just like it’s doing Access Code to Access Token conversion, hashing, setting of cookies, etc. If you do this, the whole gateway vs. individual services question isn’t even in play.


      • Adrian Ivan

        Hi Travis,

        Q1. Regarding how the Microservices validate the JWT.
        How would the Microservice get the AS Public Key? OpenID Connect mentions the Discovery Endpoint and this also solves the problem of rotating keys. Should the AS be an OpenID Connect Provider (OP)? Are there any other means to achieve the same?

        Q2. My understanding is that you also talk about a segregation of the security domains, the Reference token is used in the wild and the JWT inside the Microservices System Perimeter.
        Should the same AS/OP be used? the API Gateway exchanges the Reference for a JWT using the AS/OP and sends it to the Microservice. It means that the Microservice is in the same security domain as the AS/OP, API Gateway since it can validate/read the same JWT.
        What about the API Gateway acting as an AS in the Microservices Perimeter? It would over-sign the JWT and the Microservices would use the API Gateway Public Key to validate (not the original AS/OP key).

        • travisspencer

          Q1: Using JSON Web Key (RFC 7517), PKI (e.g., distributing the public key via AD), or by static configuration is typically how it’s done.

          Q2: I would use the same AS/OP for both security domains. I would not necessarily expose all of the AS’s endpoints to both domains, however. I would put the microservices in the green zone behind the API Gateway that is a proxy in the DMZ. I would *never* use an API Gateway in the AS role because gateways are designed to validate tokens. Most API Gateway vendors provide an OAuth capability, but I’d be very leery of those; issuing a token is a very different job than validating one, and in my experience a purpose built app for issuing tokens always works out better.

          • Adrian Ivan

            Thank you

          • dPendPickels

            I had this exact picture in my mind. So, it’s Ok to have the AS in the green zone, and expose just a couple of methods to the API Gateway in the DMZ?. This is assuming that the gateway will communicate to the AS to do auth tasks, convert tokens from opaque to JWT, etc.
            Later, the microservices receives just the JWT tokens and trusts it, because of the delegation process. Is this right?. The internal services can talk to the AS if needed, too.
            And finally, what measures can be taken to tightly secure the link between the API Gateway and the AS (because those are requests that technically hit the internal network without been totally authenticated)?. Filter IP traffic?, use an IPS/IDS?.
            Thanks, incredible article and discussion!.

          • travisspencer

            So, it’s Ok to have the AS in the green zone, and expose just a couple of methods to the API Gateway in the DMZ?

            Yes. You should secure this call by:

            * Using mutual TLS with a client cert issued to the gateway or do one-way TLS to authenticate the AS to the gateway with an SSL cert and basic auth with a long random password to authenticate the gateway to the AS.

            * Set up a skimped-down AS on some port and listening host that only does introspection (and maybe token issuance). Set up a firewall rule, so that this skimped down AS can only be accessed by the source IP of the gateway (and maybe certain server-side apps that need introspect and/or token). Don’t run introspect on any other instances of the AS. Have this cluster in the greenzone; the gateway will be authenticated before calling it, and the user will be authenticated and have a token, so you have non-repudiation of both actors.

            To split the AS up like this, you should put the authorize and authentication endpoints on a different AS cluster that doesn’t expose introspection; that node also shouldn’t have a token endpoint and should be in the DMZ. The token endpoint should be in the AS cluster that’s in the greenzone (or a 3rd if you need further segregation). I don’t know of any product that supports this dynamic nature of endpoints except Curity (which Nordic APIs is affiliated with BTW). That server was built to be deployed in just this way.

            Later, the microservices receives just the JWT tokens and trusts it, because of the delegation process. Is this right?

            The microservice trusts the token because it has a copy of the AS’s public key. The tokens are signed with the corresponding private key. The micoservice accepts this public key and places trust in the AS because of this type of topology and deployment model. So, yes, that is right.

            The internal services can talk to the AS if needed, too.

            yes, they can. They might do this to re-audience a token, for example, before calling out to some 3rd party service provider. Again though, only expose the endpoints of the AS that are needed by the microservices. This is probably only token exchange or maybe introspect.

            And finally, what measures can be taken to tightly secure the link between the API Gateway and the AS (because those are requests that technically hit the internal network without been totally authenticated)?. Filter IP traffic?, use an IPS/IDS?.

            See above, but do certainly include an IDS on the AS. Monitor the logs by plugging it into something like ELK which is hooked up to your NOC/ops center. Mount as much of the file system as read-only. Use only TLS 1.2 everywhere except the authorize endpoint where compatibility might be an issue; use strong ciphers as well. Have physical access control to the gateway and AS. Manage them over SSH and HTTPS with valid certs only. Set up a VPN to the control network, so only a few people have access to them. Etc.

  • Jacob Ideskog

    Hi Steven,
    Sure, internal services gives more flexibility in that way. Issuing a JWT directly from the OAuth server as an Access token is a valid approach for many cases, mostly internal. The thing to be aware of with that approach is that you are at risk for “creative programmers”. The Access Token is meant for the API (the microservice) so it is supposed to decode it and deal with the content of it (like RO/subject). But what can happend when you issue the JWT to the client is that there is someone on the client side finding it super useful to use the information that exists in the Access Token (AT), since it’s then readable. This is an anitpattern, since the AT is not meant for the client, the information in it is for the API or the gateway. For internal clients this is manageable, but for external clients you then commit to an API (the information inside the JWT) that you didn’t intend to from the beginning.

    On the other hand, using a JWT as AT removes the need for central Reverse Proxies and thus make an excellent candidate for internal format of the access token.

    /Jacob Ideskog

  • Ritesh Ramesh

    Should the Resource service talk to the Auth Service (token validation and decoding) internally or eternally what are the pros and cons of doing it both ways?

    • travisspencer

      The RS can talk to the AS to validate tokens if it wants to. Typically, it better that this be done by an API gateway as we discussed in the comments below. The reason for this is that calls don’t get past the DMZ before they’re authenticated. It also allows you to use opaque (ref) tokens outside of your network (i.e., plain GUIDs for tokens) and convert them on the way in to by-val tokens (e.g., JWTs). If the RS does communicate with the AS directly and no API gateway is used, the communicate should still be done in a standard-based way, specifically RFC 7662.

  • This short article and the presentation filled in surprisingly many missing pieces for me. Thank you.

    However, I’m having trouble understanding the value of the translation layer. I can see that hiding JWTs from clients eliminates a few security concerns, but it seems to me that confirming the opaque token is sufficient. The translation layer might map the opaque token to a user object, but I don’t see why that object needs to be signed behind the firewall. Why translate to JWTs and require each of the backend services to do cryptographic work to verify them?

    I’m also trying to understand why this approach might be preferable to just providing a backend translation service for opaque tokens, one that isn’t exposed externally. Doing this might eliminate the scaling issue associated with funneling all requests through a translation layer, because the services could be configured to hit different backend nodes for translation, providing more fine-tuned control over scaling. I realize that you could load-balance a front-end translator, but superficially that seems to obviate some of the benefit of using a microservices architecture.

    Mind you, all this is new to me, so feel free to challenge my assumptions.

    • travisspencer

      In addition to the security provided by giving an opaque reference token to the client (as opposed to a JWT or other by-value token) is that it decreases coupling. We were working with a customer of @2botech’s the other day, and they too were skeptical about this whole translation layer. They went with JWTs on the outside and no translation. Without hours — literally, the front-end Web devs who were building the client were using the contents, unverified, from the API’s JWT Access Token, despite being told not to. After this, the customer updated their deployment to issue an ID Token to that client and switched to using an opaque ref token for the API. This allows the two to evolve separately.

      But why translate to a JWT and not some blob of JSON or a bunch of HTTP headers that get forwarded through the gateway? 2 reasons:

      1. Trust
      2. Complexity of the back-end ecosystem

      Behind the gateway is an ever-growing set of services. Some of these are old, some are new; some are bought, some are built; some are run by you, some are run by partners. These services can be brand-new microservices or old ESBs that provide mediation and orchestration to legacy systems. The “virtual” API exposed through the gateway seems so simple from client’s point of view, but it is actual a mess of messages internally. If this isn’t your setup, be glad, but it’s most people’s. So, in this mess, how do you know who really authenticated to the client, when, and with what sort of credential? Many of the internal services need to know this sort of thing in order to make an access control decision. They can’t simply be told these things by their caller in the request. If they did, the entire internal network becomes one big “trusted subsystem” where all services trust all other services. In this type of setup, trust is lost if even one of these services is popped.

      By instead translating from an opaque token to a JWT that’s used internally, you have trust. Only one entity (that may be comprised of a 1000 different instances, each with different endpoints and configuration, mind you), can issue signed tokens that the internal services will trust. Each internal service has outsourced these important questions of identity — not to the caller(!) but to the identity server. In this type of setup, the internal API isn’t breached by simply popping its caller. Instead, the OAuth server has to be compromised in order to exploit the internal microservices.

      This also touches on your second point as well, one of providing a back-end translation service that is exposed externally. This is in fact the recommended way, but that API should conform to RFC 7662 and be provided by the OAuth server. The reason not to have have various translators is again one of trust. You want a single source of authority for asserting identity within the security domain. Having said that, the types of deployments that we do can be massive and in those it is important that the OAuth server not be a monolith. You can have different token endpoints with different clients that can only communicate with the OAuth server over various subnets. The key material, that private key that signs the JWTs is the way that trust is established. This key may be different for different parts of the back-end security domain. So, scalability is important, but I’d recommend handling it in the OAuth server and not by spreading out token issuance capabilities to different actors throughout the ecosystem.

      Feel free to ask more questions, Joe, or push back. Either these ideas are good and can take the questioning or they’re not, in which case the community owes you a huge debt of gratitude for point out the flaws!

      • Awesome response. I think I’ve got it. Services are potentially independently developed, and each service needs assurance that the client has proper authorizations. Four options come to mind:

        (1) Each service independently communicates with the AS to resolve a provided opaque token.
        (2) Each service independently communicates with a trusted local cache, which resolves opaque tokens to user data.
        (3) Each service limits itself to trusted callers and accepts unsigned user data.
        (4) Each service allows arbitrary callers and accepts only signed user data.

        Options (1) and (2) are less efficient than option (4), and options (2) and (3) impose severe restrictions on internal architecture, while option (4) does not. Option (4) seems best. If all services require authentication, resolving once at a gateway encapsulates the solution at the optimal location.

        I have been struggling for days to understand what JWTs could accomplish that session ID tokens can’t, and now I think I know. However, JWTs have two exposures that leave me uncomfortable:

        (a) JTWs can’t be immediately revoked. Long mobile app sessions essentially can’t be revoked.
        (b) A breach that exposes private keys permanently compromises users of client-provided JWTs. (Consider that breaches that expose hashed passwords and hashed session IDs leave users unexposed.)

        Your translation tier eliminates problem (b) by getting JWTs from the AS directly. I don’t understand how we are willing to live with problem (a), though. The translation tier does have the opportunity to mitigate (but not eliminate) this problem by regularly confirming even previously cached tokens with the AS regardless of expiration times.

        Thank you for the thoughtful and detailed response.

        • travisspencer

          (2) is an OK option if you do the ref token to JWT token translation and caching in a filter that’s wired up to all your APIs’ Web servers. This is an acceptable solution if all of your APIs are written in 1-3 languages. Writing that filter for >3 stacks isn’t ideal. (You might be able to get some OSS for this, or make yours open source to share the development of them.)

          That said, however, you will still need some sort of reverse proxy in your deployment, otherwise your APIs will be sitting on an open network. [shiver] Putting a DMZ between them and the wild Web will make them safer from a number of other threats. With such a network setup, you’ll have that reverse proxy so doing the translation there rather than in each API makes sense even if you can reuse the server filter across API hosts. (If you go with (4), this filter idea is how you’d validate the JWTs.)

          I have been struggling for days to understand what JWTs could accomplish that session ID tokens can’t

          I was at a customer yesterday who had an ESB and tons of legacy SOAP services. They were gonna hide all this behind an API gateway. As the token flows through this mess, a JWT will allow them to know not only who logged in but what kind of credential was used, when login occurred, was an SSO cookie used or did the user login afresh, various attributes (claims) about the user (e.g., first name, last name, customer number, etc.). This will all be possible no matter how far down the rabbit hole the little token goes. None of this is possible with session IDs unless that session store is available to all of the services (which even if it is will probably require updating legacy services to integrate with that session store and no one wants to touch those). Tokens, even if not necessary JWTs, are the best way IMO. Because JWTs are small, I would recommend them before SAML or other WS-Security security tokens or non-standard formats.

          (a) JTWs can’t be immediately revoked. Long mobile app sessions essentially can’t be revoked.

          This is a problem. There’s no denying that. In the next 12-24 months, I think you’ll see OAuth servers on the market that support Web sockets that clients can use to get push notifications about token expiration. I know of at least one that will include this during that timeframe.

          Meanwhile (and after such features are available actually), there are steps you can take to mitigate this:

          1. Issue short lived access tokens (e.g., 10-15 minutes)

          2. Issue a refresh token only to apps who have a back-end and keep their client secret on the server. Because the AT will expire every 10-15 minutes, they will need to refresh the access tokens. This will mean that a grant revocation will leave you exposed for <=15 minutes.

          3. In the introspect call that you make from the API gateway, stipulate in the response from the OAuth server how long the gateway should cache the token for. This could be < the expiration time, thus reducing your exposure even more. You could change this cache duration depending on the OAuth client. There's nothing in RFC 7662 or related specs that say you have to include a token expiration time. I would recommend only including a cache duration period, that way the gateway can't get it wrong how long it should keep the token around.

          4. Weaken the tokens over time. This is a big topic, but one you should think about and we can talk about more if it's interesting to you (or others on this blog). Essentially though, the tokens should get weaker as they age. This requires support in the OAuth server for such a thing, but gives all sorts of new architectural options.

      • Travis, you’ve been quite generous helping me to understand this model so far, but I’ve got another question for you. Suppose we have browser or mobile client Q, as well as services X and Y. Suppose both Q and X wish to use service Y. If X and Y can’t trust each other, why is Q required to use an opaque reference token to call Y, while X is not?

        I have another question, but feel free to give me a term to look up or a link. Are all the services behind the translator necessarily cohosted? It seems that they minimally need to agree on an interface for communicating the JWT, and if that interface isn’t some common standard, then the services would need to be designed to work together. I’m trying to understand the logical boundary between opaque tokens and JWTs. Is there a name for a collection of services that share JWTs under a common protocol?

        • travisspencer

          No app or service trusts any other app or service in the landscape we’re discussing here. Trust is governed centrally in the Security Token Service (STS) which is part of your Identity Management System (IMS). X never trusts Y, but both trust the STS — the OAuth server — if they are in the same security domain. If Y doesn’t trust the issuer of the tokens presented by X, it will need to convert them to tokens that it does trust by exchanging them with the STS in its security domain. If it doesn’t have a way to do this translation (because there is no STS in this other domain or it can’t do it itself — which I don’t recommend), the only choice it has is to deny access to X’s calls.

          The gateway that we’ve been discussing is a way to broker tokens into trustworthy credentials as calls enter a security domain. If calls are made out of that domain to another, some other token exchange is probably needed as well. This token exchange is how these security realms are segregated. Without such a thing, there is only 1 security domain. Sounds like you might have multiple.

          To your other question, no, all services are not necessarily co-hosted. They can be run from various data centers and geographies. Because they trust the same STS though, they are in the same realm, security domain, or fiefdom. You could also say that all services in one domain are the same Relying Party if there don’t have separate IDs/audiences (but they should). Being apart of a security domain doesn’t have anything to do with their location or the token format used (e.g., JWT); it all comes down to which STS they trust, and this is manifested by them accepting the STS’s public key. If the services trust the same STS, they’re apart of the same security domain. I touched on this in a blog post a long time ago: http://twobo.com/blog/2011/10/calling-an-oauth-10a-api-from.html. That involves the user explicitly consenting to their identity being used across domains. This isn’t always the case, however. The point about token exchange as you cross domains is the same though.

          For passing tokens around, you should use RFC 6750 for your REST services and WS-Security’s binary security token for your SOAP services.


  • Christopher Ickes

    Excellent post. Working on a simple POC. Brings me to 2 questions.

    1) What would be the difference between a traditional API Gateway layer and a Reverse Proxy layer? Or is this just semantics?

    2) Have not used the AWS API Gateway before. Is this a good time to try or would we be better off sticking with an NGINX solution?

    Specifically, we’re using OpenID Connect pattern w/ Authority tokens (opaque) stored client side and then API Gateway / Reverse Proxy layer with caching will issue private JWT for access to back-end services.

    • travisspencer

      1. An API gateway is a reverse proxy on steroids and is designed specifically to be a facade in front of APIs. Usually, the gateway products are from the SOA days and do lots of XML stuff, so it makes it easier to proxy those kind of services which is a lot harder (if possible at all) with a vanilla reverse proxy.

      2. One of the most important requirements of the reverse proxy is that it can cache tokens efficiently and that it can scale really well. I don’t know if AWS API Gateway supports caching, but if so, I’d say it’s an ideal candidate. With NGINX, you gonna bring in LUA or write C code. I think the C code would be easy and wouldn’t bring the baggage that LUA does. I’ve been tempted to write such a plug-in in my spare time but I never seem to have any ;-)

      When you store those opaque tokens client side, be sure to put them in a cookie that is secure and HTTP only. I would not recommend local storage because that isn’t governed by the browser’s same origin policy. If client side is a mobile app, try to use the key ring or other secure, app-specific storage provided by the OS.

      In the reverse proxy/API Gateway layer, be sure to cache expired tokens forever and don’t use the OAuth server’s cache expiration (the Cache-Control header value) if the OAuth server includes the token’s expiration. A token, once expired is expired forever, so this could be an easy, extra optimization.

      What OAuth server are you using to implement the POC? If you need something there, ping me offline (travis@twobo.com).

      When you’re done, it would be cool if you want to publish a write up here on Nordic APIs. If that’s interesting to you, drop @billcdoerrfeld:disqus a note. I know I’d like to read it, and I bet he wouldn’t object.

  • Inca

    Maybe late to the discussion.
    Thanks for the article. I have a few questions:

    What is the concern by exposing the JWT ? I’m building a system where the jwt is transmitted via TLS and then stored in a mobile application ( not web ). Then I used it, again via TLS, to access resources behind an NGINX reverse proxy with JWT validation support ( and rate limit based on client ip + JWT claims ).

    Maybe there is something I don’t understand, but if the opaque token is compromised it has the exact effect as if the jwt token is compromised.

    The concern is about exposing the JWT payload?


    • travisspencer

      Not late, just next ;-)

      You’re right: the opaque token is about not exposing the payload of the JWT in addition to increasing cohesion in the system. Specifically, the reasons to issue an opaque token to the app (mobile or Web) are two fold:

      1. Security
      2. Ensuring loose coupling between the app and API

      The opaque token provides another protection against attacks in a multi-pronged defense in depth strategy. The second reason is to ensure that the app, for which the token is not intended, does not peer into the token and become dependent upon its contents. (The access token is only intended for the API/RS).

      To meet these requirements, an encrypted JWT could also be used, but the opaque token is more efficient, easier to deploy, and simpler to develop.

      Therefore, the approach described in this article should be preferred IMO.