In the intricate world of API development, clear and standardized communication is paramount, especially when things go awry. Enter the “problem detail,” a beacon in the sometimes murky realm of error reporting. As defined by the Internet Engineering Task Force (IETF), Problem Details for HTTP APIs offers a universal standard to make API error responses more understandable, precise, and consistent.

Instead of leaving users to decipher vague error messages or navigate a maze of custom error formats, the IETF problem details request for comments (RFC) offers a unified approach to improve API error handling. Let’s dig in and look at how to use this standard!

Why a Problem Details RFC?

Before diving into what problem details look like, we should answer a basic question: Why are they needed in the first place?

APIs are rife with errors because the internet is rife with errors. The reality of the modern web is that you can’t control how every single bit will act across the transfer of data, and accordingly, APIs will sometimes result in errors. These errors must be clear, understandable, and, most importantly, resolvable. Errors must convey enough information to attempt a fix if needed — and if a fix isn’t needed, it should convey that as well.

Part of the problem with how errors are currently handled, however, is the context that surrounds them. Often, HTTP status codes do not carry enough information to convey what is happening. While HTML response bodies and documentation can carry additional context for the human on the other side of the error, this requires an extra step of contextualization. Another part of the issue is that this context is for human consumption and often provides little to no value to machines.

The Problem Details RFC is designed to rectify this problem.

Diving into The Problem Details RFC

How it Provides Context

How the RFC does this is actually quite clever. Problem details are provided using a specific type of problem as defined through an object composed of parts. This object can state a particular error case, and, in this error case, provide specific context for how the problem has arisen and how it can be resolved through contextual bundling.

The parts of the object for a Problem Detail look like this:

• First, you have the type, a URI that defines that specific error category. This can include something like “out-of-credit” to denote that the end user has encountered an error of that specific type. Other types could include things like “not allowed,” indicating that the user cannot perform the function.
• Next, there is a title. This string allows developers to summarize the problem for the end user, communicating more context.
• Next, we have several optional items:
  • The status value is an optional integer allowing developers to continue to provide the HTTP status code generated as part of the issue. This is optional but very useful for situations like the 404 error where there is no additional context besides “there’s just nothing here.”
  • The detail value allows for a much more in-depth and detailed summary of the issue and is the perfect place where additional context or even problem-solving tips can be given to the end user.
  • Finally, the instance value allows developers to identify the specific instance number of each occurrence, tracking the number of issues and generating individual tracking over time. This helps to identify problems resulting from a specific client environment or situation that might otherwise be lost in more general reporting.

Practical Example of Problem Details

So, what does this look like in practice? Let’s look at an example problem detail. Imagine you are trying to purchase a pass to a Nordic APIs event. When you try to make a purchase, you are presented with this problem detail:

{
 "type": "https://nordicapis.com/error/no-payment",
 "title": "No Payment Method",
 "detail": "You have no payment method set - please add a method.",
 "instance": "/user/apiGenius/errors/no-payment-001",
}

This practical example gives the user a great amount of information in a very small piece of text. The user would be able to glean the following:

• The type is no-payment, suggesting an issue with processing this order.
• The title is “No Payment Method”, communicating that there’s no method for making this payment.
• The detail informs the user that the real issue at the heart of the problem is on the user’s side. Since no payment method has been saved, no payment can be made.
• The instance lets the user and the admin know that this error has been logged into their user log area and can be reviewed at a later date.

The RFC notes that additional nested responses can also be used to communicate multiple instantaneous errors. In this example from the RFC, a validation error is sent in the form of a problem detail:

  {
   "type": "https://example.net/validation-error",
   "title": "Your request parameters didn't validate.",
   "invalid-params": [ {
                         "name": "age",
                         "reason": "must be a positive integer"
                       },
                       {
                         "name": "color",
                         "reason": "must be 'green', 'red' or 'blue'"}
                     ]
   }

Here, two problems in the form of an invalid-params extension express that both the age and color entries were incorrect. The additional reason information allows the user to be informed as to their options for resolution.

Recent Updates to Problem Details for HTTP APIs

RFC 9457 is the most recent revision of this RFC and is backward compatible with former versions. While it technically obsoletes the former RFC, it does not introduce new breaking changes in formats or features.

“RFC 9457 was created as an update to the popular RFC 7807 problem report format,” said Erik Wilde, a co-author of the RFC. “It is a backward-compatible change, i.e., it does not define a new format and is not adding new features. It addresses some issues that we were seeing with adopting HTTP error reports. Most importantly, it has improved language around problem types, and it introduces a registry for problem types.”

Registry for Common Problem Types

The most significant update for this RFC is the addition of a common problem type registry. The RFC uses very specific language for referencing the registry, stating that “registrations MAY use the prefix ‘https://iana.org/assignments/http-problem-types#‘ for the type URI,” and gives explicit expectations for how and why a problem might be registered.

“When evaluating requests, the designated expert(s) should consider community feedback, how well-defined the problem type is, and this specification’s requirements. Vendor-specific, application-specific, and deployment-specific values are unable to be registered. Specification documents should be published in a stable, freely available manner (ideally located with a URL) but need not be standards.”

This addition does two things. First, it allows developers to register common problem types for reuse across projects. When well-defined, these problem types can be replicated regardless of the language or implementation, as the registered problem types represent a common problem that needs additional context. Second, by defining the evaluation criteria for such requests, specific issues related to vendors, applications, and deployments are abstracted away. This ensures that the “universal problems” are, in fact, universal and not overly specific to a single implementation.

Clarification for Multiple Problem Issues

A specific use case not fully addressed in the first RFC is the issue of multiple problems. When using an API, multiple problems may be encountered simultaneously. These issues might feed into one another, or they could be separate, isolated issues. The new RFC recommends first communicating the most urgent or relevant problem in the response instead of creating batch problem types.

It should be noted that there is a piece of advice nestled in the change that does mention the idea of a “batch” problem type. This may be appropriate in some cases, but as the RFC notes, this is difficult to map into HTTP semantics, and as such, the advice is to communicate the most pressing problem first.

Using Type URIs that Cannot be Dereferenced

Finally, the new RFC provides specific guidance for how type URIs that cannot be dereferenced should be handled within the context of the general type specification. The update introduces guidelines for the type member in JSON, specifying that it should contain a URI reference identifying a problem type. If absent, it defaults to about:blank.

Resolvable type URIs are recommended, potentially providing human-readable documentation, while non-resolvable URIs like tag URIs are allowed. The use of relative URIs is cautioned against due to potential confusion and inconsistent handling among implementations.

The Value of Contextual Communication

Error codes are the most direct form of communication an end user will first encounter when an error occurs. Accordingly, it is a trusted source of information that is invaluable. Finding the balance between usability and brevity is a balancing act that has always been difficult to walk. Therefore, using problem details to provide both in an easy-to-parse, easy-to-control method is a great solution.

Consider the use case of the average user. Let’s say you go back to that hypothetical example of trying to buy a Nordic APIs ticket. If the system uses problem details, this is the flow that we can expect:

1. The user attempts a purchase but is denied.
2. The user receives clear communication as to the cause of the issue.
3. The user also receives a method to attempt a fix and a rough idea of how to accomplish it.
4. The user is able to get their fix implemented and goes on to make a purchase.

This is a sensible flow of events, but equally important, it has low friction. Let’s take a look at the alternative path that has more friction in the way of completing the transaction.

1. The user attempts a purchase but is denied.
2. The user looks at the error and sees a simple error code of 403 Forbidden.
3. The user searches the site to determine why they are getting a 403.
4. The documentation states several potential causes and instructs the user to look at the HTTP body response, so the user again makes the attempt and looks at the body of the response before the page is automatically reloaded by their mobile browser.
5. The user decides that this is all too much effort and decides that they never really wanted to go to the event anyhow. Cart abandonment ensues!

It’s a bad user experience, but you may think the likelihood of someone giving up that quickly is a bit overblown. The reality is that this is a more common occurrence than you might think. According to the Baymard Institute, 70.19% of all online shopping carts are abandoned — and that’s within a low friction environment! Consider how many people will jump the API ship if they can’t even figure out a simple problem like the above without doing six or seven discrete steps.

Conclusion

API error communication is vital to the health of any system. It’s not just good enough to report an error — you must report it in a clean, easy-to-understand, and useful format. Failing to do so will create a poor user experience but could also have long-term effects on adoption, customer retention, and customer satisfaction. Luckily, the problem details RFC offers a solution and would be an excellent addition to any API.

What do you think of this RFC family? Do you see any other use cases that could benefit from its implementation? Let us know in the comments below!