AI has massively revolutionized the API landscape since ChatGPT went mainstream in 2022. Developers are already using AI to generate code from scratch for their APIs. APIs also allow developers and users to interact with large language models (LLMs). As such, the growth of AI and APIs is inextricably intertwined.

Recently, alarms have been raised about potential security issues due to the proliferation of AI and APIs. Wallarm recently issued a report, the 2024 API ThreatStats Report, which discusses some of the security risks due to AI and APIs, ranging from the inherent risks of simply having more APIs to AI being used to write malicious code. Can AI improve API security, though?

In January 2024, cybersecurity expert Corey J. Ball, author of Hacking APIs, generated huge buzz in API circles with a tweet announcing the arrival of Hacking APIs GPT, an AI assistant for assessing and enhancing API security. Considering other LLM’s difficulty in living up to their hype, however, is Hacking APIs GPT all that it’s cracked up to be? To find out, we’ve delved deep to bring you an in-depth review of Hacking APIs GPT!

Hacking APIs GPT Review

Hacking APIs GPT, a specialized version of ChatGPT, is essentially an LLM designed specifically for working with APIs. Hacking APIs GPT can do everything from generating API documentation to writing code for launching an API using Python and Flask from scratch. However, its primary purpose is to analyze API security.

Hacking APIs GPT can perform many types of API security reviews, but it excels in a few specific functions. Hacking APIs GPT is designed for endpoint analysis, OpenAPI review, JWT examination, and fuzzing payloads. These are the areas we’re mostly focusing on in our Hacking APIs GPT review, but we’ll offer some general thoughts and overall observations as well.

Hacking APIs GPT Endpoint Analysis Review

Certain API endpoints can be more vulnerable than others. If you’re not careful, a vulnerable endpoint can become a serious security risk for your entire network. This fact makes the Hacking APIs GPT endpoint analysis function especially exciting and useful.

For our review, we asked the GPT about a few different APIs of varying complexity to see how it responded. To start, we fed it the HTTP Cat API, a relatively simplistic API that’s popular for trials and experiments. Here’s a brief example of the Hacking APIs GPT response when asked to evaluate HTTP Cat’s endpoints.

GET /{status_code}.jpg
Example: https://http.cat/404.jpg
This endpoint returns a cat image representing the specified HTTP status code.
**Potential Vulnerabilities and Mitigations**:
1. URL Manipulation:
* Description: Since the endpoint directly takes a status code in the URL, improper handling could lead to unintended behavior.
* Mitigation: Ensure the server strictly validates the status code input. Only valid HTTP status codes should be accepted to avoid potential abuse or injection attacks.

Not only does Hacking API GPT detect potential security risks, but it also mentions possible solutions. This alone is reason enough to get excited about the LLM’s usefulness.

Next, we evaluated the code for a dummy API written in Python using Flask with a simple three endpoint structure to get a sense of how it performs on a simple API that isn’t processing data yet. Here’s a brief example of its output:

**Endpoint Analysis and Potential Vulnerabilities**:

1. **Endpoint 1**: GET /hello

Description: This endpoint returns a static welcome message.
Potential Vulnerabilities:

Minimal Risk: Since this endpoint does not process user input or interact with any external resources, it has minimal risk. However, ensure that no sensitive information is accidentally included in the response.

2. **Endpoint 2**: POST /data

Description: This endpoint accepts JSON data via POST and returns it.
Potential Vulnerabilities:

Input Validation: Ensure that the incoming JSON data is validated. Without validation, malicious data could lead to unexpected behavior.

Injection Attacks: If the data is later used in database queries, command execution, or other critical operations, ensure it is properly sanitized to prevent injection attacks.

Large Payloads: Limit the size of the incoming JSON payload to prevent Denial of Service (DoS) attacks via large payloads.

We then asked Hacking APIs GPT to analyze the Instagram API to see its effectiveness on larger, more complicated APIs. Here’s an example of its response:

Access Token Management:

Description: Access tokens are critical for securing user data and ensuring proper permissions.

Mitigation: Ensure tokens are stored securely and refreshed regularly. Implement scopes to restrict token usage to necessary permissions only.
OAuth and Authorization:

Description: The OAuth flow needs to be secure to prevent unauthorized access.

Mitigation: Use secure redirect URIs and ensure the OAuth flow is implemented correctly to avoid vulnerabilities like token leakage.

Finally, we asked the LLM to evaluate the Optus OpenAPI spec from 2022 to get a sense of how it reacts to compromised endpoints. Here’s what it had to say.

**Key Issues Identified:**

Lack of Authentication and Authorization:

The API endpoints did not require proper authentication or authorization, allowing attackers to access sensitive data without valid credentials. This is a severe case of broken authentication, violating basic API security principles​ (APISecurity.io)​​ (SidSecure Cyber Security)​.

No Rate Limiting:

The absence of rate limiting enabled attackers to make numerous requests in a short period, facilitating massive data exfiltration. Rate limiting is essential to prevent abuse and mitigate the risk of Denial-of-Service (DoS) attacks​ (APISecurity.io)​.

Use of Shadow-IT Systems:

Some API endpoints were hosted on shadow-IT systems, which are unauthorized and unmanaged. This can lead to significant security gaps as these systems may not adhere to organizational security policies​ (SidSecure Cyber Security)​.

Final Thoughts on Hacking APIs GPT Endpoint Analysis

Hacking APIs GPT’s endpoint analysis is one of its most useful features. It’s reason enough to check out the LLM, as it can help discover potential API security risks you might’ve overlooked. You’ll need to be at least slightly familiar with APIs to truly understand what it’s saying, but that’s true of almost any API security product.

Hacking APIs GPT OpenAPI Analysis Review

Now we’ll review Hacking APIs GPT’s analysis of OpenAPI specifications to get a sense of how it works with APIs that might not be live yet. To start, we’ve analyzed the Basic Structure YAML file from OpenAPI to see how it works on a simplistic OpenAPI specification that is almost guaranteed to be correct. Here’s a representative example of its response.

**Analysis of the OpenAPI Specification**
:
Basic Information:
OpenAPI Version: 3.0.0
Title: Sample API
Version: 0.1.9
Servers:
Production: http://api.example.com/v1
Staging: http://staging-api.example.com

**Endpoint Analysis**

Endpoint: /users
* Method: GET
* Summary: Returns a list of users.
* Response:
* 200: A JSON array of user names
* Response Schema: An array of strings
* Security Considerations:
* Authentication and Authorization:

Issue: The specification does not mention any form of authentication or authorization for accessing the /users endpoint.

Recommendation: Implement authentication mechanisms such as OAuth 2.0, API keys, or JWT to protect the endpoint. Define security schemes in the OpenAPI specification to ensure that only authorized users can access the data.

As you can see, the LLM first interprets the OpenAPI specification and extracts the necessary information. Then, it evaluates the endpoints for potential risks and recommends some best practices to make the API more secure. Many of the OpenAPI YAML’s issues seem to come from it being a template, as a live API would likely address most of these concerns.

For this evaluation, we asked the LLM to evaluate the Trello OpenAPI specification to see how it performs on an extensive and popular API. We asked Hacking APIs GPT to assess the OpenAPI directly rather than uploading 15,000+ lines of code. Here’s a brief overview of its findings.

**Potential Vulnerabilities**:

1. Insufficient Authentication:

* Ensure all API endpoints are protected with appropriate authentication mechanisms to prevent unauthorized access. Publicly accessible API keys without corresponding tokens can lead to security breaches.

2. Token Exposure:

* Tokens must be securely stored and never exposed in client-side code or logs. If a token is leaked, it should be revoked immediately to prevent unauthorized access.

3. Data Sensitivity:

* The API allows access to sensitive user data, including email addresses when the account scope is requested. Proper data handling and masking should be implemented to protect sensitive information.

4. Rate Limiting Bypass:

* Ensure that rate limiting cannot be bypassed by using different IP addresses or creating multiple accounts. Implement comprehensive rate limiting policies.

There are some useful thoughts here, but they also feel a bit like generic API security best practices. They’re good to keep in mind but are not entirely revelatory. However, it still seems worthwhile to run your OpenAPI specification through the LLM as part of the development process. You can’t ever have too many pairs of eyes on a piece of code!

Hacking APIs GPT JWT Analysis Review

JSON Web Tokens (JWTs) have become a popular method of verifying user identification. To start, we used a hypothetical JWT to see how the LLM performs. We asked it to evaluate the JWT:

eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyLCJleHAiOjE1MTYyNDI2MjIsInJvbGUiOiJhZG1pbiJ9.DyDq9P7O9lg9VhHX8D0M9E3cvjK5-Jj7WxaK7Yeq2As.

The response begins by interpreting the JWT’s structure and breaking down the token into the header, payload, and signature. It then identifies a potential security risk, noticing that the JWT has expired.

Now, let’s see how it performs on a real example. We asked Hacking APIs GPT to analyze the following JWT from Auth0’s The JWT Handbook:

eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiYWRtaW4iOnRydWV9.TJVA95OrM7E2cBab30RMHrHDcEfxjoYZgeFONFh7HgQ.

Hacking APIs GPT breaks down the JWT and analyzes each part. It determines that the user’s name is John Doe, with a user ID of 1234567890, and he has administrative privileges. The token is signed using HMAC SHA-256, but it’s impossible to verify that without the secret key.

Hacking APIs GPT Code Analysis Review

Let’s finish up by making sure Hacking APIs GPT is telling the truth. One of the biggest drawbacks of using LLMs for any practical purpose is that they often make things up. To make matters worse, they hallucinate with complete certainty. This can leave undetected errors in anything generated by an LLM, which you might not catch unless you go through it line by line, which defeats the purpose of using an LLM in the first place.

To measure its accuracy and truthfulness, we’ve run a Flask app through Hacking APIs GPT with a known error to see how it responds. Our code accepted the wrong method type for one of the endpoints, asking for POST instead of GET. Encouragingly, the LLM caught the error immediately and, even better, offered solutions to fix it.

Final Thoughts on Hacking APIs GPT

After running it through its paces, Hacking APIs GPT lives up to the hype. It stands to revolutionize the API industry, opening things up to people with little technical experience. It’s also highly useful for seasoned developers, helping to debug your code and catch any potential errors.

Hacking APIs GPT certainly isn’t the only API vulnerability scanner on the market. There are numerous excellent tools for analyzing OpenAPI schema and endpoints for security risk. Yet, Hacking APIs GPT may be the most fully featured, as it can also write code from scratch and suggest improvements. If you’re looking for one AI tool for working with APIs, consider Hacking APIs GPT.

Perhaps most useful of all, Hacking APIs GPT is one of the best tools for learning how to create and work with APIs. It’s like the world’s smartest API tutorial, answering questions and problems put forth in plain language with clean, working code using the latest best practices. Anyone interested in learning how to work with APIs can reverse engineer its output and learn how to write their own code.

It should also help lay some fears about AI taking over all of the coding jobs to rest. While it’s definitely capable of producing clean, working code from scratch, you still have to know what to ask for in the first place. It sometimes takes fine-tuning to get things up and running, just like any other API product. With that being said, Hacking APIs GPT is definitely one of the most game-changing API tools we’ve encountered in a while, though. It’s well worth trying out for yourself, no matter how much experience you have working with APIs.