In recent months, AI agents have been the talk of the town. Agentic AI is arguably the next evolution in the generative AI hype cycle, as it has to do with empowering AI assistants with autonomous capabilities that could revolutionize user experiences. However, where things stand today, there’s a disconnect between the hype surrounding AI agents and their actual utility.

According to Anushrut Gupta, Senior Product Manager, Generative AI, Hasura, AI agents still tend to produce inaccuracies or unactionable results in enterprise settings. AI assistants tend to underperform, he says, whether you’re asking Google’s Gemini which files a colleague has edited, looking up sales cycles in Salesforce’s Einstein, or querying Amazon’s Rufus for basic user shopping histories.

Part of this gap concerns data because, as the saying goes, “garbage in, garbage out.” Not only does AI need relevant data to produce contextually accurate and actionable results, but it also needs easy, real-time access to that data on demand. As such, Retrieval-augmented generation (RAG) systems have emerged to give large language models (LLMs) a data retrieval mechanism, contextualizing the user experience with real-time information.

APIs have a considerable role to play in RAG. But it doesn’t end there. APIs can take AI agents beyond simple retrieval requests to make mutations on the backend, interfacing with internal services or third-party software-as-a-service (SaaS). As such, APIs are well-positioned to help realize the promise of agentic API, connecting disparate sources and executing actions on behalf of users. That said, there are some hurdles. So, how do we connect all the dots?

The State of AI Agents

One potential roadblock today is a lack of easy API accessibility and integration details that LLMs can decipher. As Zednec “Z” Nemec pointed out at Platform Summit 2024, AI agents are currently struggling with API integration.

APIs aren’t always fully documented, making API metadata challenging to discover. Not all APIs provide a public OpenAPI specification or an explicit GraphQL schema. If documentation is incomplete or fields are missing, it’s often far easier to screen-scrape such data than to go through hoops to make a standardized integration.

Beyond accessibility, AI agents themselves face a few challenges in enterprise environments, particularly in choreographing real, actionable workflows across applications. “Orchestrating between different domains and data sources and connecting them to an LLM orchestrator is not easy,” says Hasura’s Gupta. This process may require accessing data from disparate sources and contending with unique requirements per integration, such as authentication and authorization, pagination, rate limiting, and computing processing time delays.

Ask Amazon’s Rufus how much you spent in 2024, and it’ll yield a pretty irrelevant result. (Admittedly, it’s still in beta at the time of writing). In a business context, getting actionable results from an agent would require much more specific prompts. For instance, suppose a quality support representative wants to find tickets raised from particular customers in the last week where an SLA wasn’t met and then automatically issue refunds. This would be a huge timesaver, but agents currently struggle with complex queries involving multi-step workflows and mutations, notes Gupta.

Bolstering AI Assistants: The Requirements

To make these enterprise use cases more contextually aware, AI assistants first need access to the entire data landscape. According to Gupta, this requires supplying AIs with a unified semantic schema across all data sources, whether SQL databases, vector databases, internal APIs, or third-party SaaS integrations. You’ll also probably require more unified authorization for data sources.

Some headway has been made to standardize how AI assistants connect with underlying data. For instance, Anthropic’s Model Context Protocol proposes a universal standard for connecting AI systems with data sources using MCP servers and clients. Still, according to Gupta, “MCP doesn’t solve the fundamental problems when it comes to connecting data (of multiple types, sizes, and sources) to LLMs and real-world user queries.” He points to the limitations of RAG-style systems, limited query planning setup, and a lack of clear guidelines around access control enforcement as possible hindrances of Anthropic’s Claude using MCP compared to other solutions.

Some architectural strides will have to be made as well. Some AI developers are turning to a multi-agent approach, but this is not working too well in an enterprise setting, says Gupta. Instead, he advocates empowering a single agent to think on the fly with multi-step query planning with retrieval and computing, the ability to execute and self-fix, a structured memory, and role-based access control to various domains.

Lastly, analytical computational tasks will require deterministic results, says Gupta. In other words, the same question should result in the same answer. This is important since accuracy is paramount when working with sensitive internal data like support tickets, invoices, or error rates. Therefore, AI assistants must be logical and prescriptive — and provide the ability to use different weights and nudge weights when appropriate.

Example: PromptQL

Consider the customer support scenario described above, in which a company wants to issue refunds to a certain number of customers with support tickets. Gupta suggests the following prompt: “Who are my top five customers by revenue who have at least four projects with us and at least two support tickets with us? For the second-highest-revenue project, issue a refund of 3.75% of revenue.”

Automating this complex procedure would likely require multiple database calls and API interactions for checking customers in a Postgres database, gathering tickets from a Zendesk ticketing system, inferencing on this information, and then issuing refunds through Stripe. All this is technically possible using PromptQL from Hasura, a data access agent for building AI applications.

In order to aggregate data, the agent traverses a supergraph, which is essentially a giant YAML file that combines schemas from disparate services, APIs, and databases. PromptQL then creates a query plan based on the prompt. It emulates how a human being functions, opening a Jupiter notebook and writing SQL code on the fly that describes the data requirements, computing requirements, and what “thinking” is happening under the hood. (This is akin to ChatGPT showing you its sources but going the extra mile for the developer class).

With PromptQL, there’s no orchestration layer above multiple calls to various LLMs. It’s just a single call to the LLM. The agent doesn’t act 100% autonomously — for mutations (any POST API call), it asks for human approval before proceeding. All this means that, for developers, the actions it takes and its underlying design are transparent, explainable, and repeatable.

Potential Roadblocks: Something New Beyond APIs?

Still, Gupta admits that this approach may have some potential hindrances. For one, setting up a unified data layer may require some effort. For this to work well, the metadata has to be solid, so it may take some upfront work to define relationships within the unified semantic data layer to make them more contextually aware.

It also takes upfront work to gather the necessary data structures. Using a platform like Hasura or Apollo GraphQL, much of the schema introspection and insertion into the supergraph is automated. Still, however, the process is stunted if the external APIs you’re trying to integrate with don’t have well-defined APIs. (It’s not a great sign that 75% of APIs have endpoints that drift from their documentation!)

Interestingly, APIs might not be the golden answer for AI agents, says Gupta, who notes they are by design ill-suited for this purpose since they are typically rate-limited, paginated, with sophisticated access control rules. “They fundamentally have limitations since they were designed for the web interface,” he says. “We need to think of something new for agents.

This “something new” could eschew APIs altogether in favor of direct access to databases provided by SaaS companies. “Directly connecting to databases is the best way to move forward,” says Gupta. If this trend occurs, it would open up a brand new integration method, supplanting some traditional REST APIs in the process.

Beyond typical REST APIs, data mesh architectures and event-driven architectures (EDA) could also shape the “post-API” world for agentic AI, enabling them to receive trigger-based updates in real time. But regardless of the integration method, security risks are heightened when agents have greater autonomy. As such, fine-grained access control for sensitive enterprise data will be clutch.

The Future: Empowering AI Agents With Mutation Capabilities

When you think of AI assistants, you probably think of clunky consumer-facing chatbots for things like ecommerce websites. Or, perhaps you think of trying (and failing) to get through to human customer support. Although AI chatbots often don’t live up to their promise, AI agents are quickly evolving.

AI agents are shifting from ephemeral interactions to persistent memory. They’re now integrating external data sources to bring more and more context to our fingertips. Smarter AI agents in a business setting could enable some pretty profound capabilities for HR teams, engineering managers, support representatives, and other roles.

Although API access for AI agents is currently limited, things are quickly improving. For instance, API gateways — such as Kong’s AI Gateway — could act as middleware for dynamic mutations, mitigating some of the current challenges.

But of course, web API connections are only one factor here. Many technologies could power tomorrow’s AI assistants, from foundational open models, vector databases, and other open-source software — not to mention the technology underpinning user experiences like chatbots, natural language processing, speech recognition, and more. In short, it takes a town to get AI right.

Beyond autonomically integrating real-time data on the fly, automatically performing mutations on backend workloads stands as the next generation of AI. It’s a big piece of the puzzle without a definite answer. PromptQL stands as one impressive solution. But whoever can crack this code in the most effective, standardized way is set to really push things forward, bringing the API economy into the realm for AI agents to act autonomously upon. When that occurs, the possibilities are endless.