It’s official: more developers than ever are turning to APIs to solve their problems and get things done faster. Postman’s latest State of the API Report found that 74% of respondents are now API-first, up from 66% in 2023, and that number is very likely still rising in 2025.

But there’s a worrying trend accompanying that rapid adoption — worrying, at least, if you’re an API provider: the rise of web scraping and, specifically, unofficial scraper APIs. Per Search Engine Land, AI bot scraping more than doubled (over 117%) between Q3 and Q4 of 2024 alone.

If you’re a developer who uses third-party APIs, you might see this as an exciting possibility: a shift away from official APIs towards…something else. And the battle is quietly raging, with developers increasingly bypassing authorized services and turning to web scraper APIs instead.

Apify’s State of Web Scraping report describes how API-based runs on their platform increased from 3.6 billion API calls in January 2023 to 6.8 billion by October 2024. In the same time period, their active user base rose from just over 20,000 to more than 50,000 developers.

In this article, we’ll get into some of the reasons web scraping APIs might be favored over official APIs, why that could be problematic in some contexts, and outline some of the steps you can take to limit the likelihood of consumers bypassing your API in the first place.

The Rise of Scraping APIs

In our piece exploring whether the AI revolution is leaving APIs behind, we wrote about some of the factors limiting the extent to which AI tools like chatbots can interface with APIs.

Some of these include:

• Limited or no access to APIs for developers
• APIs are sometimes overcomplicated, bloated, or difficult to call
• Legacy APIs (WS/RPC) lack thorough or up-to-date documentation
• APIs sometimes only cover a fraction of the functions available via the UI

It’s worth noting that many of these points impact human API consumers just as much as they do agentic ones. If you’ve ever been in the position of trying to use an API and it falling short of your expectations, you’ll know just how frustrating it can be.

While it’s possible that some of those users will get in touch to ask you to add certain endpoints or clarify things, plenty more won’t. Some developers are more likely to take the view that it’s easier to ask for forgiveness later than permission now, and find some other way to extract the data they’re looking for. In many cases, web scraping offers just such a solution.

Web scraping APIs are a natural evolution of manual scraping techniques, such as using Python to scrape websites. Used for everything from scraping search engine results, like SERP APIs, to product prices and sentiment analysis, there are various services out there that make web scraping very straightforward. And they’re big business.

Retailer John Lewis, for example has previously reported a 4% uplift on sales after using web scrapers. Pulling data from over 100 sites in less than a day, something they’d be unlikely to accomplish using only official APIs, they used scraping to monitor competitor pricing and influence their own pricing strategy.

The Problem With Scraping APIs

Going the API route allows scrapers to perform at scale: many scraper API pricing models grant hundreds of thousands, or even millions, of API credits, with huge numbers of geo-located proxies. But, like manual scraping, they pose a problem for website and app providers from a governance perspective. And there are plenty of high-profile cases of nefarious behavior.

Unauthorized data scraping can lead to the collection of sensitive personal information like user credentials, email addresses, or even financial data. Hackers have previously claimed, for example, that a massive user record database was scraped by abusing one of Meta’s APIs.

In jurisdictions covered by the likes of GDPR, CCPA, or other data regulations, failure to properly secure sensitive data can have dire consequences not just for user trust, but from a financial standpoint too. The problem for companies? Data scraping is not in itself an illegal practice.

Cloud providers, such as Amazon AWS and others, and various specialist companies all offer web scraping APIs. Although these tools only become dangerous when bad actors use them, that’s increasingly something that app (and API) providers need to prepare for.

How to Win Users Back From Scraping APIs

Just like there are best practices associated with securing APIs, such as data minimization, monitoring traffic, implementing robust security measures (authentication, authorization, encryption, to name a few), there are steps you can take to deter web scraping. And, given some of the headaches that could be caused by people using scraping APIs, doing so is wise.

That might include using bot-blocking services, deploying CAPTCHAs, enforcing rate limiting, rendering dynamic content using JavaScript, and so on. And, ideally, limiting the information that web browsers can crawl in the first place. For example, display only basic specs and MSRP in product listings, when possible, instead of full specs, pricing history, and supplier information.

But there’s another prong to this approach: making scraping APIs less appealing. The best way to do that? Offering an official API that’s more usable and effective.

That means:

• A sensible (and fair) pricing model that scales appropriately with usage
• Thorough and up-to-date documentation, with walkthroughs, code samples, and a sandbox environment.
• A wide range of endpoints and exposed functionalities.
• Call limits and communication around those limits, to prevent nasty billing surprises.
• Engaging with users and potential users on features they find or would find useful.

It might be worth signing up with some scraping APIs — many of them offer free credits or timed trials — and testing them on your own sites or apps. Consider how that experience compares with using your own API(s) in terms of cost, ease of use, and Time to First Hello World (TTFHW).

If you end up finding that using a scraping API is easier than implementing your own API, the odds are good that other potential users are feeling the same way. On the plus side, that might give you a better idea of what to change about your own API to make it more competitive.

APIs, AI Agents, and the Future of Web Scraping

Web scraping APIs are using AI to enhance their services in a number of ways, including adaptive learning based on past scraping sessions, dynamic content handling for identifying and responding to structural changes, and bypassing anti-scraping measures like CAPTCHA.

They’re also deploying NLP and ML for intelligent data extraction, such as identifying specific content types on pages, or understanding and extracting data from unstructured content.

And as AI gets smarter, so too will web scraping APIs — the market’s leading scraper APIs are evolving all the time, with many of them leaning on AI-powered functionalities to enhance their scraping ability and make life easier for their users.

While it’s unlikely they’ll ever entirely replace official APIs, scraping API tools are rapidly becoming a thriving ecosystem of their own, treating the open web like a giant decentralized data source. That’s a trend some web services are actively resisting by introducing comprehensive anti-scraping measures. Cloudflare, for example, recently took steps to block AI crawlers accessing content without permission or compensation by default.

This game of Whack-A-Mole will almost certainly continue as scrapers look for new ways, like those AI enhancements outlined above, to overcome anti-crawling techniques and keep things rolling. In other words, web scraping and scraper APIs will not simply disappear overnight.

For now, and for the foreseeable future, providing great official APIs should not be considered optional if you want to offset the lure of web scraping. And, if you want to ensure that those APIs stay relevant, you should be doing everything in your power to make them shine.