AI scraping bots are capturing a growing slice of web traffic, U.S. data shows

Multiple internet-infrastructure firms and a bot-tracking specialist report a sharp uptick in automated agents that surf, copy and harvest online content for AI services. Across recent quarters the share of monitored visits attributable to scraping bots rose from a rare edge case to measurable fractions — roughly from one visit per 200 to about one per 50 on some provider datasets — while a growing slice of those requests ignored site directives meant to block crawlers. At the same time, an expanding number of publishers are taking direct action to limit intermediaries that aggregate or archive web content: several prominent outlets have applied selective blocks to the Internet Archive’s crawler after concluding archived collections can be repurposed as an indirect route for model training. That tactical shift — from lawsuits aimed at model builders to constraining intermediaries that facilitate mass ingestion — underscores a strategic pivot in how rights-holders respond. Technical operators of extraction tools are adopting evasion techniques that mimic normal browser behaviour and human interactions, which blurs the line between legitimate users and algorithmic visitors and undermines simple defenses like robots.txt, basic fingerprinting and static rule sets. Defenders are responding with harder measures: higher-fidelity behavioural analytics, session and identity verification for programmatic clients, and commercial offerings that meter or charge automated crawlers. Infrastructure vendors are packaging pay-for-crawl products and access-control services, signaling a move toward explicit machine-to-machine commercial flows rather than the implicit, unpaid scraping that many publishers see today. For publishers and archivists the trade-offs are stark. Targeted blocks and throttling can reduce the risk that organized, machine-readable repositories are mined for training data, but they also erode public preservation and make archives less reliable for reporters, historians and researchers. Some publishers exploring licensing deals would route payments to organizations rather than individual journalists, raising questions about who benefits from negotiated access. The near-term landscape is likely to be messy: defenders will chase evasions, vendors will commercialize access, and some rights-holders will pursue legal remedies while others seek paid APIs. Over the medium term the internet’s economic model for content may bifurcate into human-facing experiences and authenticated, paid machine feeds, with significant consequences for revenue models, content licensing, content provenance and digital preservation.