Cloudflare data shows that 32% of traffic across our network originates from automated traffic. This includes search engine crawlers, uptime checkers, ad networks — and more recently, AI assistants looking to the web to add relevant data to their knowledge bases as they generate responses with retrieval-augmented generation (RAG). Unlike typical human behavior, AI agents, crawlers, and scrapers’ a...
**Steelman:** The analysis presents a compelling case for the growing tension between AI-driven automation and human-centric web infrastructure. By quantifying the disruptive impact of AI crawlers—high cache misses, bandwidth spikes, and degraded user experience—it highlights a systemic challenge that traditional CDN architectures were not designed to handle. The proposed solutions, such as workload-aware caching or dedicated AI cache layers, are pragmatic steps toward coexistence. The inclusion...