Recent data reveal an alarming trend: IA crawlers like Gptbot d’Opnai and Claude d’Anthropic collectively generate nearly a billion monthly requests. This activity now represents 28% of the volume traditionally associated with Googlebot, marking a significant redistribution of automated web traffic.
IA robots are characterized by their particular behavior. Unlike conventional crawlers who are leaving their requests, these new agents can make up to sixty requests for the same web page, quickly saturating server resources.
This intensity is explained by their fundamental mission: to supply data with large models of language (LLM) which require ever larger information corpus. Bytespider of Bytedance (Mother house of Tiktok) currently dominates in volume of requests, followed closely by GPTBOT of Openai and Claudebot d’Anthropic.