Cloudflare noted that it saw Perplexity’s crawlers disguising themselves using a variety of schemes, such as rotating user-agent strings, which are designed to identify the type of browser and device being used to access a site, as well as changing IP addresses across different Autonomous System Numbers (ASNs), to disguise their crawlers as browsers like Google Chrome for macOS.
Legal experts believe that ignoring robots.txt violations could escalate the broader web scraping controversy around AI-generated content. Critics argue that Perplexity AI needs to adopt more transparent and ethical data sourcing practices. The scope of the operation was sizeable. the same methods were being used at tens of thousands of websites, resulting in millions of automated requests.
The issue of Perplexity AI scraping has reignited debates about Artificial Intelligence training methods and data ownership. Cloudflare said the activity was detected through a combination of machine learning analysis and network monitoring, triggered by reports from customers whose sites had been accessed despite having Perplexity’s bots blocked.
Also Read:
