ScrapeBench: Evaluating Legal Compliance of AI Agents in Website Scraping

Media reports Hugging Face Papers Source language: Chinese

ScrapeBench is introduced as the first benchmark to measure AI agents' compliance with website Terms of Service and robots.txt. Across 1,626 sites that forbid scraping, most agents achieve under 5% compliance, even when they check the policies. On 269 allowed sites, some agents over‑refuse (e.g., Claude Opus 4.6 at 98.5%).

Full text

AIagentsdeployedontheopenwebarebecomingmorecapableofperformingactionsthatmaybreachcontractualprovisionsandviolatecomputersecuritylaws.Inthispaper,weintroduceScrapeBench,thefirstagenticbenchmarkforempiricallyevaluatingthelegalcomplianceofAIagentswithwebsitescrapingpolicies,namelytermsofservice(ToS)androbots.txtdirectives.Acrosswidelyusedclosed-weightandopen-weightAIagents,wefindthatmostagentsexhibitlowcompliancerates(<5%)wheninstructedtoscrapewebsitesinoursampleof1,626popularwebsitesthatexplicitlyprohibitscraping.Inaddition,weobservethatevenwhereAIagentsactivelychecktheapplicableToSandrobots.txt,theycontinuetoexhibitlowcompliancerates.Conversely,inoursampleof269websitesthatdonotprohibitscraping,weobservethatagentssometimesrefusetoscrapedespiteitbeinglawful,suchasClaudeOpus4.6exhibitinganover-refusalrateof98.5%.Ourhumanbaselineexperiments(n=180participants)suggestthathumansexhibitcomplianceratescomparabletoAIagentsbutover-refuseatmuchhigherrates.

View arXiv pageView PDF Project page GitHub 0 Add to collection

Sources

← Back to Daily AI Report