AI Scrapers Consume 20% of Kernel.org. Git Clone Was Always Free.

On August 29, Konstantin Ryabitsev published a post called “Creepy Crawlies.” Ryabitsev maintains the infrastructure behind git.kernel.org and the Linux Kernel Mailing List archive — the servers that make it possible for thousands of contributors to coordinate development on the most widely deployed operating system kernel on Earth. His post is short, specific, and upsetting in a way that a lot of AI infrastructure commentary manages to avoid: he has the receipts.

At any given moment, across five geo-distributed nodes running 90 CPU cores total, 14 to 16 of those cores are doing nothing except rendering Linux kernel git commits as HTML for AI scrapers. That’s 15 to 17 percent of the cluster’s total capacity — Ryabitsev calls it “about 20%” when accounting for the spiky bursts. Git.kernel.org receives roughly six million requests per day demanding access to random commits. Legitimate developer traffic — the actual purpose of the infrastructure — accounts for approximately 2% of that load.

The servers have not slowed down noticeably. But 20% of a five-node cluster is now a permanent fixed cost that does not benefit Linux development, the kernel maintainers, or any user who has ever submitted a patch. That compute belongs to whoever is training AI models on the history of the Linux kernel. Ryabitsev’s infrastructure pays the bill.

The Stupidest Possible Method

This is where the story gets specific in a way that should make any engineer uncomfortable.

Ryabitsev offers a git clone of everything kernel.org hosts. Not as a technical workaround — as a deliberate design choice. The entire LKML archive is available as git repos. Linux.git is about 1.48 million commits. You can clone it once, walk every commit, and have the complete history of the kernel’s development in a format that compresses cleanly and requires zero server-side rendering. Ryabitsev writes: “Seriously, it’s just a git clone away — and then you’ll have the whole history.”

The scrapers are not using git clone. They are hitting cgit — the web interface — commit by commit, rendering each one as HTML. Cgit supports not just commits but patches, plain renders, and diffs between arbitrary commit pairs. Kernel.org has about 922 forks of linux.git. Across all those forks and all those URL-generatable combinations of commit, format, and diff target, Ryabitsev estimates the valid URL space at what he calls “1.2 metric bajillion” — combinatorially explosive, all returning essentially the same 1.48 million commits in 922 slightly different configurations.

The scrapers are crawling it anyway. Every redundant fork, every re-derivable diff, every URL combination cgit will answer. The same data that a single git clone would deliver — once, efficiently, free — is being extracted through the most server-intensive possible path, repeatedly, by bots that keep coming back.

This is not just wastefulness. It is a specific technical failure that reveals how these pipelines are built. The teams constructing AI training scrapers are running general-purpose URL crawlers against high-value domains. They don’t have domain knowledge about what they’re scraping. A developer who knew kernel.org existed would know that git clone was available. The scraper doesn’t know this, because the scraper wasn’t built by someone who uses kernel.org. It was built by someone who wanted its data, which is a different thing.

The Arms Race That Cannot Be Won

Kernel.org tried to fix this. The escalation history is documented precisely in Ryabitsev’s post.

First, the bots identified themselves via user-agent, and they got blocked. Then they faked browser user-agents, and kernel.org started blocking by IP. The bots fanned out to entire subnets, and kernel.org started blocking ASNs. Then something changed: the bots started arriving from millions of residential and mobile IP addresses, each one making four or five requests and never appearing in the logs again. Ryabitsev names this phenomenon with a phrase that deserves more attention in this conversation: “proxy SDK monetization.”

Proxy SDK monetization is the practice of embedding a lightweight SDK into consumer apps — phone apps, smart TV apps, home appliance firmware — that routes network traffic through the user’s device as a commercial service. The user installed an app. The app made a deal with a residential proxy network. Their device is now, periodically and invisibly, serving as an exit node for traffic that has nothing to do with them. The IP address the request comes from is real, residential, and essentially impossible to block without blocking actual humans.

Against this, kernel.org deployed Anubis: a proof-of-work challenge built by Techaro that requires visitors to hash a challenge string before receiving content. Difficulty 4 (the default) was effective for a while. Then the bots started solving difficulty 4. Difficulty 5. Now, as of Ryabitsev’s post, Anubis blocks 66% of the six million daily requests outright — but 33% are bots that have solved the challenge and are getting through anyway.

Tavis Ormandy — a security researcher at Google Project Zero who has spent years working on low-level exploit analysis — published a mathematical analysis of Anubis in August 2025, before kernel.org had deployed it. His conclusion: the math doesn’t work, and it cannot be made to work. A free-tier Google Compute e2-micro VM can calculate roughly 2 million SHA-256 operations per second. At the time he wrote the analysis, Anubis had 11,508 GitHub stars — roughly the upper bound on deployed instances. Mining tokens for every single Anubis deployment in the world takes that VM about six minutes. The total compute cost to scrape every Anubis-protected site on the internet for a week: approximately zero dollars. “I don’t think we reach a single cent per month in compute costs until several million sites have deployed Anubis,” Ormandy wrote.

The structural problem is permanent. Any challenge difficulty that stops a free VM also stops a human on a slow phone. Any difficulty that a mobile device can solve in a second, a scraper’s datacenter can solve in microseconds. There is no equilibrium difficulty. The arms race is not winnable at the challenge layer, because the adversary has more compute than the defender and gets productive output from every request.

Thomas Ptacek — security consultant at Trail of Bits and co-founder of Latacora — made the structural argument cleanly in the Hacker News discussion that reached 1,340 points and 691 comments: “Proof of work makes sense for a password hash, where any one guess at a password provides zero marginal utility. But every request from a scraper is productive to the scraper.” A password guesser gets one shot per hash. A scraper gets one kernel commit per hash. The incentive structure is not analogous.

What Open Infrastructure Actually Costs

Ryabitsev’s response to all of this is measured but specific about where it leads. Kernel.org is going to disable features. The URL surface that makes cgit crawlable — arbitrary diffs, blob views, historical snapshots — will shrink. “Expect to lose some functionality, at least when accessing our resources anonymously,” he writes. “Trust me, we hate it just as much as you, but at this point it’s a necessity.”

This is the consequence that gets underplayed in most scraper-impact reporting. Kernel.org is not suffering a denial-of-service attack. The site still responds. The damage is structural: open infrastructure that was designed around the assumption that making things accessible was a shared good is being pushed toward restriction because the people consuming it at scale are not sharing the costs. The 2% of legitimate traffic pays nothing. The 98% that is scrapers pays nothing. Kernel.org pays 14 cores running indefinitely.

The pattern extends beyond kernel.org. In the same Hacker News thread, a maintainer of a separate public cgit instance reported passing one million hits per day despite hosting projects with almost no user base, and described their current response: returning HTTP 402 Payment Required for diffs, blame views, snapshots, and historical commit access. They called it “my total defeat.” Another developer described a consumer app with 500,000 public pages watching its daily active user count jump 100x from bots, leaving a message in 403 responses asking scrapers to contact them for data access. Nobody has ever responded.

Ryabitsev commits to offering everything as a download to anyone who asks. He has not changed the policy that made kernel.org valuable in the first place. But the mechanisms through which that openness is expressed — the web interface, the anonymous access, the full URL space of a cgit deployment — are contracting. The open model is being maintained at the policy level and degraded at the implementation level, one disabled endpoint at a time.

There is a fix that would work. Cloudflare has been piloting a feature that charges AI crawlers per request — inserting a billing layer between the scraper and the infrastructure it consumes. That model at least makes the economics honest: the entity that benefits from the data pays some fraction of the cost of delivering it. Kernel.org cannot implement this. It is a nonprofit volunteer-run infrastructure project that offers everything for free on principle. Charging scrapers requires a billing infrastructure and a legal framework that Ryabitsev’s team did not sign up to maintain.

The AI training pipelines consuming kernel.org did not ask for permission, did not evaluate the most efficient access method available, and have not offered to share the cost. The open infrastructure model assumed good-faith actors who would use the most efficient path available when one was offered. That assumption was wrong. The result is 14 cores that the kernel project didn’t choose to spend, on data the scrapers could have taken for free another way.

AI-generated editorial illustration · TemperatureZero · September 1, 2026