A correction, an upgrade, and a 27-hour flood
A follow-up to "How ClaudeBot fell in love with The MIRE/C³"
Last time, I told the story of how ClaudeBot found the one door robots.txt left open on cfd.mire.cc and spent weeks pulling tens of thousands of randomly-named files out of a directory listing that regenerates itself on every request. The IP verification was solid, the robots.txt analysis was solid, and the core finding — a major AI crawler locked in a self-refreshing maze, entirely within the rules — still stands.
The attribution of the volume, though, needs a public correction. I went back through the MIRE's raw logs, this time splitting every request by verified actor instead of stopping at "the ramp is genuine ClaudeBot." Three things fell out. ClaudeBot had company the whole time. A different crawler entirely was running up the numbers on the neighbouring vhost. And in the middle of all of it, The MIRE/C³ quietly changed its own economics — right before the biggest flood it has ever absorbed.
The correction: two crawlers were wearing one name tag
Here's what the per-actor split of cfd.mire.cc actually looks like.
SemrushBot got there first. It found /uploads/ in late May — 1,528 requests on May 23rd, 3,968 on May 26th — and was already grazing at 5,800–6,900 requests a day by June 6th. The previous post described June 9th as a 568-hit day of calm before the cliff. The logs say June 9th was 6,041 requests — 5,976 of them SemrushBot. When ClaudeBot arrived on June 10th, it didn't discover a quiet trap; it joined a party already in progress.
The binge was a duet, roughly 50/50. ClaudeBot opened with 4,874 requests on June 10th and settled at 6,000–7,400 a day; SemrushBot matched it, day after day, for weeks. The host total sat between 9,000 and 14,000 requests a day — the peak, June 15th, was 13,974. That's the whole binge. There was never an 18,000-request day.
So where did "sustained 18,000–24,000 a day" come from? I counted every request twice. The MIRE writes two log lines per request — Request started and Request completed — and my query tallied both. Run the arithmetic on the real numbers: June 18th–30th, the host did 9,065–12,521 requests a day, which double-counts to 18,130–25,041 — precisely the "18,000–24,000" I published. The same doubling manufactured the headline cadence: ClaudeBot's actual rhythm was one request every 12–13 seconds; count each one twice and you get the "one request every six to seven seconds" I found so remarkable. It was half as remarkable. Every inflated number in the post traces back to that one query.
And the co-star? Two misses stacked. First, I verified that the ramp was ClaudeBot (true) without asking what fraction of the ramp was ClaudeBot (about half). Second, Semrush crawls from two different published address ranges — and only one of them pattern-matches the mental model of "SemrushBot's IPs." The other half of its fleet sat in my "miscellaneous" bucket. User-agent strings are a suggestion, not an identity — I wrote that last time. It turns out attribution has the same failure mode one level up: verifying who is here is not the same as measuring who did what.
I've since cross-checked all of this against a second, independent source — the CDN edge logs in front of the MIRE, for the days that survived log rotation. They agree with the corrected numbers to within a few percent: ClaudeBot 6,283–6,905 requests a day at the edge on the fully-covered days, all /uploads/, all from Anthropic's published range. The edge logs held one last embarrassment, though: the only traffic source bigger than the bots was /livelog — the MIRE's own live-log dashboard, polling once a second while a human sat watching the bots. Two of the biggest traffic days in the trap's entire edge history, 68,000 requests each, were the observer, observing. It felt only fair to include that, given the theme of this post.
And the fabricated breach data? Semrush ate more of it. After the June 17th canary upgrade turned the .sql files into ~4MB fake dumps, the directory shipped roughly 214 gigabytes of synthetic breach data in about two weeks. Verified split: about 98 GB to ClaudeBot, about 117 GB to SemrushBot. The bot I named in the title came second at its own feast.
The upgrade: the MIRE changes its own terms of service
While the two of them chewed, the trap's economics were being rewritten in three steps.
Around June 21st, the tar got deeper. The hold-the-connection delays on the CSV and ZIP canaries were raised from a ceiling of about 15 seconds to about 30. Neither crawler reacted. The requests kept coming at the same rate; each one just spent longer stuck to the flypaper.
On July 3rd, the meals became crumbs. The MIRE started intercepting AI-crawler user agents and serving them a text-only payload — under a kilobyte — instead of the multi-megabyte fakes. The rollout is visible in the logs: the first AI/crawler UA intercepted lines appear midway through July 2nd (1,042 ClaudeBot requests caught that day), and by July 3rd every AI-crawler request got the crumb. The effect on request volume: none whatsoever. ClaudeBot and SemrushBot together kept firing ten-thousand-plus requests a day into the maze — 6,387 and 4,067 interceptions respectively on July 4th. The effect on bandwidth: 10.7 GB served on July 1st, 8.5 GB on July 2nd, 42 MB on July 3rd. A 99.5% cost reduction, invisible to the visitor, because the visitor was never reading the files anyway — just collecting them.
At the same time, the MIRE got better at holding many hands at once. Serving thousands of deliberately slow responses a day means keeping dozens of connections parked in tar simultaneously, and doing that badly is a self-inflicted denial of service. Thread-efficiency work that landed on July 3rd made slow-serving cheap at scale: the average hold went to ~20 seconds, 96% of MIRE responses were held past the 10-second mark, and the box didn't blink.
Taken together: same trap, same eager visitors, but the cost per request served dropped from megabytes-and-seconds to bytes-and-their-seconds. The payload stopped being data and became time.
The timing of all this turned out to be either very lucky or very well-earned, because about fourteen hours later, a third crawler kicked the door in.
The flood: AhrefsBot brings a fleet to the neighbouring vhost
While writing the first post, I'd been watching a steady wall of roughly 250 requests a day against the MIRE on mire.cc itself — and mentally filing it under the same ClaudeBot saga. It was never ClaudeBot. On mire.cc, ClaudeBot is close to a model citizen: 257 requests in 27 days, more than half of them just re-reading robots.txt and the sitemap. The wall was AhrefsBot, patiently grazing a swamp of synthetic pinyin tag pages — /tag/, /vodsearch/, /video/, thousands of machine-generated URLs that exist only as bait — at a median of 248 requests a day, every day, all of June.
Then, on July 3rd at 14:00 UTC, its scheduler apparently decided the backlog deserved priority. Ten requests in the 13:00 hour. 1,091 in the 14:00 hour. A peak of 1,798 — one request every two seconds — and it held four figures an hour for 27 hours straight: about 32,400 requests from 561 distinct IPs, every one of them carrying Ahrefs' cryptographically signed bot-authentication headers. This was the real crawler, at full fleet strength, chasing 35,000+ unique URLs that lead nowhere except to five more URLs each.
And it hit armor that was one day old:
- Every request answered 200 — never a 404, never a reason to stop.
- Every response the new text-only crumb: the entire 27-hour flood cost about 30 megabytes of bandwidth. On June's configuration it would have been tens of gigabytes.
- Every connection held an average of 19 seconds, capped near 30. Total time billed to the fleet: roughly 170 crawler-hours in a little over a day. That meant holding five-ish connections in tar continuously, around the clock — exactly the load the thread-efficiency work had just made trivial.
At 17:00 UTC on July 4th the flood collapsed. Ahrefs didn't leave; it dropped back to grazing — still several times its old baseline — and by the morning of July 6th it was climbing again. The swamp doesn't end, and it appears to know that now, and to be strangely at peace with it.
What I'm taking from this
Verify the actor, then split the volume. IP-verifying a user agent tells you the traffic is real. It does not tell you the traffic is theirs alone. If I'd broken the June numbers down per verified actor instead of per suspicion, the first post would have had two names in the title.
Attribution doesn't travel between hosts. Once "ClaudeBot is eating the MIRE" became the story, a 250-a-day wall on a different vhost got charged to the same suspect without a hearing. The logs acquit it — there, at least.
Three crawlers, three temperaments, one shared blind spot. ClaudeBot: one IP at a time, metronomic, weeks on end. SemrushBot: two fleets, first to arrive, biggest appetite. AhrefsBot: weeks of patient nibbling, then 561 machines for 27 hours. All three fully robots.txt-compliant. Not one of them able to recognize that "an infinite directory of freshly-named files" and "the same page forever" are the same place. The letter of the rules, followed all the way into the tar — and now the tar is nearly free to serve.
The economics have flipped, and that's the real ending. In June, being crawled by enthusiastic bots cost gigabytes a day. Since July 3rd, the same enthusiasm costs megabytes — while costing the crawlers hundreds of machine-hours. I set out to answer "does ClaudeBot behave well?" and ended up somewhere more useful: it doesn't much matter which bot is at the door, if the door only ever opens onto a kilobyte and a thirty-second wait.
The Crawl-delay experiment from last time is still on the list. It may be moot: the MIRE now is the crawl delay.