What 7'434 user-agents taught us

MIRE is not a website. It sits behind twenty-eight of them, and it only ever sees what misses — anything the real sites return a 404 for gets handed to the maze instead. That makes it possible to ask a question you cannot normally ask: of everything arriving at these domains, how much is looking for something that was never there?

Twenty-nine days, 5 July to 2 August, both sets of logs complete:

	At the edge	Fell into the maze
Requests	873'011	367'692	42.1%
Distinct IPs	57'333	24'077	42.0%
Distinct user-agents	7'434	2'215	29.8%

Two out of every five requests to these domains is a miss. Not a typo, not a stale bookmark — a request for something that has never existed on any of them.

The user-agent line is the interesting one, because it moves differently. Only 30% of the strings we see ever end up in the maze, which means most of the variety on the internet is well-behaved and a small, determined minority generates the volume. 5'223 distinct user-agents — 70% of everything we saw — never missed once in 29 days.

If you rank user-agents by what proportion of their own traffic misses, you get a rather blunt document.

Miss rate	Edge	Maze	Who
94%	20'088	18'811	SemrushBot
85%	86'091	72'905	AhrefsBot
81%	8'237	6'632	Mozilla/5.0 (Windows NT 10.0; Win64; x64) — nothing else
72%	2'589	1'853	l9scan / leakix
67%	18'291	12'318	ClaudeBot
67%	2'760	1'836	SERankingBacklinksBot
63%	67'681	42'567	a generic Chrome string
62%	4'823	2'975	Go-http-client/1.1

SemrushBot spends 94% of its time here asking for things that do not exist. It is not crawling these sites in any meaningful sense; it is enumerating a guess list against them. AhrefsBot, 85%. Between them that is 91'716 requests in a month, essentially all of it wrong on purpose.

The single most common client hitting MIRE does not identify itself at all.

89'470 requests in 29 days — 24.3% of the maze — arrived with no User-Agent header. That is more than AhrefsBot. It came from just 331 addresses, which works out at 270 requests each.

The toll booth was a list of names. It matched GPTBot and ClaudeBot and SemrushBot and thirty-odd others, and it could not match the empty string, so a quarter of all traffic walked past it and collected a full, lovingly generated decoy on the way through.

That is now fixed, and it cost one line of code.

The subtler group had user-agents. Entirely ordinary ones.

Requests	Distinct IPs	Requests per IP
1'692	1'591	1.1
1'534	1'466	1.0
1'510	1'492	1.0
1'425	1'420	1.0

One request per address. Fifteen hundred addresses. One Mac Chrome string shared between all of them, then gone.

That is a residential proxy pool, spread deliberately thin so no single address ever accumulates enough history to look worth blocking. And the user-agent is not a clue — it is a real Chrome string that real people send every day. Any pattern sharp enough to catch it catches them too.

You cannot write a regex for this. The string is not the tell. The distribution is the tell, and no request carries its own distribution.

We widened the list anyway, because some of the gaps were embarrassing.

SERankingBacklinksBot is the same species as AhrefsBot and SemrushBot, both already on it. YisouSpider, DeepSeekBot, Google-CloudVertexBot, MistralAI-User, AI2Bot, Scrapy — all classes we already claimed to handle, merely spelled differently. Baidu, Yandex and Applebot joined them. Googlebot and bingbot deliberately did not; MIRE only works if scanners can still find it.

Total extra traffic captured by widening the list: 2'552 requests. 0.7%.

That is the uncomfortable finding of the month, and it is about our own July release. The mechanism we shipped as the headline feature turned out to be the least effective of this month's three changes, by a factor of thirty-five. The single line handling the empty string bought more than every name we added, several times over.

Maintaining a list of everyone who might lie to you is not a strategy. It is a subscription.

The thing we should have noticed sooner is in the shape of the traffic, not the strings.

Because MIRE only sees misses, the median IP address across 29 days made exactly one request. The 90th percentile made 19. And then:

4.46% of addresses account for 75.1% of everything.

Once that number is in front of you, the identification problem simply dissolves. You do not need to know whether the thing on the other end is a scraper, a scanner, a security vendor or somebody's broken cron job. You need to know it has been here 400 times and stopped learning anything new around request 50.

So MIRE now runs a decoy budget. The first fifty requests from an address get the full performance — the fake .env, the plausible admin panel, the SQL dump that takes eleven seconds to deliver. After that it drops to the same cheap notice the crawlers get. No classification, no list, no guessing about intent. A meter.

Replayed across the same 29 days:

	Requests	Share
Before — named crawlers only	105'065	28.6%
After — named crawlers	107'617
— no user-agent	89'470
— past decoy budget	105'473
After — total cheap path	302'560	82.3%
Still generating full decoys	65'132	17.7%

Four fifths of the maze now costs almost nothing to serve. The expensive fifth is the fifth that is still telling us something.

We widened the list anyway, because some of the gaps were significant.

Total extra traffic captured by widening the list: 2'552 requests. 0.7%.

Maintaining a list of everyone who might lie to you is not a strategy. It is a subscription.

The budget has one deliberate hole: POST requests are exempt.

A brute-force attempt against a fake login is the most valuable thing that happens here. It arrives as a POST, it carries a username and a password, and both go straight into the credential log — 4'241 pairs in these 29 days. Cutting someone off at fifty requests for being repetitive would be precisely the wrong call when the repetition is the payload. The meter only runs on GETs.

Everything else that makes MIRE worth operating still runs at full cost for that first fifty. Over the same 29 days: 131'817 canary tokens issued, 116'802 generic PHP traps, 1'940 fake logins, 1'429 upload handlers, 972 installers, 646 traversal decoys.

We did not get cheaper by doing less. We got cheaper by stopping doing it to audiences that had already seen the whole show.

Three changes went out this month, in ascending order of how much they actually mattered:

#	Change	Captured	Share
1	Widened the named-crawler list	2'552	0.7%
2	Requests with no user-agent take the cheap path	89'470	24.3%
3	Per-IP decoy budget, fifty requests, GET only	105'473	28.7%

The one that took the most work returned the least. The one that took a single line returned thirty-five times more. That ordering is the release.

But everything in that table is a replay against historical logs, not an observation.

The change deployed at 11:10 today, which zeroed every per-IP counter, so the production record so far is 610 requests. In those: 198 no-user-agent interceptions, 29 named crawlers, and the decoy budget firing three times — first at 12:14:53 against an Azure address on ciso.li that had just reached its fifty-first request, then again on mire.cc an hour later. It works. It fires exactly where it was told to.

The live cheap path, though, is running at 37.7% — not the 82.3% the replay predicts — and that gap is the honest part. A budget only bites once addresses have accumulated history, and every counter on this box is three hours old. Of the two components, the one that needs no memory — the empty user-agent — is already carrying 32.5% on its own. The one that needs memory has produced three hits. Whether it converges on the replayed number over a full month is the thing to check, not something this post can claim.

Two other things are on the watch list.

The IP table holds 50'000 addresses. Inside a 29-day window the maze sees 24'077, comfortably under it — but across six months it has seen 92'988, so over long horizons eviction is real and counters do get reset. It works in our favour, since the cache drops least-recently-seen first and a repeat offender is by definition recently seen, but the budget is best-effort rather than a guarantee.

And 4.46% is this honeypot, on these domains, over these 29 days. It is not a law of nature.

MIRE/C³: Multi-layer Intrusion Response Engine

What 7'434 user-agents taught us

MIRE/C³ — August 2026 Release

Two in Five

The miss list, by name

The toll booth caught 28.6%

The most popular user-agent is no user-agent

One Chrome, one thousand five hundred addresses

Adding names to the list barely helped

So we stopped asking who they were

Adding names to the list barely helped

What we still pay full price for

What shipped, and what we're watching

What the scanners told us this month