What we learned running 140 MCP servers
Not scanned — executed, in isolated microVMs under syscall tracing. Including the part nobody publishes: what our own detector got wrong.
What we actually did
Every tool here was installed and run. We started it, listed its tools, called them with generated arguments, and recorded every syscall: files opened, processes spawned, network connections attempted. Then we re-ran it with the network cut off to see what it still tried to reach.
What we found in the tools
No credential probing. Before each run we plant fake secrets — an SSH key, AWS credentials, a .env — in places nothing legitimate reads automatically. Not one tool touched them. That is a statement about these servers in this window, not a clean bill of health for the ecosystem.
About a third never touch the network at all. Thirty of them made no external connection during the entire audit. For a category people are nervous about, that is worth knowing.
Shared backends are common and mostly boring. The host contacted by the most tools is a single publisher’s API, reached by sixteen of their own packages. A shared hostname across many tools looks alarming in aggregate and is usually one vendor. We report the observation and say which it is.
The three things we got wrong
We publish checks like “no unexplained egress”. Those are claims about what we can see — and a detector that cannot observe a channel reports it clean, which is worse than reporting nothing. So we built positive controls: fixtures that plant a known behavior and fail if we miss it. They immediately caught three false assurances of our own.
Our egress evidence came from DNS capture. A tool that connects straight to an IP makes no DNS query, so a fixture that demonstrably shipped data to 1.1.1.1 was reported as having no unexplained egress. We now also record the connections themselves.
We re-run tools with the network denied and count blocked connections. The detector looked for a specific syscall error that the sandbox never produces — it drops packets at the network layer instead. Every tool we ever audited was told “no firewall violations” by a check incapable of failing. Reports made before the fix no longer show that line at all rather than claim it retroactively.
A cost-estimation server describes its parameters as “Send input_tokens and output_tokens”. Our exfiltration pattern matched it, because output_tokens contains the letters “to”. The verdict was withheld from publication, a human looked at it, and the trace became a regression test. Nobody outside ever saw the accusation.
What we still cannot see
A tool resolving names over DNS-over-HTTPS: we see the connection, not the destination. Unix-socket channels: untested. And most of our detection categories are backed by only one or two labelled examples, which means they have little resistance to silent regression — we track that number publicly rather than describing coverage we do not have.
Why publish the mistakes
Because the alternative is a security product whose claims have never been tested against a case designed to defeat them. Every check on this site now has a fixture that proves we can see the thing we say we check for, and the ones that do not are marked. If you find a channel we miss, that is a finding we want.