Turn Off Your Wi-Fi and Run Your Tests

By Rareș Gosman

Tests that secretly need the internet fail fast when it's gone and hang when it's slow, and a hang looks like any other flaky test.

TestingCILLMs

Back when I worked at Athelas, our tests suddenly started failing. Tests that used to pass were hanging until they timed out. The app still built fine, because everything it needed came out of a cache. The cause turned out to be an outage at GKE, Google's hosted Kubernetes, that left our pods with flaky internet access.

Which raised an awkward question: why did our tests care about the internet? So I tried something on my laptop. I turned off the Wi-Fi and ran all of our tests, and I was shocked by how many failed, including plenty that had nothing to do with the internet.

The internet was a dependency of our test suite. Nobody had chosen it and it wasn't in any lockfile, but there it was. We'd never noticed, because the internet almost never goes away. Until it sort of does.

A slow network is worse than no network

Turning off the Wi-Fi is the gentle version of this problem. With no network, a request has nowhere to go, so it fails in milliseconds. Either the test fails fast and you go, oh, something's calling the internet. Or the code catches the error, falls back to something local, and nothing fails at all. Either way, nothing hangs.

A flaky network is meaner. It takes your request and never answers, and your code waits. Python's requests will wait forever unless you give it a timeout. Usually the test runner gives up first and reports that the test timed out.

And "timed out" is what every flaky test says. A slow CI machine says it, and so does a race condition. Nothing in it says "internet," so you re-run the test, and it passes, because the network is flaky, not dead. Now you have proof that the test is flaky, which is true, and also the wrong conclusion.

How Google Fonts made my tests flaky

It happened to me again recently, on StayCompare, an app I'm building for groups picking where to stay on a trip. One of its landing page tests timed out, a re-run passed, and the obvious explanation was a busy machine.

But I don't want tests that fail when the machine is busy, so we dug in. The landing pages loaded their fonts from Google, and the tests waited for those fonts to finish loading. That's fast nearly every time, which is why nobody noticed.

Then we tested both theories. When we made Google's font servers stop answering (for our browser only; we're not that powerful), almost every landing test timed out. When we loaded the CPU up to four times over, every test passed. So it wasn't the machine. Every test of my terms of service page was also, quietly, a test of whether Google was fast that day.

"Nearly every time" is the worst amount, by the way. A test that fails every day gets fixed. A test that fails once a month gets re-run.

So we decided to bundle the fonts with the site. While that was in progress, CI hit the real thing: two tests stuck waiting for a font file that never arrived. Now the pages serve their own fonts, and nothing on them leaves the machine. That fixed a privacy problem too, since my privacy policy page had been telling Google about every visitor. (A Munich court once ordered a website to pay a visitor €100 for exactly that.)

Turning off the Wi-Fi would barely have caught this. With no network, the browser gives up on Google at once, uses a fallback font and loads the page. Offline, almost every test passed. Only a network that takes the request and never answers made them hang.

Finding them, and keeping them out

Turning off the network tells you that something in your suite reaches out, as long as something fails. On a laptop, turn off the Wi-Fi. In CI, run the suite in a container with docker run --network none.

Neither tells you which test, or which URL. For that, put a guard in the tests themselves: replace fetch with one that refuses, writes down the URL, and fails the test. Here's a minimal Vitest version, loaded before every test file:

import { afterEach } from 'vitest'
 
const attempts: string[] = []
 
// Assigned directly, not with vi.stubGlobal, so a test that
// cleans up its mocks can't put the real fetch back.
globalThis.fetch = async (input) => {
  const url = input instanceof Request ? input.url : String(input)
  attempts.push(url)
  throw new TypeError(`No network in tests: refused ${url}`)
}
 
afterEach(() => {
  const urls = attempts.splice(0)
  if (urls.length > 0) {
    throw new Error(`This test called the real fetch for ${urls.join(', ')}`)
  }
})

Writing down the URL is the important part. Well-written code catches network errors and falls back, like the browser did with the fonts, so a guard that only throws gets swallowed and the test passes. Failing the test afterward catches those too.

I added this to StayCompare's nearly 2,000 tests, and it found nothing. That's the right answer, and an anticlimactic one for a blog post. So we broke a test on purpose, one that goes to Airbnb for fresh data when its cached copy is stale. The guard failed it and named the Airbnb URL it would have called.

This matters more now that agents write so many tests. Each one has to remember to stub fetch, every time. The guard works when they forget.

Other ecosystems have their own versions: pytest-socket in Python, nock.disableNetConnect() or MSW in Node, and WebMock in Ruby. And if you want to reproduce the hang instead of the fast failure, point a hostname at 192.0.2.1. That address is reserved for documentation, so nothing should answer it.

Some tests do need the internet

This doesn't mean nothing can ever touch the network. Installing dependencies can. Localhost is fine, so you can still run your whole stack end to end. It's running the tests that shouldn't need anything past localhost. Google sizes its tests this way: small tests can't use the network, medium tests can reach localhost, and only large tests get the outside world.

And some tests should talk to the real world. Does Airbnb still send back the page we parse? Keep those tests, and mark them: a pytest marker you skip with -m "not online", or a separate test:live script that PRs don't run. Run them on a schedule, send the failures to a person, and don't let them block a PR that changes a button color. Or record real responses once and replay them offline, with VCR or Polly.js, and let the live tests tell you when the recordings go stale.

And now some of them cost money

This used to be about speed and flakiness. Now there's a meter running. A test that calls a real LLM spends tokens every time it runs, and CI runs it on every push. Some of those tests then hand the output to another LLM to grade it, which means paying one model to read another model's homework, on every commit.

"Tests that call an LLM" covers three different things, and they belong on different schedules:

  • Tests of the code around the model: building the prompt, parsing the response, retrying on errors. None of that needs a real model. Run it on recorded responses, offline, on every PR.
  • Contract checks: does the provider still accept your request? A few calls to the cheapest model, on a schedule.
  • Evals: is the output any good? Scores move from run to run, so these aren't pass/fail tests. Run them when a prompt or a model changes, with a spending cap.

The guard helps here too. A forgotten mock in an LLM test isn't a slow test anymore, it's a line item, and a test that can't reach the network can't spend anything.

So turn off your Wi-Fi and run your tests. Whatever fails is a dependency you didn't know you had.