Terrarium Testing

Your mocks inherit your misunderstandings

Here is a bug I have shipped, and you probably have too.

Service A sends a payload to Service B. A’s tests pass. B’s tests pass. The integration suite, wired up with WireMock, passes. It deploys, and something downstream is missing a field.

The reason this class of bug survives a full test suite is structural, not sloppiness. A mock encodes what you believed the other service does. You wrote the belief into the code and then you wrote the same belief into the mock. If the belief is wrong, the mock is wrong in precisely the way that makes the test pass. Your test suite and your bug are siblings.

The only thing that catches a mutual assumption is putting the real components in a room together and making them talk. We’ve always known that. We just haven’t done it much, because building the room was expensive.

That’s the part that changed.

The pattern

Stand up the actual services. Point them at each other. Then hand the whole assembly to an agent that knows what the system is supposed to do, and let it go hunting — with the source in front of it and permission to change it.

I’ve been calling it terrarium testing: you build a small sealed world, populate it with the real inhabitants, and watch what they actually do.

Compared to what you already run:

• Integration tests verify a list of cases someone wrote down in advance. The list is the artifact, and it holds what its author thought to put there.
• Fuzzing generates inputs syntactically, from the shape of the input alone. What a tenant ID means, and why two requests only matter out of order, are outside its view.
• Terrarium testing generates them semantically. An agent that has read the code forms a hypothesis about what the author assumed, then constructs the specific input that would violate it. Nobody enumerates the cases up front. What’s specified is a situation and a stance.

This is exploratory testing, in the Cem Kaner sense — simultaneous learning, test design, and execution, with each step chosen based on what the last one revealed. That practice has always had the best yield per hour and the worst scalability, because it required a skilled human’s full attention. That constraint just went away.

Two things are new, though:

The agent can modify the subject. A human exploratory tester observes. An agent can comment out the retry to see if the deadlock disappears, add a log line at the suspect branch, bisect by ablation. That’s diagnosis, and the output is an explanation with evidence rather than a ticket that says “sometimes returns 500.”

It explores semantically. It’s read the resolver, the schema, the validators. It knows which integration point matters and which is boring. It aims.

Why the subject has to be local

The first objection is that you already have a place where the real services run together, and it’s called the dev environment.

Exploration doesn’t survive a slow loop. It’s a search, and how many hypotheses you test is a function of what one cycle costs. At fifteen minutes — commit, pipeline, deploy, wait — you test two ideas and rationalize skipping the rest. At thirty seconds you test sixty, including the stupid ones, which is exactly where the field-dropping bugs live.

You can’t experiment on something you can’t change. Every one of those diagnostic edits is a write against a running process, and dev doesn’t allow it. Each hypothesis has to be laundered through a commit and a deploy first, and at that price people stop forming hypotheses and settle for reading logs.

Nobody else is in it. No half-deployed branch of someone else’s, no stale sidecar, no feature flag still on from an experiment last quarter — rebuilt-from-known-state is what makes a red result trustworthy. And because it runs from a working tree, a coordinated change across three services gets exercised before any of them merge, rather than after you’ve committed to the design.

Dev is still where your deployment gets tested. Ingress, IAM, TLS, secrets, resource limits, real data volume, a third party’s actual sandbox — none of that is in a terrarium, and plenty of incidents happen there. But dev has been carrying a job that isn’t its own: it became the place you found out whether the services agreed, back when there was nowhere else to find out. Take that question local and you leave dev a narrower one, so a failure there tells you more than it used to.

How you actually do it

Two sessions. In the first you build the world by hand. After that you just ask for it.

Session one: build the world by hand, then write it down

Open an agentic session in a workspace containing the repos. Then, roughly:

You: Figure out how to run backend-service and graphql-service locally. graphql-service should point at the local backend-service.

The agent goes off, reads the READMEs, finds the run commands, gets both processes up.

You: Now make a GraphQL query and confirm it hit backend-service.

This step is the actual milestone, and it’s the one people skip. Two green health checks mean two processes are running. A request that traverses the whole path means the wiring is real. Insist on it.

The agent tries, and hits a wall — it needs auth.

You: Use our synthesize-local-jwt script to get an auth token.

It does, and the call goes through.

That interruption is what the first session is for. The local lore that isn’t written in any repo — which script mints a token, which migration has to run first, which env var nothing works without — is the expensive part of standing up a terrarium, and it’s exactly what the agent cannot discover on its own. Every time you have to step in, note it. Those interventions are the outline of your runbook.

You: Write a markdown doc with everything necessary to do this again.

You: Now generalize that into a local skill.

Runbook first, then the skill distilled from it. Keep both. The skill is what gets loaded and executed; the runbook is what someone reads in six months when auth changes and the skill breaks, so they can repair it instead of rediscovering everything from zero. It should record the dead ends too — what the agent tried first and abandoned, which ports and variables actually mattered.

None of this is configuration. It’s the hour you’d spend onboarding a new engineer, and it’s the same material — the things nobody wrote down because everyone already knew them. You’re teaching your agent skills. What’s different from onboarding a human is that the teaching persists, and in a form the next human can read too.

It’s worth having the doc note what the terrarium doesn’t contain: which dependencies stayed mocked, what state had to be seeded. A working setup looks complete from the outside, and the next person will otherwise over-trust a green run.

Session two: invoke it against a change

Later, in a session where you’ve just built a new endpoint across both services:

You: Use the terrarium-testing skill to make sure this works end to end.

It stands the topology up from the skill and starts working:

Agent: Both services up. A query through graphql-service reaches backend-service — wiring confirmed.

Happy path works. The mutation writes, the query reads it back.

Widget.description is String! in the schema. Checking whether the service can actually produce one every time.

WidgetService.create() doesn’t require a description and the column is nullable. Creating a widget without one.

Querying it. The whole widget comes back null, not just the field — the non-null violation propagates to the parent. Any client selecting description loses the entire object.

Confirming it’s the schema and not the resolver. Making description nullable in the SDL, restarting graphql-service.

Same query. Widget comes back. Reverting the SDL.

Nobody wrote that test because nobody thought there was a case to write, and every mock of that service returned a description — because the person who wrote the mock believed descriptions were always there.

Note what pointed it there: the context, not the skill. The skill is a world-builder and stops there on purpose: it says how to stand the topology up and nothing about what to try. The schema was in front of the agent because the happy path needed it, and a String! on a field the service doesn’t always fill is the sort of thing that stands out once it’s on the screen. Two ingredients: one is written once and reused, the other is different every run.

What the terrarium added isn’t the noticing. An agent reading any codebase produces a dozen that looks wrong reactions, and most of them are noise. What changed is that a hunch can be settled in thirty seconds by creating a widget and querying it back. Make sure this works end to end is a small enough instruction to leave room for that. What you leave unspecified is part of the specification.

This is why the pattern is strongest right after you’ve made a change — you’ve just spent an hour deciding what the thing was supposed to do, and all of it is still loaded. It still works cold: a disagreement between two components is observable without knowing which side is right, and an agent that reads the repo’s own guidance can assemble a good deal of the intent on its own. What no amount of reading supplies is the endpoint that coherently does the wrong thing: the filter that excludes archived records because that’s what the code says, when the ticket said include them. Reading code tells you what it does, not what it was supposed to do. The fix is cheap: bring the intent in — you wrote it, or you paste the ticket, or you point at the issue.

Sometimes the ticket is the whole input. Point the same skill at a bug report instead of a change:

You: Use the terrarium-testing skill and try to reproduce the bug in JIRA-1234.

Same world, different stance. Failing to reproduce is also a result — the terrarium holds all of your code and none of your infrastructure, so a bug that won’t appear in one has already told you something about where it lives.

Ask for a report, not a verdict

A run ends with the agent telling you whether the thing worked. That’s the least of what it learned, and the terrarium is about to be thrown away, so the report is the only part that survives. Ask for it deliberately.

Three things go in it.

What it found. For anything it found, the agent can write the test itself. A bug that’s been reproduced against a real topology already has everything a regression test needs — the setup, the sequence, the assertion. The terrarium is disposable. The test isn’t.

What it tried. Not a coverage report. A list of concrete situations — this sequence of calls, this payload shape, this ordering, this state left over from the previous request — each with the reason someone thought it was worth trying. This is the thing downstream QE spends its first day inventing: real input combinations that were actually exercised against a real topology. A human QE can take it as a starting point rather than a blank page; an automated suite can promote the interesting ones into permanent tests. Either way the expensive part of test design has already been done by something that had read the source.

What it couldn’t reach. The most valuable part, and the part you have to ask for by name, because a run doesn’t volunteer its own limits.

• What the terrarium structurally couldn’t exercise. Real third-party behavior, production data shapes, timing under load, anything that stayed mocked or absent. “I could not test the payment provider’s actual 402 semantics — I stubbed it, and here’s the response I assumed” directs human attention far better than fifty green checkmarks.
• Where the requirement was ambiguous and the agent picked an interpretation. Those are the places a spec conversation is cheaper than a bug.
• Which enumeration was walked — the behaviors the requirements list, the branches the code contains, or both — so the reader knows what “done” meant.
• How to rebuild the topology, which by this point is just a pointer to the runbook.

A report that ends “here is what I tried, here is what I couldn’t reach, and here is where I was guessing” is a document someone can act on.

What you get

Not a deploy gate. Nobody should read a terrarium run as proof of correctness.

The cost is almost entirely one-time. The first terrarium takes an hour, most of it spent discovering that Service B needs a migration and a seeded credential. Terrariums two through fifty take a couple of minutes, and none of it is your attention. The difference between those two numbers is a markdown file.

That buys a change in what you can honestly claim when you open the PR. The default is: I believe this works because the tests I wrote pass — tests that share an author, and therefore a blind spot, with the code. Afterwards it’s: I watched this work in a real topology, exercised by something that could invent cases I wouldn’t have thought of.

That’s a smaller claim than “verified.” It’s also a much better place to be than finding out in staging on Thursday.

Go back to the bug at the top — Service A, Service B, the field that goes missing. A terrarium doesn’t guarantee you catch it. What it removes is the thing that guaranteed you wouldn’t: the mock that agreed with you. Both services are up, they’re talking to each other, and something that has read both sides is trying to make them disagree. Your test suite and your bug are no longer siblings.

Fractal mountainscape from Xmountains, a terrain generator by Stephen Booth.