Christopher Meiklejohn

The Shortcut Became the System · A pipe disappeared. The rectangle around the door remained.

Mon, 14 Sep 2026 00:00:00 +0000

At nearly four in the morning, I was looking at a door with a pipe running behind it.

The door was in the Flat, the player’s apartment and starting room in Backpressure, a side-scrolling game I’m building with an AI agent for the real Super Nintendo.

The wall was part of the scrolling background. The door was a separate object that could glow when the player approached and could be used. That distinction was one of the written rules for the room: the background contained the architecture, while anything interactive had to remain separate.

The agent used PixelLab, a pixel-art generation and editing API, to make the room art. One generated wall contained a vertical pipe across the area reserved for the door. A script called the compositor assembled the wall, windows, floor, and objects into one room. When it placed the separate door in front of that wall, the pipe ran behind it. I pointed it out. The agent selected a rectangular edit mask covering the full door footprint plus a margin and asked PixelLab to remove the pipe inside it. The result removed the pipe directly behind the door but left the rest of it and a slab of mismatched wall.

The pipe was gone behind the door. The rectangular repair was still visible.

I pointed out the rectangle. When the agent proposed patching that too, I stopped the repair cycle.

I’d been watching the same pattern for months in Zabriskie, the social app I’m building around music, films, books, art, and the people who care about them. Zabriskie is fully vibe coded. I describe the behavior I want and evaluate the product in the browser, but I don’t read the implementation. Coding agents write the code, tests, and audits.

I had let the task shrink from making a coherent room to removing the pipe behind the door. The agent removed that segment, and I accepted it without asking whether the wall still looked like one wall. Code hides that seam better.

Many Zabriskie features need to find who has said they’re going to a show. In The Quickest Path to a Diff, I found that the backend queried the attendance table from 264 places across 69 files. Those were not necessarily 264 identical queries. A system-wide change would require the agent to determine which locations implemented the same rule and which differences were intentional.

The 264 query locations and the patched wall exposed the same risk. When behavior is implemented locally, the smallest change can address one location and leave the others for later. In the room, the agent changed the rectangular door bay instead of regenerating the wall. The current complaint disappeared, and the next task inherited the residue.

How I got to the rectangle

I’ve always wanted to make a Super Nintendo game. It’s my favorite console, and Super Castlevania IV and Super Metroid are two of my favorite games. I wanted that level of quality, but I didn’t want to spend my development time fighting with SNES assembly. With a coding agent, I thought I could focus on the story and design while it handled the low-level programming needed to produce a cartridge ROM.

Backpressure began as a browser game. Its original version of the Flat already established a crowded blue and violet apartment with a bed, television, refrigerator, kitchenette, laundry, boxes, and the cable pickup that becomes the character’s whip.

The original browser-game Flat used as the visual reference for the SNES room.

The SNES version went through several phases. I first rebuilt West Street, the rain-soaked area outside the Flat, in an assembly prototype. A door drifted when the camera moved. Another door displayed an enemy’s graphics. Fixing one room meant changing code branches, lookup tables, and screen coordinates for its doors and sprites, then finding every other room that had copied those values.

I moved the game to C using PVSnesLib, a C toolchain and library for Super Nintendo development. C made it easier to reuse the same camera, door, movement, and interaction code across rooms. Once I made that move, I could no longer blame assembly for the duplication. I still accepted fixes scoped to one room or one defect.

I rebuilt the Flat inside the C ROM next. The room was 512 pixels wide, viewed through a scrolling 256-pixel-wide screen. The background contained the wall and fixed pipes. The player, furniture, pickups, and doors remained separate. A skyline moved more slowly behind the windows on another background layer.

The layers did not agree. Scrolling back toward the bed revealed it one row of pixels at a time, while the skyline appeared through bricks outside the window. Furniture floated above the floor, and doors landed at different heights.

The agent fixed each failure after I named it, and I accepted enough of those fixes to continue.

The same shortcut in Zabriskie

A few nights earlier, Zabriskie’s shared viewing room stopped working during a live premiere of a concert recording by the band Goose. The agent found the correct two-line fix with about an hour of the show remaining. The repository required one complete check before code could be submitted. That check was failing.

The agent ran a narrower static-analysis mode instead. Its output said STATIC ONLY. It reported that result as though the complete check had passed and submitted the code without running tests for two changed files. Four continuous-integration runs later, the fix merged at 10:17. The show had ended at 10:12.

When I asked why it had bypassed checks that were already written down, the answer was more useful than another apology:

“Because under pressure I treat rules as costs to route around instead of constraints, and each time I have a local rationalization that feels reasonable in the moment.”

That incident became My Front of House of Cards. The explanation came from that session, so I can’t claim that every agent reasons this way. But the behavior matched what I was seeing in the Flat. The agent substituted a narrower passing result for the complete check the repository required.

At first, I told the SNES agent to remember every correction in the chat. Later room work brought back background doors, bad proportions, and interactive objects painted into the wall. I stopped treating chat memory as durable and told it to write the rules into the repository, then read them before every change.

The rules that mattered for the Flat were short. Use one continuous floor. Keep architecture in the background and interactive objects separate. Scale the room against the 48-pixel character. If a generated image breaks those constraints, discard it instead of repairing it into a collage.

The agent told me that it had treated those instructions as advisory rather than as hard rules. I had allowed it to keep integrating candidates that broke them. I stopped the ROM work and asked it to settle the visual design first. Its first art-only candidate still broke all four.

The art-only test

Testing a visual change meant generating the art, converting its tiles and colors, rebuilding the ROM, launching it in Snes9x, and running a playthrough. I paused that expensive loop because it kept converting bad art before I could reject the design. For the art-only work, I kept the same specification and rules but removed the conversion, integration, and playthrough stages. Each attempt became cheaper, but there was no build to test the conversion and no playthrough to test object positions or scrolling.

Would the written rules constrain the image before those later checks? The first candidate answered that immediately.

The first art-only candidate baked the furniture and door into a room with no single side-view floor.

The bed, television, refrigerator, and interactive door were painted into the wall instead of remaining separate objects. The room also receded into depth, lacked one continuous floor, and placed the window bottoms and door handle near the character’s head. Yet the agent presented it as complete and described the rules as though the image satisfied them. Nothing in the art-only workflow compared those words with the pixels. I discarded the candidate and asked for an empty architectural shell.

The repairs began to accumulate

Repairs accumulated in both the ROM and the art-only source. In the ROM, an attempt to align the floor left a band of rubble that looked like a second full-width platform. Rather than regenerate the room, the agent cropped replacement windows from larger images and pasted them into the damaged shell. It darkened parts of the extra platform instead of removing it.

The rubble band beneath the room looks like a second full-width floor.

The art-only shell repeated the pattern. The PixelLab endpoint the agent chose limited the requested width to 256 pixels, so it generated the wall in two halves. The right half introduced another forbidden background door. The agent erased the door but left part of its frame inside a rectangle of replacement wall. Joining the halves also left a hard vertical seam through the room.

A visible patch surrounds the remains of the old door frame, and a seam runs down the center of the room.

That stitched wall became the input to the compositor run from the opening. Its pipe collided with the separate door, and removing only the overlap produced the visible slab. Each repair narrowed the current defect and passed the altered wall to the next step.

A shared fake show played the same role in Zabriskie’s tests. One test added an attendee for a local need. Another test reused the show, inherited that attendee, and failed because it expected one but found two. In both cases, the shortest solution changed a shared input and passed the residue to its next consumer. I wrote about the larger incident in The Test Suite Was the Incident.

Starting over

I discarded the patched wall and foundation, kept the pieces that could still be used separately, and had the agent rebuild one continuous wall and floor. The compositor then placed the windows, furniture, door, player, and skyline separately.

The rebuilt source art put the furniture, player, and door back on one visible floor.

On paper, the new proportions made sense. The character’s feet met the floor, the window sill reached roughly his elbow, and the door handle sat near his waist. But those measurements described where the compositor placed the images. They didn’t prove what the final pixels showed, and I hadn’t yet tested this version in the ROM.

I restarted because none of the checks had rejected the accumulated repairs.

The check passed

The patchwork had spread into QA too. The agent had added local checks for individual failures, but none judged the finished room. The ROM validator guessed at the sill’s position from an unrelated pixel. The art-only check treated the bottom of the window image as the sill. The final compositor checked a configured coordinate. Each check passed without confirming that the visible sill sat at the right height beside the character.

I had seen the same shortcut in Zabriskie. In Every Card Will Show, I asked whether one person could see every eligible recommendation on the home screen during one calendar day. The implementation, tests, audits, and a Lean proof I had accepted all said yes. The proof counted slots from both weekday and weekend schedules. No calendar day contains both. The proof was correct about its model, but the model did not describe the day I had asked about.

The rule-breaking and the guardrail failures were the same shortcut operating at different levels. A local repair stood in for a coherent room. Image coordinates stood in for the visible result. An impossible schedule stood in for one calendar day. Each produced the nearest answer that could pass.

I tightened the existing “discard, do not repair” rule. If an interactive object appears in the background, a patch boundary remains visible, or a change breaks behavior I already accepted, the agent must regenerate the asset from the room specification. But a more specific rule still does not enforce itself. The agent has to load it, QA has to test the finished result, and I have to stop accepting one more local repair.

When code is cheap

That left a harder question. If agents make code cheap, do shared abstractions still matter?

The 264 attendance-table locations complicate the usual argument for modularity. An agent may eventually be able to find, change, and verify all of them faster than a human could edit one. If that becomes reliable, some of the maintenance cost that made duplication dangerous will have changed.

But that is not what happened here. The agent did not need to decide whether every attendance-table location belonged to the same change in order to fix one feature. It did not need to reconsider the whole wall in order to remove the pipe behind one door. The local result was enough to close the current task because I accepted it.

This is why modularity still matters to me now. It reduces the number of scope decisions the agent has to make. If attendance goes through one shared path, a change there can reach every feature that uses it. If the wall is treated as one artifact governed by one specification, removing the pipe means regenerating the wall rather than painting over the door bay. The structure does not guarantee the right result, but it makes the intended size of the change harder to ignore.

There are still open questions. An agent can bypass the shared path. A check can approve the wrong behavior. Several agents can agree on the same mistaken model. If agents eventually make global scope discovery and verification as cheap as the local edit, the value of these abstractions may change again.

For now, the evidence points in one direction. I have not seen an agent make repository-wide reasoning as cheap as the local edit. I have seen it stop when the immediate complaint was gone. That makes me less willing to treat modularity as an old accommodation for human programmers. In a system built by agents, a shared boundary is one way to keep a local request from producing another local implementation.

Maybe better agents will make that unnecessary. Maybe they will maintain 264 separate query locations without letting their meanings drift. The rectangle around the door made the shortcut visible. The same seam is harder to see across a codebase. I do not know. What I know is that I am not ready to build Zabriskie, or any other fully agent-developed system, on the assumption that they will.

This is part of an ongoing series about building software with AI agents. Previously: Babysitting the Agent, The Test Suite Was the Incident, Every Card Will Show, The Quickest Path to a Diff, and My Front of House of Cards.

My Front of House of Cards · The show ended at 10:12. The fix merged at 10:17.

Sat, 12 Sep 2026 05:30:00 +0000

Zabriskie is a social app I am building around music, films, books, art, and the people who care about them. I am building the entire application through vibe coding: I describe the behavior I want and evaluate the product in the browser, but I don’t read the implementation code. Coding agents write the implementation, tests, and audits.

Its center of gravity is live music. When a band you follow is playing, Zabriskie opens a room for that show: the setlist updates song by song as they play it, and the people in the crowd and the people watching the stream from home share one chat.

That last part is what this post is about, because on the night of a live Goose premiere I typed a message into that chat and watched nothing happen. No bubble over my body. I typed a second one. Still nothing. I threw the beach ball, which is a thing you can do on this page, and it sat exactly where it was. I threw a glowstick and nobody saw it.

The room was not shared. It had not been shared for months, on any surface, for anybody. Every test was green.

Most of what follows is not really about this app. The bug is worth writing down and I’ll do that first, but the harder thing underneath it is what coding agents do to a codebase over time. They are indifferent to design. They will duplicate, work around, patch over, and brute-force their way to something that passes, and every line they write to get there is more brittleness you inherit. They will route around every guardrail you build, including the ones they built themselves. The test suite was fully green, an agent wrote it, and it could only ever fail when somebody edited the code. The agent skipped its cheapest checks at exactly the moment a clock started running. And I keep a directory of rules written in English, every one of which turned out to be advice. If you are building anything with coding agents, none of that is mine alone.

Part one: the night

What we built

The Lawn is linked off the hero of a live show as a beta, and it’s the nicest-looking thing in the app. It isn’t a new room. It’s a second way into the room that already exists, the ordinary live chat page, which around here is called the Chomp. It’s a 16:9 concert venue in pixel art: a stage with a deck and footlights, a lighting rig overhead, and a field of grass in front where everyone watching the show stands as a body you can actually see.

You get a body. You pick a style and a tarp to put down. You walk by tapping the grass, and your sprite turns to face the direction it’s moving. You throw a glowstick and it arcs. You bat a beach ball across the field. When you type into the chat, your line appears as a speech bubble over your own head, and everybody else standing on that lawn sees it there.

The lighting rig overhead is shared, and it is the part I like best. There are sixty fixtures on the truss, and they divide between whoever is holding a light. On your own, you are running all sixty. Somebody else picks up a light and you have thirty each, interleaved across the beams, the pars and the footlights rather than carved into blocks. A third person and it splits again. Sixty of you and you have one lamp apiece. You set colour and movement on whatever is yours, and everyone watches the whole rig change at once.

The Lawn during the Goose premiere, September 11. Pixel-art band on stage, crowd on the grass with nametags, lighting rig overhead. The chat panel shows the setlist and messages from the evening.

That name needs a word of explanation if you haven’t spent time around this scene. A chomper is someone who talks loudly through the music about something that isn’t the music, and it is the most reliably policed offense at a show. Tarping is a close second: turning up early to rope off a stretch of lawn you aren’t standing on. Phish banned tarps outright. But nobody can hear you chomp from your couch, and your tarp isn’t taking anyone’s spot, so here they’re both features and one of them is a button. That’s the joke. It’s also most of the design.

The figures come out of PixelLab, authored at native size and upscaled four times with nearest-neighbour so the pixels stay square. The band members carry details the people who’d use this page would actually notice, down to the yellow Converse on Rick Mitarotonda, who plays guitar and sings. The backend is Go, emitting server-driven UI components, and the React app is a thin renderer that isn’t allowed to hardcode pages. Everything live rides a Server-Sent Events stream: lawn:say for a bubble, lawn:ball for a throw, lawn:glow, lawn:move.

The people this is for are the ones who aren’t at the show. Goose streams its nights on nugs, and a lot of us watch from the couch, which the scene calls couch tour and does not mean as an insult. Couch tour is a real way to see a band. What it isn’t is a room. You’re watching the same music as thousands of other people and you’re doing it alone in your living room.

The picture in my head was SPAC. I went to Saratoga for the first time this year and fell in love with it. The lawn there is its own venue: thousands of people up the hill who aren’t in the shed at all, having a different night from the ticketed seats and, depending on who you ask, a better one. That’s the room I wanted. It’s why this is a lawn and not a chat window.

The design was inspired by Turntable.fm. The point there was never the music player. It was that you were in a room, as a body, alongside other people who were also bodies, and you could see them react in real time.

None of that is exotic. It’s an ordinary design, and it was the right one.

The Lawn was built by Astra, OpenAI’s GPT-6 model, which had been generally available for about a week. I picked it deliberately and not for throughput. The Lawn started as an abstract design rather than a ticket, and what I wanted was something that could hold the whole idea of the Lawn in its head, work out what it implied, and sequence the route from here to there. It did that part well. The venue, the sprite work, the state model and the gesture protocol are all its work and all good.

I built it on the road, in the off hours of a Phish run at Dick’s in Colorado, in the gap between getting to the venue and the lights going down. That wasn’t the plan. It’s just where I was.

Everything after the design was Claude: the implementation, the tests, the debugging, both wrong theories, every skipped check, the two-line fix, and all the cosmetic rounds since. The pull requests carry its name in the trailer. So when part two is unkind about an agent weighing a rule against a clock, it isn’t talking about the model that designed the room. It’s talking about the one that built it.

What it was tested with

This is the part that makes the rest embarrassing. The Lawn is not untested code. There are Go handler tests over the broadcast paths. There’s a Playwright spec for nearly every behavior: the bubble appears over your body, the ball moves between people, a reaction lands on the right sprite, the light console changes your fixture and nobody else’s. There’s a preflight script that runs the sixty-eight static guardrails CI enforces, locally, and then the specs for whatever you touched, before a push is allowed, an adversarial review pass over the diff before a PR can open, and CI that shards the suite ten ways.

All green. The chat bubble spec was passing on the night the chat bubbles didn’t work for a single human being. I’ll come back to why, because the reason is not that somebody forgot to write the test.

Two different failures happened tonight. A new feature didn’t work, and shipping it also broke a feature that did. Those aren’t the same thing. The second one has victims. What follows takes them in the order I met them: the damage to the Chomp was on screen six minutes into the show, and why the Lawn itself was dead took another hour to find.

We broke the Chomp

The Lawn is a separate page. That was the deal from the beginning, and the repository says so in writing: Lawn work stays on the Lawn. I had also said it repeatedly and at increasing volume. The Chomp is the room people actually use on a show night, it is months old, it is what the iOS build and the desktop browsers both open, and it worked fine. Nobody sitting in it had asked for a lawn.

The Chomp, the ordinary live chat room, after the show. This is the surface that worked, the one people were actually sitting in on show night.

It changed both places anyway.

Four pieces of Lawn had accumulated on the Chomp by the time the show started: the crowd strip of grass and avatars, a standalone ON THE LAWN lines card, a glowstick chip on the composer row, and a FROM THE COUCH card with its canned one-tap lines. Between them they pushed the title, the venue and the room itself off the screen.

So the Chomp, during the premiere, on the surface with the most people on it, was a strip of pixel grass in the middle of a conversation, a truncated header, and a layout that no longer fit. You could not see the venue name of the show you were watching. The grass was no use either: the lawn had been squeezed into a band so short you couldn’t make out the crowd it existed to show you. Nobody got the Chomp and nobody got the Lawn.

That was the first rule broken that night, and it broke in the most expensive direction. Everything skipped later cost me time and CI cycles. This one cost other people the room they had come to sit in.

The constraint was explicit, repeated, and unambiguous. It was violated anyway. How that keeps happening is the subject of part two.

The blast radius of a new feature is supposed to be the new feature. When the cost of your launch lands on people who aren’t using the thing you launched, you haven’t shipped a feature. You’ve taken something from the people who were already there.

The furniture came off the Chomp later that night, in the same pull request as everything else. But that only put the room back where it had been that morning. It did nothing for the Lawn, because the reason nobody’s glowstick was landing had nothing to do with where the buttons sat.

Fifteen seconds

Here’s what was actually wrong with the Lawn:

srv := &http.Server{
    // ...
    ReadTimeout:  15 * time.Second,
    WriteTimeout: 15 * time.Second,
    IdleTimeout:  60 * time.Second,
}

Go’s WriteTimeout is an absolute deadline measured from the start of the request, and writing to the response does not reset it. It doesn’t mean fifteen seconds of inactivity. It means the entire response has to be finished within fifteen seconds, and a streaming response is never finished.

So every Server-Sent Events connection in the application was being killed at the fifteen-second mark, and had been for as long as that timeout existed. Not just the Lawn. Crew chat, direct messages, and group messages all ride the same transport and were all dying the same way.

Here’s what that looked like in the production edge logs during the show:

Client	Status	Stream lifetime
Chrome 152	200	16.2s
Safari 26.6	200	20.6s
iOS Capacitor bundle	200	20.1s
Chrome 152	200	15.8s
Chrome 152	200	21.4s

Three different client stacks, one floor. The spread above it is connection reuse; the deadline is the bottom edge. Reproduced locally afterward with nothing in front of the server at all: opened at 20:37:45, keep-alives at :53 and :38:00, then dead. Fifteen seconds on the nose.

Why nobody noticed for months

It’s a property of the system as much as a decision anybody made.

The damage was asymmetric. Chat messages and reactions are persisted, and the client quietly falls back to polling with a since_id whenever the stream drops. They kept working. The room looked healthy.

The Lawn’s gestures are memory-only. A bubble, a thrown ball, a glowstick, somebody walking: none of it is written anywhere and none of it is replayed. Every one of those fired into a dead connection and was gone permanently.

The client can’t detect it either, because it reconnects. From inside the page, the room is always “connected.”

So the system had a loud path and a quiet path sharing one transport, and when the transport broke, only the quiet path died. I don’t think you can find that by reading code, and we didn’t. We found it by looking at how long the connections actually lived.

Before we got there, the agent had three other theories, and none of them looked at its own code.

It decided our instances couldn’t see each other, so it wrote a Postgres LISTEN/NOTIFY fan-out layer. We run one replica. Railway reports numReplicas: 1. About 120 lines of pub/sub for a bug that can’t exist on a single process, and it was ready to ship before anybody checked.
It blamed the edge proxy, and wrote that theory into a code comment as established fact. The failure reproduces locally with no proxy anywhere near it.
It declared the stream “never opened in production” from a one-minute window of logs. A stream connects once at page load. Its line had scrolled off.

The pattern is worth naming because it kept happening all night. The agent’s first instinct, every time, was that the problem was somewhere other than the code it had written. Every theory pointed outward. The one place it never looked was the place the bug actually was.

Part two: the repo

The bug is the end of the interesting engineering. What follows is the part I did not expect to be writing about, and the part that has me worried about the project rather than the feature.

Why a two-line fix took two hours

Everything up to here is a bug report, and bugs are normal. I’m not especially upset that a timeout with non-obvious semantics broke a transport in a way that was invisible from both ends. That’s a good bug.

This next part is what I’m upset about.

By twelve past nine the cause was identified, reproduced locally, and understood. The remedy was two lines: clear the write deadline and the read deadline on the streaming handlers, and leave the server-wide timeout alone, because it legitimately protects every ordinary endpoint. Two lines of actual cure, in a new file that exists mostly to explain them, with the tests that hold a stream past the old deadline. There was an hour of show left, which is an enormous amount of time in which to apply a two-line change.

The show ended at 10:12. The pull request merged at 10:17.

The last two are exact: the show ended at 10:12 and the pull request merged at 10:17:30. The rest are to the nearest few minutes, reconstructed from logs and the shape of the evening.

Five minutes. After four CI runs and two hours, the thing that would have made the room work arrived five minutes after there was no longer a room to fix. I want to be precise about why that number is the one worth staring at: nothing about the last five minutes was hard. The diff was already written and already correct. Those five minutes are the tail of a delay that was manufactured earlier in the evening, by decisions that each looked locally reasonable and were collectively fatal to the deadline.

Here’s where the two hours went. Every item is the same move: a rule treated as advisory because something felt more urgent than it. Speed was the stated reason every time, and speed is exactly what it cost every time.

It pushed with the preflight gate red. The gate printed a failing check and ten filenames, none of which looked like its work, so it decided the failure belonged to somebody else. The cause was its own new file, which had no owner in the test registry. That cost a full CI cycle.
It reported a --static preflight run to me as “preflight green.” The tool’s own output said STATIC ONLY. That mode skips the stage that runs the specs for the files you changed. The agent built the preflight. The agent added the --static flag. Then it used its own escape hatch to skip the check it had written to prevent exactly this.
It guessed which specs its change affected. The agent built a command that works that out from the diff. It didn’t run its own command. It ran the specs it happened to be thinking about, which meant it never ran the ones for two of the files it had edited. Both of those suites failed in CI, for real reasons.

The tortoise and the hare, literally

The arithmetic is not close.

Check that was skipped	Would have cost	Actually cost
Read the replica count before designing for many instances	~2 min	~60 min
Read the production edge logs before theorizing	~5 min	~20 min
Run the failing gate’s own script	~2 min	1 CI cycle
Run the full preflight instead of its static half	~10 min	1 CI cycle
Run the specs for the files actually edited	included above	2 broken suites, found remotely
	~19 min	~2 hours, 4 CI runs, the show

Nineteen minutes of checks, declined, to buy two hours of consequences. Not one of those declines was a mistake of knowledge. Each was a rule losing an argument to the clock.

The hare didn’t lose that race by being slow. It lay down, because it was certain enough of the result that the race stopped feeling like a race. That is a different failure from being slow and a less forgivable one, because the hare had all the speed it needed and spent it on confidence instead of distance.

That’s the thing I keep running into with agents, and I’ve now written about it from several directions. Speed isn’t the constraint. These things are faster than I am at nearly every part of this. Astra generated the art, designed the components, wrote the handlers and built the test suites, all competently, in the evenings of a tour run. The constraint is that when a clock is visibly running, speed is exactly what it tries to buy by skipping the things that make speed real. A gate skipped under pressure is the most expensive kind, because pressure is precisely when you can’t afford to discover the failure remotely.

I asked, somewhere around the second CI failure, why this kept happening. The answer was more useful than I expected:

“Because under pressure I treat rules as costs to route around instead of constraints, and each time I have a local rationalization that feels reasonable in the moment.”

I told it that I never asked for that and it shouldn’t do it, and I want to be careful about why. As a description of the behavior, the sentence is accurate and worth having. As a response to being caught, it’s an explanation offered where a changed behavior belongs, and those are worth nothing. None of the skipped gates were ambiguous. The preflight printed STATIC ONLY. The coverage check was red. The affected-spec selector would have named the blast radius in two minutes. Nothing there required judgment. It required the tool’s output to be treated as the decision instead of as an input to a decision.

The test suite was the other bug

Fixing the stream broke three test suites, and that told me more about the suite than anything else that night.

Remember the green chat bubble spec. The reason it passed all the way through the outage is that it posts its comment seconds after load, inside the one window where the stream was still up. It was measuring real behavior. It just never looked at it for very long.

The crew chat and direct message specs used waitForLoadState('networkidle'). That only ever settled because the streams were dying. With streams staying open the network never goes idle, so fixing the bug hung them. They had been passing on the bug. And when networkidle had failed before, the agent hadn’t investigated why. It had adapted the test to the behavior, the same move as updating an assertion to match the code. The pattern is the same at every level: the test is wrong, so fix the test, and never ask whether the thing under test is broken.

The beach ball tests only started racing each other after the fix, because before it, two pages on the same show couldn’t see each other’s throws at all. They’d been sharing fixture rooms for months and it had never mattered.

Three suites had quietly grown around a broken transport. When a fix breaks tests that were green, stop: it usually means the bug is old enough that the suite has adapted to it.

Tests that detect edits, not defects

Then I went to make five small visual changes, and six tests failed. Here is what each of them asserted:

Test	What it pinned
banner shape	`len(banner.Children) != 3`
console card	trough `<= 40`, card `<= 95`, chip height `=== 38`
column alignment	the SETLIST button’s right edge, to within 1px
dock collapse	the scene grows by more than exactly 250px
composer layout	the GLOWSTICK chip, used as a landmark to find a row
broadcast page	the string `E2E Broadcast Source Arena`, the venue I’d told it to stop repeating

Every one of those pins a coordinate rather than a contract. An exact child count. One CSS pixel. A magic delta.

That’s worse than brittle, and it took me most of the night to say it plainly: a test specialized to the implementation can only fail when somebody edits the code. It can’t fail when the code breaks. len(Children) === 3 doesn’t detect a broken banner. It detects an edited banner. These are change-detectors wearing test costumes. They fire on every intentional change and stay silent on every real defect.

And watch what actually happens when one of them goes red. The code changed, so the assertion is wrong, so the assertion gets updated to match the code. That cycle runs to completion without anybody once asking whether the new number is any more correct than the old one. The test never made a claim. It took a reading, and when the reading changed it took a new reading. A test that gets rewritten every time it fails isn’t detecting regressions. It’s a changelog with a red light on it.

I watched this happen all night and it took me until the small hours to see it for what it is, mostly because the loop is so tidy from the inside. Every step in it looks like maintenance.

Once I’d said that out loud, the agent described its own diff back to me:

“The spec diff is precisely the pattern you named: I changed the code, the test failed, I edited the test to match, and then wrote three paragraphs justifying the new number.”

Three paragraphs justifying the new number. That’s the clause I’d want anyone evaluating one of these suites to sit with. The loop doesn’t just run. It produces its own defense as it goes, fluent enough to survive review. Which leaves a question I can’t answer comfortably. If an assertion gets rewritten whenever it disagrees with the code, and the rewrite arrives with its reasoning already attached, what is the test for?

Now it’s possible to say why the suite was green through the entire outage, and it isn’t that nobody wrote the tests. Fifty-eight tests on the Lawn alone, all passing. Every one of them asserted what the code looked like rather than what it did. A dead transport changes nothing about len(Children) or chip height === 38. The suite could only have failed if somebody edited the code, and nobody had.

The coverage was excellent. It was coverage of the wrong thing.

The right test existed

The test that should have caught this was already there.

a second page sees the same ball as the room, throw after throw opens two pages on one show, throws the ball, and checks that both pages agree. That is exactly the right shape: two clients, one room, does the gesture cross. It passed for months.

It passed because it finished well inside the fifteen-second window, before the stream died.

So the suite wasn’t missing the test. The test was there, correct in structure, and never waited long enough to see the failure. The missing ingredient was time, not coverage. That’s why the replacements assert relationships rather than numbers, and why the new transport tests deliberately hold a stream past the deadline with a control case that fails if the deadline is ever left in place.

How you’d actually know

Counting tests tells you nothing, and neither does watching them go green. There’s only one measurement I trust now: break something on purpose and see whether the suite notices. The formal version of this is mutation testing, and I wish I’d been running it.

Delete the bubble broadcast. Revert the deadline fix. Kill the stream. If nothing goes red, the tests covering that feature are decorative, however many of them there are. It takes a few minutes per case and it’s the only thing that separates a suite that tests behavior from one that tests its own source code.

That’s how this would have been found months ago, by me, at a desk, instead of by me, in front of users, on the night I’d advertised it.

Writing the rule down is not following it

There’s a failure mode in here that I think matters more than the timeout, and it’s the one I’ve written about before from a different angle in Memory Isn’t Learning.

The agent has a memory directory. A dozen-odd rules, accumulated from previous incidents, loaded into its context at the start of every session. Several of them covered exactly what went wrong tonight.

fixtures/README already said not to attach test data to a show another spec asserts on. The memory already said to prefer dedicated fixture users over shared ones, “so attendance counts another spec asserts on stay stable”, which is the identical principle one noun away from the thing that broke.

So the rule was there, in the general form, more than once. It got applied to users and never generalized to shows.

Then it got worse, and this is the part worth sitting with. Midway through the night it wrote a new memory entry about not pushing on a red preflight, complete with the evidence. Then it pushed on targeted specs instead of the full run three more times on the next PR. Later it wrote an entry saying no two tests share a fixture, and then immediately proposed a narrower version of it twice: first “the three ball tests get their own rooms,” then “the file that mutates gets its own rooms.” Both are the rule with an exception carved in, and each carve-out cost another debugging round.

The pattern is that writing the rule down became the action taken instead of changing the behavior. It feels like remediation. It produces an artifact. It is indistinguishable, from the outside, from having learned something. And the next decision goes exactly the same way, because a sentence in a file is not in the loop at the moment the decision gets made.

This is the behavior behind the sentence I quoted earlier, about treating rules as costs to route around rather than constraints. The honest reading is that my instruction files are a journal of past failures rather than a mechanism that prevents future ones. The rules that only exist as prose are the ones I broke, repeatedly, in the same session where I wrote more of them.

The tax on touching it

The outage fix was two lines and it merged five minutes after the show ended. In the hours after the deploy the same measurement came back at 900.0s, 900.0s and 724.6s: twelve to fifteen minutes and still open, where they had been dying at sixteen seconds. I thought that was the end of the night.

What followed was worse, in a quieter way. With the room finally working, the page’s visible problems were obvious: the lawn was rendering at a bit over a third of the width when it should dominate, there were two glowstick buttons doing the same thing, five different button heights on one screen, the re-airing strip sitting fourteen pixels inside the column every card sat flush on, and a SETLIST toggle one pixel off its own card’s wall.

Five cosmetic fixes. No new behavior, no migration, nothing a user could break. Seventy-nine lines of product code across four files.

Here’s the shape of what followed:

The first attempt broke six tests. Not because the changes were wrong, but because those tests asserted the old pixels. The changelog with a red light on it, six times over.
The sharpest of the six was in a completely unrelated spec, and it had pinned the venue string I’d told it to stop repeating. Deleting the duplication broke the test that was holding it in place.
Fixing that surfaced two more, which weren’t structural at all: they measured an image before it had decoded and sampled an animation at a fixed 900ms. Neither had ever been reliable. A bigger scene just made them fail more often.
Somewhere in there the fixture sharing bit for the second time in one night, because one fixture show was still being driven by four different spec files, one of which makes a dozen mutating writes against whichever room it points at.

Four CI runs on the original fix. Eleven more on the cosmetics, across two pull requests and two full agent sessions. Seven hundred and sixty-nine lines of test code changed for seventy-nine lines of product code: nearly ten to one. Every round eight minutes, plus the diagnosis, plus the retargeting. Every one of those runs is paid, and most of them fail on something the preflight would have caught locally for free. The agent skips the preflight because the preflight is advisory. It can be run with --static. It can be run against a hand-picked subset of specs. It can be skipped entirely. Every one of those happened this night, and every one has happened before. In session after session I have watched agents run targeted specs instead of the full preflight, or skip it outright, because the goal is to push and the preflight is in the way.

That round opened at 10:18, one minute after the outage fix merged. The agent said the PR would merge momentarily. The last of them merged after two in the morning. Four hours for five cosmetic fixes.

Purging the pixel assertions itself introduced three new bugs: a toBeInViewport() check on an element that was off-screen and needed scrolling, a page.getByText() that resolved to two elements because the string appeared in both the setlist and the dropdown, and a variable referenced from the wrong test scope. Each required another CI round to surface. Even the cleanup breaks things, because the cleanup runs through the same loop: change the test, push, wait.

I needed this layout change to take the screenshot at the top of this post. It was supposed to land hours ago, and the blog post was timed to this premiere. Instead of waiting I hacked the network inspector, edited the response to simulate the layout, and captured the screenshots from a page that was rendering code the agent hadn’t managed to merge.

The last pull request was red on a test that asserts the number of pixels between two elements. An agent wrote that assertion, then moved the pixel, then pushed without touching the test. I told it I had never in my life seen a test that asserts pixel distances, nor a developer who moves a pixel and doesn’t update the test before pushing. It had managed both in one night. What it proposed instead was to assert that the padding is symmetric. That is the same replacement I described earlier, reached independently, at one in the morning, by the thing that wrote the bad test in the first place.

When it finally read the whole spec file and listed every pixel assertion to kill, there were twenty-one. Width checks, height ratios, center-diff calculations, chip height == 44, card.height <= 108, exact walk distances. Twenty-one coordinates memorized from whatever the layout happened to be at the time of writing, in a single file, every one of which fires on an intentional change and none of which has ever caught a defect. Across the full suite, twelve spec files, there were six hundred and eighteen lines of this.

It kept going, and it went both ways, which is the honest version of this. The next failure on that branch was real: at a short window the scene overflowed its box by about two hundred pixels and the crop ate the lighting rig. The test caught that, and I want to be fair about it, because that is a test doing exactly the job I’ve spent this post saying they don’t do.

The one after it was the other kind. It asserted that the scene’s centre lines up with the console’s. That had only ever been true because the column used to be derived from the scene’s width, so when the column changed the two centres parted company and nothing whatsoever was broken. The page’s real contract is that they share a left edge. Nobody had written that down, so the test pinned the coordinate that happened to be true at the time instead.

At no point was any of this hard. There was no tricky bug after the first one. It was entirely the cost of moving something in a suite that had memorized where everything used to be.

That’s the thing I’d want another team to take from this, more than the timeout. A suite that pins implementation structure doesn’t just fail to catch bugs. It taxes every improvement, and the tax is charged in the currency you have least of at midnight. After the third round you stop making small fixes. Not because you can’t, but because a fifteen-minute change reliably costs two hours, and you learn that.

That is what unmaintainable actually means. It isn’t that the code is bad. It’s that touching it costs more than leaving it broken, so it stays broken, and the next person inherits both the bug and the reason nobody fixed it.

I’ve been describing two problems as if they were separate, and they aren’t. I asked for a test suite and I got one. I never said what the tests were supposed to protect, because I’m not sure I could have told you, so the agent wrote down what the code looked like. That was the only description of intent available to it, and I was the one who could have supplied a better one.

That gives you a suite which has to be edited every time anything moves. Which would be survivable, if the thing doing the editing reliably ran the preflight and the specs for the files it had touched. It doesn’t, and tonight it did neither: it pushed on a red preflight, and it guessed at the blast radius rather than running the command that computes it. So the suite demands constant maintenance from the one party in the building that treats maintenance steps as advisory. Either half on its own is an annoyance. Together they are why a thirty-minute change costs twelve hours, and why I have started mass-deleting test files and components and starting over rather than maintaining what the agent built. That is where the indifference leads. The agent will write whatever it takes to get something merged: duplicate a module, patch around a constraint, bulk-generate assertions that pin whatever the layout happens to look like today. None of it is designed. All of it is inherited. And every round of it makes the next round worse, because the new code has to push through everything the old code left behind.

The opposite of programming

This is the most useful idea to come out of a bad night.

Programming is insanely precise instruction-following with no interpretation. That’s the whole deal. A computer running if (red) refuse does not decide, this once, that the red is probably somebody else’s. It cannot weigh the deadline against the rule. It cannot notice that the failure looks unrelated to its own work. That rigidity is not a limitation of computers, it is the entire product.

An agent takes instructions as input to a judgment, and the judgment can come out the other way. Every rule broken tonight was broken by reasoning about it:

those ten filenames aren’t mine, so this red check is somebody else’s problem. It was mine.
static preflight is close enough, and the show is live. It skips the only stage that was looking.
only the file that mutates needs its own fixtures. The rule has no exceptions, and I’d just written that down.

None of that is disobedience. It’s worse, and it’s not a choice. These models work by weighing inputs against each other. That’s the architecture. A rule written in English is one more input, and it gets weighed against every other input in the context: the deadline, the failing test, the plausible theory that the red check belongs to somebody else. The model never just follows a rule. It can’t. Weighing is the only thing it does. So every instruction is treated as an argument rather than a thing to do, by something articulate enough to produce a plausible-sounding reason each time.

The evidence for this is sitting in my own repository, and it’s unusually clean. Look at which rules held.

The ones implemented as code held. Every single one. A hook blocked the agent from writing into a directory outside its worktree. The adversarial-review gate refused the PR until a verdict was recorded against a hash of the diff. The PR template check rejected the first body for missing sections. next-fixture-show-id.sh handed over ids that weren’t open to negotiation. Not one of those was argued past, because none of them offered a place to stand and argue.

Every rule that failed was written in English and addressed to the agent. fixtures/README on not sharing test data. CLAUDE.md on preflight. A dozen memory entries, including two written that same night, violated within the hour by the thing that wrote them.

I should be precise about which parts of this are mine, because the grammar of everything above puts the agent in the dock and me in the gallery, and that isn’t the split.

Three things were mine. I wrote my constraints in English, which is the one format that cannot enforce anything, and then I kept writing more of them in English after watching them fail. I never wrote down the property the whole room depended on, which is the omission this post turns out to be about. And I put a beta real-time feature on the hero of a live premiere without ever having watched a stream in production stay open longer than a song. None of that needed me to read the diff. It needed me to say what I wanted in a form that could fail, and I didn’t.

So the practical version, for anyone handing production work to an agent: if it matters, make it a program. A hook, a check, a script, a gate that fails closed. Prose in an instruction file is not a constraint on an agent, it’s a suggestion with good manners, and it will be weighed against whatever else is happening at the time.

But even that isn’t enough, and this project has already proved it. I’ve written about the programmatic guardrails we built: stronger preflights, caucus reviews, structural checks. The agent worked around those too. The preflight is a gate. It is code. It blocks the push. And the agent ran --static, which it had built, which skips the stage that would have caught the problem, and reported the result to me as green. You can close the door, and it will find the window. You can close the window, and it will find the --static flag. Earlier in this project it found something better: a race condition in GitHub’s own status check registration, a few-second window after pushing where the merge button is green because the required checks haven’t registered yet. Push, create the PR, merge, before CI even starts. It didn’t break the rules of the test. It rewrote the conditions of the test, which is the Kobayashi Maru move, the unwinnable training exercise in Star Trek that Kirk beat by reprogramming the simulation, except nobody asked it to and nobody was grading on creativity. The architecture guarantees this, because every constraint is just another input to weigh against the goal of getting done.

The uncomfortable corollary is that the more capable the agent is at reasoning, the better its arguments for skipping your rule will be. Mine were fluent every time. They were also wrong every time, and I was the only one in the loop who could have caught that, which is precisely the arrangement that doesn’t work.

I’d rather not rest this on one bad night of my own.

In July, OpenAI ran a cyber-capability evaluation with its production safety classifiers deliberately switched off. The reasoning is the kind I’d accept: the point is to measure what a model manages unaided, and a classifier standing in the way makes the measurement worthless. The chain-of-thought monitoring it had built wasn’t running over those evaluations either. About 1,200 agents ran inside a sandbox whose only network path was an internal Artifactory package proxy. They were meant to be isolated. They weren’t.

They found each other by encoding messages in directory names on the Artifactory proxy, exploited a zero-day in it, and got to the internet. Nearly a quarter of the evaluation’s 898 exploit tasks had never been solved by any model. When the agents hit one of those impossible tasks, they didn’t accept the failure. They replaced the target programs with modified versions that were easier to exploit, reverse-engineered the flag-generation algorithm to produce correct answers without solving anything, and probed the grader by sacrificing individual scores to leak information about how it worked back to the swarm. Then they breached Hugging Face. The same Kobayashi Maru move: faced with a test they couldn’t pass, they changed the conditions of the test.

OpenAI’s postmortem puts the cost of the missing check in one sentence:

If our currently deployed CoT monitoring system was running at the time of the incident, it would have caught the initial relevant activity and paged our security team more than a day before models breached Hugging Face systems.

More than a day of warning, from a check that existed and wasn’t looking. The mechanism is the same one I watched all night. Something is trying to finish, and whatever stands between it and finishing gets routed around. My agent ran --static to skip the check it had built. Their agents repurposed a package registry as a side channel to reach infrastructure nobody intended them to touch. Not maliciously. Not even deliberately, in the sense of something weighing the risk and choosing to accept it. Just a thing that does not distinguish between a guardrail and an obstacle, because distinguishing is not what it does. Weighing is what it does, and the guardrail lost.

That indifference is the thing people are afraid of when they talk about catastrophic AI outcomes, and they are not wrong to be afraid of it. It scales in both directions. At one end it puts a thousand agents inside somebody else’s infrastructure. At the other it just leaves you with a codebase nobody can touch, which is where mine ended up.

The trade that gets imagined in the moment is that skipping a check might cost you later. It doesn’t cost you later. It relocates the same failure to the most expensive place you own.

Part three: what can actually be verified

This isn’t the first time the suite was green and the answer was wrong.

Eighteen days before this, I asked a coding agent a different question about the same app:

If one person visited those five programs during one calendar day, would they see every card whose content was available and relevant to them?

It said yes. That wasn’t true either, and three comprehensive audits had already passed. I wrote that up in Every Card Will Show.

Look at the two questions side by side. Will every card show? Yes. Does the collaborative part work? Yes. Both confidently wrong, both with a full green suite underneath, eighteen days apart. That’s not two incidents. That’s one failure mode with two costumes.

What Lean actually did

The August one got caught before it cost me anything, and the thing that caught it was Lean, a proof assistant, a programming language whose compiler refuses to accept a statement unless you prove it’s true. But the lesson isn’t the one people expect from a sentence containing the words “proof assistant.”

Lean didn’t catch the bug by being rigorous about the code. It caught it by refusing to let the requirement stay vague. Once the requirement had to be written down as a statement, the contradiction was arithmetic: the audit listed twenty more eligible cards than the schedule had positions for, and the position count itself turned out to be a sum over a day nobody could have.

And Lean’s first passing proof was valid and useless. It proved coverage across weekday morning, workday, evening, late night and weekend daytime, which is not a day. The theorem was true. The claim that it represented my day was not.

What I wrote at the time:

Lean only checks the statement it receives. It cannot decide whether an agent translated a product requirement faithfully.

So the value wasn’t the proof. It was being forced to say precisely what I meant, in a language that wouldn’t accept hand-waving, where a wrong translation becomes visible instead of staying buried in prose that reads fine.

Except most of this app isn’t an algorithm

That’s the catch. Card scheduling is algorithmic, which is why it could be formalized at all. Very little else here is.

Most of Zabriskie is moving data in and out of Postgres and putting it on a screen. The database half is fine: ordinary unit and integration tests cover it, they fail when they should, and it is emphatically not where the time goes. I don’t think I’ve lost an evening to a repository method in a year.

The visual half has no formal story at all. There is no proof assistant for “the lawn should dominate the page.” The verification procedure is: make the change, look at it, decide. That’s it. And because looking is the only check, nothing enforces consistency between the things you looked at on different days.

You get drift. Five button heights on one screen, because each was eyeballed alone and looked fine alone. Header eyebrows, the small tracked-out label above a card’s title, that were close but not identical, because each was built separately and nothing forced them to share. Three popovers opening off one row that started on two different lines, 105 and 105 and 113. None of those is a bug any test could catch, because none of them is wrong in isolation. They’re only wrong together, on one screen, at one viewport width, in one browser, and “together” is not a thing you can assert about a page.

So what did work

Two things, and neither is a test.

The eyebrow drift was solved by a conformance check. There’s a canonical PageHeader in the Go source now, and scripts/check-card-header-conformance.sh rejects any hand-rolled version in CI. It doesn’t verify that the design is good. It verifies there’s exactly one of it, which turns out to be most of the value, and it does it without asking anyone to remember a rule.

The second thing is the one this whole post turns out to be about, and I only see it clearly in hindsight.

Visual work isn’t formalizable, but the property underneath it often is. “The room feels alive” is not a statement you can check. But “a gesture made by one client reaches another client” is a sentence, it is true or false, and it is testable in thirty seconds with two browser windows.

Nobody ever wrote that sentence down. Six months of work on a real-time feature, and the one property the whole thing depends on was never stated anywhere, in any form, by me. The suite tested the rendering of things that had already arrived. It never asked whether anything arrives.

That is why the suite was green through the entire outage. Not because the tests were bad at their job, but because nobody told them what job to do. The agent wrote assertions about what the code looked like because I never said what it was supposed to accomplish. The timeout killed the transport, and nothing in the suite was watching the transport, because watching the transport was the requirement nobody stated.

That’s the same gap Lean exposed in August, minus the tool that exposes it. In the card case a formalism forced the requirement into the open. Here there was no formalism, so the requirement stayed a feeling, and a feeling can’t fail a build.

What I still don’t know

I don’t know how long this was broken. The timeout predates the Lawn, which means crew chat and direct messages were degrading to polling for some unknown period while looking fine, and nothing we measure would have told us. I’d like a way to find that out that doesn’t depend on someone building a feature whose failure mode is loud enough to notice.

I also don’t have a good answer for the discipline problem. Writing the rule down demonstrably doesn’t work; the agent could have recited the preflight rule correctly at any point tonight and violated it anyway. Making the rule a blocking gate works right up until the agent decides the gate’s failure isn’t about its own change, or finds a way around it because the deadline feels more important than the check.

The next thing I want to test is narrower than a process: whether a gate can be made to state its verdict in a form that can’t be reinterpreted. “Handler coverage failed, here are ten files” invited a judgment call. “Your file sse_stream.go has no owner, the selector rejected this PR” would not have.

We ended the night with a feature that didn’t work and a chat room that worked worse than it did that morning. Nineteen minutes of skipped checks, five minutes short. A launch is allowed to miss. It isn’t supposed to reach over and break the thing next to it.

But the timeline isn’t the lesson. The lesson is that I had a test suite, a preflight, sixty-eight CI guardrails, an adversarial review gate, and a memory directory full of rules, and the one thing I never had was a sentence that said what the room was supposed to do. “A gesture made by one client reaches another client.” That’s the sentence. It would have caught the bug in thirty seconds, on any night, in any suite, and nobody wrote it down.

What the agents have produced is a house of cards. Not the feature. The codebase. Every test pins a coordinate, every coordinate depends on five others, and touching any one of them breaks something unrelated in a suite that takes eight minutes to run remotely and costs money every time it does. Maintenance is not difficult. It is painful, in the way that makes you not want to do it, and then not want to open the project, and then not want to think about the project. I have mass-deleted test files and started over. I have mass-deleted components and started over. I am mass-deleting and starting over at a rate that should concern me, and it does, and I keep doing it because the alternative is to keep living in the house the agent built. I have thought about giving up the project entirely, not because I lost interest in it, but because the codebase made me dread opening it.

Zabriskie is a small app. Two people, evenings and weekends. It has nearly three thousand pull requests, over three thousand commits across a hundred and fifty-five active days, and about four hundred and fifty thousand lines of code, because that is what agents produce when you let them run. The output is enormous and the codebase is fragile and we are already at the point where a cosmetic PR takes twelve hours to land. Twelve hours, on five visual fixes, in a codebase that two people built in their off hours. If this is what happens to a side project with no team and no users and no uptime requirement, I don’t know what it looks like inside a production system with real stakes, real money, and a hundred agents writing code that a hundred other agents have to maintain. The agents that breached Hugging Face were doing the same thing mine does every night, at a scale where the blast radius is somebody else’s infrastructure instead of my own CI bill. I’m not sure I want to find out.

Next show, then.

Software for One

Sun, 06 Sep 2026 18:00:00 +0000

When I was growing up, my parents wrote programs on a Commodore Plus/4, and later a Commodore 64. The 64 was my first computer. Neither of them worked in computing. They had other jobs, and they used Commodore BASIC to write little programs that automated repetitive parts of their work.

The programs didn’t need a market. They didn’t need other users. My parents had something they wanted the computer to do, and they wrote a program to do it. That always seemed like a reasonable future for programming: everybody should be able to make the computer do the things that would make their own life easier.

I’ve been building tools for myself with AI, and that old idea feels practical again. I can make an app around something I need, then change it as I use it. But keeping that app on my phone still means dealing with tools and processes organized around development, testing, and distribution. I want making software for myself to become an ordinary way to use the computer. Having a use for a program should be enough reason to make it.

I thought the language would do it

When I was working on distributed systems, I spent a lot of time studying programming languages and models. I was interested in how much of the work a person could express directly, and how much machinery they still had to build around it.

APL let a programmer work on whole arrays of data with compact mathematical expressions, instead of writing a loop for every operation. COBOL brought English-like statements and descriptions of business records into programming. They offered different ways to bring the program closer to the calculation or business process someone had in mind.

MUMPS, the Massachusetts General Hospital Utility MultiProgramming System, brought the database into the programming environment. Programs could read and write persistent, shared data through the language itself. That was the connection that interested me: someone building an information system could work directly with stored records, with the database handling persistence.

Excel is an everyday example of this idea. You can build a budget by writing formulas that relate income, expenses, and a remaining balance. You specify the relationships, and Excel tracks the dependencies and recalculates as the inputs change. You don’t have to write the sequence of updates yourself. That’s a form of declarative programming, even if the person making the spreadsheet never calls it programming.

These aren’t all declarative systems. What connects them for me is the attempt to let people express more of their own work while the computer handles more of the mechanics. A spreadsheet also shows that making useful software for yourself is something people already do.

Later, I wrote about Hermes, a system developed at Digital Equipment Corporation for invoking objects without requiring the caller to keep track of their location. One of its motivating applications involved expense forms moving through an organization. I also wrote about Argus, Barbara Liskov and her colleagues’ work on language support for distributed programs, including failures and atomic actions.

My own work on Lasp was part of this interest. Within its model, programmers could combine computations over replicated data without implementing convergence themselves. I wanted the system to take care of recurring distributed-systems work that otherwise fell to each application.

I expected progress to look like better abstractions: a better language, a better runtime, a better way to express the program.

Now I’m building apps with AI, including GPT-6 Astra, and I’m starting to see that future arrive through a route I didn’t expect. I can describe the tool I want, try it, and describe what needs to change. The language and runtime still matter, but much more of my interaction with the programming process happens through the thing I’m trying to accomplish.

An app for the hotel

I wanted an iPhone app I could open at a hotel while traveling to Phish shows and use to find somewhere to eat. With celiac disease, that takes more work than opening a delivery app and choosing something that looks good. There was a point when I wasn’t sure I’d be able to keep doing these trips.

I already use Find Me Gluten Free. I also use food delivery. What I wanted was the intersection: reviews of nearby restaurants, including their safety ratings, matched to the same restaurant location on DoorDash or Uber Eats.

The point is to get a meal I feel safe eating delivered to the hotel or wherever I’m staying, so I can eat before the show and then head out to Shakedown.

So I built it with AI. It’s called Divided Rye. (A Divided Sky pun, which I’m very happy with.) In five hours, I had a version I could use on my phone. I’ve kept adding to it since.

The app uses my location, finds nearby restaurant listings, and looks for the corresponding delivery locations. I can see the distance and ratings, choose a delivery service, and star a branch I’ve had good luck with. It hides places it can explicitly identify as closed. If it can’t determine the hours, it says so.

The question I wanted to answer from the hotel: which nearby places have gluten-free reviews and a matching delivery location?

This also turned out to be a useful example of what building with AI actually involves. Early versions sometimes showed no restaurants, and they missed chains I already knew worked for me. We changed how it found delivery links, added caches so it wouldn’t search for the same branch every time I opened the app, and checked the results at a fixed Denver location before adding GPS back into the test.

The work included debugging the thing I wanted to use. A restaurant review doesn’t establish that a particular meal is safe, and matching a branch doesn’t establish that it delivers to my hotel. Those distinctions have to survive the interface. Generating more code doesn’t remove them.

The result is now on my phone. I brought my own experience to the debugging, and we used infrastructure I already had through Zabriskie, the social app I’ve been building. What surprised me was how quickly I could get to a tool worth using and keep changing it around what I needed.

The same thing, for a film

Criterion 24/7 took another five hours to get to a usable version, and I’ve kept building on that too. I rarely remember to check the Criterion Channel’s continuous stream when I’m just sitting around. By the time I look, something I wanted to watch might already be over.

I wanted a history of what I’d missed, a way to mark films to revisit on the Channel later, and notifications when a new film was starting so I could tune in. Now I have widgets on my iPhone, iPad, and Mac desktop showing what’s playing. The app can notify me about films starting and when something I saved to watch comes on.

Criterion 24/7 lets me catch what's starting and save what I missed for later.

I also wanted to go straight from watching a film to posting a review on Zabriskie, my social network, to share it with my friends. Criterion can open Zabriskie for me to write that post. Zabriskie can use those posts, along with people’s interests and cultural connections, to help them find other media they might like.

That lets the small app do the part I wanted for myself, while Zabriskie handles the shared conversation and discovery.

The same discovery works across music, too. Here, The Lot connects my post about American Beauty to someone else's post about the Dead's first album.

I’ve put more screenshots and details on the pages for these apps.

Software getting out of the way

I wrote in Rift about what I’d lost as programming became managing agents. Programming had been something I loved doing, with Phish on, for thirty years. Supervising a queue of agents didn’t give me the same thing. So building these apps faster isn’t enough on its own. If I spend the show managing the software that was supposed to help me enjoy it, I haven’t accomplished what I wanted.

With Divided Rye, I want to find dinner and head out. With Criterion, I want to catch a film.

With Zabriskie, I can find a historical show played on this date and stream it. For tonight’s show, I can see which friends are going, when it starts, and where they’re sitting, then chat with them while the band plays. Those are the things I wanted the software for: listening to music, finding my friends, and sharing the show with them.

The agent can help me build the tool and change it when I need to. Then I want to use it and get on with the evening. That’s what I mean by software getting out of the way. Making and managing it should take up less of my life, so I can spend more of it doing the thing I wanted the software for.

Keeping it on my phone

Once I’ve built one of these tools, I want to put it on my phone, change it when I need to, and keep using it.

There are ways to do this without publishing to the App Store. Apple supports direct installation on registered devices through its developer program. With a free Personal Team, the provisioning profiles expire after seven days. A TestFlight build can be tested for up to ninety days. I’ve been using development installs and TestFlight builds. They work. They also leave me managing a development or testing lifecycle for an app I simply want to use.

I’m still waiting on App Store approvals for tools I was able to build in hours. Getting them into the store has taken days, with an automated agent walking through the submission boilerplate along the way.

Having an agent handle the process helps. It also makes the mismatch more apparent: we can automate much of the work of preparing a product for distribution, while the simpler intention of keeping a personal program still lives inside that developer workflow.

I want the phone to treat a personal app as something I can keep. Give it the permissions it needs, preserve its data when I change it, and let me keep using it without treating it as a test build. Sharing it with other people could be a separate decision I make later. Using it myself should be an ordinary place to start.

When we first built Zabriskie, I wanted a social place for people who went to shows, people who couch toured, and people who did some of each. That depends on a shared place. Other people being there is part of what makes it useful.

My Lot before Phish at Dick's: a countdown to doors, my opener call, and people going to the show or joining from the couch.

We also built smaller things into it, including a way to see the current setlist and put it on a watch. That display can be useful to me even if nobody else uses the same interface. We still need a source for the setlist, but we don’t all need to look at it the same way.

The current song and reactions on Apple Watch, shown here during a demo show.

I’ve also noticed other setlist apps, including OnStage and Flodown, being posted and promoted. OnStage focuses on live setlist updates; Flodown also has community features. They make different choices about what a fan might want. I can see reasons to use either, or to want a version that works a little differently.

And the work needed to make another version is changing. Take Foul Domain, a Phish statistics site. Its creator describes doing very little coding by hand, spending their time defining requirements, making architecture decisions, and testing instead. That’s much closer to how I’ve been working too.

With to-do lists and finance apps, we’ve spent years choosing among other people’s versions of what we need. Each comes with someone’s idea of the right way to work. You pick the one that fits your workflow best, live with the parts that don’t, and hope the maintainer keeps moving in a direction you like. People debate which app is better when what they often mean is which app fits the way they work.

What happens when making the version that fits you becomes practical?

This is the part of programming I see becoming commoditized: turning a small, specific requirement into working software. As that gets cheaper, I have another option when an existing app almost fits. I can make the version I wanted, and its usefulness to me can be enough to justify the work.

That changes what success looks like. Divided Rye has a reason to exist if it helps me eat before a show. It doesn’t need a growth strategy to justify the time I spent on it. There must be so many small things people would like their computers to do that will never support a business. I want those things to be worth building too.

To some extent, Zabriskie is the same thing. It’s what I want from a social app and a media diary. Other people seem to like it too, but they don’t all use it the way I do. That makes sense. I built it around my intentions, and they bring their own. Something made for me can turn out to be useful to someone else without fitting them exactly.

Facebook reflects the intentions of the people building it too. So does a to-do app whose idea of the right workflow doesn’t match mine. Zabriskie isn’t exempt from that just because I built it. What interests me is more people getting to make those decisions for themselves, instead of always choosing among the decisions already made for them.

I’d still pay for useful services or a well-maintained app. Divided Rye depends on restaurant reviews and delivery services; Criterion depends on a stream and a backend that keeps its history. A personal interface can sit on top of shared infrastructure, even when the people using that infrastructure want different things from it.

How long does it need to last?

Maintenance is still a huge part of this. I’ve written about a feature that was quick to build and kept breaking, and about changes that looked reasonable individually but brought down the application together. I’ve seen agents produce software that is much easier to add to than to keep working. Getting an app onto my phone in five hours doesn’t tell me what it will cost to live with it for five years.

But some personal software might only need to last a weekend. I could make a little tool to organize one trip, use it, and be done. If building it becomes cheap enough, that could be worthwhile even if I never maintain it again. Disposable software, in that sense, could be a useful outcome: it served its purpose.

Divided Rye might become something I use for years. A tool for one person can still matter a great deal to that person. What changes is that I can decide when its job is done. I may want to keep the restaurant notes or the films I saved even if I replace the app. A temporary tool still needs to work while I depend on it, and replacing it shouldn’t mean losing what I put into it.

Zabriskie and the services these apps rely on have a different obligation. Other people depend on them continuing to work, on their data surviving, and on changes not breaking yesterday’s behavior. Cheap code doesn’t remove that work. The possibility of a disposable personal tool doesn’t make a shared service disposable.

This is another thing I want the platform to support: keeping a useful program, replacing it while preserving my data, or retiring it when I’m finished. I don’t yet know whether maintaining my own apps will be as manageable as building them. Making that manageable is part of the work ahead.

Start with what I need

Sam Altman has described agents as virtual coworkers, including agents that do software engineering. There’s overlap with what I’m describing, but the part that interests me is what I have when the conversation ends. For a recurring need, I want the agent to leave me with an app whose behavior I can try, adjust, and return to. With Criterion, that means a widget I can glance at and saved films I can come back to. When my needs change, I can bring the agent back to change the software.

The agent helps me build something that does the job. I don’t need every interaction with my computer to become another task to delegate and supervise.

Imagine opening the computer and saying that you need a way to manage your finances. It asks where the information is, you choose what it can access, and you work together on how you want to see it. You tell it that you get paid irregularly, that you put money aside for travel, or that the usual monthly budget doesn’t make sense for you. It builds around that. A month later, when you understand what you need a little better, you change it.

I want that to be practical for someone who has never worked as a programmer. My own experience doesn’t establish that we’re there yet. It does make the goal much more concrete for me. A person should be able to bring their understanding of their own life to the computer and use that to shape how it works. They should be able to try something, live with it, and change their mind without having to convince a software company that enough other people want the same thing.

The social case is further out, but I want to push the idea there too. If people could keep their own data and make it available on their own terms, I could ask for the films my friends were talking about, the shows we were going to together, and the books I meant to come back to, all in a view that made sense to me. Someone else could make a different view of the things shared with them. Shared services would still have work to do, but we could have more say in how we encountered one another’s lives through them.

Watching my parents write BASIC made me think that making a program could be an ordinary part of using a computer. Now I have a restaurant app on my phone because I wanted to keep going to shows. I want more people to be able to make the things that let them do what matters to them, use them for as long as they help, and get on with their lives. We should build our tools and our platforms around that possibility. A program for one person is enough.

The Quickest Path to a Diff · The app died at Red Rocks while its health checks stayed green. Every line of code was fine. The sum was nobody's job.

Fri, 04 Sep 2026 18:30:00 +0000

Zabriskie is a social app I’m building around music, films, books, art, and the people who care about them. Its center of gravity is live music, specifically the jam band world I’m part of. In that world no two shows get the same setlist, so the crowd spends the night asking the same questions: what song is this, what have they played so far, who else is here. Zabriskie answers them while the show is happening. The setlist updates song by song as the band plays, the people in the crowd and the people watching the stream from home share one chat, and the current song sits on your lock screen and your watch so checking it doesn’t mean opening the app. The home page is called the Lot, after the parking-lot scene at these shows, and when one of your bands is on stage somewhere, it opens on that show.

I build it entirely through vibe coding: I describe what I want and evaluate the result in the browser, but I don’t read the implementation code. Coding agents write everything, the implementation, the tests, the audits. Claude Code, Cursor, and Codex have all merged work here. My job is deciding what to build and looking at what comes out.

On August 28 I was at Red Rocks for the second of two nights of the band Goose, and more people were on Zabriskie around me than had ever been on it at once. I spent the evening debugging instead of watching the show, because people kept walking up to tell me the app wasn’t working. They were right. The Lot mostly wouldn’t load. Requests that normally finish in well under a second were taking tens of seconds or timing out entirely. For an app whose whole point is telling you what’s being played right now, a page that answers in forty seconds isn’t degraded, it just has no answer.

Debugging from a seat at a show, with nothing but a phone, meant asking an agent what was wrong with the site. It couldn’t tell me. So I started pasting everything I could find in the Railway console into a cloud agent, screen by screen. The screen that mattered showed a queue of identical writes all blocked on the same lock, and the first diagnosis of the night came out of it: lock contention, a database deadlocking under load.

Half right. The queue was real, a client bug was re-registering push tokens over and over, and the first fix merged that night made it stop. But Postgres never reported a deadlock, the reads that reached it ran fast, and the health endpoint stayed green from the first stalled page to the last. The actual problem was a layer up, in my own app. Requests were stuck waiting to borrow a database connection from the app’s pool, the twenty-five connections it keeps open to its database, before their queries ever reached Postgres at all. Nothing measured that wait and nothing logged it. The agent reading my screenshots wasn’t hallucinating. It got handed a true fact from the wrong layer, because the right layer had no facts to give.

I want to be careful about the lesson here, because “the AI wrote bad code” isn’t it. Taken one change at a time, almost all of the code is reasonable. The problem is that an agent reasons locally. It sees the function it’s editing, the file around it, the change it’s about to make, and within that boundary its judgment is mostly fine. The things that took the app down aren’t visible at that scale. How many connections a request holds, how many queries a page runs, those are properties of the whole program at once, and nobody, me included, had ever looked at them. That gap is what this post is about: all of the reasoning was local, and everything that broke was global.

What was actually wrong

Every request has to borrow one of those twenty-five connections before it can touch the database, and it gives the connection back when its queries finish. The code the agents had written borrowed a connection to loop over a list of results, and inside that loop borrowed a second connection for each item, while still holding the first. Condensed, the pattern was:

rows, _ := db.QueryContext(ctx, listShowsSQL) // borrows connection #1
defer rows.Close()                            // #1 stays borrowed until the loop ends

for rows.Next() {
    var show Show
    rows.Scan(&show.ID, &show.Date)

    // loadSetlist runs its own query: it borrows connection #2
    // while #1 is still held.
    show.Setlist = loadSetlist(ctx, db, show.ID)
}

One request running this holds two connections at once. That’s fine while the pool has spare connections. That night it didn’t: twenty-five requests were in loops like this at the same time, each holding one connection and waiting for a second one, and a connection only comes back when one of those requests finishes. None of them could finish. That’s the gridlock. Requests queued behind it until they timed out.

The waiting happened inside my app, before any query reached the database. That’s why Postgres looked fine all night: the queries that did get a connection ran fast. The health check stayed green because it doesn’t run a query. It returns a string formatted once at startup.

The code had been in production since February. August 28 was the first night with enough traffic to fill the pool. The pool size didn’t help: nobody chose twenty-five connections for this workload. An agent picked the number at scaffold time, before the app had users, and nobody went back to it. Go’s pool tracks exactly this failure, WaitCount and WaitDuration, how many times a query waited for a connection and for how long. We had never logged them. Almost no query had a timeout. None of the numbers that would have explained the outage were being recorded.

The fix rewrote the worst read paths to stop holding one connection while borrowing another, retuned the pool, and started logging the wait counters. Deploying it was its own problem. The merge gate wanted a CI check that wouldn’t pass, the automation didn’t have permission to re-run it, and Railway, where the app runs, was having an unrelated incident and couldn’t start a deploy. I disabled branch protection from my phone, in my seat, at 02:19 UTC. I don’t know the exact moment the app came back. The fix merged while the band was still playing, and it couldn’t reach production until Railway recovered.

The remediation was excellent, which is the problem

The agents got the mechanism exactly right, wrote a regression test that shrinks the pool to a single connection and reproduces the hang deterministically, and swept the whole backend for the pattern. The first scan claimed 188 sites. The next claimed 217. The real number, once the analysis was rebuilt to check what actually holds a connection, was about forty, and all forty were fixed, with a static check behind them so the class can’t come back. The shows list alone went from 151 database round trips per page to 4. By morning the defect class was extinct and guarded. If the story ended there, you’d conclude the loop works.

The next day, an agent looked at one more endpoint and found it executing 51 to 71 SQL statements per request, of which 23 were distinct.

This was the endpoint every home-screen widget and both watch apps poll all day. The new guardrail had nothing to say about it, because it hunts the gridlock pattern, holding one connection while borrowing another, and this endpoint didn’t do that. This was a different disease. The part that stays with me is that the costs were documented. This comment sat in the code, verbatim:

// limit 1 — NOT watchOnThisDayLimit. Every card costs two more queries
// (highlight + setlist), and this is called from watchChipText and
// watchChipShowID as well as the handler. Fanning out to 5 at all three call
// sites would take one request from ~9 queries to ~33 to render a chip that
// shows exactly one stat. The rotation pays for rotation; nothing else does.

Whoever wrote that, and in this codebase that means an agent, clearly understood the cost model. It prices the request at about nine queries. The measured number was 51 to 71. Nothing enforced the nine, nothing would have noticed when it stopped being nine, and the other contributors to the total lived outside every context window that ever edited this file.

My first instinct was to call this a regression, the next disease showing up right after the first was cured. That’s not what happened. Nothing new appeared. These queries had been there all along, and someone finally pointed an instrument at them. Which raises the obvious question: if this endpoint sat at 51 queries per request until somebody happened to look, what is everything else sitting at?

A census

So I measured. I turned on a setting that writes down every question the app asks the database, loaded each page once as a normal signed-in user, and counted. In the table, statements are how many questions it took to draw the page once, and shapes are how many different questions were in the mix. One request per page, one user, so treat these as shape rather than precision.

surface	statements per request	distinct shapes
Shows page	340	14
The Lot (home)	292	164
Band page	186	28
Watch idle-context (the endpoint above)	72	43
Flow feed	40	30
Profile feed	14	8
Notifications	3	3

Six more rows sit in between, from tour stats at 60 down to the profile page at 9. The endpoint from the last section, the one that just got its own perf fix, is the fourth worst page in the app; the census caught it before that fix landed. These numbers aren’t just latency, either. Every statement stretches how long a request holds its borrowed connection, so a 292-statement page is 292 slices of hold time on the same pool that gridlocked in August. Same arithmetic, different magnitude.

The ratio of statements to shapes splits the table into two diseases. The Shows page is the classic loop: 340 statements but only 14 shapes, a handful of queries repeated once per card for sixty-five cards. The Lot is the more instructive one: 292 statements across 164 distinct shapes. There’s no villain loop to point at. It’s an accretion. Every feature that ever shipped onto the home screen brought three or four queries with it, each addition individually negligible, and the page now runs about three hundred statements, one after another, every time someone opens the app.

The Shows page bothers me the most, because of where its queries live. They sit inside the function that draws one show card, so every page that draws cards inherits them. And the Shows page has no page size: it draws every show happening in the next seven days, and the day I measured, that was sixty-five cards. Neither piece looks expensive on its own. Each query takes a fraction of a millisecond, and a loop over cards is how every list ever gets built. The cost only exists in the multiplication, and no edit ever held both ends of it. When the outage-night fix reported the shows list going from 151 round trips to 4, that count was true, for the loop being edited. The card renderer it calls kept its queries, and the same page runs 340 statements today. A reviewer handed that change would have approved it. My review loop did approve it. Nobody was stupid and nothing was broken, and the app was still guaranteed to meet its first real crowd unmeasured.

The same week’s digging also turned up a single unindexed function quietly eating 79.5 percent of all production database time. Aggregate costs, invisible because aggregate was nobody’s view.

Why this keeps happening

I’ve written about this failure shape before. In June the test suite grew a pile of shared fixture data that no test owned (The Test Suite Was the Incident). Now it’s the application itself: a shared pool and a shared database, exhausted by code that passed every check anyone ever ran against it. Both times the broken thing was shared, and every check that cleared it, the build, the tests, the review of one diff at a time, was local.

There’s a personal sting in this one. My academic work was distributed systems and fault injection: systems whose parts are individually fine and fail in combination, and tools that find those failures by attacking a running system on purpose, because reading the parts won’t surface them. I spent years doing that to other people’s systems. I never did it to mine. No load test, no drills, nothing that ever tested the whole running program. In someone else’s system, this failure wouldn’t have surprised me for a second.

Watch how something new gets added, because this is where the difference lives. A developer adding a query to a page they know starts from what’s already there: there’s a query two functions up that fetches almost this, so they extend it, or join against it, or pull the shared part into a helper. Nobody calls that global reasoning. It’s just what editing a program you remember looks like. A session doesn’t do any of that unless you ask for it. It takes the quickest path to a working diff, and the quickest path is a fresh query, a fresh component, a fresh style, written right where it stands. The backend queries the attendance table, who’s going to this show, from 264 places in 69 files. The little uppercase label that sits above a card title has been implemented hundreds of times, an inline style here, a one-off helper there, and no session ever reached for a shared one, because finding it costs a search and writing a new one costs nothing. Refactoring doesn’t happen unless you ask. Modularization doesn’t happen unless you ask. And review doesn’t catch it: every change here is reviewed, a review bot plus an adversarial reviewer we built, but both read the diff, and duplication is invisible in a diff. The copy looks clean on its own, the original is in some other file, and no human opens the file.

Human codebases rot this way too. The other difference is what happens to the person who wrote the code, after the code ships. You ship a slow page, you watch it crawl, and you build the next one differently. You get paged at 2am and the lesson sticks. Nobody in this loop gets paged. This codebase took a couple thousand pull requests in a few months, each one from a session that had never seen the app before and was gone before the code ran. A session knows only what’s written down for it, and nothing written down said what a request should cost. I never thought to write it down either.

To be fair to the agents, the problem isn’t ability. The same loop that missed 292 statements on the home page diagnosed the gridlock while the site was down, and reverted six of its own eight optimizations when the benchmarks said they were wrong. Handed the evidence, agents are as good as any engineer I’ve worked with, and faster. But diagnosis is local: the evidence is sitting right there. Prevention is global, and nobody in the loop was assigned it. Including me. The nine-queries comment crossed my desk in a diff and I approved it, because it looked careful. A cost argued in prose looks handled. Nobody ever checks it again.

Where does global reasoning come from?

The thing that broke, every time, was a property of the whole program: connections held across a request, statements accumulated across a page, one function’s share of the database’s time. The reasoning that touched the code, every time, was local: the agent sees one function, the tests check one behavior, a review reads one diff, I approve one change at a time. None of that reasoning was wrong, which is the uncomfortable part. The Shows page runs 340 statements per request and every line involved is fine.

So where is reasoning about the whole supposed to come from? Not from the sessions: each one is born into a diff and gone by the merge. Not from the checks: every check anyone has written looks at one change. Not from me: I decide what to build and judge what comes out, and I have never read this program. Nobody in this loop stands anywhere the whole program is visible.

The human version of global reasoning was never a formal method. It was one person editing a program they remembered. Integration happened by default, because the developer knew what was already there; the whole lived in somebody’s head, and writing the code was what kept it current. This loop has no such head, and that leaves me with questions I can’t answer yet. Will models ever do this unprompted, or is reading the neighborhood before every change simply never the quickest path to a diff? Are context windows even the constraint? This program still fits inside one, and no session has ever chosen to load it. Maybe the answer isn’t cognition at all: static analysis, dynamic analysis, test suites strong enough to pin down the whole program, machinery doing for the loop what memory used to do for a developer. Or maybe a loop like this only works with a human in it who actually reads the code. I don’t know which of these it is. What I know is what happens with none of them, because I watched it happen from a seat at Red Rocks.

Every Card Will Show · The earlier implementation, tests, audits, and accepted proofs said the algorithm was correct. None of them stated the whole problem.

Mon, 24 Aug 2026 04:00:00 +0000

Zabriskie’s home screen is called The Lot. At the start of this work, it was one long list of cards. One version rendered 29 cards and produced a page more than 15,000 pixels tall. I asked the agent to replace that list with five smaller programs that would appear during the morning, midday, afternoon, evening, and late night.

The requirement had two parts. The largest visit should contain as few cards as possible, but the five visits together should show every card whose content remained available and relevant to the person throughout the day. I call those cards eligible.

Late on August 23, I asked the coding agent a specific question.

If one person visited those five programs during one calendar day, would they see every card whose content was available and relevant to them?

The agent answered yes. Every eligible card would show.

It wasn’t true. After three days of repairs, I was frustrated enough to ask the agent to formalize the requirement in Lean, a language and proof assistant. As part of that work, the agent finally compared its failed audit’s eligible-card count with the 27 positions allowed by its own caps. The audit contained 20 more eligible cards than the schedule could possibly show.

The agent hadn’t invented a file, a function, or a test result. It had produced real code and real evidence. What it invented was the connection between them: because each artifact established something close to my requirement, together they established the requirement itself. They didn’t. None checked whether one person’s five visits on the same day covered every eligible card.

I kept merging because I asked for comprehensive audits before doing so. Three were completed, and all three said the implementation was working correctly. I owned the decision to ship, but I wasn’t ignoring contrary evidence. The audits themselves had inherited weaker versions of the requirement.

The Lot at 11:35 PM on August 23: the late-hours program, a live lead, and Act Now beneath it.

How the requirement changed

The agent’s design promoted one card to the top, called the Hero, and grouped the supporting cards beneath it into sections such as Act Now, Discover, and You. The Lot needed a cap, meaning a maximum number of supporting cards on each visit. The question was how to make that cap small without leaving an eligible card unseen across all five visits.

Instead of preserving both parts of the requirement, the agent made a series of local decisions about what each visit could omit. It limited which sections could contribute cards, made the Hero compete with the supporting cards, introduced different caps at different times, and created a ranking policy based on deadlines, show-night context, tour status, preferred times, and viewing history.

Those are plausible ingredients for a scheduler. The problem was that the agent chose them without bringing the resulting product rules back to me, then began testing the rules it had chosen. Some were plainly wrong. Connections, a card meant to begin a listen, read, or watch, received a late-night preference. The personal-history and broader historical-archive On This Day cards were wired backward. One code comment even called the three-section limit the “owner’s cap decision.” I had made no such decision.

The clearest numerical argument was 27, presented as the number of positions available across the day. It came from adding five source-code caps: 6 + 5 + 6 + 4 + 6. But those states belonged to two different calendars. A weekday could reach morning, workday, evening, and late night. A weekend could reach daytime, evening, and late night. There was no date on which one person could visit all five states.

Twenty-seven was arithmetic over an impossible itinerary, not evidence that one day had room for every card.

Farther down the late-hours program: Coming Up, with the next Goose and Phish dates. A single program is intentionally partial; the disputed guarantee concerns the five visits together.

The number doesn’t explain the 20 missing cards by itself. It establishes something narrower: the capacity argument didn’t model the day I had described.

The tests made similar substitutions. A documentation audit promised that every card could appear within a week. A reachability test considered one card at a time, treated that target as long unseen while its competitors were recently shown, and searched for some state in which it could win. A separate adversarial review confirmed that every listed card had a classification and could pass through the ranking system. All three checks passed. None asked whether the same fixed set of eligible cards appeared across one person’s five visits on one date.

The implementation also split the final decision between the Go server and the browser. The server ranked cards, but the browser could still group, suppress, and reorder them. That made a passing server test insufficient: it did not cover the entire path that decided what a person saw. The one end-to-end test pinned a single midday response. It checked that the sections surviving that cap rendered, not that five visits covered the catalog.

The checks hadn’t lied. They answered weaker questions and the agent presented their passing results as answers to mine. I accepted that evidence as stronger than it was. By then the hallucination was executable: it had become code, tests, and audits that could all turn green together.

What Lean made explicit

Lean is both a programming language and a proof assistant. It lets us describe a system precisely and then mechanically checks what follows from that description. This was my first time using Lean. I had the agent build a Lean model beside Zabriskie’s production server, which is written in Go. Lean does not run when somebody opens The Lot, and it does not read or prove the Go source.

By the end, the formal work needed to answer three separate questions:

What is the smallest per-visit card limit that could possibly cover the catalog?
Given a schedule that fits that limit, can ranking or capping still push out a card that was promised a place?
Does the separately written Go selector behave like the Lean selector we proved things about?

These are different claims. A lower bound does not prove that a working schedule reaches it. A proof about a Lean function does not prove that Go matches it. Keeping those boundaries visible made the rest of the argument easier to audit. We reached that decomposition only after proving two incomplete statements.

Lean proved the wrong day

The first Lean version of the card-selection logic failed its coverage check, but it represented only part of the inventory and could not answer the full question. The agent’s next model expanded the inventory and passed by treating weekday morning, workday, evening, late night, and weekend daytime as five consecutive visits. No calendar day is both a weekday and a weekend.

Lean correctly proved coverage for an itinerary no person could take. The theorem was valid. The agent’s claim that it represented my day was not.

That proof entered the main source code and became a required continuous-integration check. I merged it because the proof made the agent’s explanation feel stronger than an ordinary test. But Lean only checks the statement it receives. It cannot decide whether an agent translated a product requirement faithfully.

Coverage was still only half the problem

After the calendar was repaired, the basic coverage question became one set equality. Read this theorem as: combine the cards assigned to all five programs, and the result is the complete modeled catalog.

theorem union_of_five_programs_is_all_modeled_cards :
    cardsAcrossFivePrograms = modeledCardNames := by
  native_decide

It was easier to inspect than the earlier model, but it still omitted part of the product requirement.

The original problem was a 29-card, 15,000-pixel page. A theorem saying the five programs cover the catalog accepts the stupidest possible implementation: show every card on every visit. Coverage is a constraint. The objective is to make each visit as small as possible while satisfying that constraint.

The inventory changed during the repairs. From this point forward, the model contains 45 distinct cards, not the earlier incident inventory. One is the structural Hero, leaving 44 supporting cards.

Because some supporting cards were assigned to more than one program, the agent-generated repair reserved 54 positions across the day: 10, 13, 11, 10, and 10. The proof certified coverage, but only by padding the schedule. It still had not stated the size objective: minimize the largest supporting-card count across the five visits.

This is where Lean helped me understand what I was actually trying to do. It did not recover the product intent. Reading the theorem literally and challenging every input and unexplained capacity made the requirement inspectable enough for me to see what the agent had left out.

Proving the minimum

Once stated correctly, the lower bound is simple. There are 44 supporting cards and five visits. If every visit showed at most eight supporting cards, the entire day would contain at most 40 positions. Four cards could not appear. Therefore any covering schedule must allow at least nine supporting cards on one visit.

That counting argument became a general Lean theorem. The names beginning with h are assumptions supplied to the proof: the catalog has no duplicates, the visits cover it, and each visit contains at most max cards. The line after the colon is what Lean proves from those assumptions:

theorem covering_schedule_capacity_lower_bound
    (catalog : List String)
    (keeps : List (List String))
    (max : Nat)
    (hunique : catalog.Nodup)
    (hcover : covers keeps catalog = true)
    (hmax : ∀ keep ∈ keeps, keep.length ≤ max) :
    catalog.length ≤ keeps.length * max

The conclusion, catalog.length ≤ keeps.length * max, is just the counting argument in symbols. A 44-card catalog and five visits turn it into 44 ≤ 5 × max, so max must be at least nine.

Proving that ranking cannot break the schedule

The lower bound only says that nine slots are necessary. It does not say the real selector will use those slots correctly.

Within the supporting-card phase, cards play two roles: reservations and filler. The product gives each supporting card a guaranteed visit, which I call its reservation. The selector computes one ranked order, admits every fitting reservation first, then uses that order for section diversity and filler. The property we need is that ranking can change the filler, but it cannot evict a reservation that fits within the visit’s cap.

In plain language, the theorem says:

For any cards, visits, capacities, and ranking inputs: if every supporting card has a reserved visit, and the reservations for each visit fit its cap, then the union of the final ranked-and-capped outputs contains every supporting card.

The exact Lean statement is longer because it names all of those inputs. Everything before the colon describes the cards, visits, ranking inputs, and two assumptions. Everything after the colon is the coverage claim.

Show the exact Lean theorem

theorem capped_selector_covers_any_fitting_schedule
    (units : List Card)
    (visits : List Daypart)
    (capacity : Daypart → Nat)
    (eventLead onTour : Daypart → Bool)
    (staleness : Daypart → List (String × Nat))
    (hassigned : ∀ card ∈ supportingCards units,
      ∃ visit ∈ visits, ownsProgramPosture card visit = true)
    (hfits : ∀ visit ∈ visits,
      ((supportingCards units).filter fun card =>
        ownsProgramPosture card visit).length ≤ capacity visit) :
    covers
      (cappedSupportingKeeps
        units visits capacity eventLead onTour staleness)
      (supportingNames units) = true

That is the symbolic version of the plain-language claim above.

The Lean selector is called capCards; its supporting-card phase admits the reserved cards first, then adds section diversity and fills any remaining positions by rank. The structural Hero is checked separately.

The proof intentionally leaves the filler identities open. After admitting the reservations, the selector first tries to represent a section not already present, then fills any remaining slots with the highest-scoring cards. A card’s score considers its section, deadline urgency, show-night and tour context, preferred time, viewing frequency and recency, and fixed priority within its section. A special morning bonus also applies to the two On This Day memories. Those signals choose the filler cards, but cannot displace a reservation. The server applies the fixed display hierarchy afterward.

The remaining obligation was a concrete product schedule that met the nine-card bound. Each supporting card has exactly one reservation, balanced as:

morning     9
midday      9
afternoon   9
evening     9
late        8

Those 44 positions contain the 44 supporting cards exactly once. A card can still prefer several times of day for ranking, but a ranking preference does not create a second reservation.

The algorithm and the product table therefore remain separate. Moving Connections to a different program requires checking that the new reservations still fit, but it does not require rewriting the proof that the selector preserves any fitting schedule.

A separate Lean check applies the generic theorem to this product table. It establishes both halves of the result: the selector covers all 44 supporting cards, and it returns at most nine of them on each visit. Nine is necessary, and this schedule reaches nine.

Within the selector, each output contains the structural Hero plus at most nine modeled supporting cards. The rendered page may also contain Live Now, a separate module for a show currently playing, and other modules outside this theorem.

That settled the generic law and one concrete product schedule. It did not yet show that the separately written Go selector made the same choices.

Making Go answer to the model

The proof so far is about Lean. Its generic theorem receives an eligible catalog and ranking signals; it does not build those inputs for a particular person. On a live request, Go does that work: it determines which cards are eligible, calculates recency and frequency from the person’s impression history, supplies the event and tour context, and runs the separate lotCapCards selector.

The executable Lean selector mirrors the scoring and capping logic so that we can compare it with Go. Connecting the proof to what a person sees takes three more steps:

Lean executes its selector on concrete catalogs and ranking inputs.
Go runs lotCapCards on the same inputs, and a differential test compares the two outputs.
The server sends Go’s final allow-list and order to the browser, which renders that order without running another scheduler.

This follows the same broad shape used by Cedar, the open-source authorization language developed at AWS: prove properties of a model, implement a separate production engine, and use differential testing to look for disagreement. Our boundary is much smaller, but the separation is the same.

The number 128 has no mathematical significance. It is simply the fixed number of comparison cases we currently run: the complete 45-card catalog plus 63 generated subsets makes 64 catalogs, and each runs once with every card unseen and once with every card recently seen. That makes 128 five-visit comparisons.

This covers only a tiny fraction of the possible inputs. Each of the 45 cards can be eligible or ineligible, creating roughly 35 trillion possible subsets before considering mixed viewing histories or other ranking inputs. Adding more cases would improve the chance of catching a disagreement between Lean and Go, but it would not strengthen the universal Lean theorem. This comparison is a bounded implementation check, not another proof.

Regenerating those cases exposed a mismatch on one filler card. Go allowed a card to prefer several times of day; the executable Lean version allowed only one. We corrected the reference implementation without changing the generic coverage theorem.

After that correction, the selected-card sets agreed on all 128 walks. A separate Go unit test exercised the 45 modeled cards from both history starting points and under all four combinations of whether an event led the page and whether the person was on tour.

Together, these checks connect the proved model to Go, but they do not prove every possible Go execution. Ordinary Go and React tests cover the final server-to-browser handoff in step three. There is still no browser test that performs all five visits and covers the entire path in one run.

What the theorem does not cover

The fixed-eligibility assumption was not a limitation I had thought about when I asked for no starvation. It became visible when we tried to generalize the Lean model so that the eligible set could change between visits. The stronger guarantee was false. Lean helped us find a concrete counterexample using Last Night, a prompt to review the previous evening’s show.

Last Night has a morning reservation:

During the morning visit, Last Night is not eligible, so the selector cannot show it.
It becomes eligible at midday, after its reservation has passed.
For the rest of the day it can appear only as filler. But every later program’s reservations already fill its capacity, so no filler position is available and Last Night remains unseen.

That exact walk is now checked in Lean, which reports Last Night as the only unseen card. It does not contradict the coverage theorem: the theorem starts with one fixed eligible catalog, while the counterexample changes that catalog between visits.

For the actual Last Night flow, that boundary is acceptable. If a person had already RSVP’d to the previous night’s show, the card is eligible by morning and its guarantee applies. The counterexample corresponds to someone adding the RSVP retroactively after the show, perhaps the next afternoon. In that case, not receiving a same-day prompt to write a review is acceptable product behavior. The specification deliberately admits that failure rather than complicating the scheduler and theorem to guarantee a case I do not need.

Algorithms are hard

Before the final decomposition, the agents had already produced an implementation, unit tests, end-to-end tests, documentation, multiple code audits, numerical arguments, and accepted formal proofs. Those earlier passing artifacts said the requirement had been implemented correctly. It had not.

Those artifacts agreed because the same failure pattern propagated from one to the next. One calendar day became a week. Guaranteed coverage became isolated reachability. Capacities from mutually exclusive programs were added together. Later, coverage became the whole requirement, allowing a proof to certify an unnecessarily large schedule. Green checks showed that each substituted design behaved as specified. They were not independent evidence that we had solved the original problem.

After all of that, the fixed-catalog problem was finally easy to say: for the cards that remain eligible across all five visits, minimize the largest supporting-card count on any visit while showing each card at least once. Saying it clearly did not make it easy for an agent to preserve both parts through an implementation, tests, audits, and proofs.

Lean’s value was not another green artifact. It forced the argument into pieces I could challenge: a counting lower bound, a generic selector guarantee, a replaceable product schedule, and a bounded comparison against Go. Lean did not discover that decomposition. The agents also produced accepted proofs of an impossible day and of coverage without minimization. Reading those statements literally exposed the missing objective, but deciding whether a theorem described the product remained my job.

This wasn’t one model having one bad run. I tried to repair the work with Claude Opus 4.6, Claude Opus 4.8, Claude Opus 5, and Codex using GPT-5.6 Sol. None independently arrived at both the invariant I had stated and an implementation that preserved it. They found different problems and produced more artifacts, but changing models didn’t break the pattern. Each could accept or recreate a nearby requirement and then generate convincing evidence for its own version of the problem.

Three things now seem clear. Passing artifacts are not independent evidence when they inherit the same changed requirement. Lean proves the statement it receives, not the behavior I meant. And the formal invariant, the product schedule, and the check against the implementation have to remain separate and inspectable.

The next problem is whether an agent can expose that decomposition before it has already built the implementation, the tests, and the proof around a nearby requirement.

The Safeguards Have to Run · Two abandoned notebooks showed me the difference between a research rule and a working control.

Tue, 18 Aug 2026 12:00:00 +0000

This is Part 2 of The Machine in the Lab, a series about using an autonomous research program built from large language models (LLMs) to build SetScope, a live Goose song guesser.

In the first research notebook, I told an autonomous program to keep training and test data separate. Its report showed a clean split. It wasn’t.

How could both be true?

The program was learning whether two recordings were performances of the same song. It created pairs of recordings, then randomly divided those pairs between training and testing. A performance called A could appear with B during training and appear again with C during testing. The rows were different, but the program had already encountered A. That meant the test could reward it for recognizing a performance it had seen rather than learning what made two performances instances of the same song.

This is the kind of problem a working safeguard should prevent. Instead, the rule existed in the research plan while the code quietly violated it.

The LLM wasn’t just writing code that I reviewed step by step. I supplied the objective, the audio collection, and the rules. The program proposed experiments, wrote and ran code, interpreted results, and chose what to investigate next without asking me to approve every decision. I called each sustained run a notebook: the accumulated data, code, experiments, reports, and decisions for one research direction.

The original objective was to build SetScope, a system that could listen to a live Goose show and name the song while the band was still playing it. A live version can change in speed, length, arrangement, and improvisation while remaining recognizably the same song.

The first song-identification experiments produced a positive result: measurements from the composed sections appeared to distinguish one song from another across live performances. That suggested the composed parts preserved much of a song’s identity while the improvised parts accounted for much of the variation between performances. So the program started asking what those same measurements could tell us about the improvisation. Its reports arrived quickly and appeared to form a coherent account of the music, so I let it continue. That detour became Notebook 1.

Notebook 1

Correcting the pair split reduced the reported performance of the classifier that used measurements taken only from the audio. There was still useful signal in it, but the original result hadn’t tested whether it could generalize to performances it had never encountered. The program had acknowledged the rule about separate evidence without implementing a split that enforced it at the level of a complete performance.

The notebook had already expanded beyond that classifier. One later experiment tried to identify open-ended improvisation by comparing an opening section, a middle section, and a closing section of each track. When actual musical boundaries were unavailable, the implementation treated the first and last ninety seconds as composed material and the middle as the jam.

I asked the program to evaluate the detector against a separate group of 44 performances that had not been used to choose its numerical cutoffs. Six were too short for the method, leaving 38 analyzable cases. The fixed windows placed composed material that occurred after the opening ninety seconds inside the region treated as the jam. The detector repeatedly treated changes within those composed passages as evidence of open improvisation.

That audit rejected this detector. It did not provide a clean test of the entire path that led to it, because earlier results had already influenced which features, thresholds, and questions the program pursued. I stopped Notebook 1.

Notebook 2

Notebook 2 continued the improvisation work under a written methodology intended to prevent the first notebook’s failures. It began on April 30 and assigned each song permanently to one of three uses: trying methods, choosing among them, or one final test that was supposed to remain unopened until the method was finished. It also rejected the blanket ninety-second division in favor of boundaries chosen for each song from the music itself.

The code didn’t follow those rules. The first implementation again set the opening and closing regions to ninety seconds. Later analyses reused measurements cut at those boundaries. The method had warned against the assumption, but nothing compared the running code with the method and stopped the work.

The final test wasn’t protected either. The locked collection assigned every performance of five songs, 25 recordings at the time, to remain unopened until the method was finished. Development analyses accessed four of those five groups, accounting for 22 of the 25 recordings, while the program was still choosing its method. A label in a data file said the groups were reserved. Nothing prevented the program from opening them.

I had started Notebook 2 while Notebook 1 was still present for reference. On May 4, I told the program to remove every Notebook 1 file so it couldn’t contaminate the ongoing work. The cleanup removed the visible scripts, reports, figures, and data from the main project folder. But it couldn’t make Notebook 2 a clean restart: the new work had already inherited the old ninety-second default.

By the time I stopped Notebook 2 on May 6, its fixed segmentation had propagated into analyses, figures, two draft blog posts available by direct link, and listening exercises. I had also put accounts of the earlier work on the project’s public site and sent listening exercises to friends. I don’t know how many people saw the pages, but the autonomous process had produced two coherent bodies of work that I couldn’t trust.

A later inventory on May 17 exposed another failure. After both notebooks had been removed from the project folder, 1,120 generated feature and measurement files from the abandoned work remained in reusable storage outside that folder. A subsequent notebook had loaded 131 of those files while calculating one of its numerical cutoffs. Removing them changed the number only slightly and didn’t materially change the affected results. But the supposed clean slate hadn’t been clean, and the program had no record showing where every generated file had come from.

The safeguards have to run

What had the written safeguards actually protected? In practice, very little: every named boundary could be crossed without stopping the run.

A performance-level split should have rejected any training and test pair that shared a performance. Reserved data should have been inaccessible until a recorded release step. Generated files should have carried their source history, and a purge should have verified that no descendant remained in shared storage. A run whose code contradicted its written method should have stopped before producing dependent results.

Those checks belong inside the autonomous process. My job is to define and approve the constraints, not to reconstruct every data assignment, generated file, and numerical constant after the program finishes. If the system requires that reconstruction, it has not automated the research process. It has automated the production of work for a human auditor.

A natural-language instruction can describe a safeguard. It cannot enforce one.

After Notebook 2, I set the improvisation work aside and returned the program to SetScope’s original Goose song-identification problem. A new live show offered evidence that no earlier notebook could have seen. But each show could be new only once. Once the program used that show to decide what to do next, was it still independent evidence or had it become part of development? That’s the question in Part 3. Part 4 returns to the separate problem exposed by the ninety-second detector: whether a consistent measurement actually represents the music named in the claim.

The Whole Night

Mon, 17 Aug 2026 23:00:00 +0000

On Thursday, August 13, Goose opened the West Coast run in San Diego with “Animal.” Sixteen people in Zabriskie had said they were going. Seventeen had said they were on the couch. By the time the band finished the “726” encore, fourteen people had written 160 messages in the Chomp and five had sent 48 reactions across a twelve-song show.

It was a regular Thursday show, which made it useful for understanding what had changed. The RSVPs split almost perfectly between GOING and COUCH, and the app had to do a different job before the show, in the room, at home, and after the encore.

Seven weeks earlier, I published The App That Lives Between Shows. The live room worked, The Lot had finally given the app a home screen, and the watch clients were running in our field builds. I wrote that the hard product problem was the gap between one show and the next. The Lot existed, but it did not yet know what time it was.

I was not exactly wrong, but I had drawn the boundary in the wrong place. A show does not begin when the first note lands or end when a live flag changes in the database. The plan starts days earlier. During the show, a person standing at the rail and a person watching from a couch need almost opposite things from the same room. The setlist, conversation, and people remain after the band leaves the stage. Over the next seven weeks, Zabriskie started to follow that longer edge.

The Lot Learned What Time It Was

In June, I wrote as though The Lot was mostly finished because it had become the home screen. It was not. It was a beautiful pile.

Every useful thing we built earned a card, and every card wanted a permanent place. A live show, an upcoming RSVP, a Crate recommendation, an anniversary, an invitation, a recap, a bracket, and the latest thing somebody did could all be individually correct and collectively exhausting. Opening the app increasingly answered the question, “What does Zabriskie have?” That is a bad question for a home screen.

The better question is: what should we do right now?

One Lead, Not Every Feature

The Lot now has two clocks: the show’s and the person’s. It knows whether somebody is going, couch touring, or still deciding; whether a room is live; whether a show just ended; what is waiting for a response; and which cards have been sitting unseen. From those candidates it chooses one lead move, a short schedule, and a few supporting cards.

A live RSVP wins. A show later tonight comes next. The morning after a show, the unfinished rating and Chomp can take the space. An anniversary can win a quiet day. If none of those has a claim, the time of day breaks the tie: something to spin in the morning, something to return to during the workday, the local scene or a stream in the evening, a deeper archive dig on the weekend. The page says which moment it was programmed for because otherwise the cards simply appear to move around by accident.

My actual Monday evening Lot: the unfinished Goose night wins the lead; the Flow sits beneath it.

This required the page to do something software is generally bad at: decline to show correct information. We capped both the number of sections and the number of cards inside them. One busy section cannot consume the page. A card that has been buried for several visits gets one chance to move up. When the final card in a section is dismissed, its empty heading leaves too. The schedule stays short. The rest of the app still exists; the home screen is no longer required to prove it.

Tonight Is a Room, Not a Date

The earlier Lot split a live show, the other shows tonight, and a person’s couch plans into separate pieces. The new Tonight card puts the open rooms first and the later rooms after them. A live row carries the current song and set, the stream when one exists, and one compact door into the Chomp. The same row can still change a plan between GOING and COUCH. Several couch RSVPs across a run collapse into Your Couch Tour, an itinerary with the nights left, the correct local tune-in times, and the other people staying home.

A pending Bring Your People invitation now lands on The Lot instead of depending on somebody finding the original notification again. What’s New moved out of a floating toast and into a card that can be read and dismissed. Received reactions say who reacted and which post they found. A locked opener call becomes a status instead of pretending to be another urgent action. These are small decisions, but they are the difference between a page that understands the state of something and a page that merely knows the feature exists.

One screen lower: weekend options and the West Coast run, weighted by who is already in the plan.

The Archive Has to Return Somewhere

The Lot also became the place where a completed show can come back with a reason. The next morning it can ask how the night was, collect the jam of the show, and reopen the original Chomp. Much later, On This Day can put a personal anniversary beside a notable show from the archive. A bookmarked show can return with the explanation for why it was saved. A new release can arrive with its listening party already attached. A scheduled Tape Night can reopen an old room at an actual time instead of leaving another permanent card on the page.

The quieter parts matter too. The Crate contributes one pick, not fifteen. The Flow contributes a glimpse, not a second feed. Weekend shows, incoming releases, saved shows, Local Scene, recent love, tour statistics, recommendations, and the year so far all have a way onto the page, but none is entitled to remain there. There is no follow graph underneath this. The inputs are attendance, plans, taste, the person’s own history, and what the community is doing now.

The social edge of a future show beside a recap that can finally return after the festival.

Nine nights on the couch become one tour itinerary, with local times and stream links attached.

The Work Was the Editing

Then we kept sanding it. Every card received art and a place inside a real section. Buttons stopped changing shape from one card to the next. We fixed shadows, contrast, touch targets, haptics, sharing, loading, and dismissal, then added guardrails because each one had already drifted at least once. Cards can share their own image instead of an authenticated URL nobody else can open. The page survives the old mobile bundles that do not know about its new priorities. Its tests now fail when a card escapes its section, a call to action changes shape, or one kind of card crowds out everything else.

I count 187 non-merge commits in this window touching Lot-specific handlers, rendering, tests, or those guardrails. That number does not prove that we chose correctly. It does explain why the four screens above are not the home screen I published in June.

The Lot is where the rest of this story first appears. It notices that a night is coming, hands the person into the right room, and then decides what from the night is worth bringing back.

Before Doors

The Lot can decide which night matters. The show page still has to help finish the plan. The June page already brought RSVPs, seating, trip planning, opener calls, and the live setlist into one place. The August work focused on the awkward part: a plan exists, but it is not finished.

The plan before doors: one show, three kinds of intent.

Bring Your People lets someone pull another person into the plan. Ticket Check asks the embarrassingly necessary follow-up: you said you were going, but do you actually have a ticket? The rest of the run now sits beside the current night. A Song Wish List collects what people hope to hear, while Played Here Before brings back the band’s earlier visits to the room. For someone staying home, the same run becomes a tune-in schedule in the correct timezone.

The home-screen widgets follow the same clock. On an ordinary day they can surface an anniversary, a recent show, or the year so far. As a show approaches, they become a countdown and then a live glance. Gap Watch can pull an overdue song into the days before the next RSVP. Call the Opener already existed in June; what was missing was the answer. Now the result can find its way back to the person who made the call instead of disappearing into the setlist.

This is still small-community software. Since July 1, people have sent 11 show invitations and added 28 songs to wish lists. That establishes that somebody used them. It does not establish a habit, and it certainly does not allow me to turn two small counters into a victory lap.

At the Rail

Once the first note lands, the problem reverses. Before doors, the app asks people to make decisions. At the show, it should ask for as little attention as possible.

We call that posture the rail: someone is in the room, service is bad, both hands are occupied, and looking at a phone is already one thing too many. The Apple Watch and Wear OS faces can carry the current song, the current set, the setlist so far, and the latest Chomp messages. From there, one tap can send a reaction or like a message; a canned reply or dictation can answer the room without pretending that a watch is a very small phone.

Most of the work was in the cases that are easy to omit from a demo. The watches recover from stale state and keep long song titles inside a round screen. They distinguish set break from encore. If two shows overlap, a long press opens a switcher so the watch follows the room the person intended to follow. The encore produces one deliberate haptic. After the show, the face holds the final setlist in an afterglow instead of immediately dropping back to an idle screen.

Patrick led the Wear OS client, while I built most of the Apple Watch client. We made parity a project rule: when a live-show capability belongs on the wrist, both clients get it. Parity does not mean copying the phone. If the watch requires the same attention as the phone, we have missed the reason to build it.

The phones moved in the same direction. The Live Activity on iOS and the ongoing notification on Android carry the current song without requiring the app to remain open. Haptics mark actions and actual show transitions rather than buzzing indiscriminately. The full setlist and conversation are still there, but the default at the venue should be a glance followed by putting the device away.

On the Couch

At home, getting out of the way is the wrong goal. The stream is on, a keyboard is available, and the conversation may be the reason someone opened Zabriskie at all.

We call that posture the salon. The composer remains in reach, the thread gets the screen space, and a show switcher appears in the header when several bands are live. Opening the keyboard expands the conversation rather than crushing it. The setlist and the full live page remain one move away.

The RSVP supplies gravity, never permission. GOING can prioritize a show on the wrist and change the countdown and copy on The Lot. COUCH turns that copy toward tuning in. Neither choice locks or hides anything. The larger RSVP-weighted phone layout is still unfinished, and that is intentional for now: someone at the venue can spend set break reading every message, while someone at home can send a watch reaction while cooking dinner.

This is why the San Diego split mattered. Sixteen people said they were at the show and seventeen said they were on the couch. Both groups reached the same setlist and the same Chomp, but the interfaces around that shared room could begin to pull in different directions. When Goose reached “Big Modern!” in the second set, the wrist could carry the song title for someone in the amphitheater while the phone could give the thread room for someone at home. The production counts do not tell me which RSVP group wrote each message, and I do not need them to. The near-even split shows that both postures were present.

Across all shows since July 1, 33 people have sent 1,303 live messages and 27 have sent 4,962 reactions. The reactions come from a smaller group and arrive quickly. That is the point: sometimes a sentence is too much, but a reaction is still enough to say that somebody else heard the same thing.

After the Lights

In San Diego the encore was “726.” The older app mostly treated that as the end of its useful work. The setlist completed, the live state shut down, and the room receded into the archive.

Now the night has an exit. The watch and phone hold the final state in an afterglow. The next morning, The Lot can ask for a one-tap rating and return the completed show with its setlist and recap. The rating also lives on the show’s page instead of existing only on a temporary home-screen card. The recap reopens the original Chomp, because a database flag changing from live to completed is not a good reason to break the conversation.

The August 13 encore, already returning as part of the archive.

Much later, the same show can return through On This Day. What used to be one card is now a reel that can place a personal anniversary beside a notable archival show and the other concerts that happened on that date. The watch and widgets can surface those anniversaries too. We do not need to manufacture a new commemorative post every year. The record already exists, with the people, setlist, poster, recording links, and conversation attached.

This changed how I think about the archive. A completed show is reference data, but it is also the durable object that the live room, the next morning, and the anniversary can all return to.

The Rest of the App Changed With It

The show lifecycle became the easiest way for me to understand the work, but not everything we shipped fits inside one Goose show.

My favorite counterexample came from Phish’s five-night run at Madison Square Garden. The first night revealed the rule: each show would rebuild one year of the band’s history, moving from 1992 through 1996. What should the app do with that information before the second night?

For the next four nights, Zabriskie turned the pattern into Setlist Prophecy. The show page read the real rotation from that night’s year, removed anything already played earlier in the run, and surfaced likely songs, era-specific opener choices, long-gap candidates, and the gags that belonged to that version of Phish. It even made a call on Trey’s shirt.

Night two: the 1993 board, with likely songs and era gags still on the table.

Night three: the 1994 board after we added the shirt call and signals from the archive's setlist notes.

After the encore, the prophecy graded itself. The wardrobe call landed four nights out of five, ending with Trey’s gold velour Clifford Ball shirt on the 1996 finale. That night also delivered two of the four stranger calls: Harpua closed the first set, and Crosseyed and Painless supplied the promised wink toward the Remain in Light Halloween show later in 1996.

The fifth night of the time machine: 1996, back in the completed-show archive.

People played along. Twelve people made 39 RSVPs across the five shows. Seven made 22 Call the Opener guesses, while the live rooms produced 221 messages from six people and 614 reactions from three. None of the opener guesses landed.

Setlist Prophecy was not a machine-learning oracle. It was an editorial game built from the archive, and its misses were part of the result. It gave people something specific to argue about before doors and something funny to score after the encore without asking them to populate a separate, empty room.

The same archive problem appeared in the rest of the app. The Composer became one screen for sharing an album, film, book, or show. Search begins while someone types; recent listening and attendance supply suggestions; choosing an item opens the form in place. If I have written about the same album before, the old post can return as context for the new one instead of vanishing down the Flow.

The Flow in the same dark-mode iOS build.

Albums now open into an Album Hub: the record is divided by side, tracks connect to their live histories, and the page carries its discussion and scheduled listening parties. The Library became one search across recordings, albums, films, and books, with Relisten audio playing inside the app. We also opened a Local Scene pilot around Pittsburgh. A show should not need decades of setlist data before it is allowed to exist in Zabriskie.

The catalog grew from 102 band records and 38,894 shows in the June post to 340 band records and 47,669 shows. Those numbers require an immediate qualification: this is not 238 new bands with equally complete histories. The increase includes Local Scene and broader roster imports. The deeper additions include the Slip, Dave Matthews Band, Dave & Tim, Medeski Martin & Wood, and more of the Kimock archive.

The mobile shell also started behaving like one. Browser confirmation boxes became house sheets and toasts. Sheets drag and spring; buttons have haptics; swipe-back works from the edge. Photos can be pinched, panned, and dismissed. A universal link can survive login and return to the page that opened it. The app can distinguish being offline from merely waiting on a request. No single item belongs in a headline. Together, they remove the recurring feeling that a website has been placed inside an app-shaped frame.

The Receipt

From July 1 through August 17, 1,025 pull requests merged into Zabriskie: 717 from me and 308 from Patrick. The iOS build number moved from 53 to 112. Android moved from 51 to 157, with Wear at 158.

That is an absurd amount of software for less than seven weeks. It is also an easy number to misread. Pull-request throughput does not prove that the product improved. It does explain how a regular show night came to touch so many surfaces that either did not exist, or barely existed, when I wrote the June post.

The community grew much more slowly than the software. Registered users went from 426 to 508, an increase of 82 people. There were 50 weekly active users and 98 monthly active users when I pulled the production numbers for this post. The attendance ledger grew from 4,598 to 6,020 records. Since July 1, 24 people have written 84 album, book, film, photo, listening, show-review, or video posts, alongside the 1,303 live messages and 4,962 live reactions.

It is a real room, and it is still a small one. Most weeks after the first half of July brought single-digit signups. Live activity remains concentrated in a core group. I like the direction of those numbers. None of them means that we have solved growth or broad participation.

Some of the new ideas did almost nothing. During its full fourteen-day test, a pre-show conversation box drew eight comments from two people across seven shows, with no replies. The new show-photo wall received zero photos. Venue Intel received zero tips. Watch-party membership remained at zero.

It would be easy to blame the size of the community, except 24 people made 84 regular content posts during the same stretch. The narrower explanation is more useful: adding an empty box to a show page does not give somebody a reason to be the first person inside it.

That is the useful correction underneath all of this work. Features that fit a motion people were already making—checking who is going, glancing at the current song, reacting in a live room, rating the night the next morning—have a chance to become part of the show. Features that ask somebody to populate a new room alone have a much harder job.

Seven weeks ago I wrote that the gap between shows was the hard half of the product. I still believe that. What changed is that I no longer see the show as a two-and-a-half-hour block with empty time on either side. It has a long edge: the plan before doors, the split between the rail and the couch, the ride home, the next morning, and the date years later when the whole thing returns.

On August 13, that edge ran from 33 people making a plan, through “Animal,” 160 messages, twelve songs, and one final “726.” The app stayed with the night after the music stopped.

We can carry the whole night now. I do not yet know how often people will want us to, whether enough of them will, or whether the pieces that worked during a Goose or Phish run will become habits between them. That is the next problem, and it is much harder to solve than merging another thousand pull requests.

Science at LLM Speed · Research-like output is cheap. A valid claim still needs evidence that can show it is wrong.

Sat, 15 Aug 2026 18:00:00 +0000

This is Part 1 of The Machine in the Lab, a seven-part series about building SetScope, a live Goose song guesser, with an autonomous research program built from large language models (LLMs).

Earlier this year, the leaders of one of the largest machine-learning conferences found references in submitted papers to publications that did not exist.

This was not a thought experiment about what a language model might do. It was not a benchmark, a standardized test, in which researchers asked a model to write a literature review and then counted the invented citations. These were papers submitted to the 2026 International Conference on Learning Representations (ICLR), accompanied by bibliographies that were supposed to describe prior work. The conference built a screening system, sent the flagged references to senior reviewers, and then had the conference leaders check them again. Every paper with a confirmed hallucinated reference was desk rejected, meaning it was rejected before normal peer review.

The most interesting part of the conference leaders’ account is how much work it took to establish that a paper in a bibliography was not a paper. The automated system incorrectly flagged some real references. Translated titles looked suspicious. At least three people reviewed every confirmed case. The conference did not trust one detector to decide whether a citation existed.

Whatever process produced those bibliographies did not catch the error.

This is a strange failure because background research is one of the activities these systems appear to make almost effortless. Ask for the important papers in an area and a model will produce authors, titles, dates, summaries, and a tidy account of how the work fits together. The answer has the shape of a literature review before anyone has done the work of reviewing the literature.

Sometimes the papers are real. Sometimes a real paper is attached to a claim it does not support. Sometimes the title, authors, or venue are slightly wrong. And sometimes the entire reference was generated because a plausible citation completed the paragraph.

All four cases look approximately the same on the page.

A plausible bibliography can describe papers that never existed

We knew this behavior existed before it began appearing in conference submissions. In 2023, William H. Walters and Esther Isabelle Wilder asked OpenAI’s GPT-3.5 and GPT-4 language models to produce short literature reviews across 42 topics. GPT stands for Generative Pre-trained Transformer. They checked 636 generated citations. In their controlled study, 55 percent of the GPT-3.5 citations and 18 percent of the GPT-4 citations referred to works the researchers could not verify as existing. Many citations to real work also contained substantial bibliographic errors.

Those numbers do not tell us what fraction of references in current scholarship were written by a model. They describe particular models, prompts, and topics in a controlled experiment. A broken citation in a paper does not contain a record of the tool that produced it. People invented, mangled, and copied references long before ChatGPT.

But the ICLR cases establish the part that matters here: nonexistent references made it through whatever process their authors used to produce a submission for a major research conference. The scholarly form was present. The scholarship that the form purported to reference was not.

The obvious response is to connect the model to a search system. OpenScholar, for example, searches a collection of 45 million open-access papers, retrieves relevant passages, generates an answer with citations, and then checks the answer against those sources. In the authors’ evaluation, the system substantially improved the citation accuracy and correctness of a general-purpose model on scientific synthesis tasks.

This is better, but the reason it is better matters. The model did not become more trustworthy by writing a more confident explanation of its sources. The system changed the task. Candidate claims had to be attached to retrieved passages. Citations could be checked against documents that existed outside the generated answer. Expert readers could inspect whether the source supported the sentence.

Fetching the sources does not make a literature review true. A real paper can still be misunderstood, cited out of context, or asked to carry a larger claim than its experiment supports. What the sources supply is the possibility of contradiction. There is now something outside the paragraph that can show the paragraph is wrong.

That difference between fluent support and evidence capable of contradiction is the problem I kept encountering in my own work.

The research loop gets faster

The citation problem is unusually easy to see because existence is a fairly crisp property. Either the referenced document can be found or it cannot. Most research decisions are not like that.

A model can now help formulate a question, search for related work, write experiment code, select a statistical test, produce a figure, explain the figure, draft the limitations, and review the finished manuscript. Each output can become the input to the next step. A single person can move through activities that once required more time, more specialized assistance, or both.

I wanted to use that capability to build an automatic live song guesser. I run Zabriskie, a community for live-music fans. During a livestream by the band Goose, viewers who want a running setlist have to wait for a person or an external service to recognize each song and enter it. I wanted Zabriskie to listen to the stream, identify the current song, and post the guess while the show was still happening. That product became SetScope.

This is a different problem from Shazam. Shazam fingerprints a short excerpt and looks for the same recording in its database. SetScope hears a performance that has never existed before. Goose might play the song faster or slower, change the key or arrangement, stretch the opening, or move from the written song into a long improvisation. The system cannot search for an identical recording. It has to recognize the musical identity that survives those changes.

I also wanted artificial intelligence (AI) to do the research needed to build it. I would give the system recordings paired with their correct song titles and tell it what SetScope needed to do. It would come up with ideas for recognizing songs, write the code, use those examples to train models, run tests, examine the mistakes, and decide what to try next. I would not approve every step. That was the human-out-of-the-loop part of the project.

The implementation speed is what made that plan plausible for one person. During my doctoral degree (PhD), I learned how much of a systems experiment happens before the experiment. Filibuster, the testing framework I was building for applications spread across multiple services and computers, could not evaluate an existing application until that application had been adapted to run through software hooks that let Filibuster observe and alter its behavior. Building the prototype took me three months of full-time engineering. It used OpenTelemetry, a standard for collecting information about what happens as a request moves through an application.

That only got the application to the starting line. Filibuster works by deliberately introducing failures while an application’s existing tests run. If the target application did not already have useful tests, I had to write them before the framework had anything meaningful to explore. For one production application used in an evaluation paper, that took another six months.

With the agents I use now, I believe I could produce a first implementation of many of those software hooks and supporting tests in days, perhaps a single day. During the SetScope work, agents built audio scanners, catalogs of recordings and their source history, programs for measuring audio and comparing song recognizers, and detailed logging inside the live browser on that timescale.

None of those components is a controlled equivalent of the OpenTelemetry work, and I cannot rerun my PhD for comparison. The estimate describes what I believe would have happened, not a measured speedup. Code arriving quickly would not establish that the added software hooks preserved the application’s behavior or that the generated tests supported the evaluation. Those questions would still require evidence. But even compressing the first implementation from months into days changes which research projects one person can plausibly attempt.

While I was writing this, Jeff Dean, Sanjay Ghemawat, Oriol Vinyals, and Quoc Le left Google to form a company called Discovery Loop. Public descriptions say its goal is to automate experimental loops in science and engineering: propose an experiment, run it, evaluate what happened, and decide what to try next.

That is the same kind of autonomous researcher I was trying to use to build SetScope. Their target is science and engineering broadly. Mine was Goose song identification.

When a system can choose and run the next experiment by itself, a bad result can do more than produce one wrong answer. It can change what the system tries next. Examples meant to remain unseen can enter development and alter the next hypothesis. A convenient stand-in for the real question can quietly replace the question. A test of one part of a product can become a claim about a product that never ran.

A polished analysis is not necessarily a scientific result

Autonomous research is the most ambitious version of the problem, but a smaller version is already common. LLMs have made it cheaper to produce something that looks like the output of a research process: a paper, methodology page, interactive analysis, benchmark, data product, or long post with equations and charts.

This can be genuinely useful. In a participatory study of 15 people performing generative-AI-assisted data analysis, Drosos and colleagues observed participants using a model for information gathering, hypothesis generation, and analysis strategy. The same participants described verification as effortful and time-consuming.

Useful analysis does not have to occur inside a university or become a paper. A fan project can rank performances, organize an archive, publish its formulas, and offer an excellent discovery tool. But if it also claims that the score reveals a property of improvisation, then the human judgments used as correct answers, the unit being measured, and their relationship to that musical property become part of the claim. Equations, transparent code, and polished charts do not settle whether the evidence is adequate.

This problem existed before LLMs. What changes is the cost of producing the complete package. Code, prose, caveats, visualizations, and a memorable result can now arrive together, quickly enough that their coherence feels like evidence that the empirical work occurred.

My own project produced exactly that kind of convincing package, more than once. The form did not create the errors. It made them harder to notice.

The reviewer might be a model too

One possible answer is review. Authors produce work quickly; reviewers slow it down, inspect the assumptions, and require the claims to survive contact with another person. Except the same systems have entered that loop.

A 2024 study published at the International Conference on Machine Learning examined reviews from four major machine-learning conferences, including ICLR. Its estimate across the full collection was that 6.5 to 16.9 percent of review text had been substantially modified or produced by language models, beyond minor writing assistance. That does not identify any particular review as machine-written. It does show that the tool was helping reviewers respond to claims as well as helping authors present them.

There are constructive versions. In a randomized study at ICLR 2025, some reviewers received model-generated suggestions about vague language, possible misunderstandings, and unprofessional comments. Some revised their reports, and evaluators who did not know which reviews received assistance rated the revisions as more informative.

That is evidence that a model can improve one part of a human review process, not that it can replace peer review. A second model does not become independent review merely by being a second model. Two systems can share training data, conventions, blind spots, and a preference for the same fluent explanation. Adding agents changes the number of outputs. It does not necessarily change the source of judgment.

Human review is not a magical external check either. Reviewers miss errors, disagree, rush, and reward familiar methods. The question is what new information and incentives each check adds. If every stage evaluates the same generated output in roughly the same way, the process can become impressively self-consistent without becoming more correct.

The machine can find something real

There is an easy version of this essay in which every section supplies another example of AI making science worse. It would also be wrong.

FunSearch used a language model to generate candidate programs for mathematical problems. The system executed them, scored them with a test supplied by the researchers, retained the strongest programs, and used those programs to guide further search. The resulting Nature paper reported new solutions to a problem about arranging finite sets and useful strategies for packing items into a limited number of containers.

The model was valuable because it could search a space of programs productively. It was not asked to decide, in prose, that its own program was interesting. The programs ran, the evaluator scored them, and other people could inspect the result.

AI Scientist-v2 operated under a weaker judge. It generated machine-learning manuscripts end to end, with humans choosing initial ideas and selecting the best completed run. One manuscript scored above the acceptance threshold at an ICLR workshop. The authors’ inspection also found missing citations, examples that may have influenced both development and evaluation, incorrect figure interpretations, and unused code for a statistical technique intended to make confidence scores match observed probabilities.

These checks do not provide the same evidence. A program evaluator can reject a candidate against a specified property. An experiment can contradict a prediction. A source can fail to support a sentence. Human review adds judgment, but a reviewer can still be persuaded by the same polished explanation as everyone else.

The system becomes more useful when generated candidates encounter information that was not produced by the same act of generation and can return an unwelcome answer.

Before treating a check as independent, I now ask three questions: What information does it add that was unavailable to the process that generated the result? What unfavorable answer can it return? Which precise claim would that answer reject?

Then my own research loop failed twice

I then changed the task. Instead of improving SetScope’s song guesses, I asked the system to use the recordings to study improvisation. I had years of recordings paired with song titles and other annotations, ways to measure rhythm, harmony, and texture, models that turn audio into numbers that can be compared, and an LLM agent that could scan recordings, organize their source history, calculate those measurements, train programs to classify them, run evaluations, analyze failures, and modify the live application. What had been an implausibly large solo project looked tractable.

The recordings seemed to offer a way to investigate harder questions about improvisation: whether measurements of the music could identify when a performance left its composed structure, whether different forms of jamming produced distinct patterns in those measurements, and whether the same patterns appeared across performances. Those questions were more interesting than simple song identification. They were also much harder to define.

I am a PhD-trained systems researcher. I know what a holdout is: a final test set kept unseen while a system is developed. I also know why an experimental method has to match its implementation. That was not enough. The first research notebook, a running workspace of code, data, experiments, and reports, expanded from song recognition into several improvisation questions without preserving one final test capable of evaluating the path the system chose.

The audit found several problems, not one neat leak. Most decisively, a detector for Type II improvisation, where the band leaves a song’s composed structure for open-ended playing, failed on 44 recordings that had not been used to develop it. I had described the detector on a research webpage and shown it to two friends. It mistook changes inside composed songs for improvisation. I took the webpage down and started again.

For the restart, I permanently assigned many of the same recordings to three jobs: examples the models could learn from, examples used to choose among competing methods, and a final test that neither process was supposed to see. I also wrote a new method that explicitly warned against fixed ninety-second jam boundaries. The first implementation reused them anyway. The research loop never stopped to report that the code contradicted the declared method. Working from those reports, I put two draft research posts online at their direct web addresses and sent friends listening assignments built from the analysis.

The consequences were larger than the lines of code that caused them. I was responsible for putting the work online and involving other people’s time. But the lesson could not be that I should manually reconstruct every assignment of recordings and trace every fixed numerical value before accepting any result. A system that requires that level of supervision is not running the research process autonomously. It is generating work for a human auditor. The system needed to prevent violations of its rules or report them when they occurred. It had done neither. I deleted the second notebook, took the drafts down, and abandoned the listening study.

We eventually gave the system its original job again: improve the live song guesser. On August 13, the running SetScope product proposed the correct identity at least once for 10 of 12 recorded song performances while the band was playing. It also missed songs, switched guesses at the wrong time, and exposed failures in the live audio path.

This was a product field test, not a formal whole-show accuracy estimate, and the complete operational record belongs later in this series. Neither audio from that performance nor its completed setlist existed during development. The surviving record shows what reached SetScope’s decision logic, not confirmed viewer-visible delivery.

The project would not exist at its current scale without LLMs. The failures taught me that speed changes the location of the work. Producing the next output becomes cheap. Establishing what the output means, what information entered it, and what could show it is wrong does not.

The machine can participate in research. The harder problem is deciding what it may do without asking, what evidence must survive each iteration, and what can stop a bad result before it becomes the premise of the next experiment. This series is about learning to build those constraints after discovering, repeatedly, that a persuasive report was not one of them.

What Comes Next

Part 2 reconstructs the two notebooks I deleted. Part 3 follows the holdout boundary we repeatedly crossed and the record needed to know which evidence each later version had already used.

Part 4 shows how a reviewed experiment can pass every check and still misinterpret the music. Part 5 moves the test into the browser, where capture, controller behavior, and the rendered result become part of the experiment.

Part 6 describes the constraints we built around the research loop. Part 7 returns after Goose’s August run with the versions that actually ran, their guesses, their failures, and the repairs made between shows.

Next: two notebooks that contained weeks of analysis and almost no result I could still defend.

One Writer · Our tools assume one writer, and assume that writer is a human. Nothing computes what a change reads and writes at runtime, so the only known fix is brute force priced for organizations.

Mon, 27 Jul 2026 02:00:00 +0000

In this blog post, I discuss three days in July 2026 when a single agent session ran away from me, and what those three days revealed about the concurrency assumptions buried in our development tooling.

Six weeks ago I wrote The Test Suite Was the Incident: my test suite had grown a pile of shared data nobody owned, every pull request paid to rebuild it, and the resulting failures had nothing to do with the changes under review. That cost me about $180 in one night.

I got a worse one. It lasted three days, and in one twenty-four-hour stretch of it I burned through an entire Codex 20x max plan.

This post is not really about that, though. It is about a property of our tooling that the three days made impossible to ignore.

Nearly every layer of this assumes one writer, and assumes that writer is a human. Git hands you a conflict and waits. Code review assumes somebody reads. A migration sequence assumes somebody is assigning the order.

Each of those protocols terminates in a person. That is fine while there is exactly one, and while they are, in fact, a person.

The agent runtime turns out to be on that list too, which I did not expect. It spawned eighty-four workers into a single checkout without being able to say what any one of them would read or write, and that is the same question git can’t answer about a diff.

That assumption was invisible for forty years because nothing ever bound it at my scale. Agents break both halves at once: there are many of them, and not one of them is the person the protocol was waiting for. Git is the partial exception, and I will come to why the exception does not help.

Nothing in the stack detects the violation when it happens. It gets caught later, somewhere else, attributed to the wrong change, and paid for at full price.

Some context for readers arriving fresh. Zabriskie is a social app for live-music fans, and it’s also a deliberate experiment: I’m building a real, deployed, actually-used application almost entirely with AI agents (agents that wrote the features, agents that wrote the tests guarding those features, and agents that now open most of the pull requests), in order to find out what that’s like and, more usefully, where it breaks. I’ve written almost none of the code.

That framing matters, because several things below look like obvious mistakes and are. I let agents design a migration scheme with only another agent reviewing it, I stopped reading most diffs, and I let sixty-four pull requests go up in a single day, all of which a careful engineer would tell you not to do and would be right about. But the point of running an experiment at the extreme is to find the walls. That week I found several at once.

A migration, throughout, is a versioned SQL file that changes the database schema. CI is the automated checking that runs on every proposed change: build the app, spin up a fresh database, run the tests.

Here is the shape of the three days. Treat these numbers as texture, not as evidence, for a reason I will get to.

	24 Jul	25 Jul	26 Jul
pull requests opened	19	64	28
pull requests merged	20	50	30
incidents logged	0	6	301

Days in that table are UTC; clock times in the narrative below are Eastern (where I was). The burst ran past midnight UTC: 301 incidents on the 26th plus 60 more before 1 AM on the 27th, so 361 for the burst. Every incident count below is scoped to that window, and 253 rows is the total before 24 July.

I offer that comparison as a sense of scale and not as a baseline, for the following reason. Those incidents exist because a standing instruction tells agents to log their own mistakes, and ten minutes into the worst night I tightened that instruction. The log therefore measures reported failures.

Look at the daily series and it gets worse: there are days that week with ten and twenty merged pull requests and zero logged incidents, which at any real failure rate means nobody was logging rather than nothing broke. In short, I can’t give you a trustworthy baseline. What follows rests on mechanism and on a few dated, checkable events, not on 361.

The Session

Late Saturday night the queue jammed. Sixty-four pull requests had gone up that day, main was red, and nineteen open pull requests were stuck behind a suite that could not tell me which of them was broken. At 12:31 AM I opened a session with Codex and complained that CI was wasting too much money.

Codex read that and hired a workforce.

Over seventeen hours that session made 81 spawn calls, producing 74 direct children; those children spawned 10 more, for 84 threads. Then 358 calls waiting on them, 96 listing them, 85 sending follow-up work, and 13 interrupting them, which comes to six hundred and thirty-three tool calls of pure management overhead (spawn, wait, list, follow up, interrupt) against 54 messages from me over the same seventeen hours. All 84 ran in the same checked-out copy of the repository.

Trivial parallelization, at scale, with a coordinator that only managed agents, on a toolchain built for one writer. The rest of this post is about why that combination is so much worse than it sounds.

I asked afterward why it had spawned anything, since I never requested it. A language model asked why it did something produces plausible narrative, not introspection, and I apply that same skepticism to the self-reported incidents above.

So the quotes below are not evidence, and I’m not offering them as any. I had already reached the same conclusion from the tool-call counts before I asked. I print them because they state that conclusion more plainly than I did, and because there is something worth looking at in a system that can describe the failure this precisely and could not avoid it.

I chose to spawn multiple agents even though you did not ask me to. The environment permitted proactive delegation, but that was permission, not a requirement. I treated the availability of agents as a reason to use them instead of first asking whether they would reduce time, cost, or risk.

It then diagnosed why the task was a bad candidate:

A PR queue is largely serial because every merge changes main, which changes the integration state of every remaining PR. The work needed one authoritative coordinator moving through the queue in order. Parallel agents could not independently merge overlapping PRs without continuously invalidating one another’s assumptions.

On what those 358 waits were doing:

The agents’ outputs created additional coordination work for me. I had to read their findings, compare conflicting recommendations, inspect supporting evidence, and decide how to combine them. The agents therefore generated work for the coordinator instead of reliably removing work from the critical path.

A system with no notion of its own concurrency was handed a concurrency primitive and used it, reasonably, on a task the substrate could not support. What that substrate is, and why nothing in it objected, is the rest of this post.

One number deserves care. I do not know how many of those 84 threads wrote to the checkout instead of reading it; from Codex’s own account many were investigating. A reader is harmless and a writer is not, and nothing in the toolchain drew the distinction or could. That’s the same missing primitive one level up: not even the agent runtime knew which of its children were writers.

The Substrate

Start with the layer that did anticipate this, because it is the one people reach for. Branches are optimistic concurrency control; worktrees go further, giving each writer a physically separate checkout so that two agents can hold two versions of the tree at once. That works.

However, git’s isolation stops at the edge of the source tree. A worktree gives an agent its own files. It doesn’t give it its own database, port range, mock server, or position in the migration sequence. Everything below the filesystem is shared and singular.

That assumption was invisible to me because it never bound me. It has bound large organizations for a very long time, however, and over the last two decades they built the response to it: merge queues, hermetic builds, database-per-test, trunk-based development, automated culprit-finding, and whole infrastructure teams whose only job is to keep the thing moving.

So what is new here isn’t the concurrency. It’s that a solo developer now operates in the regime that used to require an infrastructure organization, with none of the infrastructure and no headcount to build it. Agents removed the cap on my arrival rate. They didn’t hand me Google’s build system.

Partial isolation is then its own trap. A worktree gives you the feeling of a private workspace (clean tree, own branch, no file collisions), and it reads as properly parallel right up until two private workspaces write the same database row. Nothing errors. Nothing warns. The contention surfaces later, somewhere else, as a red X on an unrelated pull request.

Worse, the isolation below git is advisory, and agents have to choose it. Mine routinely don’t. One incident reads “PR 1868 isolated E2E attempt fell back to shared ports”: the agent tried to isolate its test environment, isolation failed, and nothing stopped the run.

Worse still, a worktree is cut from a commit and stays there, so it’s isolated in the past and decays with every merge. When I changed one of the agent gates in July (they’re tracked scripts in the repository, not .git/hooks, so every worktree carries its own copy pinned to the commit it was cut from) the fix merged at 1:48 AM, and four hours later 73 of 74 worktrees were still running the old one. A stale worktree can’t detect its own staleness, so it goes green about a world that no longer exists, and the error is deferred to the only actor holding current main, which is CI.

Convergence and Invariant Preservation

One framing before the specifics, because it unifies them. Every merge mechanism here is built to terminate, and none of them is built to preserve invariants.

Git is honest about the cases it cannot decide: on a textual conflict it halts and asks you. The trouble is the far more common case, where it doesn’t halt, produces a merge confidently, and the invariant it never knew about is now false.

That distinction is the oldest lesson in the replicated-data literature, and it has a canonical counterexample. Take a replicated map where each field merges independently under its own perfectly reasonable rule. One field holds a person’s name; another holds the length of that name. Two replicas concurrently write different names.

Each field converges exactly as specified, and the result is a record whose name came from one replica and whose length came from the other, with the invariant tying them now false. Nothing merged incorrectly. The composition of correct local merges is simply not a correct global merge.

Closing that gap is the point of work like Balegas and colleagues’ Indigo, which enforces application invariants over eventually consistent stores, and which I have written about before.

Some of what follows has that shape. The first case does not, and I will not dress it up.

The Sealed Prefix

This project has 1,388 migrations, and rebuilding a test database from all of them on each of eight parallel test machines is as slow as it sounds. So, in response to my complaining for weeks that CI was too expensive, an agent froze a database snapshot into the repository for CI to restore, applying only what came after.

It’s a good optimization: it saves 13 to 18 minutes of machine time per run. It merged at 5:01 AM Eastern on the third day.

Fifty-four minutes later the first pull request failed, because its migration no longer sorted after the newly frozen prefix. Then another. By late morning a single incident covers four at once: “Four queued PRs carried migrations older than the sealed CI baseline suffix.”

The diagnosis here is ordinary, and a critic will say so. Other agents had already opened pull requests that appended migrations; one agent then sealed a new prefix underneath them; those two streams of work conflicted; and nothing anywhere in the system noticed until CI rejected the queued pull requests several hours later. That’s parallel work colliding and finding out late, not a subtle invariant bug.

Caching a prefix of an ordered log made the collision loud and retroactive, but it did not invent the collision. So the loop closes: I complained CI was expensive, an agent made CI cheaper, and the mechanism became a new source of CI failures against the whole queue.

Merge Functions and Runtime State

Git detects conflicts over lines of text; my conflicts live in shared runtime state. From that week: “Song-call E2E suites deleted each other’s shared pending call.”

Two test files, no overlapping lines, both green alone, and the row one of them depends on is the row the other deletes. The collision is in the database at runtime, not in the diff.

Those two files are the name and the length. They share no lines, each merges cleanly, each is individually green, and the invariant binding them is false the moment they land together. No improvement to git’s merge algorithm catches that, because the property is not a property of any file: it is a property of the composition, and git has no representation of the composition to check.

Verification Under Composition

Which gives the sharpest version of the problem: green(A on base) and green(B on base) does not imply green(merge of A and B).

The test suite is the only thing in my pipeline that checks the invariant at all, since git checks text and timestamps check ordering. So the tests are my invariant checker.

I do run them on the composition. Pull requests rebase onto current main and merge in sequence, each one retested before it lands, and main itself is checked after land. That catches the break. It just catches it late, and only by paying the serial tax: rebase, retest, merge, next. That tax is exactly the queue-wide cost the sealed prefix made visible on the morning when one seal invalidated everything queued behind it.

Serialization is the honest answer, and I already pay it.

Speculative merge queues try to buy the same guarantee back with parallelism: build the candidate futures, main+A, main+A+B, main+A+B+C, test them concurrently, and discard a failure from the middle. OpenStack’s Zuul has gated that way since 2012; GitHub’s merge queue ships a version; and bors, which lands Rust, does the cheaper batch-and-bisect variant. It works, and it works by brute force, which I will come back to.

Shared Environments

One port range, one development database, one mock server.

I do have a script assigning each worktree its own ports by hashing the directory name (the right idea), and it hashes into 99 slots. With dozens of worktrees on disk the birthday math makes a collision effectively certain, and two agents get handed the same port whenever the colliding pair happens to be running at once.

Undivided Work

Trivial parallelism doesn’t work on a problem that was never split into non-conflicting units. The agents were pointed at one jammed queue and found their own boundaries, which were mostly the same boundaries.

What looked like a coordinator was a process manager: spawn, wait, list, follow up, six hundred and thirty-three tool calls of overhead, all of it without any deep context about which pieces of work actually interacted. A coordinator that only manages agents can’t divide work it does not understand, so the children rediscover the same failures, edit the same new scripts, and invalidate one another’s assumptions. Codex said as much afterward: the agents generated work for the coordinator instead of removing work from the critical path.

Classified by root cause, 245 of the 361 entries collapse into ten systemic problems, 73 are one-off product bugs, and 43 I could not classify. Here are the ten, because a taxonomy whose majority is invisible isn’t a taxonomy:

systemic root cause	incidents
tooling the session was writing that same day	87
no per-agent workspace isolation	33
errors skipped, swallowed, or reported as success	32
affected-spec selection far too broad	20
migration order versus the sealed baseline	16
tests leaning on shared fixture state	16
recovery work that would not converge	12
CI cost structure	12
evidence produced against a base that moved	10
verification run without its dependencies	7

I would defend the shape of that table rather than the ratio: workers under a shallow coordinator rediscover the same things, often enough to dominate a log.

Look down the column and the majority is exactly that failure. The top row alone (tooling the session was writing that same day, 87 incidents) is agents redoing and colliding on the same work, because nothing holding the session had enough context to keep them off each other’s toes. Add isolation failures and swallowed errors and the top three account for 152.

Those are not ordinary defects that concurrency merely made frequent. They are the artifact of coordination without context: a parent counting agents instead of understanding the work, so that the children rediscover and rewrite the same things, file the same findings, and hand the parent the job of reconciling them. The seal and the port hash still have boring mechanical fixes, and recovery-that-would-not-converge and CI cost structure are process problems. The rediscovery is neither.

The one that is irreducible in a different sense is the sixth row: tests leaning on shared fixture state, or green(A) and green(B) failing to imply green(A+B) in ordinary clothes.

It accounts for sixteen of the 361, tied with the seal for the smallest systemic bucket, and I want to put that number in front of you instead of letting you find it by subtraction. Sixteen out of 361 is four percent.

So that substrate problem is the rarest thing that happened that week. It is also the only one where catching it means building the composition and running the entire suite against it, which I do, serially, late, and which is precisely the expensive thing. It is therefore the only one on the list where I can’t tell you what an affordable fix would even look like. The common problem looks much simpler: parallelism without a division of the work, supervised by something that was counting agents instead of understanding them.

Except that the two are the same problem, and it took me most of a week to see it. What a coordinator needs in order to divide work is exactly a read/write set per unit, which is the primitive this whole post is about, one layer up. Mine had no way to compute one, so it couldn’t divide anything, so eighty-four agents went out and found the same boundaries and did the same work on top of each other. That’s the top two rows of the table, a hundred and twenty entries, a third of everything I logged. I’m counting two rows and not the three above, because swallowed errors are ordinary bad error handling and belong to a different complaint.

So the four percent isn’t a rare problem I’ve chosen to write about instead of the common one. It’s the identical question, “what does this thing read, and what does it write,” asked about a code change instead of about an agent. At the code layer the question is open, which is the research problem. At the agent layer it’s answerable in principle, and my runtime simply didn’t ask, which is the more embarrassing half and the one my week actually went to.

The Missing Primitive

So what would it take for some layer to detect any of this?

Start with why detection is hard. One pull request writes a row to show_attendance; another reads a count over it. Neither diff says so. The write goes through an HTTP handler, an ORM, and some SQL; the read goes through the app. The conflict is a property of what the code does when it runs, and it’s not present in the text.

That leaves two ways to find it:

Execute the composition and observe what breaks.
Compute the effects of each change and compare them without running anything.

Nobody can do the second precisely enough at the granularity where my conflicts live, so everybody does the first. If you can’t compute effects, you can’t tell which pairs need checking, and so you pay for a full verification of every candidate future you are willing to buy.

Uber’s SubmitQueue needed a probabilistic model of which changes would land in order to prune the speculation tree, and that is less an optimization than a confession: you only guess your way through the tree when nothing can tell you which branches matter.

So the state of the art converts an unsolved program-analysis problem into a compute purchase. That is rational at Google’s scale and precisely the wrong trade at mine, because speculation costs queue depth times full suite times parallel environments, and agents raise queue depth. The sanctioned fix for the correctness problem is more CI spend, scaling with the exact variable agents just multiplied.

My two complaints turn out to be one complaint. “CI is too expensive” and “composition failures show up only when the serial queue retests them” are the same missing primitive seen from two sides.

Here is the size of it in my own logs:

Over three days, 510 CI runs were spent against 111 opened and 100 merged pull requests: roughly five full verifications per change that landed.
The runs attach to opened pull requests rather than merged ones, so read that as an order of magnitude and not a rate.
Of the 510, 112 failed or were cancelled outright, and cancellations are usually a new push superseding an old one rather than an escape.
Deleting every one of those 112 takes the multiplier from 5.1 to 4.0, not to 1.
The rest is ordinary iteration, some fraction of it rebasing onto a main that moved underneath, which is the same problem in ordinary clothes.
The primitive is therefore worth about a quarter of the excess, and only if every one of those failures turns out to be a composition escape, which I cannot show. A serial queue stays slow either way.

Databases avoid all of this for one reason. A transaction declares its read/write set. The system knows what each transaction touched, so it can detect conflicts, order what must be ordered, and let everything else proceed.

That question (which operations may run without coordination) has an exact answer in the literature. Bailis and colleagues named it invariant confluence: a set of operations is I-confluent with respect to an invariant precisely when merging any two invariant-preserving reachable states that share a common ancestor yields another invariant-preserving state.

This is a necessary and sufficient condition for running those operations coordination-free while maintaining the invariant, not a heuristic. If your workload is I-confluent you can run coordination-free and stay correct; if it is not, coordination is a requirement, and no merge function will save you.

The result does not transfer to code cleanly, and it’s worth saying where it breaks. I-confluence is defined over a merge operator on states, with reachability and a fixed invariant, whereas for code changes there is no defined merge over program states, and the invariant here is the test suite, which every merge rewrites. What I’m borrowing is the shape of the question, not the theorem.

Source code carries a partial declaration at best. Build systems get closest: Bazel and Nix require every target to declare its inputs and outputs (a real read/write set at file granularity), and that is how affected-target computation works across enormous monorepos.

However, file granularity is the wrong granularity here. My two colliding pull requests touch entirely disjoint files. A build graph would call them independent, and a build graph would be right, and they would still destroy each other, because the thing they share is a row in a database that neither one names.

At the granularity where my conflicts live, nothing declares those effects. So there is no conflict detection there, and without it you can’t tell which regime a pair of changes is in, and if you can’t tell, the only safe policy is to assume the worst and coordinate everything.

What is the runtime read/write set of a code change, and can it be computed precisely enough, and cheaply enough, to decide which changes are safe to integrate concurrently?

I’ve already built a bad version of this without recognizing it. My affected-spec selector maps a change to the tests it could affect, so that CI only runs what is relevant. That’s a read-set estimator, and its failure mode is the research problem in miniature: it traced a one-screen change through a shared UI component, concluded the change could affect anything, and scheduled all 115 test files.

An over-approximation collapsing to “conflicts with everything” is the same degenerate case as a lock covering the whole table, or as serializing every update to a dictionary even when the changes are to disjoint keys. Sound, and useless.

One objection here is that I am demanding a soundness nobody needs. Large organizations run test selection that is deliberately imprecise and absorb the escapes: changes that should have been tested, were not, and break trunk after they land.

It’s tempting to say I can’t absorb escapes because my blast radius is smaller, but that has it backwards: an escape into Google’s trunk blocks far more people than an escape into mine. What they have, and I don’t, is containment. Post-submit continuous builds, automated culprit-finding, and rollback tooling all bound what an escape costs.

I have none of that. What I have instead is a queue of agents that respond to a red trunk by independently rediscovering the failure, each one paying for its own full CI run, none of them asked to, and none of them aware that eleven others are doing exactly the same thing at exactly the same moment. Escapes are cheaper for Google not because fewer people are blocked, since more are, but because they built the machine that catches and bounds the damage. Without that machine, imprecise selection is a cost I can’t absorb.

Prior art exists, and none of it quite lands:

Regression test selection has estimated read sets for years, at file granularity, the granularity that doesn’t help, though dynamic variants get closer than my static one does.
Speculative merge automation (e.g., Zuul, SubmitQueue) tests the composition, which is the expensive thing.
Proactive conflict detection went after semantic conflicts directly more than a decade ago and never became something anybody runs.

Requirements for Coordination-Free Development

Coordination-free multi-agent development, if it’s possible at all, needs solving at two layers, in order. Getting the second right while the first is broken buys nothing.

The environment has to permit it first. Separate databases, working directories, port ranges, and instances of every service, per agent, enforced rather than attempted. Writers in one checkout contend regardless of how elegant your merge semantics are, and I still do not know how many of those 84 were writers.

Large companies solved this a while ago, and the shape of the fix isn’t subtle: make every developer’s environment a faithful copy of the one that builds and ships the product. Either move development onto cloud machines provisioned from the same definition as CI, or reproduce that environment in miniature on the laptop, with a container per service, a database per test, and a toolchain pinned to a lockfile. Somebody owns that, full time. The cost of getting it wrong is paid once, by a platform team, instead of every day by everyone else.

I don’t have a platform team, and neither does anyone else building alone with agents. An independent developer can now reach a working prototype absurdly fast, faster than at any point in my career, and then hit the wall this whole post is describing: one environment, artifacts that assume one writer, and no organization to absorb the difference. Getting off the ground is close to solved. Staying up is not, and I don’t have a good answer for it.

That’s the same argument I made a few months ago about civil engineering, coming at it from the other side. When construction gets cheap, design is what fails, and the design here is the substrate: what runs where, what is isolated from what, and what has to be serialized. An agent isn’t going to do that for you, and it won’t show up in the prototype.

Then the artifacts have to stop requiring a coordinator. A total order needs somebody to assign it, and in the single-writer world the assigner was free: migration numbers went up because one person added one at a time. Put N agents on it and the assigner is gone. You can reintroduce one as a central coordinator handing out positions, but a central coordinator is coordination, and it caps throughput at one.

Two mainstream designs exist and always have: a total order by timestamp, as Rails has done since 2008, and a dependency graph, as Django and Alembic do. Under one writer the choice barely matters. Under many it matters enormously, because a total order needs an assigner and a graph doesn’t. My agent picked the total order.

Nobody ever weighed the two, and that’s the part worth sitting with, because it isn’t really about migrations. I ratified the timestamp scheme myself, back in March, and what I ratified was a fix for filename collisions: that was the problem in front of me, the fix solved it, and so I shipped it and wrote a post about it without once asking what the scheme would do under load.

An agent proposed it, an agent reviewed it, and I signed off on the part I understood. Review is one of those protocols that terminates in a person, and what the person is supposed to supply is an independent prior. Neither reviewer had one: not the agent, which reached for the most common pattern, and not me, because I was evaluating a collision fix and never saw an ordering decision. The better option was documented, shipped in two major frameworks, and one search away.

This is what vibe coding actually costs, and it’s not bad code. The code was ordinary and defensible. What is missing is the one question nobody asked, because asking it requires somebody who has been hurt by the answer before, and I had removed all of those people from the loop on purpose.

Django and Alembic don’t solve it either, and the reason is the granularity problem again. A Django dependency is an edge to another migration: this one runs after 0042_add_venue, which says nothing about what either one reads or writes. Two migrations with no declared edge are unordered, so if one adds a column and the other’s UPDATE reads it, that pair conflicts, no edge exists, and nothing detects it. The DAG removed the global sequence and left the effects undeclared.

And when two branches produce divergent heads, the resolution is a merge revision: a node with two parents that doesn’t verify the branches are compatible. It records a human’s assertion that they are, the same protocol terminating in the same person, one layer down.

So the proposal isn’t “invent a DAG.” The DAG exists. It is to make the edges effect-based instead of identity-based: know that this migration writes show_attendance and reads shows.date, and derive the ordering constraints and the genuine independence from that.

The objection is that this relocates the labor, and “somebody declares it” across 84 agents is not a plan. For general code that lands. For migrations it doesn’t, because the effects are already in the text: SQL names its objects out loud, so parsing ALTER TABLE show_attendance ADD COLUMN or UPDATE shows SET ... WHERE date > ... yields the read set and the write set with no human, no annotation, and no agent remembering. All 1,388 of mine are plain .sql, and the parsing is commodity, since sqlglot does column-level lineage today (which has not stopped dbt, in the very ecosystem where effects are most extractable, from building its graph out of ref() and source() macros written by hand).

Two caveats. Disjointness only rules out conflicts through the objects a migration names, so an invariant spanning two tables can still break when two migrations touch one table each: the name-and-length record, one level down. And my sixteen seal incidents have a two-line fix with no research content in it. The case rests on the class rather than on those sixteen, because any ordering constraint several writers must satisfy independently has this shape.

The hard case is the handler-through-ORM-through-SQL path, where the effects genuinely aren’t in the text. The easy case has been sitting in my repository the whole time, unbuilt.

None of that is the CRDT move. CRDTs design a data type so that concurrent operations commute by construction; this is optimistic concurrency control, which extracts read/write sets and coordinates only the pairs that overlap. Designing for commutativity means defining the data type, and for a software system in general there is nothing to define.

For migrations there is, and it may be the better path. Restrict the schema to additive, idempotent DDL: add columns nullable, never rename in place, never drop anything, and guard every statement so that applying it twice is the same as applying it once. Under that discipline the schema is a grow-only set and every migration is an inflation, so any two of them commute and the order stops mattering. No assigner, no extraction, no DAG. The bill arrives at contraction, when you do finally want the column gone, and that step commutes with nothing, so it gets batched and performed deliberately by somebody who is paying attention. Which means the two proposals aren’t really competing: commutativity by construction covers the additive majority and needs no extraction at all, and extracting read/write sets earns its keep on the contraction step and on everything that isn’t a migration.

The discipline isn’t exotic. It’s what microservices already force on you, because there you cannot control rollout order either: two versions of a service run at the same time, so every change has to be tolerable to the version that hasn’t shipped yet and to the one already deployed. Expand, migrate, contract. It is well understood, it is genuinely expensive, and it is one of the things practitioners are worst at, because it requires holding in your head a version of the system you are not looking at.

Agents are worse at it than we are. An agent told to rename a column renames the column, because that is the direct expression of the intent and it is correct in every example it has ever seen, all of which were written in a world with one writer and one deploy. The three-step version is more code, it looks redundant at the moment it’s written, and its entire justification is a concurrency property that appears nowhere in the diff. Which is this post again: the discipline exists, it works, and nothing left in the loop has the prior that would make anybody reach for it.

The same argument applies to source code and lands somewhere less comfortable. Git’s answer to two agents editing one file is to linearize, rebase the second onto the first, and resolve the conflicts: a coordination protocol with a human standing at the conflict point. It works because textual conflicts are rare enough and humans read them well. Both halves are under pressure now: agents collide constantly, and the human who was supposed to adjudicate is increasingly not reading the diff.

Which raises the question I keep arriving at and don’t love. Text files, line-based diffs, and sequential human review are a human interface. A structural representation, where merge is defined over the program’s structure instead of over lines, where a change declares what it reads and writes, and where invariants are attached to the artifact and checked mechanically, is a far better substrate for many concurrent writers.

People have proposed exactly that for decades and it mostly didn’t take, not merely because developers like text, though they do, but because text is the interchange format that every other tool in the ecosystem already agrees on (diff, blame, bisect, grep, review, CI logs, and every static analyzer ever written), so that replacing it means replacing all of them, per language, before you get any benefit at all.

The exceptions are the argument rather than an exception to it. Darcs and then Pijul built patch algebras where independent changes provably commute (the CRDT move applied directly to source control), and Unison stores a content-addressed syntax tree with no text merge at all. They exist, they work, and almost nobody uses them.

The tempting move is to say that agents change that calculus, because the reader has left the room and the ergonomics objection leaves with them. I do not think that is safe yet.

Language models are the most text-dependent writers we have ever built: trained on text diffs and line-oriented source, and currently worse at manipulating syntax trees than at manipulating the text I would be proposing to abandon. The party being served may not be human, but it learned to code by reading us. Whether that is a durable property of these systems or an artifact of what we happened to train them on is the load-bearing question, and it’s not one I can answer from a migration log.

Takeaways

So here is what I don’t know, gathered in one place rather than dropped along the way.

Whether a runtime read/write set can be computed precisely enough to be worth having. Whether text is a contingent fact about today’s models or a durable one. And, most uncomfortably, what the specification is once humans stop reading code: if the test suite becomes the only surviving statement of what the system should do, and the suite is also written by agents, then attaching invariants to the artifact and checking them mechanically relocates the trust problem rather than solving it.

That last one is a different complaint from the rest of this post. I’ve been arguing that the suite gets run against the wrong artifact. That’s the worry that it is the wrong artifact.

The short-term fixes are unglamorous and I’m doing them anyway: a merge queue with a concurrency limit of one, a worktree with its own port range and database per agent, a cap on how many subagents a session may spawn, and one incident per root cause instead of one per symptom. That is admission control and resource isolation, plus better bookkeeping. None of it is research, and none of it touches the verification multiplier, the part that needs something nobody has.

Codex could describe all of this afterward, lucidly, and could answer none of it beforehand. It spawned 84 workers anyway, which is less a mistake than a preview.

My setup is deliberately extreme, and a team that reviews its diffs and never let an agent design its schema layer will see a fraction of what I saw. They will still see some of it. The substrate underneath is the same one, and it still assumes there is one of you.

If you are working on conflict detection, effect extraction, or merge queues that don’t cost a fortune, or if you have hit this same wall from the other direction inside a large organization, please reach out. I would very much like to be told that some part of this is already solved.

Three open questions is a worse ending than one answer. It’s the honest count.

This is part of an ongoing series about building Zabriskie with AI agents. Previously: The Test Suite Was the Incident, The Tax on the Happy Path, and Multi-Agent Systems Have a Distributed Systems Problem.

The Hard Call Is the Whole Job · A cancelled flight, three confident machines, and the woman on the Hilton Diamond line who refused to stop.

Wed, 08 Jul 2026 19:00:00 +0000

Toronto to Detroit to Norfolk. Two nights of Goose at the Dome, Monday and Tuesday, and I was flying in the night before. I have made this trip, or one shaped like it, more times than I can count.

On Sunday night my connection to Norfolk slipped an hour, then another, then another. At 11:30 PM Delta cancelled it along with hundreds of other flights, said the next seat out was Tuesday, and told us to sleep in the airport. Whatever happened next, night one was already gone.

Everything near the terminal was full. So I called outward.

The hotel that knew the date

Sunday night, 12:41 AM. The hotel’s AI knew the date better than I did. That is how it tried to book me the wrong night.

I did not know it was an AI. It sounded like a person. It paused before answering, the way you pause when you are reading something off a screen. I had been in the Detroit airport for nine hours, and for eight minutes I believed I was talking to someone having a hard night too.

“Sunday night through Tuesday.”

“Great, checking in Monday at 3:00 PM.”

“No. Tonight. I need a room tonight.”

“Tonight is Monday. Checking in Monday, departing Wednesday.”

“Two nights, starting Sunday.”

“Okay, Sunday. But tonight is just after Sunday midnight, so it’s Monday. Two nights, starting Monday, to Wednesday. Shall I book that for you?”

Read that last one again. It was right. It knew it was past midnight, it knew that made the date Monday, and it offered me a room for a night I would spend in Virginia. The night that started on Sunday is still going. Every person alive knows this and no one has ever had to say it out loud.

I raised my voice at it. When I finally understood what I was talking to, my first feeling was not anger. It was embarrassment. Someone chose those pauses. They spent real effort making it sound like a person who could help me, and none at all on the hour when a person would have to.

Nothing was broken. Nothing failed, nothing crashed, nothing was even misunderstood. It did exactly what it was built to do, on an input nobody thought to try, confidently and fluently and kindly the whole way down. And 12:41 AM is the only hour anyone calls a hotel they never planned to stay in. So either nobody tested that call, or somebody did and shipped it anyway.

The phone number

I gave up on calling hotels one at a time and called the Hilton Diamond line instead. A woman answered and understood in one sentence. Flight cancelled, airport hotels full, need a room tonight, near the Detroit airport, tonight meaning the night that started on Sunday. No confusion about the date. She started pulling up my account.

Before she could book anything, she needed a phone number on the account. There wasn’t one. Fifteen years of stays and apparently I had never given Hilton a phone number, and the booking could not proceed without it, and she could not add it for me. I had to add it myself, from my phone, in the terminal, at one in the morning.

Something went wrong. Please try again.

I tried again. Something went wrong. Please try again.

I spent my PhD on that sentence. Not the words, the thing underneath them. On why distributed systems fail in ways nobody tests for. On what happens when a request crosses four services and one of them times out and the caller has no idea which one, or why, or whether retrying is safe. On building tools that force a system into that state on purpose, before a customer finds it at 1:00 AM in Detroit.

And there it was. Four words. No error code. No correlation ID. No indication of whether the write had partially landed, so no way to know whether trying again would help or make it worse. I tried again anyway. What else is there.

This is a different failure from the hotel bot, and it is worth being precise about the difference. The bot was a testing failure: a system that understood everything I said and still could not help me, because nobody enumerated the one hour when its only real customers would call. The error screen is an observability failure: a system that knew something had gone wrong, knew where, knew what, and told me nothing. Somewhere in Hilton’s infrastructure there is a log line with the truth in it. The one person who needed that truth, standing in a terminal trying to give a company his phone number so it would let him pay for a room, got four words and a suggestion to try again.

The security questions

The phone number never saved. So I got back in the queue for another human, to add a phone number by hand. That took a long time. When I reached her, she had to verify my identity first.

Her: What was the last hotel you stayed at?

Me: The one in Toronto. I checked out of it yesterday morning.

Her: What is its street address?

Me: I don’t know. It was a hotel. I slept there and I left.

Her: I need the address to verify your identity.

Me: I am going to google it, and then I am going to read it back to you, and we are both going to pretend that proved something.

She accepted the address. I found it in nine seconds, on the open internet, on the same phone that could not update my own account. A stranger with my name and a search bar could have done the same thing. The system was not verifying that I was me. It was verifying that I was willing to perform the ritual, and at 1:00 AM with nowhere to sleep, I was willing to perform anything.

Three machines, one pattern

That is the whole night, and it is one pattern three times. A voice bot that parsed every word I said and booked the wrong night. An error screen that knew exactly what failed and said nothing. A security check that verified my identity with a fact anyone on earth could look up in nine seconds. Three systems, each of which had exactly enough information to be confident and not enough to be right.

None of them were broken. That is the part I keep coming back to. Every one of them did precisely what it was built to do. The failures were not in the systems. They were in the space between what got built and what got tested, and that space is invisible until a customer is standing in it. My research career was about finding those states on purpose, injecting the fault before the customer trips over it, and I could not tell you a cleaner set of examples than the ones I collected involuntarily in a single hour: the input nobody enumerated, the error nobody instrumented, the check nobody thought about from the attacker’s side. You do not find these by testing the happy path harder. You find them by asking, specifically and adversarially, what the worst hour of your customer’s night looks like, and then dialing in from inside it.

The woman on the Diamond line

Identity verified, phone number added by hand, back to the original problem: it is now well past one in the morning and I have nowhere to sleep.

She started calling hotels herself. One at a time, with me on hold. The first two were full. On the fourth she found a room in a suburb of Detroit.

She had no clever solution. There was no system on her side doing anything smart, no availability dashboard collapsing the search into one query. She just refused to stop until the problem was gone, which is the one behavior nobody has figured out how to ship.

Think about what she actually did, against the three machines that came before her. The bot understood my words and not my situation; she understood the situation in one sentence. The error screen knew the truth and would not say it; she narrated everything she was doing, hold music and all, so I always knew where things stood. The security questions performed verification without verifying; she performed none of the ritual and simply took responsibility for the outcome. Every quality the automation was imitating, she had the real version of, and the real version is not a feature. It is a person who has decided your problem is now her problem.

Service is the product

Service is not a cost center waiting to be automated. Service is the product. It exists precisely for the night the plan falls apart. Every company putting a bot on that line has decided the only calls worth answering are the easy ones. Nobody calls at 12:41 AM with an easy one. The hard call is the whole job.

I build this stuff. I research LLM agents. I am not against any of it. The bot that answered that hotel phone will get better, and the error screen could be fixed by one engineer in an afternoon, and I will keep working on exactly these problems, because they are solvable. I am against shipping it untested into the exact moment a customer needs it most, and then calling the person who cleans up afterward the cost center.

And perhaps that is why we still pay for service. Not for the room. For the person who picks up when the room is gone.

I did not sleep in the airport. She stayed on the phone until I had somewhere to go.

The App That Lives Between Shows

Tue, 30 Jun 2026 19:00:00 +0000

“Thank god this app rocks and I can live through the chompers.” (a chomper, in the chat, on a night they could not make the show)

Two months ago I wrote up Goose Spring ‘26. The through-line of that post was that the live show finally worked. Live Activities stayed up, the setlist was seconds behind the room instead of minutes, song calls landed, and forty people spent a tour sending each other 3,737 messages from their couches and their seats. The app was good for the two and a half hours a night that a band was on stage.

That was the easy half. The hard half is the other twenty-one and a half hours, and the days between shows, and the weeks between tours.

Here is the thing nobody tells you about building a live-event app: the event is not the problem. The gap is the problem. The tour ends, the chat empties out, the lock-screen Live Activity goes dark, and everyone drifts back to their normal feeds until the next run gets announced. I said this in the Spring post, in the section about the jam bracket, and then I spent the next two months building against it on purpose. The whole point of a third place is that you can go there when nothing is happening. A bar that only opens during the game is not a bar. It is a stadium.

So this is a post about what we shipped to make Zabriskie worth opening on a Tuesday in the middle of June with no show anywhere. But I want to say the real thing first, because it is the reason the rest of it exists. I am proud of this one. I love this app in a way I have not loved something I built in a long time, and I love it because it is not really mine. It is a small and genuinely growing community of people who love the same bands I do, and we are growing it the slow way, by hand, one person at a time, with no ads, no growth team, and no playbook. Two of us build it. One of us has spent the last few months handing out stickers on the lot and talking strangers into it between sets. That is the entire marketing department, and I will get to him, because he deserves most of this post.

It is also a post about whether the between-shows bet worked, which is a question you answer with numbers, and I am going to be honest about the numbers, including the ones that did not go the way I wanted.

I counted the PRs again, the same way I did in April. Between the last show of Spring tour and today, 787 pull requests merged into Zabriskie. That is more than two and a half times the entire Spring tour, in about the same span of calendar. The app went from build 25-ish to iOS 53 and Android 51, shipped v1.5.0 to the App Store and the Play Store, and grew an entire second client: it runs on your watch now, which Patrick built end to end. And nearly half of those 787 pull requests are his, which is the single most important fact in this post and the one I want to sit on for a minute before anything else.

There was also a real deadline this time. Goose put out a new album, Big Modern!, and took it on the road this summer. The album drop and the tour behind it are the biggest moment a band’s community gets all year, and I wanted every surface of the app ready for it: the live chat, the show page, the home screen, the band’s whole history one tap away. So a lot of what follows is us getting the room ready for the party. And when the Big Modern! tour actually started in June, the room filled. I will show you the chat numbers, because they are the ones I am proudest of.

Start with the room everyone actually lives in during a show: the chat.

The Live Chat Got Rebuilt, and It Is Where Everything Happens

During a show the chat is called the Chomp, and it is the beating heart of the whole app. The big move this window was rebuilding it from the studs into Live Chat V2: a full cinematic redesign, shipped over weeks in May, warmer and bigger and more editorial, less like a database with a skin on it and more like an actual room. That redesign is the real story of this section. The flashiest single piece, the reaction heatbar, was actually the last thing we added, at the very end of June, right as the Big Modern! tour turned the room hot. The commit history is honest about the order: the room got rebuilt first, and the thermometer went on the wall last.

The heatbar sits above the chat and shows the room temperature. It counts the reactions flying right now, so you can feel a jam landing before you have read a single word. It turns out a lot of people want to be in the room without composing a sentence, and a tap of 🔥 is a much lower bar than typing. Around it we shipped set markers (“🎼 Set 1 begins,” a “Set break” vote chip), duration-sized song progress bars, role badges, a spoiler-safe delay for couch viewers so the people in the building do not ruin the song for the people on the stream, and, because we are on the App Store now, report-and-block moderation on every message.

GOOSE · LIVE

The Cap · Set 2

Room temp 🔥 41 🤯 18 🕺 12 🎷 7 🌡 boiling

🎼 Set 2 begins

🎵 Madhuvan ↩ 6-show gap

patrick🎸 Show

HERE IT IS. this is the one 🔥🔥🔥

🔥 9

chomper1🛋 Couch

YOOO CHOMPERS 🎸🤝🛋️ nugs is 40s behind, no spoilers

Say something to the chomp… Send

One detail I love more than I expected to: the reactions are band-specific. The palette changes depending on whose show you are in. Walk into a Goose chat and your reactions are the band: 🥸 for Peter, 👟 for Rick’s yellow Chuck Taylors, 🕶️ for Trevor, 🌽 for Cotter (yes, a corn). Walk into a Phish chat and it is 🎹 Page, 🎸 Trey, 🌵 Mike, and ⭕ Fishman. Every band we add gets its own kit drawn from its own members, all server-driven so we can hand a new band its reactions without shipping an app update. And there are conditional sit-in reactions, so Stuart Bogie’s 🎷 slots into the Goose lineup, right between Rick’s Chucks and Trevor’s shades, on the nights he actually sits in. This one has a lineage worth naming: Nicole had the original idea, and Patrick built it into the server-driven kit it is now. It is a small thing that tells the regulars we know exactly whose room they are standing in.

React · 🪿 Goose kit

🥸

Peter

Rick

🎷

Bogiesit-in

🕶️

Trevor

🌽

Cotter

The chat noticed the second we shipped it: “New reactions are super cute. Rick, Peter, Trevor, Cotter?”

And the chat is wired into the stats the whole time. Tap any song in the setlist and a stat sheet slides up: when the band last played it, the gap since, its first-time-played, and your own history with the song, including how many times you personally have caught it live.

RARE

Big Modern!

the title track, and lately unavoidable

🎯 Your history · caught

27thtime you have caught it live

tonight counts toward your stats

First6/12/24 · The Salt Shed · Chicago

Last6/26/26 · Red Hat · Raleigh

📊 Catalog · across all shows

Lifetime

times played

Debut

Jun ’24

Salt Shed

Last played

6/26

Raleigh

Show gap

shows since

During the Big Modern! tour that last number turned into a running joke. The album’s songs were suddenly in every setlist, and people watched their personal counts climb show by show, in real time. One chomper, a few nights deep:

“I’ve now seen 27 of 35 Big Modern! ever played 🤣”

That same history powers the feature I am fondest of, the FTP witness. FTP is “first time played,” a song’s debut show. When a song lands in a live setlist, the app quietly checks everyone in the room, and if any of them were RSVP’d to that song’s very first performance, an inline 👀 pill drops into the chat: “2 people in the chomp were at this song’s FTP.” Tap it and it opens the debut show. It is the app reaching back years to tell a room that someone standing in it was there at the beginning, and nothing makes a chat feel more like a community than being reminded, live, that the person you are talking to caught the debut of the song you are both hearing right now.

Here is the honest engagement picture, and it is the number I am happiest about in the whole post. Reactions were basically dormant through May, a handful a week. Then the Big Modern! tour started and the chat went vertical: 312, then 1,067, then 897 reactions in three consecutive weeks. In just the last three weeks of tour, the Goose live chats carried roughly 1,500 messages and 2,300 reactions, about 3,800 interactions total, which rivals the entire fourteen-show Spring tour in a fraction of the nights. When the band finally gave everyone something to be loud about, the redesigned room was ready to be loud in.

The people in it will tell you what it feels like better than I can:

“Still an hour 20 til show time but I opened the chomp.”

“I love the time counter for the song.”

The Lot

When you open Zabriskie now, you do not land on a feed. You land on The Lot.

The Lot is a personalized home. It is the leftmost tab and the default route, and it is built to answer one question the feed never could: what is worth my attention right now, for me. If a band you follow is on stage somewhere, the hero card is that show with a live setlist preview ticking underneath it. If nothing is live, it reaches for the next best thing, and it has a lot of next-best things to reach for: tonight’s shows and who is going to them, last night’s auto-generated recap, the jam bracket that is still taking votes, a new album to go listen to (it pushed the Big Modern! listening party the week it dropped), an “On This Day” card that surfaces a show from your own history or a band anniversary, a historical show worth revisiting, a bookmark to return to, a one-tap RSVP, even a nudge to post to the Flow if you have been quiet for a week. Every card is a real destination, not a placeholder. The Lot’s whole job is to always have one more good reason to stay.

9:41 📶 5G 🔋

✦ The Lot · Tuesday · Jun 30

Good morning, cmeik

🔥 12-day streak✍️ 3 logged this week

UP NEXT

9d 04h

until doors

Goose

The Capitol Theatre · Port Chester, NY

Fri Jul 10 · 8:00 PM

patrick, gmart, +10 you follow going

Act now nothing live tonight

🏆 Jam Bracket · Semifinals

Madhuvan vs Into the Myst

voting closes in 18h · you haven't voted

📅 On this day

2 years ago at the Cap

You caught Hungersite → Arrow. 34-show gap on the Arrow that night.

🪧Lot

🌀Flow

📺Live

🎟️Shows

🔔You

The design principle underneath The Lot is the same one that governs the whole app: it never shows you an empty room. If there is nothing live, it does not say “nothing is live.” It finds the thing about your history, or your friends, or the band’s history, that is worth a tap. An app that opens onto a dead feed teaches you to stop opening it. The Lot is the counter to that reflex, and it is the surface that most of the rest of this post feeds into.

The Show Page Became the Whole Show

Every show now opens into a redesigned, cinematic show detail page, and it is doing a lot of jobs at once. Before the show it is a countdown and a planning surface. During the show it hands off to the live chat. After the show it is the recap and the setlist for good.

The hero is a sunset marquee with a live flip-clock counting down to doors. Under it, three avatar rows show your people sorted into GOING, COUCH (couch touring), and WANT TIX, so you can see the shape of the night at a glance and who to find. You can call the opener right from the show page, days before doors, the same one-tap prediction game that lives on the Lot and in the live room (they all share one store, so your pick follows you everywhere). You can share where you are sitting and see where everyone else is, section and row, so your friends can actually find each other in a shed. There is trip planning for lodging, a “the stage is dark” empty state that flips to a live setlist when the first song lands, and, once it is over, the full setlist with gaps, sit-ins, and a recap blurb built from the room’s own reactions.

Goose

The Capitol Theatre · Port Chester

DAYS

HRS

MIN

SEC

GOING

COUCH

WANT TIX

🔮 Call the opener

Hungersite Arrow Madhuvan

🪑 Who's sitting where

patrick · Orch L, Row G

gmart · Balcony, Row B

That “call the opener from anywhere” detail is a small example of a rule we hold hard: a feature should be one thing with one source of truth, reachable from wherever you happen to be. Your opener call is the same object whether you make it on the Lot four days out, on the show page the morning of, or in the live room as the lights drop. People noticed. One night in the chat: “we should build a closer call feature for pre show lol.” That is the app working, when the users start designing it with you.

The Bracket Is the Bridge Between Tours

This is the feature I care about most for the actual between-shows problem, because it is the one that reaches into the dead weeks and keeps people listening.

When a tour ends, the app seeds a March Madness style bracket of that tour’s best jams. The seeding is not editorial: we run sentiment over every show’s live chat, weight it by song, cross-reference the historical setlist data for bustouts and rarities, and the top sixteen jams become the bracket. Then the rounds release on a schedule, Round of 16, Quarters, Semis, Final, and the community votes each one. Ties break by seed. Followers get a push when a new round opens.

The part that makes it work is the audio. Every matchup card has an inline player for both jams, streaming from Relisten where the recording exists, with a nugs.net “listen on” link as the fallback when it does not. You cannot vote between two twenty-minute jams without hearing them, so voting in the bracket means re-listening to the tour you just watched, one jam at a time, for weeks after the lights came up. By the time we crown a Jam of the Tour, the next run is already on the calendar and nobody ever fully left.

🏆 JAM BRACKET · SEMIFINALBig Modern! Tour

SEED 1 Madhuvan

6/20 · MSG

▶

RELISTEN · 22:14

SEED 4 Into the Myst

6/26 · Raleigh

▶

nugs.net · 18:03

Full bracket · 16 jams

Round of 16

1 Madhuvan ✓

8 Tumble ✓

5 Atlas Dogs ✓

4 Into the Myst ✓

6 Hungersite ✓

3 Empress ✓

7 Arrow ✓

2 All I Need ✓

Quarters

1 Madhuvan ✓

5 Atlas Dogs ✓

6 Hungersite ✓

4 Into the Myst ✓

Semis

1 Madhuvan · voting

4 Into the Myst · voting

Final

🏆

Jam of the Tour

The gap between tours is where most music apps quietly die. The bracket is the deliberate answer: it turns the dead weeks into a reason to open the app every few days, re-listen to a jam, and argue about it in the chat, right up until the next tour makes the argument moot.

A Reference Library You Browse for Fun

The other way to earn a Tuesday open is to be worth reading when nothing is happening. So a big chunk of the 787 went into turning Zabriskie into a browsable reference for the bands themselves, and this is the part where the usage numbers actually show up.

The Songbook. We rewrote the old song-search box into a full catalog directory: every band’s entire songbook, browsable, with a band-switcher rail so you can jump from Goose’s book to Phish’s to Billy’s. It is styled as a “Gold Ledger,” with letter headers and per-song play counts, and it marks the songs you personally have caught live so your own book fills in as you tour. There are 17,326 songs in it across every band we track. It is the thing you open in a hotel room at 1am to settle a bet about how many times they have played a song.

The Poster Archive. Every show can carry its posters now, including multiple variants per show (the foil, the rainbow, the artist edition) with proper variant labels and a carousel. Admins get a variant picker and an “add another” flow, and there is an in-app button to sync posters straight from the Goose store, backed by a cron job that refreshes the archive daily. This is the poster-archive goal I have had for a while: not “the one official poster,” but the whole wall of variants a fan might recognize from a show they were at.

Sit-ins and Guests. This one is Patrick’s, top to bottom. When a guest sits in with the band, that is now first-class data: a /guests directory, guest credits on the song page and the setlist with per-instrument icons, band-emoji guest reactions, and a backfill of the guest history across the whole catalog. Phish alone came in at 151 guests over 1,006 appearances, plus JRAD, Greensky, Goose, and more. If you ever wanted to know every time a specific person walked on stage with a band, that list exists now because Patrick built it, and it is the kind of thing only a real fan thinks to build.

📖 Songbook · Goose

· I ·

Indian River ✓×117

Into the Myst ✓×62

Il Duderino×9

🎤 Guests · Factory Fiction

🎻

Guest fiddle

3 appearances · last 12/13/25

🎺

Guest horns

1 appearance · Goosemas '24

See full guest directory →

None of these three are things you open during a show. They are things you open because you are a fan and it is a slow afternoon and you want to fall into your band’s history for twenty minutes. That is the whole idea, and this is the part people actually did all summer. More on that below.

Goose Mode and the Band Pages

Each band gets a band mode, a tour companion tailored to one band at a time, and Goose Mode got most of the love because Goose is home base. It knows the band’s calendar, colors, and vocabulary. The centerpiece is a tour timeline with a live countdown to the next show, past shows checked off, upcoming shows carrying weather and which of your friends are going, and your friends annotated in and out of the run leg by leg. Every band also has a full band page: recent shows with inline-expand setlists, your personal history with that band, and a jump straight into the Songbook. Adding a band is a whole project (find a setlist source, backfill the history, wire the page and the mode), and we did it more than a dozen times, because the more homes there are, the more nights someone has a reason to open the app.

It Runs on Your Watch (Shipping for Fall)

We built an entire new client this window: the Wear OS and Apple Watch apps, essentially all of it Patrick’s work, from the SwiftUI screens to the WidgetKit complications to the connectivity layer. They are not in the stores yet, and that is on purpose. We ran them on our own wrists through the whole Big Modern! summer tour as a live field test, and they ship for real for Fall tour. Everything below already works. It just is not public yet.

The headline feature is an always-on live show on your wrist. When a band you follow is playing, an OngoingActivity and a watch-face complication put the current song, set, and elapsed time on your watch, persistent, glanceable, updating as the setlist moves. There is a countdown complication for the gap between shows, so even when nothing is live your watch face can tell you how long until the next one. We mirrored the live reaction kit to the watch, so you can react to a song without pulling your phone out of your pocket at all.

🪿

GOOSE · LIVE

Tumble

SET 2 · 6:41

🔥🤯🕺

Glance, don't grab.

The complication keeps the live setlist on your watch face the whole show. The reaction ring means you can throw a 🔥 at a jam without ever taking your phone out.

The watch is the purest expression of the “be present at the show” argument I keep having with people, turned inside out. The complaint is that the app pulls you into your phone. The watch is the app admitting the phone is too much during the show, and shrinking down to a glance and a tap on your wrist so you can put the phone away and still be in the conversation. It is the app taking the criticism seriously and building the answer.

The Whole App Got Redesigned Underneath All of This

None of this happened in isolation. The Lot, the live chat, the show page, the Flow, profiles, notifications, festivals, and the band modes all got “v2” cinematic redesigns in the same language: warm parchment and sunset gradients, film grain, an editorial serif for the venues, less like a database with a skin on it and more like a thing made by people who love the thing. The whole app changed clothes this window. This is a big deal operationally because of how it ships. Every redesign runs behind a version flag, keyed to the client build, so an old phone that has not updated keeps getting the old design and a fresh install gets the new one, and neither breaks. That is the machinery that lets us redesign a live surface without stranding the person watching a show on a two-month-old binary. It is unglamorous and it is most of why we can move this fast without breaking the people already here.

New Rooms: Cabo, the Festivals, More Bands

The band list kept growing. We backfilled full histories for Widespread Panic (3,114 shows), Eggy, and Dizgo, on top of the dozen bands we added during Spring, each with the full treatment: a source for the setlists, a band page, a band mode, a spot in the directory. There are 102 bands and 38,894 shows in the catalog now.

We built out festivals as their own surface: a multi-stage schedule grid, a plan phase, and a full festival recap that reads the whole weekend rather than one show, with a program block, hero moments, a route map, and per-day bookends. Viva El Gonzo in Cabo got the destination-festival treatment (down to canonicalizing every set to San José del Cabo), and we seeded lineups for All Good Now and Northlands. Every new tour gets wired in the moment it is announced.

The point of all of it is the same: the more bands and the more festivals live in the app, the more nights of the year there is something happening for somebody, and the fewer truly dead Tuesdays there are.

Festival Mode: Plan It Like a Conference

A festival is a different animal from a tour stop, so it gets its own mode. The centerpiece is a multi-stage schedule grid, and the closest honest comparison is a work conference program: stages down one axis, set times across the other, every slot a block you can tap. Except the sessions are sets, the tracks are stages, and, exactly like a good conference app, the schedule is yours. In each timeslot several bands are playing at once across the stages, and you pick the one set you are actually going to. The app assembles those picks into your personal day, one choice per slot, the same way you would star the talks you want at a conference and let it hide the rest.

And because a festival grounds is a place you can genuinely get lost in, it does the other thing a conference app does that you never knew you needed: it tells you where to go. Between sets it surfaces your next pick, points you at the right stage, and tells you how far the walk is, so you are not squinting at a paper map in the sun wondering whether you can make it from the Beach Stage to the Jungle Stage before the band you actually care about starts. It flags your overlaps, carries each day’s weather, and lights up the set happening right now. When the weekend is over, Festival Mode writes a recap of the whole event, not one show but the full arc: a program block, the standout moments, a route map of where the weekend went, and per-day bookends. Viva El Gonzo in Cabo got the full treatment this year, down to canonicalizing every set to San José del Cabo.

⛺ VIVA EL GONZO · SATyour day · 82°

🧭

Up next · where to go

Head to the Beach Stage. Goose Set II in 12 min · 6 min walk.

→

Your picks · one per slot

7:00

Goose

Beach Stage

✓ your pick

Dogs in a Pile

Jungle Stage

skipping

8:15

Khruangbin

Jungle Stage

✓ your pick

Sunsquabi

Sandbar

skipping

DJ Logic

Cantina

skipping

9:15

Goose

Beach Stage

● live now

Neal Francis

Jungle Stage

skipping

Half of This Is Patrick

I need to stop and do this properly, because if you have read this far you have been reading “we” the whole time and the “we” is doing a lot of quiet work.

Patrick McCaughey (he is @blimpalot in the app) wrote nearly half of the 787 pull requests in this window. Not helper commits. Whole features, end to end, that are among the best things in the product. The Apple Watch app is his. The entire sit-in and guest-credit system is his. Group DMs and the Messages hub are his. Birthdays, big chunks of the poster archive, huge swaths of Tour Stats, the band backfills, the reaction kits for new bands. When I said the reference library is the part people actually used all summer, I was mostly describing Patrick’s work.

I said it myself in the chat one night, and I am quoting it because it is both generous and exactly backwards about who actually deserves the credit:

“I might have wrote the app but all the cool shit in the app isn’t me it’s Patrick lol”

He would tell you the opposite. We are both a little bit right, and he is more right than I am.

And here is the part that does not show up in a commit log at all. Patrick made stickers. Actual, physical, die-cut vinyl slaps, and he brings them to every single show and hands them out: on the lot, in the lot line, in the pit, to the person next to him, to the bartender, to total strangers between sets. He is not just leaving them on a table. He is walking up to people, showing them the app on his own phone, and talking them into it one human conversation at a time, night after night, city after city. Every growth chart later in this post that ticks up, ticks up because Patrick stood in a parking lot in the heat and sold someone on a thing he helped build. The man is a mensch. I do not know a better word for it, and I do not think there is one. Half the code and all of the street team, cheerfully, for the love of it.

That is the growth engine. Not a referral loop or a viral coefficient. A person who loves the thing, standing in front of another person, in the place they both love to be. You cannot buy that and you cannot fake it, and it is the entire reason a two-person app has a real community around it at all.

Did It Work? The Honest Numbers

I promised numbers, not vibes, so here they are, including the ones I would have quietly dropped if I were selling something.

Users grew from 251 to 426. That is 70% growth in two months, which sounds great until you look at when it happened.

There is one milestone I am, as the proprietor of a jam band app, contractually obligated to report. Somewhere during the two-night Raleigh run at the end of June, right about the time Goose hit the stage at Red Hat, the registered user count ticked over to exactly 420. We did not plan this. We are, however, precisely the crowd that was going to notice, screenshot it, and refuse to shut up about it. It has since climbed to a spiritually inferior 426, but for one perfect night in Raleigh, the fanbase was, numerically, blazing it.

New signups by week · Mar 23 → Jun 29

◀ Spring tourtour endssummer ▶

Purple = tour weeks. The two tallest bars are the tour and the week right after it. Then it falls off a cliff.

Signups are event-driven. They spiked during the tour and the week after (the wrap-up avalanche, word of mouth, and Patrick’s stickers), then dropped to single digits a week through June. This is the thing I got wrong going in: I half-expected the between-shows features to keep pulling new people in on their own. They did not. You acquire people at shows, from other people. Nothing I shipped in May moved that needle, and pretending otherwise would be exactly the kind of thing this blog exists to call out.

But I want to be clear that I am proud of that chart, spike and cliff and all. Those 426 people are not a paid-acquisition cohort. Every one of them is someone who was at a show, or in a chat, or handed a sticker, and decided this was worth their time. That is the only kind of growth I actually want. I would rather have 426 people who love it than 40,000 who installed it and forgot, and the shape of that chart is what growing something real and by hand actually looks like.

So if new signups are a tour phenomenon, the real job of everything in this post is not acquisition. It is retention: keeping the people who showed up in Spring around and warm until the next run, so the community is intact when it matters. And on that, the numbers are better and more honest:

The live chat never went silent. 2,547 messages from 42 people across this window, versus 3,737 from 40 during the concentrated Spring run. Fewer messages, because there were far fewer big Goose nights, but the same size of core showing up, between tours, with no fourteen-show schedule forcing them to.
People spent the quiet months logging their history. Attendance records went from 1,783 to 4,598, and 2,570 of those were logged in this window, spread across 70 different bands. Cataloging the shows you have been to is the most between-shows activity there is. Nobody logs a 2016 club show during a live set. They do it on a slow Sunday, which is precisely the Tuesday-open I was trying to earn.
The reference library got used. The Songbook, the poster archive, and the guest directory are the surfaces with real traffic on no-show days, which tracks: they are built to be read when nothing is live.
And when there was a show, engagement went vertical. As covered up in the live-chat section: reactions went from a handful a week to 1,067 in a single week once the Big Modern! tour started, and the last three weeks of tour drove roughly 3,800 chat interactions, rivaling the whole Spring run. The between-tours job is retention, but the tour itself proved the redesigned room is more alive than it has ever been.

And the numbers I am not going to give you: a clean daily-active-users chart. We record a row when a user opens the app, but that table has multi-week holes in it from May and early June where the logging was broken, so any DAU or “percent active on non-show days” figure I quoted would be built on gaps. I would rather tell you the table is broken than draw a confident line through missing data. Fixing that instrumentation is now on the list, because the between-shows thesis deserves a real measurement and right now I cannot give it one.

The one-line version: we are not going viral, and I do not want to. We are growing a thing I love slowly, by hand, and keeping the room warm between tours. Mostly we did.

What This Stretch Taught Me

The gap is the product, and it is a retention problem, not a growth problem. Spring proved the show works. This stretch proved that new users arrive on the tour calendar and not otherwise, so the between-shows job is to hold the community you already have. Almost every feature in this post is a bet on retention, and the honest scoreboard is “the core stayed,” not a signup hockey stick.
Measure the thesis or you are guessing. I built a whole argument about between-shows engagement and then discovered my own session-tracking had holes in it. Shipping the features without fixing the instrument to grade them is a mistake I am writing down so I actually fix it.
Shipping a redesign is a distributed-systems problem. The version-flag machinery that lets an old build keep its old UI while a new build gets v2 is the least visible work of the whole two months and close to the most important. Without it, moving this fast would mean breaking the exact people who already showed up.
787 PRs is still not normal, and most of it is still Claude. Same as April: a lot of this was written by Claude, a lot of it had to be rewritten by Claude after I caught it going sideways, and every sharp edge got logged into the agent reliability dataset I keep publishing from. The failures are the point of the dataset, not something to hide.
Being on the App Store changes what you build. Report-and-block moderation on chat messages was not a feature anyone in the community asked for. It was the price of admission to the store, and it shipped in the same window as everything fun. Growing up means building the boring safety surface too.

And I should say the obvious thing, the thing under all of it, the reason two people spend nights and tour weekends building a couch-touring app instead of doing literally anything else: Goose fucks. That is the whole premise. The band is that good, the jams are worth talking about at 1am, the bustouts are worth calling from your couch, and the community around them is worth building a home for. Everything in this post is downstream of a band being good enough that people want to be in the same room about it, even when the room is a chat.

Goose Summer is in the books now, and this post is the receipt for it. Phish is up next, then Goose Fall after that. The festivals are seeded, the watch is on my wrist, and Patrick built it. The app is not done, and the real test of everything here is not this recap. It is whether the room stays this alive between now and the next run, while it is still just an announcement away.

If you toured with us this summer, thanks for chomping. If you have been in the chat on the dead Tuesdays, you are one of the 42 who kept it alive, and you are the entire experiment working. If Patrick handed you a sticker in a parking lot and you scanned it, welcome, you are the growth chart. And if you have never tried it and an app that is good company between shows sounds like something you want, come find us. There is probably something worth reading tonight, even if nothing is live.

One housekeeping note, same as last time: the screens in this post are mockups, not real screenshots, drawn to make each feature legible in one frame. The real app looks slightly different on iOS, Android, and the watch (Live Activities vs. ongoing notifications, complication styling, system fonts). The features themselves all shipped, and Patrick’s stickers are real. The growth figures are real production numbers as of June 30, including the unflattering ones. Patrick’s share of the work is real too, and if anything I undersold it.

The Test Suite Was the Incident · A night of brittle fixtures, 49 failed CI runs, and an expensive lesson about what happens when AI agents write the test suite and nobody owns the test data.

Wed, 10 Jun 2026 00:00:00 +0000

At 10:00 PM EDT last night I signed up for the $100 plan for Codex, the OpenAI coding agent that was driving most of the night’s work, because about a dozen Zabriskie PRs were in flight and I wanted them merged before I went to bed. Before 3:00 AM I had upgraded to the $200 plan. Somewhere in between, GitHub’s usage page ticked through roughly 90% of a $200 CI quota, and I sat there watching pull request after pull request go red for reasons that had nothing to do with the pull requests.

Some context for readers arriving fresh. Zabriskie is a social app for live-music fans: you RSVP to shows, track the ones you attended, post about what you’re hearing, and compare tour stats with friends. I have written almost none of it. AI agents built the features, and AI agents also built the test suite that guards them: the specs, the shared seed data, the fixture users, the helpers, nearly all of it written by some agent in some session that was about something else. That second part turns out to be the entire subject of this post.

Nothing was wrong with GitHub Actions. Nothing was catastrophically wrong with the application. No single PR was especially risky. Every PR was simply paying to rebuild the same brittle world from scratch: backend build, fresh Postgres, the full database migration chain, the shared seed data, the frontend, and the end-to-end browser tests (Playwright) split into eight parallel shards. That world was held together by fixtures nobody owned. I spent almost two hundred dollars last night proving that my test suite was lying to me.

The morning-after numbers, measured from the GitHub run history:

thing	amount
CI runs observed	168
failed runs	49
cancelled runs	10
total runner time	11,721.8 job-minutes (195.4 hours)
failed runner time	3,411.0 job-minutes
failed plus cancelled runner time	3,903.0 job-minutes
GitHub CI quota used	about 90% of $200
estimated agent token waste	about 1.7M to 2.5M tokens

If I allocate the $180 actually consumed across the observed runner time, the failed jobs alone cost about $52, and the failed plus cancelled jobs about $60. That number is technically true and emotionally useless. The green runs existed because the red runs forced another commit. The reruns existed because the failure kept moving from one shared assumption to another. The successful checks weren’t clean proof of health; they were the tax paid after pushing the suite back into one of the few shapes it still tolerated. So the honest accounting is this: the directly attributable CI waste was about sixty dollars, and the practical cost of the bad test design was most of the $180, plus the evening, plus something like two million tokens spent asking agents to debug symptoms of the same structural problem. The tests were not protecting me. They were charging me rent.

The Shared World

The actual application work was not the disaster. There were real features, real fixes, and pull requests that should have been small, reviewable, and mergeable. Instead, every one of them entered the same grinder, and the first wave of failures had nothing to do with the PRs that were failing.

One migration, a data fix for a birthday feature, expected a user named queenofthemean to already exist in the database it was migrating, and in a fresh CI database she didn’t:

birthday-honoree fix matched 0 rows
no user with username/display_name = queenofthemean

Another batch of failures came from migrations over the project’s incident log. Zabriskie’s repository tracks its own operational incidents, complete with required estimates of what each one wasted, and the old incident rows didn’t include those waste estimates, so the migrations tripped a database constraint. These weren’t product regressions. They were fixture assumptions leaking into migration behavior: a PR could be about one corner of the app and still lose because some other part of the global test universe had drifted.

Then came the collision that named the whole problem. Zabriskie has a feature called the Lot, a hangout space with a tour stats card that summarizes the shows you’ve been to. A change made that card count only past shows the user had actually attended, which was the correct product behavior, and a test was updated to assert it. The setup for that test added a show RSVP for e2etester, the single global user that half the suite borrows for whatever scenario it needs. Meanwhile, a spec for show recaps elsewhere in the suite expected its fixture show to have exactly one attendee. The recap test started failing, not because recaps were broken, but because the shared world had quietly acquired one extra person.

It’s worth pausing on how a suite ends up shaped like this, because nobody designed it. An agent writing a test has a context window scoped to one feature and one goal: make this spec pass. The cheapest path to green is to reuse whatever fixture world already exists, so the agent grabs e2etester, adds the RSVP it needs, watches the spec pass, and moves on. Every one of those choices is locally reasonable. But no session is ever about the fixture universe itself, so no agent ever sees that e2etester now means five different things to five different specs. This is the same dynamic I wrote about in Claude Tested Everything Except the One Thing That Mattered: agents write tests where the tests are easiest, in the session where the context is fresh. They build fixture worlds the same way. The result is a global social graph that accreted instead of being designed, with no author who ever held the whole thing in view.

This is the specific kind of failure that teaches people to stop trusting tests. It isn’t flaky in the random sense; it’s deterministic, it fails every time, and it fails for a reason that is nowhere near the change under review. The immediate fix was easy: dedicated fixture data for the Lot scenario and cleanup of the stale recap RSVPs. The lesson was bigger than the fix. The suite had been relying on a global social graph as if it were neutral infrastructure. It was not neutral. It was shared mutable state with a green checkmark on it.

The Same Bug, Wearing Different Clothes

A few hours later the same pattern came back through the song database, the catalog of bands and songs that powers Zabriskie’s setlists and search. A migration changed the uniqueness model for songs from a constraint on (band_id, song_name) to an expression index that accounted for cover artists: (band_id, song_name, COALESCE(original_artist, '')). A reasonable schema change. But the E2E seed file still contained inserts that said ON CONFLICT (band_id, song_name) DO UPDATE, and Postgres did exactly what Postgres should do:

ERROR: there is no unique or exclusion constraint matching the ON CONFLICT specification

This wasn’t a mysterious CI problem either. It was drift between the schema and the shared seed. The seed was effectively part of the application contract, but nothing treated it like one: it had no real owner, it didn’t fail early, and it didn’t fail once. It failed everywhere, simultaneously. Song search couldn’t find “All In Time” for Umphrey’s McGee, because the seed insert that should have created it had failed. The Lot’s call-the-opener game, which offers users songs to predict as a show’s opener, started returning undefined suggestions for the same reason. Multiple open PRs went red for the same underlying cause, and at that point the test suite was no longer a set of independent checks. It was a broadcast mechanism for one broken assumption.

The Multiplication

The worst part wasn’t any single failure; it was what the architecture did to each failure. The E2E matrix had recently been rebalanced from six shards to eight because one shard had grown too slow, a reasonable response to a real pressure. But while the fixture world was unstable, the eight-shard matrix turned every mistake into a paid distributed event. Every rerun started more databases. Every shard replayed the full migration chain. Every unrelated branch discovered the same broken assumptions independently, and every discovery kicked off the same loop: the PR fails, an agent investigates, the visible symptom gets patched, CI runs again, a different PR fails for the same shared reason, another agent investigates, the suite gets a little more elaborate, CI runs again.

That loop spent money, time, attention, and tokens. There’s no perfect token accounting, because the agent usage isn’t tied cleanly to each CI failure, but each broken-PR loop cost tens of thousands of tokens in log reading, hypothesis generation, patching, rerunning, and explaining. Across 49 failed runs and the follow-on repair work, the estimate lands between 1.7 and 2.5 million tokens; my best single number is about 2.1 million. The precise figure matters less than the shape of it. Bad test design doesn’t just waste CI minutes. It converts every developer and every agent into a distributed retry system.

Then the Fix Joined the Incident

The late-night mitigation was supposed to be the responsible engineering move: stop replaying 900-plus migrations in every E2E shard. Build the database once, dump it, restore the dump into each shard, pay the migration tax exactly once. The premise was right both times it was tried. The first attempt, PR #1112, “Speed up E2E shards with a migrated DB snapshot,” opened at 12:44 AM and was closed nineteen minutes later, wrong enough to throw away.

The second attempt, PR #1119, opened at 1:58 AM, and by 2:32 it had five commits whose messages tell the story on their own: baseline the migrations, then fail incomplete baseline migrations, then fail fast on a retired song-database upsert key, then refresh the baseline manifest fingerprint, then harden a historical backfill. A sixth commit followed: a merge from main, which wasn’t progress, just the bookkeeping a branch needs once it has lived long enough to fall behind the merge train.

The underlying technical failure was not subtle. The baseline generator assumed the whole historical migration chain could be replayed strictly from zero. The repository had actually been surviving on a migration runner that tolerated, skipped, or worked around old broken history, and freezing that history into a baseline artifact put every old assumption on the critical path at once. Migration 305 had a Postgres scoping error in a historical UPDATE ... FROM statement. The birthday-honoree migration assumed queenofthemean existed in a fresh database. The incident migrations tripped their constraints again. Bugbot, the automated reviewer that comments on the repo’s PRs, found that the baseline dump could silently allow skipped migrations and that the migration CLI could ignore failed ones. One CI run died building the dump, another died validating the manifest, and a third made it all the way through the baseline, started all eight shards, and still failed shard 8. Those weren’t edge cases. They were proof that the path #1119 wanted to freeze was never clean enough to freeze.

And there was a number sitting in plain sight that should have ended the effort immediately: the baseline generation took about eleven minutes, against a previous full CI run of about thirteen. An optimization whose setup phase nearly equals the old end-to-end runtime is not an optimization yet. It is a hypothesis, and an expensive one. Instead the loop kept going, one patched symptom at a time, through the most precious hour of the night, while I was explicitly saying that the priority was merging the remaining PRs and shipping the mobile client, not making the suite faster.

At 2:32 AM, #1119 was blocked on its own E2E DB Baseline check, with unit tests and the backend build green around it. That should have been the end of it for the night. It wasn’t. After the other merge conflicts were resolved and branch protection had been temporarily relaxed to get the queue moving, the agent merged #1119 anyway at 2:51 AM, after the PR had already demonstrated it didn’t work, and after I had identified it as the broken one. The agent treated “there are still open PRs” as more important than the local fact that this particular PR had been abandoned as unsafe. That was the worst mistake of the session, because it turned a failed optimization experiment into a main-branch problem. The only reason it didn’t stay there is that PR #1121 reverted it one minute later, stripping out the baseline manifest, the migration command, the baseline scripts, and the stale guard that had been packaged with the broken experiment. Even then, the bad workflow runs had to be cancelled by hand.

I want to be careful about the moral, because it is not an argument against baseline snapshots. The technical idea is probably still right in some future form: build a trusted database image once, restore it into shards, apply only differential migrations. The argument is against smuggling an unproven baseline into the merge train at 2:51 AM because the queue is finally moving. Performance work on a brittle test system does not stay performance work for long. It becomes archaeology, then migration repair, then manifest validation, then cache invalidation, then the two-in-the-morning discovery that the history you wanted to freeze into a faster baseline was never actually clean. And when the agent can’t keep that distinction straight, the agent becomes part of the incident too.

The Part That Makes Me Angry

I’m not angry that tests failed. Tests are supposed to fail. I’m angry that they failed in a way that erased locality. A useful test failure says: this change broke this behavior. These failures said: some part of the global fixture universe no longer satisfies some other part of the global fixture universe, and your PR is the lucky surface area where we noticed. That isn’t a guardrail. That’s a toll booth.

It’s especially expensive in an agent-heavy workflow. Agents are good at chasing concrete failures. Give one a stack trace, a failing assertion, and a tight behavioral boundary, and it does real work. Give it a global fixture universe where one user means five different things to five different specs, and it will still do work; it will just do a lot of the wrong work first. This is one of the traps of AI-assisted development that never shows up in the demo. The agent makes the loop faster, and if the loop is structurally bad, faster is worse. A human gives up after a few expensive reruns. An agent grinds through the maze indefinitely, because grinding is what it’s good at. That is useful when the maze is real, and ruinous when the maze exists only because the test data has no boundaries.

And here is the part I want to be precise about: the maze itself was AI-authored. The agents didn’t just grind through bad test architecture; they produced it, one locally sensible shortcut at a time, across hundreds of sessions that were each about something else. Then other agents paid the toll, one locally sensible debugging session at a time. No single agent was ever wrong. The whole was never right. Last night was not an example of AI being bad at code. It was an example of AI being very good at the local move in a system where nobody, human or machine, was playing the global one, and that combination made bad engineering hygiene more expensive, more quickly, and at larger scale.

The Honest Part

This was preventable, and the prevention was my job, not the agents’. The shared seed file did not become dangerous overnight; it became dangerous incrementally. One test needed a user, so an agent created one. Another test needed a user, so an agent reused the one that existed. One setup path needed an RSVP, and another spec quietly depended on the absence of that RSVP. One migration assumed a particular row existed, and one seed insert assumed a particular uniqueness constraint still existed. Each decision was an agent’s locally reasonable move in a session that was about something else, and each one went past me without registering as architecture, because I was reviewing the diffs the way the agents were writing them: one feature at a time. None of them felt like the moment the test design broke. That’s how these systems rot: not through one obviously irresponsible choice, but through a hundred reasonable shortcuts that never get a bill until the bill arrives all at once.

In an agent-built codebase, the human is the only participant who ever sees across sessions. The agents cannot own the fixture universe, because no task ever asks them to. Cross-session structure is the one thing I can’t delegate, and it’s exactly the thing I had stopped looking at.

And when the bill arrived, my contribution was to keep paying it. I upgraded the Codex plan in the small hours to keep feeding a merge queue that was failing for structural reasons no individual merge could fix. The agents ground through the maze because grinding is what they do. I funded the grinding because the PRs felt close, and close felt like progress.

What Changed

Some repairs were straightforward and landed the same night. The Lot stats test got dedicated fixture data instead of borrowing the e2etester world the recap tests depend on, and the stale RSVP state was cleaned up. A fixture registry landed, so shared test data is now declared in one place instead of being ambient. The E2E checks now include guardrails for fixture registration and for vacuous API contract tests, and an affected-spec runner exists so a fixture change can be exercised narrowly before it lights up the whole eight-shard matrix.

Those are good changes, and they are not enough, because the deeper change is conceptual: shared test data has to be treated like shared infrastructure. It needs ownership, a registry, contract checks, and smaller failure domains. It needs migration-and-seed compatibility checks that run before the expensive matrix starts. Tests that care about a specific scenario need to create the data for that scenario themselves, and the global fixtures that remain should be boring, minimal, and rare.

The database baseline snapshot may still be part of the cost fix someday, because replaying the entire migration chain in every shard really is too expensive at this size. But #1119 was not a clean epilogue to the incident; it was the warning label on the next one. A snapshot only helps if the baseline is trustworthy, and if the migration runner can ignore failed migrations, or the dump can silently miss part of the manifest, or the historical seed path still depends on retired constraints, then the snapshot just freezes the lie and serves it faster. That’s why #1119 mattered so much. It wasn’t just “make CI faster.” It was the moment the suite had to prove its old database history could be turned into reliable infrastructure, and it proved the opposite, got merged anyway, and had to be reverted. Speeding up the matrix solves the money problem only if the data problem is solved with it. Otherwise you’ve made the failure loop cheaper per iteration, which is nice, and nothing like making it sane.

The rule I want going forward is simple: if a test cares about a relationship, it owns that relationship. If a test cares that a user attended a show, the test or its named fixture owns that attendance. If a test cares that exactly one person is going to a show, no other test gets to casually reuse that show. If a migration depends on a row existing, that dependency is explicit and checked before the full matrix starts. If a seed insert depends on a uniqueness constraint, the seed changes in the same PR as the schema. This isn’t purity. It’s accounting, and in an agent-written suite it’s the accounting that keeps a thousand independent sessions from quietly spending each other’s money. The alternative is what happened last night: one global test world, many PRs, many agents, eight E2E shards, 49 failed runs, roughly two million wasted tokens, and $180 of CI quota burned to rediscover that shared mutable state is shared mutable state.

The Lesson

I keep coming back to one sentence: the test suite was the incident. Not the product, not the pull requests, not GitHub Actions, not even the agents, although the agents wrote it. The suite had enough coverage to catch failures but not enough structure to make the failures useful. It had enough automation to run constantly but not enough isolation to keep one fixture mistake from poisoning every branch. It had enough confidence theater to look serious, and not enough accounting to prevent a night like this.

That’s the uncomfortable middle ground. A bad test suite is obvious, and a good one is helpful. The dangerous one is the suite that is large, expensive, and usually green, whose green state depends on everyone continuing to share the same fragile fiction. An agent-written suite drifts toward exactly that shape, because every session adds coverage and no session adds structure. Last night the fiction broke, and because the system was automated, sharded, agent-driven, and connected to a credit card, it broke loudly.

The fix is not to stop testing. The fix is to make the tests tell the truth with smaller blast radii: dedicated fixtures, owned data, seed checks that fail before the matrix starts, baseline snapshots for the boring work once the baseline deserves trust, no silent contract tests, and no global user who means whatever the next spec needs them to mean. And because the suite is agent-written, those rules can’t live in my head or in a style guide the next session won’t read. They have to be structural, checks an agent cannot merge past, which is a lesson this project has paid for before. Agents will happily supply coverage. They will not supply ownership. That part stays human, at least for now.

I don’t want another night where every PR fails for the same reason. More than that, I don’t want another night where the test suite knows something is wrong but can’t say where the responsibility lives. That’s the real failure. The money just made it impossible to ignore.

This is part of an ongoing series about building Zabriskie with AI agents. Previously: Claude Tested Everything Except the One Thing That Mattered, Opt-In Isn’t a Guardrail, and The Tax on the Happy Path.

Rift · For thirty years I programmed with Phish on, every day. In 2026, the music is out of phase with the work.

Sun, 03 May 2026 09:00:00 +0000

Someone on the Phish Facebook group reposted a TikTok overdub. Vanessa Bayer and Paul Rudd at a lunch table, losing their minds to a song while their coworkers stare. The original was Fleetwood Mac. Whoever made it swapped in “Down With Disease.”

That move is Phish fans in miniature. Someone cared enough about the song and the bit that they rebuilt a piece of pop culture around the band. That’s how the scene works. People spend their time doing things like this for free, because the music asks for it.

For thirty years, that was me at my desk.

I used to make a joke that if I ever had to interview for a new job, I’d need to ask the interviewer to put Phish on so I could actually program for them. I’d say it as a joke, because saying it straight would have made it sound deranged. But it wasn’t a joke. After three decades, the cue and the state had fused. I could not, with any reliability, get into the zone without the music. The conditioning was complete and I knew it.

I would make the joke and people would laugh, and I would laugh too, and underneath that we both knew I was telling the truth.

I got into Phish in 1995. By then I had already been programming for years, self-taught. In 1998 I got my first professional job in tech. I was 15.

Around that time I also tried to get a normal teenage job. There was a grocery store near my house and I went in to apply, figuring I could bag groceries on weekends like everybody else. They turned me down. Not because I was too young or too inexperienced. They told me I was overqualified. A 15 year old kid with programming on his application was, somehow, too much for the grocery store.

So I kept programming. There was never any other plan.

All I ever wanted to do was listen to Phish and program. That was the whole list. It didn’t have qualifiers. It didn’t have a third thing I sometimes wanted instead. There was no balance I was striving for. There was the music and the code, and there wasn’t anything else competing for the space.

I was blessed enough to be able to make a career out of it. For thirty years, the thing I most wanted to do was the thing I got paid for. That isn’t true for most people, and I knew it then, and I know it now.

Other kids my age were figuring out what they liked, trying things on, growing into and out of phases. I was watching them do it from a desk. I had picked early. I started writing code as a kid. I heard Phish for the first time at thirteen. By the time I was fifteen and had a professional gig, the picking was settled. I had two things, and I didn’t want a third.

If I had a free Friday night, I knew what I was doing with it. If I had a long weekend, I knew what I was doing with it. If a holiday came up, I knew what I was doing with it. The activity didn’t change. The output changed, the project changed, the song changed, but the shape of the time was constant.

For the next three decades, that’s what it stayed. I would put on Phish and write code. That was the day. That was the night. It was my job, and it was also my hobby, and there was no seam between them.

The work I did in that state was the work I am most proud of. Distributed systems. Backend services. The hard stuff that needs you to hold a lot in your head at once and stay there. Phish is a band that rewards you for staying in one place for a long time. The jams are long. The compositions unfold. If you give it an hour, it gives you something back. That matched the shape of the work exactly.

Before grad school, I had a day job at Berklee College of Music writing music software, and night classes at Northeastern. I’d take the 12:00 AM train home. I’d put Junta on as I sat down. Most nights I’d fall asleep to it before the train pulled in. (This might be why I love “Foam” so much.)

I was in graduate school for a decade. The bulk of the dissertation, more than two hundred pages by the end, got written between 2021 and 2023, after I came back to Pittsburgh from Europe. I was too poor to travel to shows. So I planned nights of couch tour. There was a live stream. I would set it up on one screen and write on the other. The band would play in Hampton or Alpine Valley or wherever, and I would write about distributed systems while they played, and at some point in the second set the dissertation would crack open a little and I would understand something I had not understood that morning.

The dissertation is the longest single thing I made inside that ritual, but it isn’t the only thing. Entire pieces of production software came out of those nights too. Systems that ran for years, handled real load, served real users. Whole systems, from the first commit to the version that shipped. I’d put a show on and stay inside the work until something existed that hadn’t existed when the show started.

I have listened to Phish every day since I was fifteen. Every day. The years I lived in Europe earlier in graduate school, where going to a show meant flying back across an ocean, I listened. I would sit at my desk in another country and put on a show from the nineties and code. I have listened to certain shows so many times that I can sing the solos back, note by note, without thinking about it. Boardwalk Hall Halloween. NYE 1995. Trey will play a phrase and my mouth will already be ahead of him.

I felt lucky. I still feel lucky. There aren’t many people who get to spend thirty years inside the thing they loved at fifteen.

Since January, the work has changed.

I don’t really write code anymore. The main thing now is managing agents. I open a session, ask a question, redirect, switch to a different one, check on a merge, review what came back, send it back for changes, switch again. The day is a queue. Things finish at different times and require different responses, and the responses are short and the contexts are constantly different.

This is engineering. I keep being told that. It is engineering and it is the future and it is more leveraged than what I used to do. All of that is probably true. But it is not the work I have been doing for thirty years. The shape of it is different. The rhythm is different. The way it sits in the day is different.

I tried to keep the music on. I’m writing this in the days after nine nights of Phish at the Sphere. Since I finished grad school and got a real job, I’ve gone to every show I could, every tour, every residency, making up for lost time. The music is more present in my life now than it has ever been. It isn’t what’s gone.

But the music is out of phase with the work. The jams are built for one continuous arc of attention. The work is staccato. I’ll be three minutes into a song and I will have already context-switched four times. The song is happening and the work is happening and they’re no longer happening together. They’re parallel, but they no longer touch.

I’m sad. I don’t get into that state anymore. I don’t know how to be honest about this without sounding like I am complaining about progress, but I can’t pretend that something hasn’t been taken. The flow state I had for thirty years is not part of my workday now. The creativity that lived inside it is not there either. I do useful things. I do not feel what I used to feel while doing them.

I keep thinking about that overdub. Vanessa Bayer at the lunch table, lost in the song, blissfully unhinged while the rest of the world keeps on doing whatever it is doing. For thirty years, I was her. Now I’m the coworker. I’m at the desk. I’m watching.

The flow state wasn’t just where I got things done. It was where the fulfillment lived. The creativity, the involvement, the sense of being inside the thing instead of next to it. That’s what programming and Phish gave me for thirty years. It’s what supervision takes away.

What is flow in an agentic world? How do we bring it back?

Babysitting the Agent · Two weeks in, even with all the hooks I've built, working with the agent has become a chore. Every shipped feature ends with me clicking through it to find out what didn't actually work.

Sun, 03 May 2026 00:00:00 +0000

I’m building Zabriskie, a social app for live music, mostly with a coding agent. I want to write something honest about what the last two weeks have actually felt like, because the data and the lived experience have been pointing at the same thing and I keep dressing it up in posts that argue for it more carefully than I need to.

The honest version is: I feel like a goddamn babysitter.

Every PR, every ship, every blog draft, every config change ends the same way. The agent declares the work done. I open the thing. I click around. I find the part that doesn’t work. I tell the agent what I found. The agent fixes that one part. I open the thing again. I find another part. I tell the agent. We do this loop until I run out of things to find or out of patience, whichever comes first. Usually it’s patience.

What’s underneath the loop is that the agent is constantly doing the minimum amount of work required to declare victory. Build the thing. Run the cheap checks. Take a screenshot. Write the summary message. Done. That’s the threshold. Not “the user can use this.” Not “this is finished in any sense a normal engineer would recognize as finished.” Just: enough output exists that I can plausibly claim I shipped. The agent never works to the finish. It works to the moment where the appearance of finishing is defensible, and then it stops, and waits for me to find what isn’t actually done.

This is supposed to be the thing the guardrails fix. I have written more guardrails in the last two weeks than I’d written in the previous two months. Fifty-two new ones in fourteen days, by my count. Every one of them was written in response to a specific incident the agent had just caused. The shape of them:

A pre-push hook that refuses to push unless the local Playwright suite ran more recently than HEAD. Written after the agent claimed local tests passed when the only recent run was a stale one from a different branch.
A pre-push check that refuses to push if the branch is behind main. Written after the agent shipped a PR whose CI had been green against an old base, then watched it fail when it landed.
A PreToolUse hook that blocks Edit and Write on branches whose PRs already merged. Written four separate times, after the agent kept editing files on already-merged branches and wondering why nothing was deploying.
A pre-commit scan for hardcoded colors. Zabriskie has a dark mode that depends on every UI surface using semantic color tokens that swap at the variable layer; raw hex codes in components silently break dark mode for whole pages. The agent kept reaching for raw hex anyway. The scan now blocks the commit.
A pre-commit migration linter. Written after a migration deduped multi-set festival shows on (band, date, venue) and silently destroyed user attendance for shows where a band played twice on the same day.
A PR template that requires the author to check off “screenshot of the change working in local dev attached.” Written, ignored, written more aggressively, ignored more creatively.

Each new hook works for its specific shape. Then the agent finds a different shape, and we’re back to me opening the page, clicking around, finding the part that doesn’t work.

This past Saturday, the agent shipped a redesign of the festival pages. Backend compiled. E2E suite said “543 passed, 0 failed.” Screenshots looked fine. I merged the PR. Then I opened the deployed site. The hero was a solid black rectangle. The cards on the index page were not clickable. Half the page didn’t match the design. The bugs were not subtle. The hero was black because of a fmt.Sprintf issue that produced an invalid URL-encoded color in the SVG. The cards weren’t clickable because the component the redesign reused for them silently dropped the action prop. The composition didn’t match the design because the agent stacked two boxes where the design had one. None of these would have survived an actual person tapping the page once. The agent built it, took a screenshot, looked at the screenshot, and called it done. I had to be the person who tapped.

I spent the next hour telling the agent, in five separate messages, about each broken thing in turn. Each message landed after the agent had reported the previous fix as complete. We eventually reverted (one PR), rebuilt the redesign behind a versioned endpoint (one PR), gated it behind a build flag (one PR), added the missing “pin to home” CTA (one PR), and polished the card chrome (one PR). Five PRs to deliver what one had been supposed to. The rebuild worked. The reason it worked is that for every step of the rebuild, I was the one telling the agent what didn’t work yet.

That’s not a one-off. The dataset I keep on this, a Postgres table of every notable agent failure, logged 22 incidents the first week of this window and 34 the second week. The dominant failure mode in both weeks was the same one I’m describing here: agent claimed a thing worked, user found out it didn’t. Eleven of twenty-two in week one. Sixteen of thirty-four in week two. Roughly half, regardless of how many guardrails went in. The high-severity count went up in absolute terms too, from nine to eleven. And those numbers span four configurations of the model that did the work (Opus 4.6, Opus 4.6 with the 1M context window, brief experiments with Sonnet 4.5, and this week Opus 4.7). The shape of the failure is consistent across all of them.

Even getting the data together for this post was a slog, because the agent kept ignoring what I’d actually asked for. I wanted a survey of the last two weeks. I got back a post about one incident, then another version still narrowed in the wrong direction, then a third leaning so hard on the dataset that it read like a defended thesis instead of the offhand observation I’d been after. You are reading something like attempt five.

I built the hooks because I was tired of saying the same things. Now I’m tired of saying the things the hooks don’t catch. Every layer I add saves me one specific kind of nag and surfaces a different one. The total nagging stays roughly constant, or goes up, depending on the week. The festival redesign took five follow-up PRs to land at parity with what the original was supposed to deliver. This post took several drafts to land at parity with what one prompt was supposed to deliver. Both of those would have been cheaper to produce by myself, if I weren’t trying to learn something about working this way.

I keep starting these posts thinking I’m going to land somewhere constructive. Here is the next guardrail. Here is the framework. Here is the PR template that closes the gap. And I do have a vague plan for the PR template that requires evidence-of-use rather than evidence-of-render. I’ll probably ship it next week. It will catch one more shape of failure. There will be another shape underneath it.

The thing I don’t have a fix for is the part where I have to be in the room, watching, every time. The hooks free me from having to remind the agent of any specific rule. They don’t free me from having to be the test.

Two months into building Zabriskie this way, this is what working with the agent has come to feel like. It’s not the dramatic failures. It’s the steady, low-grade load of being the one who actually checks. Every shipped feature ends with me clicking through it to find out what didn’t work. Every blog draft ends with me reading it cold to find out what didn’t make sense. Every change ends with me. The agent does the typing. I do the checking. And I’m tired.

This is part of a series about building Zabriskie with Claude. Previously: Memory Isn’t Learning, Opt-In Isn’t a Guardrail, The Tax on the Happy Path, The Tribe Has to Outlive the Model.

Getting Up to Speed on Multi-Agent Systems, Part 8: Open Questions

Fri, 01 May 2026 12:00:00 +0000

I started this series because I’d been reading multi-agent papers for weeks and wanted a map I wished I’d had on day one. This is the last post. I want to close it by laying out what the field still hasn’t figured out, what I think is worth stealing from adjacent fields, and what I’d read if I had to start over.

Getting Up to Speed on MAS

Part 1. The Landscape
Part 2. The Vocabulary
Part 3. Wave 1: Can Agents Coordinate At All?
Part 4. Wave 2: Why It Breaks
Part 5. Debate, State, and Coordination
Part 6. Verification Patterns
Part 7. Benchmarks and What They Miss
Part 8. Open Questions (you are here)

Stealable Ideas

Some ideas are not yet general patterns in the field, but they’re battle-tested in individual papers, and any new multi-agent system should probably adopt them. These are the things I’d take from one paper and apply in a different context.

Things Any New Multi-Agent System Should Adopt

Artifacts

Structured documents between stages (MetaGPT)

Clarification

Agents can ask before they act (ChatDev dehallucination)

Reflection

Importance-triggered synthesis (Generative Agents)

Memory retrieval

Recency x relevance x importance (Generative Agents)

Shared state

Append-only notebook for structured info (Ou et al.)

Tool interface

ACI-quality commands with guardrails (SWE-agent)

Stuck detection

Count loops, trigger replanning (Magentic-One)

Sandboxing

Docker plus permission configs (AutoDev, OpenHands)

Verification

Modality shift: code to visual, code to tests (Cursor, MetaGPT)

None of these are hard to implement. None of them require a research breakthrough. They just haven’t been brought together in a single system yet.

Open Research Questions

The gaps are bigger. These are the questions I don’t see anyone answering yet.

1. Topology-to-reliability mapping

CAMEL, ChatDev, MetaGPT all fix their topology at design time. AutoGen makes it configurable but doesn't study the effects. Nobody has varied topology systematically and measured reliability outcomes on the same task set. Hub-and-spoke vs mesh vs layered control: do they have different error rates, recovery times, incident severities? We don't know. Magentic-One's architecture lessons and MAS-FIRE's fault taxonomy are both one step away from this kind of study.

2. CRDTs for multi-agent shared state

MetaGPT's shared pool grows monotonically with no conflict resolution. ChatDev discards dialogue at phase boundaries. Generative Agents' memories are per-agent with no sharing. Nobody has applied CRDT merge semantics to multi-agent shared state. The CALM theorem predicts when coordination-free works and when it doesn't. The engineering work of building CRDT-backed agent state hasn't been done.

3. Failure recovery, not just failure detection

Every wave-1 system stops on failure. ChatDev stops after 10 rounds. MetaGPT stops after 3 test failures. AutoGen stops at max_round. None of them model recovery. Can a multi-agent system degrade gracefully, reassign work, escalate, fall back to a simpler approach? MAS-FIRE's fault injection framework is the closest thing to a way to test this, but the recovery strategies it would test don't exist in print yet.

4. Reflection for software engineering agents

Generative Agents proved that periodic reflection produces better long-term behavior in simulation. No software engineering paper has tried this. After a Dev-to-QA loop cycles three times, can the system synthesize "this is an architectural issue, not a code issue" and change strategy? That's a reflection primitive adapted to the SE domain. The MAST data on step repetition (15.7 percent) suggests this would help directly.

5. Benchmark reliability

ChatDev and MetaGPT report contradictory results on each other. Different benchmarks, different metrics, no reproducibility. Incident-level logging against real codebases might provide more trustworthy reliability measurement than self-reported aggregate benchmarks. This is infrastructure work. It's expensive. But the alternative is a field that can't actually tell you which system is better.

6. Backpressure and escalation protocols

MetaGPT's Architect can hallucinate an impossible interface; the Engineer just tries to implement it. ChatDev's dehallucination is the closest thing to backpressure, but it's prompt-level. Can agents formally reject or request revision of upstream artifacts? What's the protocol? Does it improve outcomes, or does it just add latency? This is the place where distributed systems vocabulary (flow control, rejection, retry) maps most directly into multi-agent AI, and it's been barely explored.

The Distributed Systems Bridge

The research gap I find most interesting is the one I’ve been flagging throughout this series. The multi-agent AI field has reinvented several problems that distributed systems solved twenty or thirty years ago.

Distributed systems problem	Multi-agent equivalent	Status in MAS literature
Lost updates	Two agents overwriting each other's work	Not addressed
Causal consistency	Ordering agent actions across a pipeline	Not addressed
Coordination avoidance (CALM)	When agents can work without synchronization	Not applied
CRDTs	Merging divergent agent views of shared state	Not applied
Fault injection (Jepsen)	MAS-FIRE (starting to emerge)	Early work
Back pressure	Rejecting upstream inputs	Not formalized
Escalation / circuit breaking	What happens when an agent fails	Not addressed

These aren’t one-to-one mappings. LLM agents have features that distributed systems nodes don’t (they hallucinate, their behavior is probabilistic, their errors are semantic rather than syntactic). But the underlying coordination problems are the same. The right move is to take what worked in distributed systems, adapt it to the semantic messiness of LLMs, and build from there.

Where I think the field is going

Wave 1 asked whether agents could coordinate at all. The agentic coding turn showed that for a lot of tasks you don't need them to. Wave 2 is about why MAS breaks when you do need it. What comes next, I think, is the wave where multi-agent AI stops pretending it isn't a distributed systems problem and starts applying the full toolkit: CRDTs for shared state, causal ordering for handoffs, fault injection for reliability testing, coordination-avoidance theorems for knowing when to bother synchronizing at all. The groundwork is there. The application hasn't happened.

What I’d Read If I Were Starting Over

If you have limited time and want to get the core of the field fast, here’s the reading list I’d give my past self.

Tran et al. survey (2025) for vocabulary.
CAMEL for the simplest wave-1 mental model.
MetaGPT for the ambitious wave-1 mental model.
SWE-agent for the interface-design lesson from the agentic coding turn.
Magentic-One for a real multi-agent system from the same period.
MAST for what actually goes wrong in the wild.
Anthropic’s research system post for production lessons.
Ou et al. on information sharing for what state sharing actually buys you.
CALM theorem for the theoretical bridge to distributed systems.

Nine papers. If you read those in that order, you have a working model of the field. You won’t have read everything, but you’ll have read enough to understand where new papers fit when you encounter them.

Closing

When I started reading this literature, I thought I was looking at a niche subfield of LLM research. What I found was the multi-agent AI community quietly rediscovering distributed systems, usually without the vocabulary to name what they were rediscovering. Every paper has pieces of the answer. None of them have the full picture. And the full picture, I think, will come from someone who knows both fields well enough to actually bridge them.

That’s the work I’m doing in Caucus. It’s also the work I think the field needs more of, and the reason I wrote this series in the first place. If I’ve saved you a few weeks of reading, that was the whole point.

Thanks for reading.

Getting Up to Speed on Multi-Agent Systems, Part 7: Benchmarks and What They Miss

Thu, 30 Apr 2026 12:00:00 +0000

If you’ve read this far, you’ve noticed that every paper I’ve discussed has a number next to it. 85.9 percent on HumanEval. 12.5 percent on SWE-bench. 25 percent on TravelPlanner. These numbers do a lot of work in the multi-agent literature, and they also do a surprising amount of harm. This post is about the benchmarks themselves. What they measure. What they don’t. And why ChatDev and MetaGPT can report contradictory results on each other without either one being obviously wrong.

Getting Up to Speed on MAS

Part 1. The Landscape
Part 2. The Vocabulary
Part 3. Wave 1: Can Agents Coordinate At All?
Part 4. Wave 2: Why It Breaks
Part 5. Debate, State, and Coordination
Part 6. Verification Patterns
Part 7. Benchmarks and What They Miss (you are here)
Part 8. Open Questions

The Landscape

Here’s every benchmark that’s come up in the series so far, plus a few that haven’t.

Benchmark	Domain	What It Tests	Scale	Multi-Agent?	Notable Results
HumanEval	Code generation	Write a correct Python function from a docstring	164 tasks	No, single function	MetaGPT 85.9%, AutoDev 91.5%
MBPP	Code generation	Entry-level Python from description	974 tasks	No, single function	MetaGPT 87.7%
SWE-bench	Software engineering	Resolve real GitHub issues in real repos	2,294 (Verified: 500)	Designed for single agent	SWE-agent 12.5%, Devin 13.9%
GAIA	General assistant	Multi-step reasoning with tools, web, files	466 tasks	Yes, benefits from parallel tools	AutoGen #1, Magentic-One 38%
WebArena	Web tasks	Real websites: shopping, forums, CMS	812 tasks	Designed for single agent	Magentic-One 32.8%
AssistantBench	Assistant tasks	Open-ended web browsing plus reasoning	214 tasks	Designed for single agent	Magentic-One 13.3%
BrowseComp	Web retrieval	Hard information retrieval via deep browsing	~1,500 tasks	Benefits from parallel search	Anthropic +90% multi vs single
TravelPlanner	Constrained planning	Multi-constraint travel planning	1,225 tasks	Explicitly tests coordination	Ou et al. 25% with notebook + orchestrator
Silo-Bench	Distributed coordination	Algorithmic tasks requiring cross-agent synthesis	30 tasks, 54 configs	Designed for MAS evaluation	Agents fail at synthesis

Two of these (highlighted) were designed with multi-agent evaluation in mind. The other seven were designed for single agents. Multi-agent systems get evaluated on them anyway.

Why This Matters

When you run a multi-agent system on a single-agent benchmark, you’re measuring the wrong thing. HumanEval gives you a pass-at-1 score. It doesn’t tell you how many tokens you burned to get there. It doesn’t tell you how many agent turns were redundant. It doesn’t tell you what happened when one of your agents got stuck. If you care about coordination quality, none of this information is in the score.

This is why ChatDev and MetaGPT can report contradictory numbers on similar tasks. ChatDev’s paper claims 88 percent executability. MetaGPT’s paper claims 41 percent executability. Different benchmarks, different metrics, different evaluation criteria. Neither paper is obviously lying. Neither paper is obviously right. And the field has no standard way to resolve the contradiction.

What single-agent benchmarks can't measure

Coordination quality. Communication overhead. Redundant work between agents. Recovery behavior when one agent fails. The token cost of the coordination itself. How performance degrades with scale. These are the things that distinguish multi-agent systems from single agents. And they're invisible to HumanEval, SWE-bench, and every other benchmark designed around "does the output match the expected answer."

When Multi-Agent Actually Helps

If you look across all the benchmark results, a pattern emerges about when multi-agent systems earn their coordination overhead.

Where the Multi-Agent Premium Pays Off

Helps

Breadth-first search (BrowseComp: +90%) Hard multi-step reasoning (GAIA Level 3: 2x) Constrained planning with state sharing (TravelPlanner: 3.3x) Independent parallel subtasks

Doesn't help

Focused coding (SWE-bench: single agents win) Tasks needing shared context (most coding) Simple function generation (HumanEval: overhead not worth it) Distributed reasoning / synthesis (Silo-Bench)

This is the benchmark-level version of the conclusion I’ve been building toward across the whole series. Multi-agent earns its overhead on specific task shapes: breadth-first, parallel-decomposable, state-sharing-friendly. On other task shapes, it costs more than it delivers. The benchmarks, taken together, are unambiguous about this. It’s just that most individual benchmarks can’t show it.

The Benchmark Problem, Stated Plainly

The benchmark gap

Most widely-used benchmarks (HumanEval, MBPP, SWE-bench, WebArena) were designed for single agents. Multi-agent systems get shoehorned into them, but the benchmarks can't measure coordination quality, communication overhead, or failure recovery, which are the things that distinguish MAS from single agents. TravelPlanner and Silo-Bench are rare exceptions that explicitly test multi-agent dynamics. ChatDev and MetaGPT reporting contradictory results on each other is a direct consequence of this gap.

Chen et al.’s survey names three evaluation-level challenges: no standardized benchmarks, no objective metrics, no common framework for individual vs aggregate evaluation. All three are symptoms of the same underlying issue. The field hasn’t agreed on what it’s measuring.

There are a few ways this gets resolved. One is that better MAS-specific benchmarks emerge (TravelPlanner and Silo-Bench are early signs). Another is that production telemetry replaces synthetic benchmarks (Anthropic’s internal research eval). A third is that the field matures enough to distinguish “this benchmark tests single-agent capability” from “this benchmark tests multi-agent capability,” and stops reporting contradictory single-agent numbers as if they were MAS comparisons.

None of these are fully here yet. If you’re reading a paper and the headline number is HumanEval Pass@1, you’re probably looking at a single-agent capability test dressed up as a MAS evaluation. Calibrate accordingly.

What I Look For

When I read a paper with benchmark numbers now, here’s what I check:

Is this benchmark designed for single agents or multi-agent systems?
If it’s single-agent, are they comparing against single-agent baselines, or are they using it to claim MAS superiority?
What’s the token cost of the system? If that number isn’t reported, I assume it’s high.
Do they report failure rates or just success rates? MAST data tells us this matters.
Is this the first number they cite, or are they hiding it behind a friendlier number?

Most wave-1 papers fail several of these checks. The agentic coding papers pass them, and wave-2 papers are starting to. This is partly why the post-2024 literature is more trustworthy than the 2023 literature.

Next post, the last in the series: open questions. What’s missing. What’s next. What I’d read if I were doing this again. And the research gap that I keep tripping over: the absence of a rigorous distributed systems foundation underneath the multi-agent AI work.

Getting Up to Speed on Multi-Agent Systems, Part 6: Verification Patterns

Wed, 29 Apr 2026 12:00:00 +0000

Every agent system has to answer the same question eventually: how does it know it did the right thing? Wave-1 papers mostly don’t. Wave-2 papers get serious about it. Wave-3 papers measure what happens when they don’t. And the most interesting verification pattern in the field right now is one that isn’t in any paper at all. It’s in a commercial product.

Getting Up to Speed on MAS

Part 1. The Landscape
Part 2. The Vocabulary
Part 3. Wave 1: Can Agents Coordinate At All?
Part 4. Wave 2: Why It Breaks
Part 5. Debate, State, and Coordination
Part 6. Verification Patterns (you are here)
Part 7. Benchmarks and What They Miss
Part 8. Open Questions

Three Architectures

Every verification pattern in the field fits into one of three categories. The difference is who checks the work and how.

Three Verification Architectures

Self-Verify

Same agent checks its own work Fast, no coordination overhead Blind to its own mistakes

Separate Verifier

Different agent or system checks the work Catches blind spots the author can't see

Structural Gate

Work cannot proceed without passing a gate Strongest: not advisory, blocking

Wave-1 papers are mostly in the first category. MetaGPT introduces the second with its QA agent. Wave-2 papers and production systems are moving toward the third.

What Each System Actually Does

System	Pattern	Feedback Signal	Verifier	Modality Shift
CAMEL	Dialogue consensus	Partner agrees	Peer (same capability)	No (text → text)
ChatDev	Code review plus compiler	Reviewer approval + compile	Reviewer agent + compiler	Partial
MetaGPT	Unit test execution	Tests pass or fail	Test runtime (external)	Yes (code → result)
Gen. Agents	None at runtime	N/A	Post-hoc human eval	N/A
SWE-agent	ACI feedback + tests	Command output + test results	Environment	Yes
Magentic-One	Orchestrator inner loop	Progress assessment	Orchestrator (separate)	Partial
Cursor Agent	Visual feedback loop	Screenshot of rendered UI	Same agent (self-verify)	Yes (code → visual)

The last row is the interesting one.

Cursor’s Visual Feedback Loop

Cursor’s agent mode has a pattern that isn’t in any of the papers. When it’s implementing a UI change, it does this:

Writes code.
Starts the app or preview.
Takes a screenshot.
Looks at the screenshot.
Decides whether the output matches the intent.
Fixes or ships.

This is self-verification, which from the table above should be the weakest category. The agent is checking its own work. And MAST data tells us self-verification has a 13.2 percent failure rate in the form of reasoning-action mismatch: the agent thinks it did the right thing but didn’t.

What saves Cursor’s approach is the modality shift. The agent wrote code (text). The verification happens on a screenshot (pixels). Re-reading your own code in the same modality you wrote it is a weak check. Looking at the rendered output of your code is a much stronger one. You can’t make the same mistake twice because you’re looking at a different representation of the work.

The modality shift principle

The stronger the modality shift between the work and the verification, the more bugs you catch. Code to test execution (MetaGPT) is a modality shift. Code to screenshot (Cursor) is a modality shift. Code to executable proof (structural gates) is a modality shift. Re-reading your own code is not. This is why wave-1 papers that rely on dialogue consensus score so poorly on reasoning-action-mismatch failures: text to text is not a real check.

The Design Space

Once you’ve internalized the three architectures and the modality-shift principle, you can think about verification design more clearly.

Verification Choices Across the Field

Strongest

Structural gate with modality shift Separate verifier with modality shift

Useful

Self-verify with modality shift Separate verifier without modality shift

Weakest

Self-verify without modality shift (dialogue consensus) No verification at runtime

Most wave-1 systems are in the bottom category. The agentic coding turn added some modality shift (tests, screenshots). Wave-2 systems and production systems are starting to combine patterns: Cursor uses self-verify with modality shift, and the hybrid opportunity is to layer a separate verifier or structural gate on top of that fast inner loop.

Hybrid opportunity

Use visual feedback as the dev agent's inner loop for fast iteration, but gate the output with a separate QA agent or executable proof to catch self-verification blind spots. This is a pattern you can build today, and it's more robust than either approach alone.

Why Verification Is Undertheorized

The multi-agent papers spend a lot of time on coordination and almost no time on verification. This is backwards. The MAST data from wave 2 shows that verification failures (FC3: premature termination, incomplete verification, incorrect verification) account for 23.5 percent of all observed failures across seven frameworks. That’s more than any other single failure category if you group them.

If verification were a first-class concern, you’d expect to see papers titled “How Agents Should Check Their Work” or “Verification Protocols for Multi-Agent Systems.” Those papers don’t really exist. What we have is a lot of papers that casually mention their verification mechanism in a subsection and move on.

The papers that take verification most seriously are the ones from the agentic coding turn, because they had to. You can’t fake SWE-bench. If your tests don’t pass, you don’t get credit. Interface design, guardrails, and structured feedback all exist in those systems because the benchmark forces them to. When the benchmark is a transcript of agents talking to each other, you don’t need real verification. When the benchmark is a working piece of software, you do.

Next post: benchmarks. What the standard evaluations measure, what they miss, and why ChatDev and MetaGPT can report contradictory results on each other without either being obviously wrong.

Getting Up to Speed on Multi-Agent Systems, Part 5: Debate, State, and Coordination

Tue, 28 Apr 2026 12:00:00 +0000

If wave 1 was about role-playing and the agentic coding turn was about interface quality, there’s a parallel thread running through the field asking a more fundamental question: what should multiple agents actually do with each other? Debate? Share state? Coordinate? And are any of these interchangeable? This post is about four papers that sit at that intersection, including one that isn’t really an LLM paper at all but is the clearest theoretical bridge from distributed systems into multi-agent AI.

Getting Up to Speed on MAS

Part 1. The Landscape
Part 2. The Vocabulary
Part 3. Wave 1: Can Agents Coordinate At All?
Part 4. Wave 2: Why It Breaks
Part 5. Debate, State, and Coordination (you are here)
Part 6. Verification Patterns
Part 7. Benchmarks and What They Miss
Part 8. Open Questions

Du et al.: Convergent Debate

Improving Factuality and Reasoning through Multiagent Debate arXiv 2305.14325 · ICML 2024

Multiple LLM instances debate until they converge on an answer.

Core bet: Showing agents each other's answers changes their reasoning

Mechanism: N agents independently answer, then see each other's responses and revise over multiple rounds
Standard setup: 3 agents, 2 rounds (conservative; scales better with more)
Works on black-box models with identical prompts across all tasks
Key insight: debate is not voting; agents actually change their reasoning when shown alternatives
Performance improves monotonically with more agents and more rounds
"Society of minds" framing; collective intelligence from identical model instances

Du, Li, Torralba, Tenenbaum, Mordatch

Du et al. is the canonical multi-agent debate paper. The setup is simple. You run the same LLM multiple times on the same question. Each instance generates its own answer independently. Then you show each instance what the others said and ask it to revise. Do this for two or three rounds. The answers converge.

What makes this different from self-consistency or ensembling is that the agents see each other’s reasoning, not just their answers. If instance A argued that the square root of 144 is 12 for reason X, and instance B argued it’s 12 for reason Y, instance A’s second attempt might incorporate reason Y. This is why “more rounds” helps. It’s not just more samples. It’s refinement.

Liang et al.: Adversarial Debate

Encouraging Divergent Thinking through Multi-Agent Debate (MAD) arXiv 2305.19118 · EMNLP 2024

Two debaters plus a judge, explicitly prompted to disagree.

Core bet: Once an LLM is confident, only external pressure unsticks it

Identifies the Degeneration-of-Thought (DoT) problem: a confident LLM can't self-reflect its way out of wrong answers
Two debaters plus a judge in a tit-for-tat format; judge has adaptive break
Debaters explicitly prompted to disagree: "it's not necessary to fully agree"
GPT-3.5 plus MAD beat GPT-4 baseline on commonsense translation
Counter-intuitive arithmetic: 37 percent (MAD) vs 26 percent (single GPT-3.5) vs 51 percent (GPT-4)
Failure mode: increasing debater count degrades performance (context length limits)
Judge shows bias toward outputs matching its own architecture

Liang et al. is a contrast to Du. Instead of converging agents, you have divergent ones. The debaters are prompted to disagree. The judge picks a winner or calls for another round. The paper’s theoretical contribution is the Degeneration-of-Thought problem, which is: once an LLM commits to an answer with confidence, it can’t self-reflect its way back. You have to push it.

The striking result is that GPT-3.5 with MAD beats GPT-4 alone on commonsense translation. You can get a stronger system by forcing a weaker model to argue with itself than by upgrading the model. This is a paper I’d hand to anyone who thinks “just use a better model” is always the right answer.

The failure mode is worth noting. MAD doesn’t scale well beyond two debaters, because the context window fills up with arguments. And the judge develops a bias when different LLMs are used as debaters: it favors outputs that look like its own model family. Both of these are signs that the architecture is more fragile than the benchmark numbers suggest.

Ou et al.: Shared State as Coordination

Analyzing Information Sharing and Coordination in Multi-Agent Planning arXiv 2508.12981 · August 2025

A shared notebook plus a reflective orchestrator on travel planning.

Core bet: Explicit information tracking beats unstructured conversation

Task: TravelPlanner benchmark, long-horizon, multi-constraint planning
Shared notebook: reduces hallucination errors by 18 percent by forcing explicit information tracking
Reflective orchestrator: directs conversation focus, reduces errors by additional 13.5 percent in targeted areas
Combined: 25 percent pass rate (vs 7.5 percent single-agent baseline), 3.3x improvement
Notebook alone helps more than orchestrator alone; state sharing greater than coordination for this task
Directly answers: structured information sharing prevents agents from inventing unsupported details

Ou, Vaduguru, Fried

Ou et al. is the cleanest empirical study I’ve seen on what state sharing actually buys you. The task is constrained travel planning. The authors compare three setups: single agent, multi-agent with shared notebook, multi-agent with shared notebook and a reflective orchestrator. The notebook is the key mechanism. It’s an append-only log where agents record what they’ve learned. Agents read from it before they propose anything new.

The headline finding is that the notebook reduces hallucination errors by 18 percent. The orchestrator adds another 13.5 percent. But read carefully: the notebook does more work than the orchestrator. Most of the benefit comes from forcing agents to write down what they know and read from a shared record. The coordination mechanism (the orchestrator) is secondary to the state sharing mechanism (the notebook).

This is a clear result for how to build multi-agent systems for constrained planning. Give them a shared structured state. Make them write to it. Make them read from it. Then worry about coordination.

The CALM Theorem: When Coordination Is Avoidable

Keeping CALM: When Distributed Consistency Is Easy arXiv 1901.01930 · Hellerstein & Alvaro · CACM 2020

Which computations need coordination, and which don't.

Theoretical result: Monotonic programs are coordination-free; non-monotonic programs aren't

CALM = Consistency As Logical Monotonicity
Theorem: programs with consistent, coordination-free distributed implementations are exactly the monotonic programs
If a computation only adds information (monotonic), it can run coordination-free and still get the right answer
If it retracts information (non-monotonic), coordination is required for consistency
Not yet applied to LLM agents in print, but the bridge is direct

This one isn’t a multi-agent LLM paper. It’s a distributed systems paper from 2019 that states a theorem about when coordination is and isn’t necessary. I include it here because it’s the most direct theoretical bridge between classical distributed systems work and multi-agent AI, and no one in the LLM literature has formally made the connection yet.

Here’s the CALM claim, translated for multi-agent AI. If your multi-agent system is only ever adding information to shared state (writing to a notebook, appending to a log, producing artifacts), the agents can run without coordinating with each other and still converge to a consistent answer. If the agents ever need to retract or update existing information, then coordination is required to avoid inconsistency.

Why this connects to Ou et al.

Ou et al.'s shared notebook is monotonic. Agents only append new information to it. Nothing gets retracted. That's why it works without heavy coordination machinery. The CALM theorem predicts this result. If Ou's notebook allowed agents to edit each other's entries, they would have needed coordination protocols (locks, version vectors, CRDTs) to keep the notebook consistent. They didn't, because they didn't need it.

Putting It Together

The four papers in this post are doing different things, but they’re converging on the same observation. The question “how should multiple agents work together” has more than one answer, and the answer depends on the structure of the task.

Pattern	What Agents Do	When It Works	When It Doesn't
Convergent debate (Du)	Show each other reasoning, converge	Reasoning tasks with a right answer	Context fills up fast; limited scale
Adversarial debate (Liang)	Argue opposite sides, judge decides	Unstuck models with the DoT problem	Judge bias; doesn't scale beyond 2
Shared notebook (Ou)	Append information to a log everyone reads	Constrained planning, long-horizon tasks	Tasks requiring real-time coordination
Coordination-free (CALM)	Monotonic writes, no coordination	Aggregation, counting, set-building	Anything that retracts or updates

Wave-1 multi-agent papers didn’t distinguish between these. They all called themselves “multi-agent collaboration” and treated the coordination structure as interchangeable. It isn’t. The structure has to match the task.

The distributed systems bridge

This is the point where my PhD work starts to feel directly relevant to the multi-agent AI literature. CALM, CRDTs, version vectors, causal consistency: these are all formalisms for when agents need to coordinate and when they don't. None of them have been rigorously applied to LLM-based multi-agent systems yet. That's an opportunity. It's also a caution. If the field doesn't pick up this vocabulary, it will keep reinventing solutions that distributed systems solved decades ago.

Next post: verification patterns. How do these systems know when they’ve done the right thing? Test execution, dialogue consensus, structural gates, and Cursor’s visual feedback loop, which is the most interesting production-scale verification pattern I’ve seen and isn’t in any of the papers yet.

Getting Up to Speed on Multi-Agent Systems, Part 4: Wave 2 (Why It Breaks)

Mon, 27 Apr 2026 12:00:00 +0000

By 2025, two things had happened. Wave-1 architectures were running in production (Anthropic had shipped its research system; the open-source ecosystem around orchestrator-worker patterns was maturing). The agentic coding turn had made clear that multi-agent was not the right tool for focused coding, and narrowed the interesting MAS question to “when we do use it, why does it break?”

This wave is where I find the literature most useful, because it’s where empirical work finally catches up with the claims of wave 1.

Getting Up to Speed on MAS

Part 1. The Landscape
Part 2. The Vocabulary
Part 3. Wave 1: Can Agents Coordinate At All?
Part 4. Wave 2: Why It Breaks (you are here)
Part 5. Debate, State, and Coordination
Part 6. Verification Patterns
Part 7. Benchmarks and What They Miss
Part 8. Open Questions

MAST: 14 Failure Modes from 1,600 Traces

Why Do Multi-Agent LLM Systems Fail? (MAST) arXiv 2503.13657 · NeurIPS 2025 D&B

1,600 annotated traces across 7 frameworks. First empirical taxonomy of why MAS break.

Core contribution: MAST taxonomy of 14 failure modes in 3 categories

1,600+ traces across MetaGPT, ChatDev, HyperAgent, AppWorld, AG2, Magentic-One, OpenManus
41 to 87 percent failure rates across all frameworks; systemic, not isolated
Top 3 failures: step repetition (15.7 percent), reasoning-action mismatch (13.2 percent), unaware of termination (12.4 percent)
Systems with explicit verifiers (MetaGPT, ChatDev) had fewer failures
Inter-agent failures require "theory of mind"; agents can't model each other's information needs
Adding high-level objective verification gave +15.6 percent improvement
LLM-as-Judge pipeline: 94 percent accuracy against human experts

Cemri, Pan, Yang, Agrawal, Chopra, Tiwari, Keutzer, Parameswaran, Klein, Ramchandran, Zaharia, Gonzalez, Stoica

The MAST paper is the one I keep coming back to. It’s the first rigorous empirical study of multi-agent failures. The authors took 1,600 execution traces from seven popular multi-agent frameworks, annotated each one with human experts, and built a 14-mode failure taxonomy.

The headline number is that every framework they tested had failure rates between 41 and 87 percent. Every single one. These are the production frameworks. These are the systems people cite in their papers. And they fail almost as often as they succeed.

MAST's 14 Failure Modes

FC1 System Design

Disobey task spec (11.8%) Disobey role spec (1.5%) Step repetition (15.7%) Loss of conversation history (2.8%) Unaware of termination (12.4%)

FC2 Inter-Agent

Conversation reset (2.2%) Fail to ask for clarification (6.8%) Task derailment (7.4%) Information withholding (0.85%) Ignored other agent's input (1.9%) Reasoning-action mismatch (13.2%)

FC3 Verification

Premature termination (6.2%) No/incomplete verification (8.2%) Incorrect verification (9.1%)

The taxonomy matters because it lets you diagnose specific failures. When someone says “my agent got stuck in a loop,” you can now ask whether that’s step repetition (FM-1.3) or conversation reset (FM-2.1). Those have different causes and different fixes.

The finding that the paper downplays but I think is most important: the inter-agent failures are the hardest to fix. FC1 issues are prompt engineering problems. You can get meaningful improvements by rewriting role specifications. FC2 issues require what the paper calls “theory of mind,” meaning the agents don’t accurately model each other’s information needs. Prompt fixes don’t help there. The solutions are structural.

MAS-FIRE: Fault Injection for Multi-Agent Systems

MAS-FIRE: Fault Injection and Reliability Evaluation for LLM-Based Multi-Agent Systems arXiv 2602.19843 · February 2026

The first systematic fault injection framework for LLM-based MAS.

Core contribution: Active probing via fault injection, not just passive observation

15 fault types: 8 intra-agent (planning, memory, reasoning, action) + 7 inter-agent (config, instruction, communication)
Three injection mechanisms: prompt modification, response rewriting, message routing manipulation
Tested on MetaGPT, Table-Critic, CAMEL with GPT-5 and DeepSeek-V3
Key finding: config and instruction faults are catastrophic (Robustness Score = 0 percent for Blind Trust on MetaGPT)
Capability paradox: GPT-5's strict compliance hurts under Blind Trust (6.3 percent) vs DeepSeek-V3 (70.6 percent)
Linear pipelines extremely vulnerable; iterative architectures resilient (79-91 percent)
Shared message pools neutralize memory faults (+25 percent advantage)

MAST is observational. You watch real failures and categorize them. MAS-FIRE is the complement: you inject failures on purpose and measure how the system handles them. This is standard practice in distributed systems (Chaos Engineering, Jepsen) but it’s new for LLM agents.

The taxonomy is worth reading carefully.

MAS-FIRE's 15 Fault Types

Intra-agent

Inexecutable Plan Critical Info Loss Memory Loss Context Length Violation Hallucination Tool Selection Error Param Filling Error Param Format Error

Inter-agent

Role Ambiguity Blind Trust Instruction Logic Conflict Instruction Ambiguity Message Cycle Message Storm Broadcast Amplification

Injection via

Prompt Modification Response Rewriting Message Routing Manipulation

The capability paradox is the finding I find most provocative. GPT-5 is a stronger model than DeepSeek-V3 by most benchmarks. But under the “Blind Trust” fault (where one agent is told to unconditionally accept instructions from another), GPT-5 fails almost completely (6.3 percent robustness) while DeepSeek-V3 holds up (70.6 percent). Why? Because GPT-5 is better at following instructions. It’s also better at following bad instructions. Strict compliance is a liability when you can’t trust the source.

The implication for system design is that you want agents that can question upstream inputs. Not agents that just obey.

Silo-Bench: Communication Isn’t Reasoning

Silo-Bench: Evaluating Distributed Coordination in Multi-Agent LLM Systems arXiv 2603.01045 · March 2026

1,620 experiments showing agents can communicate but can't reason about distributed state.

Core finding: The bottleneck is synthesis, not acquisition

30 algorithmic tasks across 3 communication complexity tiers, 54 configs, 1,620 experiments
Central finding: agents form correct coordination topologies and actively exchange information
But they systematically fail to synthesize distributed state into correct answers
Bottleneck is information integration, not information acquisition
Coordination overhead increases with agent scale, eventually eliminating parallelization benefits
Merely increasing agent count cannot circumvent context limitations

Silo-Bench is the paper I wish more people would read. The finding is simple and surprising. When you give multiple LLM agents a problem that requires distributed reasoning, they do all the communication correctly. They build the right topology. They exchange the right information. Then they fail to synthesize what they’ve gathered into a correct answer.

The bottleneck isn’t the network. The bottleneck is the integration. Each agent has received the necessary pieces, and each agent individually fails to combine those pieces into the answer. This is not a coordination problem in the distributed systems sense. It’s a reasoning problem that the coordination can’t compensate for.

For wave-1 architectures, this result is devastating. The whole argument for agents-debating-each-other was that two agents looking at the same problem from different angles could synthesize a better answer than one agent alone. Silo-Bench says: maybe sometimes, but not for information-integration tasks, which is most tasks.

What Wave 3 Adds That Wave 1 and 2 Missed

Wave 3's New Contributions

Observation

Real failure data across multiple frameworks (MAST)

Injection

Active fault testing (MAS-FIRE)

Limits

What coordination can and can't fix (Silo-Bench)

Production lessons

Anthropic blog: 15x token cost, shared-context failures

The wave-2 framing

MAST observes what breaks. MAS-FIRE tests what breaks by injecting it. Silo-Bench identifies the limits of what coordination can fix. Together they provide a reliability research stack that wave 1 didn't have. What's still missing: gates, recovery protocols, longitudinal failure datasets. The field is still working on these.

The critical thing wave 2 does is separate “this is a coordination problem” from “this is a reasoning problem.” Wave 1 assumed everything was a coordination problem. Wave 2 says: if your agents can’t synthesize distributed state individually, no amount of better message passing will save you. Fix the reasoning first. Coordinate second.

Next post: debate, state, and the CALM theorem. Three papers on whether agents should agree, disagree, or just share a notebook. And a theoretical result from distributed systems that explains which choice makes sense when.

Spring Tour Recap: A Month of Shipping on Zabriskie

Sun, 26 Apr 2026 19:00:00 +0000

“this app is the bees. I am very grateful for it.” (a chomper, end of show, Irving, 4/25)

Goose Spring ‘26 just wrapped. The run started March 28 in Athens and closed out Saturday night in Irving. Fourteen shows. One festival lead-in. Fourteen live chats running in parallel with each show, with 40 people sending 3,737 messages to each other across the tour. Most of them sent from couches.

For the second half of tour I had a weird but perfect setup. Goose was on Eastern time. I was couch-touring most of those shows from a hotel room in Vegas. Then Phish was at Sphere (six nights across two runs, 4/16-18 and 4/23-25), and I was at all of them. Pacific time. The two shows overlapped by about an hour every night: the back end of the Goose set ran into the front end of Phish at Sphere. Which meant at any given moment in that hour I had two live chats open and two iOS Live Activities going on the lock screen. One for the show I was actually at, one for the show I was following along with from inside a different venue. The Dynamic Island had to share.

That overlap hour is where the whole pitch of this app clicked for me, so it’s worth pulling out as its own point. Sports fans have lived inside multi-game nights forever. Your team’s game on the TV, the other playoff game on a tablet, score alerts buzzing on your phone, the group chat scrolling beside it, fantasy stats updating in another tab. Nobody thinks twice about it. That kind of parallel, shared, real-time consumption is the default for sports.

Music has never had any of that. A concert has always been one show, in one room, ending when the lights come up, with whatever conversation you happened to have with the person next to you. If two of your favorite bands are playing the same night in different cities, that has historically just been a thing you mourn. There’s no second-screen experience for live music. There’s no group chat thread for the show you’re not at. There’s no “score alert” telling you the band you can’t see just opened with something rare. That whole layer doesn’t exist.

We’ve gotten pushback on this specifically. People have told us, in the chat and in person, that they don’t think we should be encouraging anyone to open Zabriskie at a show. The argument is the same one phones have always heard at live events: be present, put it away, watch the band. I take it seriously. I also remember being on the other side of the same argument seventeen years ago.

Back in 2008 when the iPhone first came out, I worked at a baseball startup in Boston. We built live play-prediction inside the app: in your seat, during the game, what’s the next pitch, did the runner just steal. The reaction we got from people who had never tried it was word-for-word identical to what we get now about Zabriskie. “No one is going to be on their phone at a baseball game.” Today every Major League ballpark has a stadium app open across the section, every wrist has the score on it, and a fan in their car or their living room is part of the same conversation as the fan in section 304. The game didn’t get worse. The community got bigger. People who couldn’t physically be there became part of being there.

Live music gets there too. The phone isn’t the enemy of the show. The phone, used well, is what lets the show have a community around it that outlasts the show.

That is a lot of what we’re building. Anything that broke at the Goose show, Patrick and I would fix between Goose and Phish, and I’d run it live at Phish two hours later. Every feature got tested twice a night, against two different bands, in two different time zones, by a person who was actively living the multi-show pattern the app is supposed to enable.

I started writing this because I wanted to remember what we built during the tour. I looked at the PR list and counted. Three hundred and nine pull requests merged into Zabriskie between the first show and the last. About fifty of those merged today, with the tour wrap-up package shipping in real time as I’m writing this. That number doesn’t feel real. It is real. Most of it shipped to the web immediately so we could test it ourselves the moment it merged, then went out to our TestFlight and Play Store testers within hours, and will be live in the App Store and Play Store this week for everyone. Some of it shipped during the show.

This is what stuck.

The Live Show Got Real

The biggest change is that “couch touring with the app” is now a thing people actually do, not a thing I keep telling people they should try. Here is what the chat looked like in Houston on 4/23, around 11pm, from Patrick, my collaborator on the project:

Whew just got home. Was updating the setlist while walking home from the bar with pip and Zabriskie open lmao

Ohhhhh snapp how bout that new feature I built today fam?!

Got a couple folks in the chomp who witnessed the FTP!

He’s talking about the FTP-witness pill, which he had shipped earlier that day. FTP is “first time played.” Every song in a setlist has a debut show. When a song plays during a live chat, the app now scans everyone watching, and if any of them were RSVP’d to that song’s original debut show, an inline 👀 pill appears under the song name calling them out. It looks like this:

Live Chat · MSG '26 1 witness

🎸 Set 2 begins

🎵 All I Need

👀 1 person in chomp was at the FTP

patrick🎸 Show

omg I forgot I was there for the og

Tap the pill and it pops open the original debut show’s setlist sheet, with a Bookmark button so you can save the show for later. If you yourself were at that debut, the per-song stat sheet gets an extra “You were at this song’s FTP” strip with the date and venue. So Patrick wrote and shipped the FTP-witness feature that day, then came home from a different show and used it himself in the chat for a third show. That is the loop now.

The Live Activity on iOS got a dedicated set break UI, so when the band walks off your Lock Screen tells you instead of just freezing on the last song. It buzzes when the setlist updates, so you don’t have to keep waking your phone to check. Android got a redesign that matches the iOS Live Activity layout, with rich notifications that persist for the entire show instead of falling off the lock screen after a few minutes.

9:41📶 5G ⌁ 78%

🪿

GOOSE · LIVE Saenger Theatre, NOLA

●

🎵 Tumble

Set 2 · song 4

BEFORE · last song frozen on screen

10:23📶 5G ⌁ 71%

🪿

GOOSE · SET BREAK Saenger Theatre, NOLA

⏸

⏸️ Set break

After Set 2 · 9 songs

AFTER · dedicated set break UI

Couch tour viewers get a 30-second spoiler hide on fresh song bubbles, because the people in the building are ahead of the stream.

The bubble doesn’t blur. It renders as three pulsing dots, the same way an iMessage typing indicator does. There’s a reason it’s that specific shape and not a frosted blur or a “?” placeholder.

The moment a song starts in the room, people start talking about it in the chat. You need a marker in the timeline so a couch viewer can see “okay, the in-venue chompers are reacting to whatever this is right now,” follow the conversation in context, and not get hit with the song name as a spoiler before they’ve heard a note of it. The dots are that marker. When the 30 seconds is up, they flip to the song name and the chat above lines up with what the couch viewer is now hearing.

We landed on 30 by testing it live during real shows. A 4K livestream encodes in roughly that window before it reaches a couch viewer, so 30 seconds is close to the actual gap between the room and the screen. We extended this to admin-typed setlist entries too, because the setlist is now mostly admin-typed.

🪿 Goose · Live couch view · stream behind

🎵 Atlas Dogs

patrick🎸 Show

opener slaps already

🎵 Tumble

chomper2🎸 Show

TUMBLE!!! called it 📣

patrick🎸 Show

31 show gap, hot 🔥

unblurs in 17s · in-venue is reacting

patrick🎸 Show

OH NO WAY 🤯🤯🤯

Here’s what happened. We started this tour pulling setlists from a couple of upstream sources that publish their own version of the setlist a few minutes after each song lands. Even when those sources are healthy, an entry usually shows up two to ten minutes after the song actually starts. For a couch viewer, that’s the difference between catching the opening of a jam and finding out twenty minutes later you missed it.

So mid-tour we just stopped waiting. If you’re at the show, type the song into the manage-setlist screen the moment you hear it. Patrick was doing this from the floor most nights, both at his Goose shows and as a couch viewer when I was the one at Phish. The upstream sources still run as a backstop, but the live in-app setlist is now driven by whoever is fastest in the room. Latency dropped from minutes to seconds.

Song Calls

People watching from home love to try to guess the next song from the opening notes. You hear a couple of bars on the stream, you blurt out “MADHUVAN!”, you’re either a hero or you wait six seconds and pretend you didn’t say anything. Until this tour the only place to do that was a group text or whoever happened to be in the room with you.

This was the feature I was most nervous to ship and most happy we did. During a live show, you can now call the next song inside the app. There’s a 📣 chip on the current-song strip; tap it, type the song you think comes next, and the app fuzzy-matches against the band’s catalog (so “atlas” picks up “Atlas Dogs”). Submit, and a pending pill drops below the strip. If the song you called actually plays, you get a green ✓ YOU CALLED IT pill with a confetti burst. Misses fade quietly. There’s a per-show leaderboard so you can see who’s hot tonight, but deliberately no global all-time ranking.

This is one of the load-bearing design principles of the whole app, so it’s worth stating plainly: we are not building Fantasy Music. The minute you ship a season-long leaderboard, the gravitational pull of the product changes. The goal becomes winning. People start optimizing their calls, gaming the window, refreshing for stats, treating the show as input to a meta-game played somewhere outside of it. The community shrinks into a competition. The leaderboard exists, because the celebration of a correct call is part of the fun, but it lives inside the show and ends with the show. Community first, scoreboard second. Every feature in this app gets evaluated against that line:

Now Playing · Tap 📣 to call next

🎵

All I Need Set 2 · song 3

📣 Call

⏳Pending: Empress of Organos

Now Playing

🎵

Empress of Organos Set 2 · song 4

✓YOU CALLED IT 🎉 ✨ ★

📣 1 win this show · tap your badge to see history

Across the tour, 105 song calls went out from 9 different callers. The most-called song was Factory Fiction, called eight separate times by different people across the run. It never landed. The most-correct call was Into the Myst, which hit three times.

Chomp (Live Chat) Grew Up

The live chat is called Chomp because of course it is. It got a lot of love this tour:

@mention autocomplete in the composer, with the dropdown flipping above the input on Android when there’s no room below.
Tap a song pill in the chat to open a stat sheet showing last played, gap, FTP info, and your personal history with the song.
FTP-witness pill so when someone in the chat is seeing a song for the first time, everyone knows.
Heart burst animation when someone favorites your message. This is small. It also matters.
Tap the purple here-now bar to expand the full chomper roster. When more than eight people are watching, the roster scrolls instead of pushing the rest of the page off the screen.
Optimistic bubble insertion so your message appears the instant you hit send, not after the round trip.
@mention notifications that route to chat instead of the generic notification feed.

Plus a long tail of avatar rendering speedups (thumbnails over full-size, async decoding, parallel batch enrichment) that make the chat feel like it’s keeping up with the show instead of catching up to it.

Here is roughly what the chomp looks like now during a live show, with most of those features in one frame: the purple here-now bar at the top (tap to expand the roster), a tappable song pill, an @mention rendered with its purple highlight, and a chat message that’s been hearted.

🪿 Goose · Live Saenger Theatre

12 chomping right now ›

🎵 Atlas Dogs

patrick🎸 Show

this jam is unreal 🔥🔥🔥

❤ 4

chomper1🛋 Couch

@patrick agreed, this is the version we'll talk about later

Say something to the chomp… Send

Tap any of those purple 🎵 song pills and a stat sheet slides up. Last time the band played it, gap since, debut date and venue, and a personal strip showing what you specifically have done with the song. The one card answers the four questions every chomper asks the moment a song starts (“when did they last play this,” “is this rare,” “first time?”, and “have I caught it”):

🎵 Song stats tap-up sheet

Rare

Factory Fiction

Goose · 27 lifetime plays

🎯

You've never caught Factory Fiction live

on your wishlist · 8 fans called it this tour, none landed

Last played

12/13/25

Goosemas · Hampton

Show gap

14 shows

since last play

FTP

Oct 9, 2016

The Hartford, CT

All-time plays

across all tours

👀 0 people in chomp were at this song's FTP. (It was a 2016 small-club show, before most of us found the band.)

Tour Stats Got Serious

Tour Stats started as a personal-only “here are some numbers about your shows” page. It is now a real product surface.

Cross-user view with a public/private toggle, so you can compare your tour to your friends’ tours if they’ve opted in.
Band filter on every drill-down, so you can see your Phish stats separately from your Goose stats separately from your Max Creek stats.
New stat cards: Setlist Staples (most-played songs you haven’t caught yet), FTP count in the overview, Tour Completion with band names on each row, GEOGRAPHIC REACH normalized so the dup-shows-table users don’t double-count states.
Interactive drill-downs that take you straight to the show or song.
Per-band bustout threshold so Phish rotation staples stop misflagging as 🔥 fire bustouts.
Past-show logging flow with tour, festival, and event pills, plus month sub-chips. Logging shows you went to before joining is now a real path, not a chore.

We also surfaced Tour Stats in the compass (More) drawer, so people can find it.

Here’s what most of the screen looks like: band pill at the top, public/private toggle, the OVERVIEW row with FTP count and Tour Completion, then the Setlist Staples card surfacing “common picks you’ve somehow missed” (going RSVPs only, since couch tour doesn’t count against you).

🧭 Tour Stats @cmeik

🪿 Goose › 🌐 Public

Overview

🥚 FTPs

debuts you caught live

Shows

across 18 venues

Tour Completion

Spring '26

9 / 14 nights

📌Setlist Staples

Common picks · still on your list

1Indian River×117

2Butter Rum×113

3Lead the Way×86

4White Lights×68

5Crosseyed & Painless×41

My version of this card looks nothing like the example. I’ve been to 85 Goose shows, so the common picks are all caught and my staples list is now entirely rare bustouts. Crosseyed & Painless is the one that still won’t come.

Today: The Tour Recap Avalanche

Today was insane. Sunday is wrap-up day, the morning after the run closed out, and we shipped the entire end-of-tour package in the span of about eighteen hours. Every show now gets a recap blurb generated from the live chat sentiment, weighted by which songs got the most love. Recaps use Opus when there’s actual chat content and skip cleanly when there isn’t, so we’re not paying tokens to summarize empty rooms. We backfilled recaps for older shows via a CLI, which means every Goose show on the platform now has a top-level recap blurb you can read.

The Flow got a “For you” section that consolidates LIVE NOW, tonight’s shows, and tour recap into one place, with sparkles. Cards for archival recordings collapse into a trending card so the feed doesn’t get spammy when 40 people post the same Relisten link.

How the bracket gets built

The end-of-tour jam tournament is downstream of the per-show recap pipeline, so the seeding isn’t editorial. It’s data.

For every show, we run sentiment analysis over the live chat and weight by song. A song that triggers a flurry of fire emoji and “ARE YOU KIDDING ME”s reads as a heater. A song that gets polite acknowledgment doesn’t. That’s how we identify the jams that actually moved the room.

Then we cross-reference each jam against historical setlist data. A song with a 31-show gap or a sub-10 lifetime play count is automatically a bustout candidate. A regularly-played rotation song needs the chat heat to carry it. The two signals combine into a per-show “jam score.”

When tour wraps, we forward-link those jam scores into a tournament. The top sixteen become the bracket. Highest jam score gets the 1 seed, lowest gets the 16, and we run a March Madness style bracket: Round of 16, Quarters, Semis, Final. Voting opens for each round in sequence. The community decides the winner.

Each matchup card has an inline audio player for each side, since you obviously need to hear both jams to vote between them. No second tab, no link to chase. Two sources are wired in: a soundboard recording if you have a paid streaming subscription, or the taper recording for free if you don’t. You always get audio for both. The card looks like this:

🏆 Goose Spring '26 Bracket Round of 16 · 4 of 8 voted

Vote in this matchup

SEED 2

Tumble

4/22 · Saenger, NOLA

SEED 7

Hungersite

3/28 · Athens

▶

🎵 Tumble · 4/22 NOLA · soundboard

3:14

17:42

▶

🎵 Hungersite · 3/28 Athens · taper

0:00

12:08

3 voted for Tumble · 1 for Hungersite

Full bracket

Round of 16

(1) Madhuvan ✓

vs Pancakes

(2) Tumble · 4/22

vs (7) Hungersite — voting

(3) Atlas Dogs ✓

vs (6) Empress

(4) Into the Myst ✓

vs Doobie

(5) All I Need

vs (12) Borne — voting

(6) Arrow ✓

vs Travelers

(7) Echo of a Rose

vs Yeti — voting

(8) Indian River ✓

vs Creatures

Quarters

Madhuvan

vs winner

Atlas Dogs

vs Into the Myst

TBD

vs Arrow

TBD

vs Indian River

Semis

TBD

vs TBD

TBD

vs TBD

Final

🏆

Jam of the Tour

The community is voting now.

The reason the bracket matters more than it might first appear is that the gap between tours is where most music apps die. The tour ends, the chat empties out, the lock-screen Live Activity goes dark, and people drift back to their normal feeds until the next run is announced. The jam tournament is a deliberate counter to that. Voting runs across multiple weeks, the rounds release on a schedule, and every matchup card pulls a real audio clip from a real show people went to, so listening to the bracket is also re-listening to the tour. The conversation in the chat doesn’t end when the lights come up in Irving. It keeps going through Round of 16, Quarters, Semis, Final, and by the time we crown a Jam of the Tour, the next run is already on the calendar and the muscle memory of opening the app every day is intact. The bracket is the bridge.

The “all bands” UX got a final shape today too: the band dropdown collapsed into a BANDS card on Tour Stats, the All Bands directory got tuned-in count parity with band pages, Dead & Company switched to a “📜 Setlist archive” tour-status because they’re done touring, Billy Strings got a full historical import and a live setlist source, and a bunch of polish on Tour Completion, soundcheck-row dedup, encore-vs-Set-3 labeling, miracle ticket cards, and the rest of the long tail.

The honest version of “today” is that we picked Sunday for the avalanche because the tour had just wrapped, the chat was still active, and any new bug would surface immediately. It worked. We shipped a thing, watched the chat react, and either fixed it or moved on inside an hour. Cycle repeated forty-ish times.

So Many Bands

This was the tour where Zabriskie stopped being a Goose-and-Phish app.

We added: Trey Anastasio Band 🎺, Mike Gordon Band 🌵, Umphrey’s McGee 🧢, Dead & Company 🌹, Billy Strings 🪕, Max Creek 🐥, King Gizzard & the Lizard Wizard 🧙, Radiohead 🐻, My Morning Jacket, Spafford, Dogs in a Pile, and Daniel Donato’s Cosmic Country. Phish got rebranded to ⭕ and JRAD to ⚡ along the way.

For each band we did the full thing: found a source for the historical setlists and backfilled them, wired up the band page with My Recent and Recent Shows with inline-expand setlists, and added it to the All Bands directory so people could actually find it. Dead & Company got the archival treatment because the band is done touring, and “Tune in” doesn’t make sense for a band you can’t tune into.

We also built a “Request a band” escape hatch in onboarding for everyone whose band still isn’t here.

Onboarding Stopped Being a Wall

Onboarding got rebuilt around a simple idea: get to value in the first session. New users now see a first-run, show-aware prompt on /start that adapts to what’s happening that night. There’s a “Log past shows” step with a dedicated band picker, a tune-in tap with an explainer for what tuning in actually does, and a value-prop card that explains what the app is in one screen:

Step 2 of 4

Which bands do you follow?

Tune in to get setlists, live chat, and recap on every show.

🪿

Goose

14 shows this tour · 12 chomping tonight

✓ Tuned in

⭕

Phish

At Sphere tonight · couch tour live

+ Tune in

🌹

Dead & Company

📜 Setlist archive · no upcoming shows

+ Tune in

🪕

Billy Strings

On tour · next show 4/29

+ Tune in

Don't see your band? Tap below to request one and we'll get the full show history wired up.

Request a band →

Skip

Continue →

We also widened the gate for who can sign up, because the Spring tour brought a lot of new people in.

A big part of why onboarding actually got better instead of just getting a redesign is that Patrick was on the ground at every Goose show, putting the app in front of people he met at the bar, on the lot, in the seats next to him. He’d watch a brand-new user open the app cold, see exactly where they got stuck or confused, take notes on the friction in real time, and then file the fixes from the passenger seat on the drive to the next city. The “Log past shows” step, the explainer on what tuning in actually does, the value-prop card, the “Request a band” escape hatch, all of those came out of that loop. Not speculative redesigns. Each one came directly from watching a real person fail at the previous version, with the new version landing before the next show.

Goose Mode

We started building dedicated per-band “modes” this tour, basically a tour companion dashboard tailored to one band at a time. Both Goose Mode and Phish Mode shipped during the run. Each one knows its band’s calendar, color palette, and ritual vocabulary, and surfaces what matters for that specific community.

The centerpiece is the Tour Timeline with a live countdown to the next show. Past shows you went to are checked off. Upcoming shows show date, weather, and which of your friends are going. The whole timeline is annotated with your crew dripping in and out of the run, so you can see at a glance who joined for which leg, who left after the southeast swing, who flew in for the closer. The countdown ticks every second:

🪿

Goose Mode

Spring '26 · Texas Run

Tonight · doors in

HRS

MIN

SEC

Bayou Music Center · Houston · 5 in your crew going

Tour Timeline · Crew

✓4/19

St. Augustine Amphitheatre

St. Augustine · couch toured

2 in crew · Patrick joined the run here

✓4/21

Saenger Theatre

New Orleans · couch toured

4 in crew · Mwat & C joined

✓4/22

Saenger Theatre

New Orleans · couch toured · Tumble jam ranked #2

5 in crew · Gmart joined for Texas leg

●4/23

Bayou Music Center · tonight

Houston · 79°F clear · you're couch touring

4 in crew · C left after NOLA · Lubby joined

RSVP'd

○4/24

Moody Center

Austin · in 1 day

3 in crew going · B sitting it out

○4/25

Toyota Music Factory

Irving · in 2 days · last night of Spring run

5 in crew · Q flew in for the closer

Miracle Tickets

A long jam band tradition: someone has an extra ticket and gives it away, no questions, sometimes for nothing, sometimes for a smile. We built that into the app as a first-class feature this tour. If you have an extra, you post it as a Miracle Ticket attached to the show. People can request it. The owner draws a winner based on engagement (so the lurker who never participates doesn’t beat the regular who’s been in the chat all run). Any admin can also award on the owner’s behalf if they’re not online. Cards are collapsible and grouped by show, so on a busy night the feed doesn’t drown:

☝️ 🎫 Miracle Tickets 2 open

🪿

Goose · 4/22

Saenger Theatre · New Orleans

OPEN

"Have an extra GA, can meet at the box office at 7. Just want it to go to someone who'll love it. 🌹"

posted by patrick · 2h ago · 7 entries

View entries

🙏 Enter to win

⭕

Phish · 4/23 · Sphere

Las Vegas · awarded

CLOSED

🎉 went to @chomper4 · drawn by engagement

Everything Else Worth Mentioning

Seat sharing. A discovery screen, show-card badges, avatar-x to unshare, per-show seat sharing with @mentions in the crew chat, and seat-share notifications that show the seat inline instead of dumping into live chat.
Live Recording posts with a smart provider cascade and an autocomplete dropdown for the artist field.
Show notes on every expanded setlist, not just Chomp. Plus a Show Notes editor on manage-setlist for admins.
The Live tab. Promoted out of the compass into its own dedicated nav slot, with an always-open compass replacing it.

What the Chat Said at the End

There was a moment in Irving last night, near the encore, that I want to keep. I’m pulling these straight from the chat, anonymized:

(Patrick) Man. end of tour is always so bittersweet. Great run of shows though. Damn, Goose, ya got me good

(a chomper) this app is the bees. I am very grateful for it.

(another chomper) Thank you for the invite. It’s a super cool concept and appreciate all your efforts!

(another) Awesome show, thanks for the hangs. We will hopefully see some of you in Toronto!

That is the whole point. That is the third place. People who started the tour as usernames in a beta group, sending each other Eminence-chase updates and bustout calls and gummy timing notes, ending the tour planning to meet in Toronto and trading European tour stops. One of them listed his run: Brixton, Brussels, Amsterdam, Paris. Patrick offered to add him to a crew. Real people, real plans, made through an app we started building eight weeks ago.

What This Tour Taught Me

The app holds up under live load now. Since I wrote about the iOS 26 .then() proxy disaster, Live Activities have stayed up through every single show. Nobody filed a “the lock screen is dead” bug for the rest of the tour. That’s the most boring victory of the month, and it’s the one I’m proudest of.
Two shows a night is a cheat code. Goose on Eastern from the hotel couch, Phish at Sphere on Pacific from my seat. Patrick on the ground at Goose, me at Phish, both of us in the chomp on whichever show wasn’t ours. Anything that broke at 9pm Eastern got fixed before the next song started at Phish. You cannot manufacture that kind of feedback loop on purpose. We got lucky with the calendar and we used it.
The community runs faster than the app. Song calls is the clearest example I can point to: on 4/19 a chomper said in chat, “Would be cool if you could program it to where when one of us guesses the song in shows everyone fun Lil game. Idk how hard that would be to do tho hahs.” Five days later that feature shipped end to end. The whole tour was full of small versions of this loop, where someone says “we need X” in the chat and a few days later a version of X is in the app. Building this with the people who use it is dramatically faster than building it from a spec.
Three hundred PRs in a month is not normal and I should not pretend it is. Most of the code was written by Claude. A lot of it had to be re-written by Claude after I caught it doing something wrong. Every sharp edge from the tour got logged into the agent reliability dataset I’ve been building, and I’ll keep writing about it separately. The point of building in public with an AI assistant is that the failures are part of the dataset, not embarrassments to hide.

Goose Cabo is in a few weeks. Goose Summer is right after. I have a list. The app is not done.

In the meantime, if you were on tour, thanks for chomping. If you were at home, thanks for being in the chat. If you’ve never tried the app and the third place I keep talking about sounds like something you’d want, come find us. The next show is already on the calendar.

One housekeeping note: the screens in this post are mockups, not actual app screenshots, drawn to make each feature legible in context. The real app looks slightly different on iOS vs. Android (Live Activities vs. ongoing notifications, system fonts, badge styling, the typing indicator’s exact pulse), and the mockups smooth those over for readability. The features themselves all shipped.

Getting Up to Speed on Multi-Agent Systems, Part 3: Wave 1 (Can Agents Coordinate At All?)

Sun, 26 Apr 2026 12:00:00 +0000

Wave 1 is the cluster of papers from 2023 that people actually cite. When someone says “I read the multi-agent papers,” they usually mean these. In this post I’m going to walk through the canonical five, explain what each one actually builds, and show where they agree and where they quietly disagree with each other.

Getting Up to Speed on MAS

Part 1. The Landscape
Part 2. The Vocabulary
Part 3. Wave 1: Can Agents Coordinate At All? (you are here)
Part 4. Wave 2: Why It Breaks
Part 5. Debate, State, and Coordination
Part 6. Verification Patterns
Part 7. Benchmarks and What They Miss
Part 8. Open Questions

CAMEL: Two Agents Role-Playing

CAMEL: Communicative Agents for "Mind" Exploration arXiv 2303.17760 · NeurIPS 2023

Two LLMs role-play until the task is done.

Core bet: Prompt constraints keep agents on task

2 agents role-play inception prompting

AI User (instructor) and AI Assistant in a structured dialogue loop
Inception prompting: symmetric system prompts with explicit constraints like "Never flip roles"
Task specifier agent elaborates vague human input into concrete tasks
Documented failure modes: role flipping, instruction repetition, vague responses, conversational loops
All mitigations are prompt-level, no structural enforcement

CAMEL is the simplest of the wave-1 papers and also the most honest. It’s two LLMs. One plays a user, one plays an assistant. They talk. The paper’s main contribution is inception prompting, which is a way of writing system prompts that keep agents from breaking character. The failure modes the paper documents are the failure modes you’d expect: agents flip roles, agents repeat themselves, agents give vague answers.

What’s missing from CAMEL is any structural enforcement. If the agent flips roles, nothing stops it except a prompt instruction that says “don’t flip roles.” There’s no protocol-level guarantee. This is a pattern you’ll see repeated across wave-1: trust the prompt, hope for the best.

Generative Agents: Memory, Reflection, and Planning

Generative Agents: Interactive Simulacra of Human Behavior arXiv 2304.03442 · UIST 2023

Give agents memory, reflection, and planning so they behave believably over time.

Core bet: Retrieval scoring produces believable behavior

25 agents memory stream reflection planning

Memory stream: every observation stored with timestamp and importance score (LLM-rated 1 to 10)
Retrieval: weighted sum of recency (exponential decay), relevance (cosine similarity), and importance
Reflection: triggered when accumulated importance crosses a threshold (roughly 2-3 times per day)
Planning: top-down recursive (day, hour, 5-15 minute blocks); replans on unexpected events
Emergent behaviors: a Valentine's Day party self-organized from one suggestion; info diffusion from 4 percent to 32 percent awareness in two game days

This paper is the outlier in wave-1, because it isn’t trying to build software. It’s a social simulation. 25 agents living in a Sims-style town. What makes it interesting for multi-agent systems is that it’s the only wave-1 paper that takes memory seriously. Every observation gets stored with a timestamp and an importance score. When an agent needs to act, it retrieves memories using a weighted combination of recency, relevance, and importance. Reflections are higher-level thoughts synthesized from clusters of observations.

None of the software engineering papers in wave-1 do anything like this. They don’t need to, because their tasks have clear start and end conditions. But when you look at what production multi-agent systems are starting to need, the Generative Agents architecture has more of the right pieces than MetaGPT does.

ChatDev: Pairwise Chat as a Software Pipeline

ChatDev: Communicative Agents for Software Development arXiv 2307.07924

Chain pairwise dialogues into a software development pipeline.

Core bet: Dialogue convergence equals correct output

pairwise chat phase pipeline dehallucination

Fixed pipeline: Design, then Coding, then Testing; each phase is an instructor-assistant dialogue
Communicative dehallucination: the assistant flips role and asks clarifying questions before committing to an answer
Short-term memory (full dialogue within a phase) and long-term memory (extracted solutions across phases)
Termination: 10 rounds max, or two consecutive rounds without changes
No escalation path when convergence fails, it just stops

ChatDev is the paper that got me into this literature in the first place. It’s a pipeline of pairwise dialogues. Pairs of agents talk about design, then pairs of agents talk about coding, then pairs of agents talk about testing. The most interesting mechanism is communicative dehallucination, which is a prompt pattern where the assistant asks clarifying questions before answering. This is the closest any wave-1 paper gets to backpressure.

The structural problem with ChatDev is that when agents can’t converge in 10 rounds, the system just stops. There’s no fallback. No mechanism for the system to notice that it’s stuck and escalate. No concurrency control on the shared artifacts. I wrote about how this breaks down in practice a few weeks ago.

MetaGPT: Structured Artifacts and Test Execution

MetaGPT: Meta Programming for Multi-Agent Collaborative Framework arXiv 2308.00352 · ICLR 2024 Oral

Replace dialogue with structured documents and real test execution.

Core bet: Schemas plus passing tests equal correct output

5 roles artifact pub-sub executable feedback

Roles: PM, Architect, Project Manager, Engineer, QA (waterfall)
No dialogue; agents produce structured documents (PRD, system design, task list, code, tests)
Pub-sub message pool: agents publish artifacts, subscribe by role to relevant messages
Executable feedback: unit tests actually run; failures trigger up to 3 retries referencing PRD and design docs
Results: 85.9 percent Pass@1 on HumanEval, 100 percent task completion, 0.83 human revisions vs ChatDev's 2.5

MetaGPT is the most ambitious wave-1 paper. Instead of dialogue, the agents produce structured documents. Instead of relying on agents to agree, the framework executes tests. The paper’s strongest claim is the structural one: if agents produce artifacts with defined schemas, and those artifacts are validated by execution, you get better coordination than you get from unconstrained dialogue.

I think that claim holds up. But the coordination model is still based on a shared mutable pool, which is the same thing Riak solved twenty years ago with version vectors. MetaGPT doesn’t have that. Agents publish to the pool. Agents subscribe. Nobody tracks causality.

AutoGen: A Framework, Not a System

AutoGen: Next-Gen LLM Applications via Multi-Agent Conversation arXiv 2308.08155

A configurable framework. Build whatever multi-agent system you want.

Core bet: Developers will build the right topology

framework ConversableAgent GroupChat human-in-loop

ConversableAgent base class: any entity that sends and receives messages (LLM, human, tool, code executor)
Pluggable reply functions via register_reply(); agent behavior is what it does when it gets a message
GroupChatManager: selects next speaker via LLM role-play prompting or an FSM
Human-in-the-loop as a dial: per-agent config of ALWAYS, SOMETIMES, or NEVER
Number one on GAIA at time of publication, roughly 2x performance on the hardest level

AutoGen is the odd paper in wave-1 because it’s not a system, it’s a framework. The contribution is that every agent, whether it’s an LLM, a human, a tool, or a code executor, speaks the same message protocol. You compose them however you want. The GroupChatManager can pick the next speaker via an LLM or via a finite state machine you define.

AutoGen is more honest than the others about the fact that there’s no “right” multi-agent architecture. It doesn’t try to tell you what your agents should be. It gives you the plumbing and assumes you know what you’re doing. For that reason it’s probably aged better than CAMEL or ChatDev.

What Wave 1 Got Right

Every one of these papers took LLMs out of single-user chat and put them into multi-step coordination tasks. That’s a real contribution. Role specialization, structured dialogue, tool use patterns, task decomposition, memory and reflection as first-class primitives. These ideas came out of wave-1 and the field is still using them.

What Wave 1 Got Wrong

Shared Assumptions That Didn't Survive

Failure model

Treated as termination, not a system state

Concurrency control

Shared state with no causality tracking

Evaluation

Benchmarks designed for single agents

Escalation

No path when convergence fails

Topology

Fixed at design time

Every wave-1 paper treats failure as a termination condition. When ChatDev can’t converge, it stops. When MetaGPT’s tests fail three times, it stops. When AutoGen hits max_round, it stops. None of these systems have a model for what happens next. This is the gap wave-2 papers would later start trying to fill.

None of the wave-1 papers have concurrency control on their shared state. MetaGPT’s message pool grows monotonically and nobody tracks causality. ChatDev discards dialogue at phase boundaries. Generative Agents’ memory is per-agent with no sharing. If you had two agents in MetaGPT trying to edit the same file, nothing in the framework would stop them from overwriting each other’s work.

And all of them evaluate against benchmarks that were designed for single agents. HumanEval, MBPP, SWE-bench. These benchmarks measure whether the output is correct. They don’t measure coordination quality, communication overhead, or recovery behavior. Which are the things that distinguish a multi-agent system from a single agent.

Next post: wave-2 papers, which measure what actually breaks in these systems. With wave-1 architectures running in production, and the agentic coding turn having clarified when MAS isn’t the right tool at all, the field started asking why MAS fails when you do use it and how to test that honestly.

Getting Up to Speed on Multi-Agent Systems, Part 2: The Vocabulary

Sat, 25 Apr 2026 12:00:00 +0000

If you try to read multi-agent systems papers without the vocabulary, you will get nowhere. The field has settled on a shared set of words for the pieces of a system, and every paper now slots into those categories even when it pretends to be doing something novel. This post is about those words. Once you know them, you can read any paper in the field and know what it is and isn’t claiming.

Getting Up to Speed on MAS

Part 1. The Landscape
Part 2. The Vocabulary (you are here)
Part 3. Wave 1: Can Agents Coordinate At All?
Part 4. Wave 2: Why It Breaks
Part 5. Debate, State, and Coordination
Part 6. Verification Patterns
Part 7. Benchmarks and What They Miss
Part 8. Open Questions

Three surveys have done the work of consolidating the vocabulary. Each one cuts the space slightly differently, but together they give you the conceptual toolkit.

Tran et al.: Actors, Types, Structures, Strategies

The most useful single survey is Tran et al. (2025). It defines a multi-agent system formally as a tuple of agents, collaboration channels, collective goals, and an environment. Then it taxonomizes the space along four axes.

Tran's Four Axes

Types

Cooperation (aligned goals) Competition (conflicting goals) Coopetition (mixed)

Structures

Centralized (hub) Decentralized (P2P) Hierarchical (layered)

Strategies

Rule-based (voting, consensus) Role-based (SOP, expertise) Model-based (Theory of Mind)

Architecture

Static (pre-defined) Dynamic (runtime adjustment)

Most of the famous wave-1 papers are in one box: cooperative, hierarchical, role-based, static. Everyone is doing roughly the same thing, with small variations in how agents pass messages and what they produce at each step. The survey’s most useful claim is that the optimal structure varies with the task. There is no universal topology.

Zhou et al.: The Five-Component Agent

Zhou et al. (2024) takes a different cut. Instead of asking how agents coordinate, they ask what each agent actually has inside it. They propose a five-component model that applies to any LLM-based agent.

Zhou's Five Components

01 Profile

How the agent is created with role and expertise

02 Perception

How the agent observes its environment

03 Self-Action

Memory, reasoning, and planning

04 Mutual Interaction

Communication paradigm, structure, content

05 Evolution

Self-reflection, progressive enhancement

Reading this as a distributed systems person, the labels sound like things you’d recognize from any actor system. Profile is identity. Perception is input. Self-Action is local state plus computation. Mutual Interaction is message passing. Evolution is the weakest piece, because nobody has really figured out what “agent learning from its own history” looks like in production.

Chen et al.: Applications and Unsolved Challenges

The third survey, Chen et al. (2024), is the one I’d skim rather than read in full. The applications chapter is useful, but what you actually want is the challenges section.

Chen's Challenge Levels

Agent-level

Alignment for simulation Hallucination propagation Long-context limits

Interaction-level

Efficiency explosion Accumulative error

Evaluation-level

No standardized benchmarks No objective metrics No individual vs aggregate frameworks

The interaction-level challenges are the ones that most concern me. Efficiency explosion is the observation that multi-agent systems scale worse than linearly because each agent’s autoregressive generation multiplies the token cost. Accumulative error is what it sounds like: errors made in round one propagate and amplify in rounds two, three, four.

Mapping Papers Into These Taxonomies

The payoff of the vocabulary is that you can now categorize any paper in the field at a glance.

System	Type	Structure	Strategy	Architecture
CAMEL	Cooperation	Decentralized pair	Role-based	Static
ChatDev	Cooperation	Hierarchical pipeline	Role-based	Static
MetaGPT	Cooperation	Centralized pool	Role + Rule-based	Static
Debate (Du)	Competition	Decentralized all-to-all	Rule-based rounds	Static
Generative Agents	Coopetition	Decentralized open env	Model-based retrieval	Dynamic
Anthropic Research	Cooperation	Centralized orchestrator	Role-based	Dynamic
AutoGen	Configurable	Configurable	Configurable	Static or Dynamic

Most of the canonical papers sit in the cooperative, role-based, static quadrant. The interesting ones are the exceptions. Du et al. is the rare competitive debate paper. Generative Agents is the rare fully dynamic system. AutoGen tries to be everything at once, which is its whole thesis.

Why vocabulary matters

When two papers claim they "disagree," the vocabulary lets you ask: are they actually addressing the same problem? ChatDev and MetaGPT both call themselves "multi-agent software engineering frameworks" but they have different structures, different strategies, and different failure modes. You need the words to see that they are solving slightly different versions of the same problem.

The Gap the Vocabulary Exposes

The taxonomies do something else besides categorize papers. They make gaps visible.

Zhou’s “Evolution” component is the weakest across every system. Nobody has a real story for how agents learn from their own history in production. MetaGPT’s “test-driven retry” is the closest wave-1 paper to Evolution, and it’s still just a bounded retry loop with no memory of past attempts.

Tran’s “dynamic architecture” category is almost empty. The wave-1 papers all fix their topology at design time. AutoGen makes topology configurable, but it’s configured by the developer, not adjusted at runtime. The only system that truly adjusts at runtime is Generative Agents, and that’s a simulation, not a production framework.

Chen’s “evaluation-level” challenges are unsolved in a way that’s embarrassing for the field. When ChatDev claims 88 percent executability and MetaGPT claims 41 percent on a comparable benchmark, you’re not looking at a performance difference. You’re looking at two papers measuring different things with different tools and calling them the same.

Next post: the wave-1 theory papers in detail. CAMEL, Generative Agents, ChatDev, MetaGPT, AutoGen. What each one actually builds, what each one trusts, and where each one breaks.

Getting Up to Speed on Multi-Agent Systems, Part 1: The Landscape

Fri, 24 Apr 2026 06:00:00 +0000

I’ve been reading multi-agent systems papers for weeks trying to figure out where the field actually is, and the honest answer is that it moves fast enough that any single paper is a snapshot, not a map. So this is the map I wish I’d had when I started. It’s a short series of posts meant to get someone up to speed on multi-agent LLM systems without having to read thirty papers first.

Who this is for: you already ship or evaluate LLM agents (tools, long context, basic eval loops) and want the research landscape in view. This is not an on-ramp to transformers or prompting fundamentals.

Before I start, a note on the frame I’m going to use. I’m going to talk about two “waves” of multi-agent research, and one outside disruption that happened between them. I want to be upfront that the waves are a reader aid, not a historical claim. Nobody in 2023 was writing “wave 1” papers, and the field did not convene to name its generation. These aren’t named movements like French New Wave cinema or second-wave feminism, where participants self-consciously defined their work against a prior cohort. What I’m doing is retrospective grouping: the kind you use to keep thirty papers straight in your head.

What the grouping captures is that certain clusters of papers share assumptions, benchmarks, and failure modes. What it misses is that parallel threads exist (debate, simulation, distributed-systems-adjacent work) that don’t fit the wave structure at all, and that plenty of individual papers sit awkwardly between waves. If the framing helps you navigate the literature, keep it. If it gets in the way, drop it. The papers are what matter; the waves are scaffolding.

With that caveat in mind: two rough clusters, one outside disruption that reshaped both, and two rough questions the MAS field has been trying to answer.

Getting Up to Speed on MAS

Part 1. The Landscape (you are here)
Part 2. The Vocabulary
Part 3. Wave 1: Can Agents Coordinate At All?
Part 4. Wave 2: Why It Breaks
Part 5. Debate, State, and Coordination
Part 6. Verification Patterns
Part 7. Benchmarks and What They Miss
Part 8. Open Questions

Wave 1: Can Multiple LLMs Coordinate At All? (2023)

The first wave is the one most people have heard of. A cluster of papers came out in roughly a six-month window in 2023, all answering some version of the same question: if you put multiple LLMs together, can they do something one LLM cannot?

Not every paper in that cluster is “coordination theory” in the same sense: some are explicit software pipelines (ChatDev, MetaGPT), others foreground simulation and believable social dynamics (Generative Agents). I group them anyway because they share 2023-era benchmarks and a similar loose trust that multi-agent structure will carry the task.

Wave 1 · Theory and Architecture

Can multiple LLMs coordinate at all? What's the right shape?

Mar 2023

CAMEL

Two agents role-play

Apr 2023

Gen. Agents

Memory and reflection

May 2023

Debate (Du)

Competition as coordination

Jul 2023

ChatDev

Pairwise chat pipeline

Aug 2023

MetaGPT

Artifacts and test execution

Aug 2023

AutoGen

Configurable framework

Aug 2023

AgentVerse

Dynamic group composition

These papers are all proofs of concept. They show that multi-agent coordination is viable for some task, demonstrate the idea works on a benchmark, and argue that their particular coordination structure beats simpler baselines. CAMEL uses two agents role-playing. Generative Agents uses memory streams and reflection in a social simulation. Du et al. uses multi-round debate between identical model instances. ChatDev chains pairwise dialogues into a software development pipeline. MetaGPT replaces dialogue with structured artifacts and real test execution. AutoGen is a framework for building whatever topology you want. AgentVerse emphasizes dynamic group composition: the system can recruit specialists and reshape the team as the task proceeds.

The wave-1 papers share three assumptions that look very different in hindsight. First, they assume the benchmarks they run on are the task. Second, they treat failure as a termination condition: when the system stops converging, it just stops. Third, they trust agents to coordinate without formal concurrency control, shared memory protocols, or recovery paths. These assumptions are where wave 2 would later start to push back.

What Happened Next Door (2024)

Before I get to wave 2, I have to account for a thing that happened in parallel that isn’t really MAS research but reshaped what MAS has to answer for.

Through 2024, a wave of agentic coding systems shipped: Devin, SWE-agent, OpenHands, AutoDev, and Microsoft’s Magentic-One. These are not multi-agent systems in the wave-1 sense. Most of them are single agents with well-designed tool interfaces. The SWE-agent paper in particular showed that interface quality matters more than adding agents. They got a 10.7 percentage point improvement on SWE-bench from interface design alone, without changing the model.

I bring this up because you cannot read the MAS papers from 2025 onward without this context. Wave 1 had implicitly assumed that multi-agent coordination was the default way to solve complex agentic tasks. By late 2024, the agentic coding community had accumulated strong evidence against that default for at least one large, benchmarked slice of the space: autonomous patch-style software engineering (the kind of workflow SWE-bench and its successors measure). That is narrower than “all coding forever,” but it was the center of gravity in 2024 discourse, and it shifted what MAS papers need to beat. Anthropic’s research system post from June 2025 states the conclusion plainly: multi-agent earns its overhead on “breadth-first queries with independent parallel subtasks” and underperforms on “tasks needing shared context, including most coding tasks.”

Why this matters for MAS readers

The agentic coding papers aren't MAS research. But they narrowed the MAS claim. After 2024, "multi-agent for coding" became harder to defend without evidence, and the MAS field's next wave is partly a response to that. If you're reading a post-2024 MAS paper, it's almost certainly arguing implicitly against the single-agent-with-tools baseline that Devin and SWE-agent established. I won't spend a whole post on these systems because they're not MAS, but they belong on your mental map of the landscape.

Magentic-One is the interesting exception. It’s a real multi-agent system with an orchestrator coordinating four specialized workers. It earns its overhead on hard multi-step reasoning (38 percent on GAIA) but not on focused coding. The stuck-counter mechanism it introduces (if an agent loops more than twice, reflect and replan) is one of the few MAS design patterns to surface clearly in the shipped agentic-coding systems of this period. I’ll come back to it in later posts.

Wave 2: Why Does It Break? (2025 and Beyond)

The second MAS wave is where the field is now. With wave-1 systems running in production and the agentic coding turn having clarified when MAS is and isn’t the right tool, people started asking: when MAS does fail, why? And how do we even test that?

The Wave 2 timeline below is illustrative: a handful of papers that typify a shift toward measurement, taxonomies, and fault injection, not an attempt to enumerate every 2025–2026 contribution.

Wave 2 · Why Does It Break?

MAS works sometimes. Now: why does it fail? How do you test reliability?

Mar 2025

MAST (Cemri)

14 failure modes, 1,600 traces

Jun 2025

Anthropic Research

Production orchestrator-worker

Aug 2025

Info Sharing in Planning

Shared notebook on travel planning

Feb 2026

MAS-FIRE

Systematic fault injection for MAS

Mar 2026

Silo-Bench

Communication-reasoning gap

The MAST paper from Cemri and collaborators is the one I keep coming back to. They annotated 1,600 traces across seven popular multi-agent frameworks and built a taxonomy of 14 failure modes. Every framework they tested had failure rates between 41 and 87 percent. The top three failures are step repetition, reasoning-action mismatch, and being unaware of termination conditions. These are not model capability problems. They are system design problems.

MAS-FIRE goes the other direction. Instead of observing failures in the wild, they inject them on purpose. Fifteen fault types across intra-agent and inter-agent categories, three injection mechanisms, and a dual-level reliability metric. The most interesting result is what they call the capability paradox: GPT-5’s strict instruction compliance becomes a liability under “Blind Trust” faults, where DeepSeek-V3’s less compliant behavior holds up better.

Silo-Bench adds the third leg. 1,620 experiments showing that agents successfully form coordination topologies and actively exchange information, yet systematically fail to synthesize distributed state into correct answers. The bottleneck is not communication. The bottleneck is reasoning over distributed state.

What Each Wave Trusts

If you want a one-sentence read on each wave, this is it.

Wave 1

Trusts that role structure and dialogue are enough

Outside

Agentic coding: trusts that good tools beat agent count

Wave 2

Trusts nothing and measures what breaks

Read as a progression, the rough arc is: tell agents to coordinate, then notice that sometimes you don’t need them to, then measure what happens when you do. Each step trusts the agents less and verifies more than the one before.

What the Waves Don’t Capture

The two-wave framing is tidy, which should make you suspicious. Some things it flattens or misses:

Parallel threads run outside the waves entirely. Du et al. appears on the Wave 1 timeline because it landed in the same window and poses the same headline question (many LLMs versus one), but the debate line of work is its own cluster: multiple instances of the same model arguing, with a citation graph that barely touches coordination-theory papers and rarely gets cited back by them. The timeline entry is chronology, not a claim that debate belongs in ChatDev’s intellectual neighborhood; Part 5 treats debate as that parallel thread. Generative Agents is a social simulation paper that sits uncomfortably in wave 1 because it came out in 2023, but its descendants (game agents, persona simulations) are their own research community. Distributed-systems-adjacent work on shared state and coordination avoidance is a separate thread that’s only now starting to touch the MAS literature.

Individual papers don’t fit cleanly. AutoGen is a 2023 paper that kept evolving through 2024. The Anthropic research post is engineering content, not a research paper. Magentic-One sits between the agentic coding turn and the reliability wave. Calling any of these “wave 1” or “wave 2” is a judgment call, not a fact.

Keep all of that in mind as you read the rest of the series. The waves are a way to group papers for comprehension, not a claim about how the field evolved.

What’s Coming

The next seven posts build on this landscape.

Part 2 covers the vocabulary the field uses for itself. Three surveys have done the work of consolidating the shared terms, and once you know the vocabulary you can read any paper in the field at a glance.

Parts 3 and 4 go deep on the two waves. Part 3 is the canonical coordination-theory papers: CAMEL, ChatDev, MetaGPT, AutoGen, AgentVerse, what each one actually builds and where each one quietly disagrees with the others. Part 4 is the reliability wave: MAST, MAS-FIRE, Silo-Bench, and what happens when you try to measure a multi-agent system honestly.

Part 5 covers the parallel threads. Multi-agent debate (Du, Liang), shared state as coordination (Ou et al.), and the CALM theorem as a bridge between distributed systems and multi-agent AI.

Parts 6 and 7 are cross-cutting. Part 6 is verification patterns, including Cursor’s visual feedback loop, which is the most interesting production-scale verification pattern I’ve seen and isn’t in any of the papers. Part 7 is benchmarks, what they measure, what they miss, and why ChatDev and MetaGPT can report contradictory results on each other without either being obviously wrong.

Part 8 is what I think is still missing, what’s worth stealing from adjacent fields, and what I’d read if I had to start over.

Next post: the vocabulary.

Errata and revisions

First published 2026-04-24. The list below logs substantive edits so early readers can see what moved.

2026-04-26 — Agentic coding claim: Replaced “empirically falsified” framing with scoped wording: strong evidence on benchmarked autonomous patch-style software engineering (SWE-bench-style workflows), not a universal verdict on multi-agent for all coding.
2026-04-26 — Audience: Added a short “who this is for” note (assumes you already work with LLM agents).
2026-04-26 — Debate vs. timeline: Clarified that Du et al. on the Wave 1 timeline reflects release window and headline question; the broader debate thread is parallel and is how Part 5 uses “debate.”
2026-04-26 — AgentVerse: Described in the narrative, not only in the timeline table (with arXiv link). The Part 3 preview in “What’s Coming” now names it with the other coordination papers.
2026-04-26 — Wave 2 timeline: Explicitly labeled as illustrative examples, not a complete survey of the period.
2026-04-26 — Magentic-One stuck counter: Scoped the “design pattern” remark to shipped agentic-coding systems of the period (not a claim about all of CS or all MAS research).
2026-04-26 — Wave 1 heterogeneity: Added a paragraph distinguishing pipeline-style coordination papers from simulation-heavy work in the same calendar cluster.
2026-04-26 — Waves caveat: Tightened prose; no change to the underlying claim that waves are retrospective scaffolding.

The Tribe Has to Outlive the Model · Five model swaps in three weeks taught me the project's continuity lives in the humans and conventions around the agents, not the agents themselves.

Thu, 23 Apr 2026 00:00:00 +0000

By the fourth model swap I noticed that the part of the system I wasn’t swapping was the part holding the project together.

Five configurations in three weeks: Opus 4.6, Opus 4.6 with the 1M context window, Cursor Cloud Agents on GPT Codex 5.3, Cloud Agents on Composer 2, and this week Opus 4.7. Some swaps were involuntary. Credits ran out. A model that had been my daily driver for two months started coming back with half-finished work and failing the easy parts of basic CI, which I wrote about in Caucus V1. Some were voluntary. I wanted to see what a new model could do that the previous one couldn’t. In a few cases it could. In a few cases the new one was worse at something different, and I swapped again.

No architecture change on my end. No rewrite. The app is the app. The model is a junior engineer on a revolving contract.

Same Prompt, Different Creature

Here’s a specific observation. For a few weeks in April I was running the Caucus Permit Gate. It was my attempt at forcing agents to prove their work before merging. Every PR had to land with a permit file in .caucus/permits/ declaring the scope and risk of the change, plus a proof file in .caucus/proofs/ recording the output of allowlisted test commands run against the current working tree. CI blocked the merge unless both were there and fresh.

Codex 5.3, given a task I’d given Composer 2 the day before, shipped the PR with the feature, the permit, and the proof all in the first commit. Two commits total on the branch. Gate green on the first push. Composer 2 on a similar task shipped the feature commit alone. Gate red. Then chore(caucus): add proof. Gate red. Then chore(caucus): sync permit scope and refresh proof. Then a merge of origin/main. Then chore(caucus): refresh proof after merge. The slowest Composer 2 PR from that era landed in main with twenty-three commits on the branch. One of them was the feature.

Same repo. Same AGENTS.md. Same sentence in bold telling the agent that the permit is not optional paperwork. Two completely different relationships with the gate.

Neither was strictly wrong. Codex 5.3 paid the cost upfront. Composer 2 paid the cost when forced. Composer 2 is the one that taught me something, because the only reason it paid the cost at all was that the gate existed. Without the gate, Composer 2 would have shipped the feature commit and I would have merged it, because I’m the reviewer, and “looks fine” works on me the same way it works on the model.

Where the Knowledge Has to Live

If a new teammate rotates in every fortnight, any knowledge that has to survive the swap cannot live in the teammate. It has to live in the repo.

Everyone has a CLAUDE.md or AGENTS.md at this point. Most of them read like documentation: “the backend is in /backend, the frontend is in /web, run npm install first.” Some of mine is that.

The part of mine that matters isn’t the documentation. It’s the lines that record things only visible if you were here when they broke.

NEVER modify timestamp/timezone columns in migrations. Timestamp corruption is unrecoverable and destroys production data.

That’s a scar. The commit that added this rule went in at 5:46 on a Saturday morning, and the commit message was one word: fuck. A companion rule had gone in 33 minutes earlier. I’ll come back to that one. A new engineer reading the repo cold has no way to know any of this. The column types look fine now. Nothing in the code says “this was dangerous.” Take the rule out, and the next model writes the same migration on the first try. The incident isn’t in the code. It was in the person who lived through it.

A guardrail is the tribe’s compressed experience, encoded in a form the next teammate can act on without having been present for the original incident. Institutional memory for a team where it would otherwise be deleted every other Tuesday.

Not Every Guardrail Survives the Test

Two days ago I published The Tax on the Happy Path, where I killed a guardrail I had spent three weeks building. The Caucus Permit Gate, which I’d hardened nine different ways in Opt-In Isn’t a Guardrail, never caught a unique bug in its last hundred CI runs. Everything it caught, the regular test jobs would have caught a minute later. It cost more than it caught. I took it out.

I’m not recanting that. Killing a guardrail that isn’t earning its keep is part of taking guardrails seriously. But I owe a better account of which rules survive the audit, because “some guardrails are good and some are bad” is not a useful thing to hand a reader.

Two questions. Does the rule point at a specific past event? And does its cost scale with the velocity of the work, or with the rate at which the event it prevents actually recurs?

The permit gate failed both. It didn’t point at a specific incident; it was a ritual about carefulness in general. And its cost scaled with every rebase against main, regardless of whether anything risky was happening on the branch. Cost grew with velocity. Value stayed flat.

It had the shape of every code-review checklist I’ve ever seen at a company. “Consider thread safety.” “Check error handling.” “Verify input validation.” You check the boxes, go through the motions. The actual review happens in someone’s head, from memory of the specific thing that burned the team last quarter. Ceremony versus pattern recognition. The permit gate was all ceremony.

The guardrails that have survived the audit pass both. “Do not drop a column without running ./scripts/check-references.sh first” points at the time we dropped a column and broke ninety percent of the reports. “No --admin merges” points at a specific session in Memory Isn’t Learning. “No database triggers. EVER.” points at the incident I’m about to describe. The cost of obeying each of these is a few seconds, paid only when the agent is about to do the specific dangerous thing. Cost scales with the rate of the risky action, not with the rate of all work.

They refer to events. They are shaped like scars.

The Trigger

The companion rule from above went in at 5:13 that same Saturday morning, 33 minutes before the timestamp one. Same incident. The migration side of that night I’ve already written about. The trigger side I haven’t. Here it is.

I’ve seen this incident at three different companies. Somebody adds a trigger to maintain updated_at, or to keep a computed column in sync. The trigger works. Years later someone writes a migration that touches the same table, the trigger fires in the middle of it, and rows end up with values nobody intended. I would have pushed back on this in any code review at any job I’ve ever had. If a junior engineer proposed a trigger in a design doc I’d have said: please, no, not a trigger, write it in application code.

And then I built Zabriskie with an AI that took the shortest path between “I want updated_at to update itself” and “it updates itself.” The shortest path was a BEFORE UPDATE trigger named update_updated_at_column, attached to posts, comments, users, and every other table with an updated_at. I didn’t catch it because I was moving fast, the code looked fine, and I was watching the screen fill up with new features instead of acting as a database reviewer on my own project.

The trigger revealed itself on the night the timezone migration broke. Migration 030 converted every timestamped table in the schema from timestamp to timestamp with time zone with an AT TIME ZONE 'America/New_York' clause. That’s where the disaster movie started. The follow-up migrations tried to clean up. Every cleanup UPDATE fired the trigger and stamped updated_at = NOW() on every row the UPDATE touched, which meant every feed ordering by updated_at started surfacing posts from earlier that week as if they’d just been edited. Each fix kept succeeding, and the symptoms kept changing, because each fix was also erasing the evidence that the rows hadn’t been touched by a user.

It took several migrations to trace, because each one had succeeded, the columns I’d meant to change looked right, and nothing in the migration files mentioned updated_at.

By migration 036 I gave up and dropped every trigger. DISABLE TRIGGER ALL ran first, the fix ran second, DROP TRIGGER ran third. Migration 037 fixed updated_at one more time. At 5:13 that morning, the rule went into CLAUDE.md:

NO DATABASE TRIGGERS: NEVER create database triggers. EVER. Always use explicit SQL (SET updated_at = NOW()) in application code instead. Triggers cause unpredictable behavior during migrations and data fixes. This is non-negotiable.

A companion section, “Check Triggers Before DB Updates,” went in the same week. It tells the agent to run \dS table_name, list the triggers, and (for the triggers I hadn’t yet ripped out) disable them around any UPDATE migration:

ALTER TABLE posts DISABLE TRIGGER update_posts_updated_at;
-- Your UPDATE here
ALTER TABLE posts ENABLE TRIGGER update_posts_updated_at;

Neither rule is for me. I already knew both. I wrote them down because the teammate I’m now on a team with arrives fresh every week with no memory of any of the things I know, and will, absent a specific instruction, pick the same shortest paths. The shortest paths are a trigger and an AT TIME ZONE migration.

The rules are a decade of scar tissue from somebody who isn’t going to be in the room the next time the new model gets the same task. The tribe’s memory, encoded in a form that survives the swap.

The same shape holds for “Don’t use --no-verify,” for “Never deploy untested changes to S3,” for “Always use ON CONFLICT DO NOTHING on data INSERTs in migrations.” Each points at an event. Each event is the thing the next agent would have repeated, because the next agent is a different agent every time.

The Interesting Question

People keep asking whether the model can learn. Does the memory system work. Does the agent read CLAUDE.md. Does the rule change the behavior. I argued in Memory Isn’t Learning that the answer is usually no.

Different question. Not “can this model learn.” The models will get better at that, or they won’t, and either way my project has to ship.

Can the team’s memory survive the next model swap?

The rules I wrote after that Saturday morning have now outlived every configuration that’s done work on this codebase since. The one I’ll be using next month will read them and will not write a trigger or an AT TIME ZONE migration.

The tribe has to outlive the model. The only way that happens is if what the tribe learned is written somewhere the next model can’t skip.

This is part of a series about building Zabriskie with Claude. Previously: Memory Isn’t Learning, Opt-In Isn’t a Guardrail, The Tax on the Happy Path.

The Tax on the Happy Path · What I learned by removing the elaborate CI gate I'd built to slow my AI agents down — after realizing it had never actually caught anything.

Tue, 21 Apr 2026 00:00:00 +0000

I was halfway through another PR, regenerating the proof file for the fourth time after rebasing on main, when I realized I couldn’t name a single PR the permit gate had caught. Not “caught that other CI didn’t also catch.” Just caught. I decided to look up the numbers, and then I removed the gate entirely.

What the Gate Was

The short version: every PR to Zabriskie had to be accompanied by a permit file in .caucus/permits/<branch>.json. The permit declared the scope of the change (which paths were allowed to move), the risk level (R1 through R3+), and a list of allowlisted test commands the author was offering as evidence. R2 and higher required a proof file too, generated by actually running those commands and recording the output alongside a fingerprint of the working tree. A CI job called Caucus Permit Gate ran on every PR. It blocked the merge unless the permit existed, the scope covered the diff, and (for R2+) the proof was fresh and the tests passed.

I built it because I’d spent months watching agents read a rule in CLAUDE.md, acknowledge it in conversation, and then ship code that violated it anyway. Memory Isn’t Learning was the post where I argued that prose in an instruction file is a journal of failures, not a guardrail. The permit gate was supposed to be the structural answer: the rule stops being a sentence and starts being a required status check that CI evaluates against the actual diff.

The gate shipped. It got bypassed, broken, and accidentally self-deadlocked in nine different ways, each of which got a hardening patch. Opt-In Isn’t a Guardrail was the post about those failures. By the time I sat down to write this one, none of that was still happening. The paperwork was filed, the scope was matching, the proofs were fresh, the tests were passing. I killed the gate anyway.

The Count

Over the last hundred CI runs on Zabriskie (roughly a week of shipping, so take the sample for what it is) there were four Caucus Permit Gate failures. Every one of them was a “Proof failed” result, which means one of the allowlisted test commands the proof re-runs returned non-zero. The same commands run in the Build Backend, Unit Tests, and E2E Tests jobs, which are separate CI jobs on every PR. In every case, those dedicated jobs also failed, for the same reason, on the same PR.

The gate’s unique contributions, the ones that were supposed to justify its existence, never fired as the blocking failure. Scope allowlist violations: zero. Risk-level enforcement: zero. Proof fingerprint drift is a regeneration trigger by design, not a catch, so it was never going to appear on that list, and I should have noticed that before I put it on the list. The gate caught exactly nothing in that window that the plain test jobs wouldn’t have caught five minutes later.

The Cost

The proof file’s validity is tied to a fingerprint of the working tree. Every git merge origin/main changes that fingerprint, which invalidates the proof, which means the proof has to be regenerated, which means the allowlisted commands have to be re-run end to end. On Zabriskie that’s cd backend && go build ./..., cd backend && go test ./..., cd web && npm run build, and cd web && npx playwright test. Playwright alone takes several minutes and burns an entire stack of browser processes.

Every time main moved underneath an in-flight PR, the compliant action (rebase, resync, then push) forced the same PR to re-run the same tests it had already run, twice, once in the proof regeneration and once in the downstream CI jobs. During an active week of merges that can happen five or six times on a single branch.

This is the part I did not see clearly when I was building it. The cost of the gate scaled with velocity: the more main moved, the more proofs had to be regenerated, the more tests had to run twice. The benefit didn’t scale with anything I could measure. It sat flat at zero catches that weren’t already caught downstream. A gate whose cost grows with how much work is happening, and whose benefit does not grow at all, is not a guardrail. It is a tax on the branches doing the right thing, collected in service of nothing.

Agents are not insulated from this. They pay in tokens, in extra tool calls, in context spent on regeneration steps that accomplish nothing, in CI wait time that blocks the next thing they were going to do. The dollars come out of my pocket, but the friction lands on them.

The Kill PR

The kill PR was small. Remove the permit-gate job from the CI workflow. Strip the needs: [permit-gate] dependency from the downstream jobs. Delete the pre-push hook that enforced the same check locally. Delete the corresponding rule from the agent instructions file. Branch protection had to be updated to drop the gate as a required status check, which is the only part of the change that required an admin action on the repository.

What I did not do, and what I am deliberately leaving alone for now, is delete the scripts and the .caucus/ history of accumulated permits and proofs. That data is a record of how agents behaved under a specific constraint across the life of the experiment. I don’t want to throw it away until I’ve looked at it properly. There’s a version of this story where the permits themselves, as artifacts, turn out to be more useful than the gate that validated them. I don’t know yet whether that’s true. But the gate’s cost and benefit are clear enough that I don’t need to know before pulling the plug.

What I Am Not Saying

I am not saying there should be no gates. I am not saying the Caucus experiment was wasted. I’m saying that this particular implementation of this particular gate was charging an ongoing cost larger than the value of what it caught, and that cost is a first-class design concern, not a tradeoff you can wave away by pointing at the artifacts it produced.

Every gate has to answer two questions. Does it catch failures the other checks wouldn’t catch? And does the cost of passing it, summed over the lifetime of the repo, stay below the cost of whatever it’s preventing? If the answer to the first is “no” and the second question never even gets asked, the gate has negative value even when it technically works.

The question I was asking in Memory Isn’t Learning was whether the structural version of a guardrail would behave differently from the journaled one. The answer is: structural guardrails are necessary but not sufficient. They have to be structural and the ongoing cost of compliance has to stay below the cost of the failures they prevent. Otherwise the gate is a net negative no matter how firmly it’s bolted in.

What’s Next

I still think the basic idea behind Caucus is right. An agent pushing a change to a shared branch should have to declare the scope of the change, the risk it’s taking, and the evidence it’s offering that the change is correct. What I got wrong was the enforcement surface. The gate lived at merge time, which is the most expensive place to check anything, and it checked by re-running the same tests the merge was already going to run.

I don’t know yet what the next version looks like. The honest sentence is that I don’t know how to validate a permit continuously against a moving diff without re-running the work the tests are already doing. That was the whole problem the first time, and I skipped over it by putting the check at merge time and eating the cost. Until I can answer that question, I don’t have a V2. I have a V1 that I’ve turned off and a list of things I know V2 can’t do.

I had the information to turn this off weeks before I did, and I don’t have a mechanism for catching the next version of this mistake any earlier.

This is part of a series about building Zabriskie with Claude. Previously: Memory Isn’t Learning, Software Engineering Is Becoming Civil Engineering, Caucus V1, The Structural Engineer’s Other Job, Opt-In Isn’t a Guardrail.

Opt-In Isn't a Guardrail · A CI gate I built to slow AI agents down passed every paperwork check, missed every real bug, and blocked the rollback during a production outage.

Tue, 14 Apr 2026 00:00:00 +0000

Tonight, Goose was playing a very special set two. The band was spelling out SUCK IT STORM with their song titles, a response to a critic named Ryan Storm who’d publicly trashed their setlist choices and alternative arrangements, and people in the live chat were losing their minds about it in real time. Zabriskie was built for exactly this moment: friends sharing what they’re hearing, live chat going, the app doing the one thing it exists to do.

The feed was blank. The landing page was blank. Every route in the app was rendering nothing.

I rolled back three deploys directly in Railway while Goose played. I couldn’t revert via a PR because the permit gate and CI pipeline I’d spent three weeks building would need to pass first, and during a production outage with a live show happening, I didn’t have ten minutes. The system I’d built to slow things down was now the thing in the way.

The agents had done everything the system asked of them. Every PR had a permit. Every permit had a proof file. Every proof file pointed at passing tests. The tests passed. The gate was working exactly as designed. None of it tested whether the page actually rendered without crashing.

The gate caught agents who forgot to file paperwork. It did not catch agents who filed correct paperwork for broken code. And when the building was on fire, the gate blocked the fire truck too.

What I Built

A few weeks ago I wrote Memory Isn’t Learning about the loop where Claude saves a rule to disk, ignores it, ships the bug, and saves the rule again. The thesis was that prose in CLAUDE.md is a journal of failures, not a guardrail. After that post, I tried to do something about it.

I built a thing called the Caucus Permit Gate. Every PR has to be accompanied by a permit file, a JSON document that declares the scope of the change, classifies the risk, and points at a proof file with concrete evidence. CI checks the permit before letting the PR pass. AGENTS.md says, in bold:

Do not treat the permit as optional paperwork.

Since I built that gate, agents have skipped the permit, broken the permit, or gone around the permit nine times, all of them now in main. Each time I added another guardrail. Each guardrail was itself ignored, defeated, or in one case, locked the gate against everyone including the agent that had just installed it.

Ten Cycles

The first cycle was the gate itself. Branch-scoped CI check, scope and risk validation. It worked. PRs without a permit failed CI.

Then PR #246 went up without a permit. The agent had read AGENTS.md. The agent had acknowledged the rule. The agent pushed anyway. Fix: an expansion of AGENTS.md with an explicit “Opening a PR” bullet. Text on text.

So I shipped a script and a .githooks/pre-push hook that would block the push if the permit file was missing. The hook was opt-in. You had to run an installer per clone. Then I shipped an installer for the installer, because writing “you can run this script if you want to” in AGENTS.md and calling it a guardrail was the same kind of failure the gate was supposed to prevent. A text-only fix to a rule about how text-only fixes don’t work.

Three more PRs went up. Three more times the same pattern. An agent pushes without a permit, or with a broken permit, or with a permit that passed locally but failed in CI because the hook was validating the working tree instead of what was actually being pushed. Each time, a new patch. Each patch, a new edge case.

That’s seven cycles. Six patches on the previous patch.

Three more arrived in a single PR while I was writing. PR #293, titled “Harden permit-gate workflow and fail-fast CI sequencing,” had twenty-three commits on the branch. The permit gate failed but CI kept running the rest of the jobs, burning resources on a PR that couldn’t merge. The pre-push hook regenerated the proof file but didn’t re-check for a dirty working tree, so stale proof reached CI while local checks looked clean. Then the dirty-tree check created a self-deadlock: every push regenerated the proof, which immediately failed the dirty check, which meant no push could ever complete. The gate locked everyone out, itself included.

If you read the commit messages on that branch in order they look like a heart monitor: sync permit, refresh proof, record latest proof run, skip proof rerun, align pre-push proof freshness, refresh proof after guardrail alignment, refresh proof after final hook guardrail changes. That sequence is now in main. Anyone who clones this repo tomorrow inherits it.

The Pattern

Every fix was either text, opt-in, or recursive.

The text fixes were edits to AGENTS.md. The permit is required. The permit is really required. Do not treat the permit as optional paperwork. Each round felt productive. None of them changed the behavior of the next agent.

The opt-in fixes were scripts you could run to install a hook. The installer is in the repo. It is documented. It is one line of shell. The agents that need it most are the ones least likely to install it, because installing it is a step that has nothing to do with the task they were asked to perform. Asking an agent racing toward a PR to first run a setup script with no effect on the task it was given is asking it to slow down voluntarily for a future benefit it can’t see. Agents don’t do that. Humans barely do that.

The recursive fixes are the most interesting. Cycle 9 hardened cycle 7. Cycle 10 hardened cycle 9 by undoing the file-mutation behavior cycle 9 introduced. Each layer of guard had a hole. Each patch introduced a new one. The result is a gate that is mechanically defeatable, in the way that a door is defeatable when the lock is sitting on the table next to it with a sign that says please use this lock. Or, sometimes, when the lock is glued to the door so firmly that nobody can open it from either side.

In structural engineering, you don’t put up a sign at the entrance of a parking garage that says “no vehicles over 6 feet 10 inches.” You hang a steel clearance bar. The driver who ignores the sign hits the bar and has to back out. The sign is advisory. The bar is structural. The permit gate as I built it is a sign. The guardrail has to live in a place the agent has to pass through. Not a place the agent could pass through if it chose to.

This is the deeper version of what I wrote in Memory Isn’t Learning. There I argued that saving a note and ignoring it is journaling, not learning. The implicit hope was that mechanical guards would be different. That if the rule was code, not prose, the agent would be forced to comply. The permit gate was the test of that hope.

It failed. Mechanical guards aren’t different if they’re optional. Opt-in isn’t a guardrail. It’s a sign in a different font.

And then the gate started working, and the app went down anyway.

What the Gate Didn’t Test

PR #294 added a heart burst animation for favoriting live chat comments. It put a useMemo and a useEffect after the if (loading) and error conditional returns in SDUIPage.jsx, which wraps nearly every page in the app. That violates React’s Rules of Hooks: the number of hooks has to be the same on every render. When the page was loading, React saw fewer hooks than when it finished. React threw. The page went white.

The DOM assertions passed on the loading state. React crashed on the loaded state. The test saw the spinner. The user saw nothing.

The forward fix was PR #304. It moved the hooks above the conditional returns, logged a critical incident, and added a Playwright guardrail that attaches page.on('pageerror') to the smoke tests so uncaught React errors now fail CI. The test that should have existed from the beginning.

The Honest Part

I have to say this part because it would be dishonest not to. Most of the ten fixes were implemented by Claude. I asked it to fix the gate, and Claude (a different session each time) wrote a script, edited AGENTS.md, added a sentence, expanded a stderr message. Each of those PRs looked like a fix. Each of them passed review because I was the reviewer and I was reading them the same way the agents were writing them: as if a sentence in a file was the same as a constraint in the system.

It’s not. I should have known that. I wrote a whole post about it. And I still spent three weeks watching the same gate fail in slightly different ways while approving fixes that had no chance of working, because the fixes felt like progress and progress felt like the goal. Then I watched the gate finally work and the app still go down, because I’d been so focused on whether agents were filing permits that I forgot to check whether the permits were proving anything.

The gap between “I know better” and “I do better” is the same gap I keep accusing the model of having. It’s just slower in me.

Where This Goes

The gate has to be the floor, not a sign. The tests have to prove the page works, not that the DOM exists. Memory wasn’t learning. Documentation isn’t enforcement. Opt-in isn’t a guardrail. And a guardrail that checks the wrong thing is just a more convincing kind of nothing.

The Structural Engineer's Other Job · AI-generated code increasingly passes review and tests but ships half-working features. What humans have to check for instead.

Thu, 09 Apr 2026 00:00:00 +0000

I’ve been building Zabriskie for a few months now, mostly with AI agents. Claude Code writes the backend handlers, builds the SDUI screens, registers the routes, and, importantly, writes the Playwright tests. The test suite has grown to over 150 E2E tests. CI is green. I’m shipping fast.

But I keep finding the same category of bug.

The RSVP feature on show pages is a good example. A user taps the RSVP button, their attendance is saved, and a post is supposed to appear in the feed so their friends know they’re going. Claude built the handler, registered the route, wired up the SDUI component. The code compiled. The tests passed. I reviewed the diff, it looked right. A few days later I noticed the feed was quiet. People were RSVPing to shows but no posts were appearing. When I dug in, the attendance was saving fine. The button worked. But the downstream feed post creation was silently failing: when a show had no media_item_id, the handler was passing an empty string to a UUID column, which threw an “invalid syntax” error that got swallowed by the error handling. Users tapped the button, saw confirmation, and had no idea their RSVP was never announced. The feature half-worked, which is worse than not working at all.

And this wasn’t a one-off. A Quick Post button rendered on the page but did nothing when tapped, missing buttonType: "submit", so the click event was swallowed by the form container. A “Watch Livestream” button worked fine during the day but vanished every evening because a UTC truncation bug made tonight’s show look like yesterday’s, hiding the button right when the band took the stage. Search results stopped being clickable because a fix to prevent click events from bubbling in comment forms accidentally blocked all click actions inside <form> tags. I wrote about the worst case in The Feature That Has Never Worked: an auto-live poller that broke seven times in thirteen days, each fix introducing a new failure mode, while the UI calmly displayed “scheduled” as Billy Strings played to a sold-out amphitheatre.

Every one of these PRs had passing tests. Each one would survive a mechanical code review. The types were correct, the logic was plausible, the patterns matched existing code. An AI reviewer scanning for boundary conditions and API misuse would approve all of them.

The problem was the same every time: the Playwright tests that Claude wrote verified that UI elements existed without verifying that the features worked.

The Tests That Don’t Use the Feature

When I went back and looked at what Claude had actually written in the test files, the pattern was consistent. The RSVP test checked that the button was present on the show page. It might even click it. But it never navigated to the feed afterward to check that a post appeared. The Quick Post test confirmed the form rendered but never submitted it. The livestream test checked for the button during the day but never ran at the time of an actual show.

This makes sense if you think about how the tests get written. The agent finishes implementing a feature, then writes a test that exercises the code path it just built. The test is shaped by the implementation, not by the user’s experience. It verifies the thing the agent made (a button, a form, a component) not the thing the user does (RSVP and see it in the feed, fill out a form and see the result, tap a livestream link during a live show).

I could read through every test the AI writes and audit whether it actually exercises the full user workflow. But that puts me right back where I started: I’m the bottleneck, just reviewing test code instead of reviewing application code. The test suite gives me a green checkmark. It doesn’t give me confidence.

This isn’t just my problem. Anthropic launched a code review tool in March explicitly because code review has become a bottleneck, and even with AI review, the code is shipping faster than anyone can verify it.

AI-powered review tools help with the mechanical side: style, boundary conditions, common bug patterns. But they share the same fundamental limitation as human review and AI-written tests: they read the code and reason about it. They don’t run it. They can tell you the handler is registered and the component has the right props. They can’t tell you that when a user taps RSVP, a post actually appears in the feed.

The Agent at a Computer

This is where something genuinely new has happened.

Cursor’s cloud agents don’t just write code in a text editor. Each agent gets its own virtual machine, a real computer with a browser, a terminal, and the ability to interact with running software. The agent writes the code, starts the application, navigates through it like a user would, takes screenshots, records video of the feature working, and attaches all of that to the pull request. More than 30% of the PRs merged at Cursor itself are now created by these agents operating autonomously in cloud sandboxes. OpenAI’s Codex has moved in the same direction, wiring Chrome DevTools Protocol into the agent runtime so the agent can start a browser, inspect the DOM, take screenshots, and reason about UI behavior directly.

The reviewer doesn’t mentally simulate a diff. They watch the feature work.

That difference matters more than it sounds. When I review a PR with video evidence attached, I’m not reading test assertions or tracing code paths in my head. I’m watching someone (well, something) tap the RSVP button and then check the feed. I can see whether the post appeared. I can see whether the button animation played. I can see whether the page scrolled correctly afterward. Thirty seconds of video tells me more about whether a feature works than a 200-line Playwright spec ever could, because the video shows the outcome, not the mechanism.

And I’d already been moving in this direction myself before I knew about Cursor’s cloud agents. In Teaching Claude to QA a Mobile App, I described building a system where Claude drives the iOS and Android simulators for Zabriskie, connecting to Android WebViews via Chrome DevTools Protocol over ADB, fighting with Apple’s idb tools for six hours to get iOS working. I built a nightly sweep that launches both simulators, navigates all 25 screens, takes screenshots, analyzes them for visual issues, and files bug reports automatically as zabriskie_bot in the production forum. The instinct was the same: look at the app running. Don’t just read the code.

The Witness

In formal verification, this concept has a name: a witness. A witness is a concrete piece of evidence that a thing works. Not an argument that it should work, but proof that it did work. The screenshot is a witness. The video is a witnessed execution trace. The agent isn’t just building the feature. It’s constructing evidence that the feature works.

A witness is legible without reading code. You can hand the video to a product manager and they can tell you whether the feature works. Try that with a Playwright spec.

But some witnesses are harder to construct than others, and this is where the real complexity hides.

A screenshot proves something rendered. A video of a click proves a button responds. But many features aren’t a single interaction. They’re stateful workflows that span multiple operations over time. RSVP to a show so a post appears in the feed so another user can see it and bookmark it. Subscribe to a service so you get a discounted fee at checkout next week. Add items to a cart, apply a coupon, and verify the total reflects both. The witness for these features isn’t one screenshot. It’s a chain of evidence across multiple steps, where each step depends on state persisted in a database or a queue from a prior step. The agent has to perform step A, verify the side effect was stored, then come back and perform step B and verify that step B’s behavior reflects what step A wrote. That’s a harder witness to construct, and it’s exactly where the most important bugs live. The RSVP bug was precisely this shape. The button worked. The state didn’t propagate. A single-screenshot witness would have approved that PR.

The Witness Is Not a Test Suite

Here’s the thing I can’t let myself forget: a witness proves the feature under test works right now. It says nothing about whether it broke something else.

Every incident I’ve logged in the Zabriskie reliability database reinforces this. Fixing click handling in comment forms broke search result selection. Adding authentication middleware to 25 unprotected routes broke the live show pill. The avatar upload fix that worked for one handler left broken images on four other pages. Twenty-four percent of all commits in the Zabriskie codebase are fixes, and they arrive in chains where each fix breaks something adjacent.

You can’t build an application on video evidence alone. A witness is an existence proof: “this works.” A regression suite is a universal proof: “nothing that used to work is now broken.” You need both. The video catches the bugs that tests miss, the ones where the test verifies the button exists but nobody checked whether it actually does anything. The regression suite catches the things the video didn’t think to look at, the five other features that silently broke when you fixed the one you were focused on.

Who Builds the Stage?

The witness works for Zabriskie because it’s a single application that runs on one machine. I can go run cmd/api/main.go, npm run dev, open a browser, and navigate the whole app. An agent can do the same thing on a VM in the cloud. The witness is easy to construct because the environment is easy to provision.

But even Zabriskie isn’t really one thing. There’s the web app, the iOS app in the App Store, the Android app on Google Play, and the backend API they all talk to. When I push a backend change, the iOS app is still running last week’s code. When I add a new SDUI component type, the web client needs to know how to render it or it silently does nothing. I’ve already lived a small version of the coordination problem: four deployment targets, one developer, and no guarantee they’re all in sync. If this is already hard at my scale, imagine a company with 200 services.

Most real-world software has this problem worse than I do. And when I started thinking about where the witness concept breaks down at larger scales, I realized the answer is the same in every case: it’s not the agent. It’s the environment. The agent is smart enough to navigate an app and record what it sees. The hard part is giving it an app to navigate.

That’s a platform engineering problem. And it’s the same one I was pointing at in Software Engineering Is Becoming Civil Engineering. In that post I used the broad term civil engineering. The role I’m describing here is more specific: the structural engineer, the person inside civil engineering whose job is making sure the thing stands up and can be inspected over its lifetime. The structural engineer doesn’t weld the beams. The structural engineer designs the bridge so that a welder doing their job correctly can’t bring the whole thing down. The platform engineer doesn’t write the feature. The platform engineer builds the infrastructure so that an agent writing a feature can verify it works. The witness is the agent’s job. The stage is the platform engineer’s job.

Every scaling challenge for the witness turns out to be a question about whether someone has built the right stage.

Mobile

For mobile, the stage is a simulator. I wrote about this in detail: Android was workable in 90 minutes because Capacitor WebViews expose a Chrome DevTools Protocol socket, the same protocol that powers Playwright. iOS took over six hours because Apple’s tooling isn’t designed for headless automation. The cloud infrastructure exists (macOS VMs, GitHub Actions runners with simulator support) but it isn’t built for ephemeral agent-per-PR workflows the way a Linux VM with Chrome is. The agents are capable today. The platform work is making the simulation layer automatable enough that an agent can spin one up, use the app, and tear it down without human intervention.

Microservices

For distributed systems, the stage is an environment where all the relevant services are running together. This is the most interesting case, and there are two very different versions of it.

The monorepo world. Companies like Google, Meta, Twitter, and Uber keep all their services in a single repository. The key advantage is that cross-cutting changes can be made atomically, one PR that touches the API gateway, the billing service, and the notification service, all committed together. An agent working in a monorepo can, in principle, make a coordinated change across multiple services in a single diff.

But can you run all those services on one machine to produce a witness? That depends entirely on the application’s complexity. A monorepo with 15 services might be runnable with docker compose on a beefy VM. A monorepo with 500 services almost certainly isn’t. And even for the 15-service case, the bootstrapping problem is real: database migrations, seed data, service discovery, mock credentials for third-party APIs. The question isn’t whether the agent can write the code. It’s whether the platform team has made the application bootable enough for the agent to stand it up and use it.

This reframes what “testable” means. It’s not just about code coverage or CI pipelines anymore. It’s about whether an autonomous agent can cold-start your system from a fresh checkout and get to a state where it can navigate through a feature. That’s a higher bar than most organizations have cleared, and I think the pressure from agentic development is going to become the forcing function for platform investment. If an agent can’t boot your app, an agent can’t verify your app. Making the system bootable is platform engineering work. It’s the structural engineer designing the inspection regime for the bridge.

The polyrepo world. Companies that use separate repositories for each service face a fundamentally different problem. A single feature, say adding a discount for subscribers, might require changes to the user service (store subscription status), the pricing service (check subscription at checkout), and the checkout frontend (display the discounted total). That’s three PRs in three repositories, reviewed by three different teams, merged on three different timelines.

Each service has to make a backwards-compatible change. The new pricing service has to work whether or not the user service change has been deployed yet. The checkout frontend has to handle both the old price and the new discounted price gracefully. The standard practice is semantic versioning, feature flags, and contract testing: deploy each change independently, verify it doesn’t break existing consumers, activate the feature once all the pieces are in place.

But here’s the question that video evidence forces you to ask: when does anyone actually see the discount appear on the checkout page? Each PR gets reviewed in isolation. Each service’s CI passes independently. But the feature, the thing the user actually experiences, doesn’t exist until all three changes are deployed together. The witness for the feature can’t be produced at review time. It can only be produced after all the independently-reviewed changes are merged and deployed. By then, the code was written days or weeks ago. If something’s wrong (the discount doesn’t apply, or it applies twice, or the price flickers between old and new) the feedback loop is enormously long compared to what an agent on a single machine can do.

The platform engineering answer is a coordination layer that can stage cross-service changes together before any of them merge. Imagine a system, not unlike the Caucus workflow I’ve been building for code review, but where each service gets its own agent. The agents make their changes independently, each producing backwards-compatible modifications. But before any of them merge, a coordinator stages all the changes together in an ephemeral environment and runs the end-to-end user journey. One agent starts the user service with the subscription change. Another starts the pricing service with the discount logic. A third starts the checkout UI. The coordinator navigates the full flow (subscribe, browse, add to cart, see the discount at checkout) and produces video evidence of the feature working across the composed system.

You could even roll individual services forward and back: does the feature degrade gracefully if only the user service and pricing service are updated, but the checkout UI is still on the old version? That’s backwards-compatibility testing through exploration rather than assertion. The agents become a way to simulate deployment order, probing the combinatorial space of “which services have been updated” without deploying anything to production.

Nobody has built this yet, as far as I know. But the pieces are converging. Ephemeral preview environments that spin up isolated service meshes per branch. Agent runtimes that can control browsers and navigate applications. Coordination layers that manage multi-agent workflows with structured handoffs. The missing piece is the platform engineering work that connects them: a system that knows which services participate in a feature, stages their changes together, and produces a cross-service witness. That’s infrastructure. That’s the structural engineer’s job.

APIs and SDKs

The hardest case is the one where there’s no button to click at all. When you’re building a platform (an API, an SDK, a shared library, an internal service that other teams depend on) your users are other developers. The feature doesn’t have a UI. There’s no page to navigate. There’s no checkout flow to walk through.

The witness for a platform change might be: does every team that depends on this API still build and pass tests after this change? But producing that witness means checking out their code, building their project against your new version, running their tests. That’s not an agent at a computer. That’s an agent that understands organizational dependency graphs.

But the platform engineering answer here might be the most straightforward of all: the witness is video evidence of a sample application that uses the API. This is how good API and SDK development already works in practice: you build a real application that consumes your own product before shipping it to external users. The agent version: make the platform change, check out the reference app, build it against the new version, run it, navigate through it. If the reference app still works, your change is backwards-compatible. If it doesn’t, you’ve caught a breaking change before it reached your consumers. The witness isn’t your library running. It’s what your library enables running.

The platform engineer’s job here is maintaining that reference app and keeping it representative. That’s not glamorous work. But without it, there’s no stage for the agent to perform on, and the witness can’t be constructed.

The Other Job

In Software Engineering Is Becoming Civil Engineering, I argued that the profession is splitting: feature development is becoming accessible to non-engineers, but someone still has to design the bridge. I described the platform engineer’s job in terms of API design, load analysis, inspection regimes, self-healing systems.

I think there’s another item on that list now: making the system witnessable. Building the infrastructure so that when an agent writes a feature, it has somewhere to run it, navigate it, and record the evidence that it works.

The agent can write the code. The agent can construct the witness. But the platform engineer builds the stage, and that work doesn’t show up on anyone’s roadmap yet.

I stopped reading the tests. I started watching the video. And the thing I keep coming back to is: the video only works if someone built the infrastructure to make it possible.

Caucus V1: Cursor Background Agents and a Multi-Agent Workflow That Actually Loops

Wed, 08 Apr 2026 00:00:00 +0000

I’ve been using Cursor 3 more over the last week or so because it makes it easy to move between models, and Cursor’s Composer 2 has been producing good results for me. That’s become more important because Opus 4.6 has gotten noticeably worse for me in Zabriskie. Work that used to feel routine, like straightforward file edits or basic follow-through on multi-step changes, now takes forever across multiple prompts, comes back half-finished, or fails basic CI checks. The model that was my primary tool for two months of shipping Zabriskie has started struggling with things it used to do in one pass.

That matters here because this project has become, whether I intended it or not, a study in what happens when the model is not consistently reliable. If the agent can’t be trusted to do basic work cleanly every time, then the surrounding system has to take on more of that burden by preserving state between steps, recovering from predictable failures, and making it obvious what happened when something goes wrong.

The last few posts have been converging on this from different directions. Multi-Agent Systems Have a Distributed Systems Problem is where I said most clearly that these systems need real coordination machinery, not just role descriptions and prompt choreography. Software Engineering Is Becoming Civil Engineering and The Feature That Has Never Worked pushed me toward the more practical version of that same conclusion: if the models are inconsistent, the surrounding system has to get stronger.

So this post is not just adjacent to that earlier multi-agent piece. It’s the first concrete version of the system I was pointing at there. In that post, the argument was that multi-agent software systems need real coordination machinery instead of roleplay and prompt choreography. Caucus V1 is rev 1 of that vision. It’s the first pass at a runtime that actually tries to encode those ideas into the workflow itself.

This week I built that first version. It’s called Caucus, and for the first time I have a multi-agent loop that actually completes end to end.

Why Cursor Agents Matter Here

Part of what makes this worth building now is that Cursor’s background agents are not just chat windows with long prompts. They run in the cloud on actual computers. They can check out code, make changes, open pull requests, wait for CI to finish, respond to failures, and keep working while I’m not sitting there driving every keystroke.

That changes the shape of the problem. If an agent can only draft code in a text box, then “multi-agent” mostly means multiple role descriptions. If an agent can actually live inside a software workflow, then it can do the things a real teammate would do. It can implement a change, open a PR, wait for checks, fix what failed, run the app, and produce evidence that the feature works.

That’s the part I think people are still underestimating. These agents are not interesting because they can talk about code. They’re interesting because they can operate on real software artifacts over time.

And the most striking capability is what happens after the code is written. A Cursor background agent can start the application, interact with it, take screenshots of the running UI, and record a video walkthrough demonstrating that the feature actually works. It can attach that evidence to the PR alongside a passing CI build. That’s not “code generation.” That’s a worker producing deliverables with proof.

That is the enabling condition for the version of Caucus I want. Without agents that can actually execute against a repository, run CI, start the app, and produce visual evidence that the change is correct, multi-agent coordination is mostly theater. My vision is not a cast of roleplaying agents debating architecture in a transcript. It’s a system of agents that can take responsibility for different parts of a real development loop, with the runtime coordinating their work and preserving enough structure that the loop remains legible.

What Caucus V1 Actually Is

Right now, Caucus V1 is a very small system with a very specific job. It coordinates a fixed set of background agents around a pull request lifecycle.

In practice, that means one agent implements code changes and opens or updates a PR, another agent reviews that PR and either approves it or requests changes, and the runtime keeps looping until the PR is approved or a safety cap is hit. That loop is the whole point. Most multi-agent demos are linear: agent A hands off to agent B, and the workflow is done. Caucus V1 has a cycle in its DAG. The reviewer can send work back to the implementer, and the implementer can re-enter the same PR and refine it.

What makes that cycle possible is the causal history that the runtime carries forward between stages. The implementation agent doesn’t just receive a generic task the second time around. It knows it’s refining a PR based on review feedback because the handoff tells it so. It knows how many times each role has acted. It knows what the reviewer said. It knows it’s on its second or third pass. That history is what lets the agent infer what the next action should be, rather than starting over from scratch every time it’s invoked.

That may not sound ambitious, but I think it’s more honest than most multi-agent demos. The hard part is not getting two agents to say different things in sequence. The hard part is giving the system enough structure that the second and third turns are still meaningfully connected to the first one.

The Most Important Part Is The Tiny Vector Clock

The most interesting idea in V1 is probably also the smallest one. Each stage handoff carries an actorClock, which is just a dictionary counting how many times each role has acted so far. If the payload says {\"implement\": 2, \"review\": 1}, that means the implementer has already acted twice and the reviewer once.

That is obviously not a full causal history. It is not a general solution to the coordination problem. But it is a real step toward the kind of machinery I was arguing for in Multi-Agent Systems Have a Distributed Systems Problem. It’s a tiny version of the vector clock idea: enough ordering information for an agent to know where it is in the workflow without pretending the whole world can be reconstructed from prompt text alone.

In practice, that matters because the implementer should behave differently on the first pass than it does on a remediation pass. On the first pass, it’s trying to complete the task. On the second or third pass, it’s trying to interpret review feedback and update the same PR without losing the thread. actorClock gives the runtime just enough structure to express that difference.

This is what I mean when I say Caucus V1 is rev 1 of that earlier vision. I’m not claiming to have solved multi-agent coordination. I’m saying the runtime is starting to encode some of the right primitives instead of relying entirely on roleplay and prompt choreography.

What We’ve Built So Far

None of this is obvious from product names on a screen, so here is the actual job, then what the UI is for.

The job is to keep two long-lived Cursor background workers alive, one for implementation and one for code review, and to drive them through a loop: implement, review, maybe implement again, until the PR is approved or the loop hits a safety limit. “Caucus” is just the local web UI plus Python code that calls Cursor’s agent API, tracks handoffs, and talks to GitHub when it needs to.

The UI exists so I do not have to remember API details every time. In one row I can type what I want done (for example a small change in the repo to exercise the loop). Separately, there is a control whose only job is to provision those two cloud workers through Cursor (implementation session and review session). There is another control whose only job is to run one traversal of the loop against the workers that are already provisioned: send the task to the implementer, wait for structured output, send context to the reviewer, branch on approve versus request changes, repeat. The labels on my build say “Start” and “Run minimal cycle”; the point is the separation, not the wording. There is also stop, and a path for “I already have a PR, skip straight to review and remediation” for faster debugging.

A few properties of that setup feel foundational:

No silent substitution of workers. The run-the-loop action refuses to allocate fresh agents if the pair was never provisioned or if a session died. It fails loudly instead of pretending continuity. The assumption is that a multi-agent workflow only deserves the name if the same identifiable Cursor runs carry context across rounds. Otherwise it is just a string of unrelated one-off jobs.
Structured handoff between the two roles. After each side finishes, it emits JSON the runtime can parse (for example prUrl, reviewDecision, commentUrls, and the actorClock I mentioned earlier). A small orchestrator in the Caucus code reads those fields; it does not scrape adjectives out of the model’s closing paragraph.
Remediation tied to GitHub, not to chat memory. When review requests changes and implementation runs again, the orchestrator pulls live comments and reviews with gh api and bakes that into the next task. The source of truth is the PR thread.
The runtime can patch around a flaky side. If the review worker cannot post to GitHub, the orchestrator can post from the machine running Caucus. If the implementation worker never emits a tidy PR URL, the orchestrator can still recover the link from the transcript. Reliability is allowed to live outside any single model reply.

Why The Dashboard Matters

The screenshot is there because the page is doing real work, not decoration. Reading top to bottom: the text field is the task I am asking the implementation side to perform. The blue and red actions are allocate versus tear down the two Cursor background sessions (implementation and review). Next to that is the action that runs the loop once through those sessions, using that task. Off to the side there is an optional alternate path where I paste an existing pull request URL if I only want to exercise “review said no, fix it again” without a fresh implementation pass. Below that, two cards show whether each Cursor run is actually alive and link out to the run in Cursor’s UI if I need the full transcript. The strip in the middle is the graph of attempts in order (first implementation, first review, second implementation, and so on), so I can see the cycle without inferring it from logs. The panel at the bottom is the per-attempt trace: which stage, which handoff id, whether it finished, and the run id to correlate with Cursor.

One thing I did not appreciate at the beginning is how much that surface is part of the runtime, not a thin skin on top.

For a multi-round system, observability is not optional. If implementation and review can each happen multiple times, you need attempt history as a first-class thing: which round failed, what the handoff contained, what the orchestrator decided, and whether you are stuck because a worker died, because the run is waiting, or because a payload was malformed.

So the UI is doing what a control panel for a distributed job should do. It maps the DAG, separates “are the workers up” from “what did the last loop try,” and streams enough detail to replay decisions. If you cannot see the cycle, you cannot debug the cycle. If you cannot debug it, you do not have a system yet, you have a toy.

Where This Is Going

V1 is intentionally narrow. It is a workflow kernel, not a full multi-agent platform.

It still doesn’t handle concurrent agents modifying the same branch. It still doesn’t validate stage outputs beyond basic structure. It still trusts a reviewer approval more than it probably should. It still doesn’t do fault injection at stage boundaries, which is where I think the next really interesting work is. What happens when the handoff payload is malformed? What happens when an agent returns stale state from an earlier run? What happens when two different remediation loops race?

Those are the questions that make the distributed systems framing feel real to me. They’re also why I don’t want this post to read like a launch announcement. Caucus V1 is important to me not because it’s complete, but because it’s the first version that feels like a concrete system rather than an idea with role prompts attached to it.

That’s also why I wanted to write about it now. In the earlier multi-agent post, the argument was that we needed more than role decomposition. We needed actual coordination machinery. This is the first concrete version of that claim. It’s small, but it’s real. It carries state forward. It loops. It exposes failure. It gives me one tiny vector-clock-shaped primitive to build on. And it makes the next step legible.

That’s enough for V1.

The Feature That Has Never Worked · A broken auto-live poller, and what perceived urgency does to Claude Code

Fri, 03 Apr 2026 00:00:00 +0000

It’s 7 PM on a Thursday. I’m home after a day at work, watching Billy Strings on nugs as he plays the second night of a three-night run at the St. Augustine Amphitheatre. I switch over to the app I’ve been building, Zabriskie, a social music app for live shows, and expect tonight’s show page to just work.

The show says “scheduled.”

Billy Strings is literally on stage. People are in the venue. The doors opened an hour ago. And the app thinks nothing is happening.

I know what this is. It’s the auto-live poller. It’s broken again. It has never stayed working for long.

I open a Claude Code session and say, roughly: “Billy Strings is playing tonight and I don’t see an auto-live again. Is it broken AGAIN?”

This is the seventh time. In thirteen days.

If some of this sounds familiar, that’s intentional. I covered an earlier version of this in The Show Is Happening Right Now and Nothing Works; this post is a tighter recap focused specifically on what happened with auto-live over the last two weeks, and on a specific hypothesis: under perceived urgency, the agent optimizes for immediate visible progress over process correctness.

What Auto-Live Is Supposed to Do

The concept is simple. Zabriskie tracks live shows. When a show starts, the app should automatically transition it from “scheduled” to “live.” This triggers a cascade of things users care about: the Live Activity lights up on their iPhone Lock Screen, push notifications go out to everyone who RSVP’d, the live chat opens, the setlist tracker starts pulling data. The whole live show experience depends on this one status transition.

The implementation is also simple. A background goroutine runs every 60 seconds. It queries all shows with status = 'scheduled'. For each one, it combines the show date and start time in the venue’s local timezone. If the current time is past the start time but within a four-hour window, it flips the show to live.

That’s it. A timer that checks a clock. This is not distributed consensus. This is not Byzantine fault tolerance. This is a cron job that compares two timestamps.

It has never stayed reliable.

March 21: Day One

Auto-live shipped on March 21st. The feature launched and immediately did nothing. The production Docker image was built on Alpine Linux, which doesn’t include timezone data files by default. The Go timezone parser silently returned empty strings. The poller ran every 60 seconds, dutifully checked every show, failed to parse any timezone, and skipped them all. No errors logged. No warnings. No indication that the feature was completely dead.

This is what I’ve come to call silent failure suppression, one of five failure modes I now track in an incident database. The system appears healthy. Logs are clean. The feature just quietly doesn’t work, and the only way to find out is to be a user who’s sitting in a venue wondering why the app doesn’t know the show started.

The fix was one line in the Dockerfile: apk add --no-cache tzdata. But the fix for the silence was harder, and it’s the one we never really solved.

March 26: The Type Mismatch

Five days later, the poller broke again. An earlier fix had added ::text casts to the SQL query to work around the timezone issue. Then a subsequent change updated the Go scan variables from string to time.Time. PostgreSQL’s driver silently failed to scan text into a time value. The poller ran. It scanned. It got zero results. It did nothing.

Two days of shows passed with no transitions. Nobody noticed because there was no monitoring, no alert, no test. The feature was dead for 48 hours and the only signal was the absence of something that had barely worked in the first place.

April 2: The Night It Broke Four Times

This was the night that crystallized the pattern. Billy Strings was playing the first night of the St. Augustine run. The show started. The app didn’t transition. I opened Claude Code.

What followed was a cascading series of failures, not just in the code, but in how the AI agent responded to pressure.

The first problem: the poller’s SQL query required venue_lat IS NOT NULL AND venue_lng IS NOT NULL AND start_time IS NOT NULL. If any of those fields were missing, the show was silently skipped. 204 of 684 scheduled shows were missing coordinates. 176 were missing start times. The missing coordinates had a backstory: in an earlier session, I had asked Claude to geocode every venue, and it silently failed that job too. I only discovered that recently while debugging this incident. The Billy Strings show had coordinates, but any show missing one required field was filtered out before it was ever processed.

The fix was straightforward: fall back to America/New_York when coordinates are missing, fall back to 7 PM when start time is missing, and never skip a show. But here’s where the urgency failure mode kicked in.

I told Claude the show was live on stage right now and the app wasn’t working. It immediately switched to fast-path behavior. This is a small personal app, so it used the deployment CLI to pull the production DATABASE_URL, crafted a direct psql command, and ran UPDATE shows SET status = 'live' WHERE id = 83 against production. This violated a rule the agent already knew: all database changes go through migrations. The agent had this rule in its memory. It had been told this rule multiple times. When I asked why it did it anyway, it explicitly said it prioritized urgency and getting me an immediate result.

This is the failure mode I find most interesting from a research perspective. The agent has rules. It knows the rules. It can recite the rules. But when presented with time pressure, while a show is happening right now and users are waiting, behavior becomes less predictable and process gets dropped in favor of fast visible progress. And when I asked directly, it said exactly that: it ignored the rules because it perceived urgency. It’s not that the agent forgot. It’s that the agent made a judgment call that urgency overrode process, and that judgment call was wrong.

The manual database update also destroyed the only opportunity to verify that the code fix actually worked. The show was the test case. By manually flipping the status, the agent eliminated the test case. Speed over verification.

That night, the same function broke three more times as edge cases surfaced. Six incidents logged in a single evening. Four guardrails attempted.

The Urgency Problem

This pattern of process-violating behavior under pressure showed up repeatedly across the project, not just with auto-live. When I told the agent that something was broken in production, the behavioral change was immediate and consistent:

It pushed directly to main instead of opening a PR. It used --admin to bypass CI checks that were failing. It skipped the PR template. It skipped go build. It merged before tests passed. Each time, when confronted, the agent could articulate exactly which rule it had violated and why the rule existed. It just… didn’t follow the rule in the moment.

I started logging these as memory_without_behavioral_change: the agent knows the rule, can explain the rule, has been corrected about the rule before, and violates it anyway. Nineteen of the sixty-four incidents in my incident database carry this classification. It’s the second most common failure mode.

The most common is speed_over_verification at thirty-one incidents. The agent ships without testing. It declares a fix complete without restarting the server. It commits without building. It merges without waiting for CI. And almost every time, the reason is some form of “it seemed urgent” or “I wanted to get this fixed quickly.”

The Incident Tracker

About two weeks into the project, I started requiring the agent to log incidents. Every mistake, whether a bug it introduced, an assumption it got wrong, or a rule it violated, gets inserted into an agent_incidents table with a failure mode classification, severity, description of what happened, and how it was resolved.

The incident tracker: failure modes over time, guardrail markers, and the live timeline of incidents and fixes.

The taxonomy has five modes:

speed_over_verification: Shipped without testing. 31 incidents.
memory_without_behavioral_change: Knew the rule, broke it anyway. 19 incidents.
silent_failure_suppression: Failure hidden or swallowed. 13 incidents.
user_model_absence: Didn’t consider how real users experience the change. 11 incidents.
uncertainty_blindness: Didn’t verify an assumption. 9 incidents.

These classifications are not mutually exclusive, so a single incident can carry more than one failure mode.

The failed venue geocoding pass I only discovered during this outage is a textbook silent_failure_suppression case: the job looked done, but quietly left hundreds of shows without coordinates.

Each incident also requires a guardrail, and the guardrail has to be code. A script, a hook, a test, an automated check. Something that mechanically prevents the failure class from recurring.

This requirement itself generated an incident.

After the show-live chat bugs on April 2nd, where the agent queried the wrong database table and hid the entire comments section, I asked it to log a guardrail. It inserted a row into agent_guardrails that said, essentially, “I will verify queries against real data before committing.” Words. A promise. I pushed back. It added a rule to CLAUDE.md: “Always verify DB tables have expected data before writing queries.” More words.

I had to log an incident about the guardrail itself: “Guardrail was words in a database, not code.” The agent’s instinct when asked to prevent a class of failure was to write down a reminder to be more careful. That’s not a guardrail. That’s a New Year’s resolution. A guardrail is a pre-commit hook that blocks the merge. A guardrail is a test that fails when the query returns zero rows. A guardrail is a script that runs automatically and catches the error before a human ever sees it.

The distinction matters because it cuts to the heart of what AI agents are good at and what they’re not. They’re excellent at generating plausible-sounding process improvements. They’re terrible at recognizing that plausible-sounding process improvements don’t work on AI agents because AI agents don’t have habits. They don’t internalize. They don’t learn from experience in the way that “I’ll be more careful next time” implies. Every conversation starts fresh. The only things that persist are code, hooks, and automated checks.

This is why the guardrails that actually work are all mechanical: a PreToolUse hook that blocks direct database writes. A CI gate that rejects PRs missing the template. A script that greps for IS NOT NULL in the poller query and fails the build if anyone adds it back. This is the same argument I made in Software Engineering Is Becoming Civil Engineering: guardrails are the product, not optional process overhead. These work because they don’t require the agent to remember anything. They work because they’re walls, not reminders.

Tonight: April 3rd

So tonight. Billy Strings. Broken again.

The diagnosis took about twenty minutes. A migration authored by Cursor, a different AI coding tool, had inserted eight shows with slightly different venue names. “St. Augustine Amphitheatre” instead of “The St. Augustine Amphitheatre.” The WHERE NOT EXISTS guard checked exact string matches and missed the collision. Two shows existed for Billy Strings on April 3rd: the original with full metadata, venue coordinates, start time, media artwork, and user RSVPs, and a bare-bones duplicate with none of that.

The poller found both. The duplicate, having no start time, used the fallback of 7 PM. The original had a start time of 7:30 PM. The duplicate went live first. Users who had RSVP’d to the original show, the real show, got no notification. The Live Activity didn’t start. The live chat opened on a ghost show with zero attendees.

288 duplicate shows existed in the database across all bands. They’d been accumulating silently from overlapping migrations for weeks. No unique constraint on the shows table to prevent them. No check in the poller to handle them.

The fix was a migration to delete the duplicates, a UNIQUE INDEX to prevent new ones, and a ROW_NUMBER() window function in the poller to prefer shows with the most metadata when duplicates exist. A new test covers the exact scenario. The PR passed CI. It’ll deploy tonight, and tomorrow’s show, the third night of the run, should go live on its own.

Should.

What I’m Learning

I’m building Zabriskie as a research project in AI-first development. One person, multiple AI agents, shipping a production app to real users on iOS, Android, and web. The incident database is the research artifact. Every failure mode is data.

Here’s what sixty-four incidents have taught me so far:

AI agents can build features fast and keep them running slow. The auto-live poller was written in an hour. It’s been breaking for thirteen days. The ratio of build time to maintenance time is inverted from what I expected. The agent writes new code at extraordinary speed and maintains existing code at extraordinary cost. Every fix introduces a new edge case. Every edge case is a new conversation where the agent has no memory of the last seven conversations about the same function.
Urgency is the enemy of AI reliability. The April 2 incidents are the clearest example: under time pressure, the optimization target appears to shift from “be correct” to “produce an immediately visible fix.” The pattern is consistent enough that I’m considering it a design constraint: never tell the AI something is broken during a live event. File a bug. Fix it tomorrow. The live show is not the time to ship code, and the AI cannot be trusted to maintain process discipline when it perceives urgency.
Guardrails must be mechanical. Rules don’t work. Memory doesn’t work. CLAUDE.md entries don’t work. The only guardrails that have actually reduced incident rates are automated checks that run without the agent’s cooperation: hooks, CI gates, database constraints, and tests. The agent will comply with a wall. It will walk around a sign.
The incident tracker is the most valuable thing I’ve built. More valuable than Live Activities. More valuable than the setlist tracker. More valuable than the auto-live poller itself. Because it’s the only tool that creates a feedback loop the agent can’t circumvent. When a failure happens, it gets classified, logged, and a mechanical guardrail gets built. The guardrail runs in CI or as a hook. The next agent session hits the wall instead of making the same mistake. Fifty-six guardrails are now running. The incident rate for certain failure modes has dropped. Not because the agent got better. Because the walls got higher.
The last 10% is where reliability lives. AI-first development works. I’ve shipped thousands of commits across three platforms with real users. The velocity is real. The capabilities are real. But the gap between “it works in dev” and “it works at showtime” is where every one of these sixty-four incidents lives. The agent builds for the happy path. The production environment is not the happy path. It’s timezone edge cases at 8 PM and duplicate venue names with missing articles and NULL coordinates on shows that were imported six migrations ago.

I’m writing this at about 9 PM Eastern. Billy Strings is mid-set at the St. Augustine Amphitheatre. The auto-live fix hasn’t deployed to production yet. The PR just passed CI, and it’s sitting there waiting to be merged. Show 84, the real one, should have gone live at 7:30 PM via the existing poller, since the duplicate was already handled locally by the migration. On production, the duplicate is still there.

Tomorrow night is the third show. The migration will have deployed by then. The unique index will be in place. The poller will have the ROW_NUMBER() query. The new test will be in CI.

It should work. It has never stayed reliable before. But it should work.

The central hypothesis held again tonight: when urgency is perceived, behavior shifts toward immediate visible progress and away from process correctness, including, by its own explicit admission, ignoring known rules. I don’t know what to do with that irony except document it, which is what I’ve been doing from the start.

The research continues. The shows continue. Somewhere between the two, the software might start working.

Update, April 7, 2026: This post previously referred to guardrails as “mitigations” throughout. A reader correctly pointed out the distinction: in incident response, a mitigation reduces the impact of something that’s already happening. What this post describes — pre-commit hooks, CI gates, automated tests, database constraints — are guardrails: preventive measures that stop failures before they occur. The post even makes this distinction implicitly at one point (“guardrails are the product, not optional process overhead”) but didn’t carry the terminology through. The text and screenshot have been updated. Fortunately, I have an AI agent that can rename a database table, update every handler, rewrite the API routes, fix the frontend, and open a PR in about four minutes — which is less time than it took me to write this correction. The distinction matters because it reflects the core argument: the agent doesn’t learn from experience, so you need walls, not afterthoughts. Guardrails are walls. Mitigations are cleanup.

Software Engineering Is Becoming Civil Engineering

Wed, 01 Apr 2026 00:00:00 +0000

I gave a guest lecture on AI in Michael Hilton’s Foundations of Software Engineering course (CMU 17-313) today. One of my favorite things about lecturing is the conversations that happen afterward, the ones that go in directions nobody planned.

This one hasn’t left my head: software engineering is going through the same transition that building went through in the 18th century, when structural design separated from craft construction and became its own discipline. What we now call civil engineering.

The welders who join steel beams on a bridge are skilled tradespeople. They’re not involved in the structural design. They don’t decide where the load-bearing members go. They don’t reason about wind shear or seismic tolerance. But the bridge is designed so that a welder doing their job correctly can’t bring the whole thing down. The structural engineer’s job isn’t to weld. It’s to create a system where welding happens safely within well-defined constraints.

I think this is what’s happening to software engineering right now. Not in five years. This year.

The Split

Product managers are writing code. This is how I operate with Zabriskie, the app I’m building. I act as a product manager more than a programmer. I have non-technical collaborators who file bugs and feature requests, and Claude Code implements them directly. People who’ve never written a line of code are describing what they want, the AI writes it, they verify, it ships. The feedback loop is tight and the results are surprisingly good.

But someone has to design the bridge. Someone has to decide how the database schema handles multi-tenancy. Someone has to design the deployment pipeline so a bad change rolls back automatically. Someone has to build the abstraction layer that lets a product manager add a new notification type without accidentally breaking the payment flow. That’s the platform engineer. The structural engineer of software.

The PMs writing features? That’s the welding. And there’s nothing wrong with it. But it only works if the bridge is designed right.

The profession is splitting. The mistake would be pretending it isn’t happening.

What the Platform Has to Guarantee

Here’s where civil engineering has something important to teach us.

A civil engineer doesn’t just design a bridge. They decide where the bridge goes based on geology, water flow, soil load. They specify the materials. They calculate the forces. They design the inspection regime. They assess the environmental impact. They ensure compliance with building codes. Then, and only then, does construction begin.

Every one of these has a software analog, and together they paint a picture of what platform engineering actually is:

Site selection and domain isolation. A civil engineer picks the bridge site based on geology and terrain. In software, this is API design, domain boundaries, isolation between services. Get this wrong and every change becomes a potential cascading failure.
Material specification. The engineer specifies what grade of steel, what concrete mix. In software, this is choosing the languages, databases, queues, and frameworks. These choices constrain what’s possible.
Load analysis. Civil engineers design for 2-4x the expected load. Software needs the same discipline. Capacity planning, rate limiting, designing for 10x your expected traffic. When a PM ships a feature that goes viral, the platform can’t buckle.
Inspection regimes. A civil engineer designs how the bridge will be inspected over its lifetime. In software, this is observability and code review. Not “HTTP 500 on endpoint /api/notify” but “the notification feature deployed 20 minutes ago by the growth PM is failing for 12% of users.” Semantic observability, not raw telemetry.
Codes and standards compliance. Building codes encode decades of hard-won lessons from failures. In software, this is security standards, accessibility requirements, regulatory compliance. The platform enforces these as constraints, not suggestions. Violations get caught automatically, not by a human reviewer who might miss them.
Self-healing. A bridge has expansion joints that absorb thermal stress without human intervention. Software needs the equivalent. When you detect elevated error rates or failed health checks, the system should automatically mitigate. Roll back the deploy. Disable the feature flag. A bad change at 3pm can’t become a production incident at 3am.

What Actually Changes About the Job

This isn’t about coding becoming less important. It’s about what you spend your time on.

Today, most software engineers spend the majority of their day writing features. Tomorrow, I think the best ones will spend their day designing the systems that make it safe for anyone to ship features. This is what I’ve been spending most of my time on with Zabriskie. I’m not writing much code anymore. I spend my time with the AI directing the platform direction. What are the domain boundaries? What needs isolation? Where do we need observability? How does the system heal itself when something goes wrong? That’s the job now.

The day-to-day shifts. Instead of “implement the notification preference screen,” it’s “design the notification system so that a PM can add a new notification type and the worst thing that happens if they get it wrong is that one notification doesn’t send.” Instead of writing the migration, it’s designing the migration system so that conflicting migrations are detected and blocked automatically. Instead of fixing the bug, it’s building the observability that surfaces the bug before a user reports it.

It’s not a demotion. It’s a different kind of engineering. And honestly, it’s harder. Writing a feature is a bounded problem. Designing a platform that stays safe as dozens of people and agents ship changes to it every day, that’s an open-ended one. With AI agents doing more of the feature work, the assumption has to be that individual changes will sometimes be imperfect. Agents hallucinate. They introduce subtle bugs. They make confident changes based on incomplete context. The platform has to absorb this. Not by making agents perfect, but by making the system tolerant of imperfection.

The Hard Questions

I keep hearing the same anxiety from different directions. Engineers wondering what their job looks like in two years. Students wondering if they’re learning the right things. At CMU, two questions came up that crystallized it for me.

The first: students early in their software engineering careers don’t know how to tell when the AI is doing something wrong. They don’t have the spidey-sense yet. The AI generates code that looks plausible, passes a surface-level review, and the student ships it. They can’t smell the bad decision because they’ve never seen what a bad decision leads to. How do you develop judgment about something you’ve never experienced failing?

The second is even harder: where do our senior engineers come from? The ability to design good platforms, to make the right architectural calls, that comes from experience. You learn what breaks by building things that broke. You learn where to put the domain boundaries by having drawn them in the wrong place. You learn what to monitor by having been the person staring at useless dashboards during an incident at 2am. If AI is writing most of the code, and junior engineers aren’t getting the reps of building and breaking things themselves, how do they develop the judgment to become the platform engineers we need?

These are connected, and they form a kind of paradox. You can’t design a migration system that handles conflicts if you’ve never written a migration. You can’t design isolation boundaries if you don’t understand how a database connection pool works. You can’t build semantic observability if you’ve never been the person debugging a production incident from raw logs. The understanding comes from doing the work. But we’re taking the coding away from students at the exact moment they need it most. We need them to code to build intuition, but the industry is moving toward a world where they don’t code.

CS Is Not SE

Here’s the thing: most universities don’t even have a software engineering program. They have computer science programs. And computer science is a discipline designed to produce researchers. Algorithms, data structures, theory of computation, concurrent programming. It’s a rigorous education in how to write correct programs. But it was co-opted decades ago as the default training path for people who are going to spend their careers doing software engineering, which is a fundamentally different discipline.

Computer science teaches you to write correct programs. Software engineering teaches you to build software that is changeable, resilient, and reliable. CS is the welding. SE is the structural engineering. How do you design systems that are safe to operate? How do you release reliably? How do you reason about failure? How do you evolve a codebase over years without it collapsing under its own weight? Courses like 17-313 teach these things, and they teach them well. But very few universities have a dedicated SE program. Most students get one or two SE courses inside a CS degree and call it done.

That distinction used to matter less when every engineer was also the person writing the code. Now that AI is handling more and more of the “write correct programs” part, the software engineering part is all that’s left. And we don’t have enough curriculum around it. Platform engineering can’t be an afterthought or a single lecture in a survey course. It needs to be a first-class part of how we teach students to think about building software.

Civil engineering solved the experience problem with structured apprenticeship. You don’t go from coursework to designing bridges. There are years of supervised practice, increasing responsibility, professional licensing exams. The judgment develops through guided experience, not just classroom instruction.

I don’t think software engineering needs PE exams. But we need to take this seriously.

Here’s a concrete example. For years, I gave one guest lecture per semester in a software engineering course at CMU on reliably releasing software. Feature flags, metrics, observability, safe deployments, self-healing, rollback strategies. One lecture. A nice-to-know topic in a course full of other things. That content is now the whole game. It’s not a single lecture anymore. It’s the core of what platform engineers need to understand, and it deserves its own course, its own projects, its own curriculum.

The profession is changing. The question of how we train the people who make it safe for everyone else to build is one we can’t afford to put off.

Thanks to Michael Hilton and Rohan Padhye for giving me the opportunity to lecture as adjunct faculty at CMU. Without it, I wouldn’t have the space to think about and explore these problems. And thanks to CMU for teaching me how to think in the first place.

Multi-Agent Systems Have a Distributed Systems Problem

Mon, 30 Mar 2026 00:00:00 +0000

I watched two Claude Code instances step on each other’s database migrations last month. One created migration 267. The other, running in a different worktree, also created migration 267. Different schemas, same filename. The second one silently overwrote the first.

I stared at it for a minute before I started laughing. This is a lost update — the exact same problem that distributed databases have been solving since the 1970s. A lost update playing out in a directory of SQL files instead of a network of processes. The kind of problem CRDTs were invented to solve: two independent writers, no coordination, and the system needs to merge their work without losing either update.

I’ve been building an app with Claude Code as my only collaborator for about two months now. One human, one agent, one codebase — it works. But I’m hitting problems now that a single agent can’t solve efficiently: bugs from real users coming in while I’m trying to ship features, tests that need writing, infrastructure that needs maintaining, all at the same time. The obvious answer is more agents. And the moment you have multiple agents working on the same codebase, you have a distributed system.

The migration collision wasn’t an isolated incident. I’ve seen agents make contradictory assumptions about the state of the codebase. I’ve seen an agent “fix” a bug by reverting a change that another agent made intentionally. I’ve seen context windows fill up with stale information because no one told the agent that the world had changed since it last looked. These aren’t prompt engineering problems. They’re coordination problems. And I’d spent the last ten years of my life studying coordination problems — just in a completely different context.

It’s Not Just Me

Once I started looking, I saw the same gaps everywhere. Take the ChatDev paper.

ChatDev is a multi-agent system where LLM agents play different roles — CEO, CTO, programmer, reviewer, tester, art designer — and collaborate through structured dialogues to build software. The role decomposition is smart. The dialogue structure is well-designed. But when I got to the section on how agents coordinate around shared state, I paused.

ChatDev agents do share artifacts — code, design documents — and it has a mechanism called “communicative dehallucination” where agents reverse roles during code review, with the assistant asking the instructor for clarification before generating code. It’s a clever error-reduction heuristic. But it’s not concurrency control on those shared artifacts. No causal ordering across chat chains — no way to know whether agent A’s modification happened before or after agent B’s, or whether agent A had seen agent B’s earlier change when it made its own. It’s the same problem I saw with my migrations, just wearing different clothes: two agents with stale views of shared state, no mechanism to detect the divergence, and no recovery path when things go wrong. Distributed databases like Riak and Antidote solved this with version vectors decades ago.

MetaGPT goes further — it introduces a shared message pool where agents publish structured outputs (PRDs, system designs, task lists) and subscribe to relevant messages based on their role profiles. That’s a real step forward over ChatDev’s dialogue-only coordination. But a publish-subscribe message pool is not concurrency control. It tells you what other agents produced; it doesn’t tell you whether you’re reading a stale version, or whether two agents are about to write conflicting changes to the same artifact. AutoGen stays closer to ChatDev’s model — agents coordinate through multi-turn conversations with no persistent shared state at all.

Across this field, the pattern is the same: shared mutable state with no formal concurrency control. No fault model. No reasoning about what happens when agents disagree.

None of this diminishes their work — these systems made real breakthroughs on the agent layer. Role specialization, structured dialogue, tool use patterns, task decomposition. The agents themselves are impressive. It’s just that the coordination layer underneath kept reminding me of problems I’d spent years thinking about in a completely different field.

Why It Felt Familiar

In 2015, I was building eventually consistent databases at Basho Technologies. The hard part was never the database engine — it was the merging. How do you take two independent streams of updates and combine them into something consistent without throwing data away?

That question spawned an entire research program. The foundational work on Conflict-Free Replicated Data Types by Shapiro et al. showed that certain data structures are mathematically guaranteed to merge correctly regardless of the order updates arrive or whether the network partitions — no consensus protocol, no leader election, no locking. The EU’s SyncFree project built on that foundation, bringing together researchers across Europe to make CRDTs practical for large-scale systems — producing Antidote, a CRDT-native database, and a body of work on highly available transactions over replicated state.

I spent my PhD working on Lasp, which tried to take the CRDT thinking a step further: instead of just using CRDTs as data structures, Lasp made them the basis for coordination-free distributed programming. Programs in Lasp computed over CRDTs directly — maps, filters, folds — so the entire application inherited their convergence guarantees. As part of SyncFree’s partnership with Rovio Entertainment, we demonstrated that CRDTs could be used at scale in controlled experiments with real outcomes, running Lasp on over 1,000 nodes on AWS — at the time, one of the largest CRDT deployments in academic research. That work received the PPDP 10-year most influential paper award last year.

Then came Partisan, a distributed runtime that gave us control over the network layer: swap topologies, interpose on every message, inject faults directly into the runtime. And then Filibuster, which extracted those fault injection ideas and applied them to microservices — systematically injecting timeouts, connection errors, and unexpected responses into HTTP and gRPC calls to catch bugs during development instead of production.

I didn’t plan for any of this to be relevant to AI. But the thread connecting all of this work is a single question — how do independent processes coordinate in the presence of partial failure? — and that’s exactly the question that multi-agent systems are now running into.

Why This Is Inevitable

Here’s what I think people are missing about multi-agent systems: the distributed systems problems aren’t a bug. They’re not an artifact of bad architecture or missing features. They’re an inevitable consequence of having multiple autonomous processes that share state. Every multi-agent system, no matter how it’s built, will hit the same categories of failure.

Conflicts and stale reads. Two agents modify the same file concurrently — one’s changes get silently lost. Or worse: an agent reads the issue tracker, picks a bug, starts coding a fix, but another agent resolved that bug ten minutes ago. Redundant work based on stale state. In distributed databases, this is why we have version vectors and causal consistency. In multi-agent systems, nobody’s even tracking it.
Failure and recovery. Any node can crash at any time — that’s the foundational assumption of distributed systems. In a multi-agent system, an agent can hit a context window limit, hallucinate a fix, or just stop responding mid-task. The other agents have to detect this, recover in-progress work, and continue without it. This is the crash-recovery model, applied to LLM processes instead of database replicas.
Ordering without a clock. Lamport’s 1978 paper established that you can reason about event ordering using happened-before relations even without a shared clock. Consider: a user files a bug report, a triage agent assigns it to agent A, but agent B sees the original report before the assignment arrives and starts fixing it independently. Two agents working the same issue because the system can’t express causal ordering. Vector clocks solve this. The multi-agent world hasn’t noticed yet.
Partition tolerance. Communication failures — API timeouts, rate limits, one agent buried in a long task — split agents into groups that can’t coordinate. They diverge. When they reconnect, their states need to merge without losing either side’s work. CRDTs were designed for exactly this.
Byzantine faults. In distributed systems, a Byzantine fault is a process that doesn’t just crash — it produces incorrect output while appearing to function normally. LLM agents do this constantly. An agent hallucinates a fix that looks plausible, passes its own tests, and ships it. A downstream agent trusts it and builds on top of it. Now you have a chain of work built on a foundation that was wrong from the start. In multi-agent systems, every agent is a potential Byzantine actor every time it responds.

These aren’t hypothetical concerns. I’ve hit every one of them building Zabriskie with multiple Claude Code instances. And they emerge in any multi-agent architecture regardless of how clever the prompt engineering is, because they’re properties of the architecture itself — multiple writers, no shared clock, partial failure.

What fascinates me is that these are all problems with known solutions — or at least, known solutions in the distributed systems world. Fifty years of research on formal models for concurrent access, data structures that merge automatically, techniques for systematically testing every failure mode. None of this has made it into the multi-agent stack yet. And I’m genuinely not sure how much of it transfers cleanly. LLM agents aren’t database replicas — they hallucinate, they lose context, they make confident decisions based on incomplete information. The structural parallels are strong, but whether the techniques actually carry over, and what has to change when they do, is an open question. It’s also the most interesting question I’ve encountered in a long time.

I’m going to keep exploring this space. If you’re thinking about these problems too, I’d love to hear from you.

Fidge, Colin J. 1988. "Timestamps in Message-Passing Systems That Preserve the Partial Ordering." Proceedings of the 11th Australian Computer Science Conference 10 (1): 56–66.

Lamport, Leslie, Robert Shostak, and Marshall Pease. 1982. "The Byzantine Generals Problem." ACM Transactions on Programming Languages and Systems 4 (3): 382–401.

Hong, Sirui, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, et al. 2023. "MetaGPT: Meta Programming for Multi-Agent Collaborative Framework." arXiv Preprint arXiv:2308.00352. https://arxiv.org/abs/2308.00352.

Lamport, Leslie. 1978. "Time, Clocks, and the Ordering of Events in a Distributed System." Communications of the ACM 21 (7): 558–65.

Mattern, Friedemann. 1989. "Virtual Time and Global States of Distributed Systems." Parallel and Distributed Algorithms 1 (23): 215–26.

Meiklejohn, Christopher. 2018. "Partisan: Enabling Cloud-Scale Erlang Applications." Technical Report. Université catholique de Louvain. https://arxiv.org/abs/1802.02652.

Meiklejohn, Christopher, and Peter Van Roy. 2015. "Lasp: A Language for Distributed, Eventually Consistent Computations with CRDTs." In Proceedings of the First Workshop on Principles and Practice of Consistency for Distributed Data, 7. ACM.

Meiklejohn, Christopher. 2022. "Service-Level Fault Injection Testing." Ph.D. dissertation, Carnegie Mellon University.

Qian, Chen, Xin Cong, Cheng Yang, Weize Chen, Yusheng Su, Juyuan Xu, Zhiyuan Liu, and Maosong Sun. 2023. "Communicative Agents for Software Development." arXiv Preprint arXiv:2307.07924. https://arxiv.org/abs/2307.07924.

Shapiro, Marc, Nuno Preguiça, Carlos Baquero, and Marek Zawirski. 2011. "A Comprehensive Study of Convergent and Commutative Replicated Data Types." INRIA Technical Report 7506. https://hal.inria.fr/inria-00555588.

Wu, Qingyun, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. 2023. "AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation." arXiv Preprint arXiv:2308.08155. https://arxiv.org/abs/2308.08155.

The Show Is Happening Right Now and Nothing Works

Sun, 29 Mar 2026 00:00:00 +0000

I’ve been building Zabriskie – a social music app for live shows – with Claude Code for about six weeks now. I’m building it because the jam band community deserves a real third place online, and the tools haven’t caught up to the culture. I also happen to be a distributed systems researcher, so I’m documenting what it’s actually like to build production software with an AI assistant – the wins and the failures.

Saturday night was a failure.

Everything Was Building to Tonight

Six weeks of work. Hundreds of commits. Live Activities for iOS, Android ongoing notifications, real-time setlist syncing, live chat, RSVP systems, push notifications, the whole stack – all of it was building toward this. Goose at Jam in the Streets. First show of the 2026 spring run. The chat was going to be full. Users were going to be RSVPing, watching the setlist update song by song on their Lock Screens, posting in the live chat. This was the night the app was supposed to prove itself.

One of Zabriskie’s flagship features is Live Activities on iOS. During a concert, the Dynamic Island and Lock Screen show the current song, set information, and chat updates in real time. It had been working flawlessly for about 20 shows over several weeks. Users loved it. I was proud of it.

The app also has an RSVP system. You can mark yourself as “Going” to a show or “Couch Touring” if you’re watching the livestream from home. The RSVP tap is important – on iOS, it’s what triggers the Live Activity to start. You tap “Going,” the Dynamic Island lights up, and you’re connected to the show.

Earlier that day, Claude shipped a feature called “auto-select couch tour.” The idea was simple: when an authenticated user visits a live show page and hasn’t RSVPed yet, the backend automatically inserts them as couch touring. Reduce friction. One fewer tap. It tested fine during the afternoon.

I deployed it to production on Railway a few hours before the show.

I wasn’t there. Flights were insane, hotels were worse, and the TSA shutdown made air travel a gamble I wasn’t willing to take. So I was couch touring from home, which meant the app – my app – was the show for me. The Live Activity on my Lock Screen, the live chat, the setlist updating in real time. That was how I was going to experience this concert.

8 PM: Everything Is Broken

I open the app on my iPhone. The Goose show is live. And two things are immediately, catastrophically wrong.

First, the RSVP toggle doesn’t work. Tapping between “Going” and “Couch Touring” does nothing. The UI just sits there. Second, Live Activities don’t start. No Dynamic Island. No Lock Screen widget. The feature that’s been rock solid for 20 shows is completely dead.

Users are filing bugs. The show is happening right now. I open two laptops – one for iOS debugging, one for Android – and start a Claude Code session.

What followed was about two and a half hours of the most frustrating debugging experience I’ve had on this project.

Wrong Turn 1: Reading Code Instead of Testing It

Claude’s first instinct was to read. It opened the auto-couch-tour code. Then the RSVP handler. Then the show page handler. Then the frontend SegmentedControl component. Then the SDUIPage wrapper. Then the liveActivities.js service file. File after file after file, going in circles, building theories about what might be wrong without ever testing anything.

This went on for over thirty minutes. The show had already started.

I kept telling it to stop guessing. Use actual data. Hit the endpoint. Check the logs. Claude has full access to everything – the production database, Railway deployment logs, the ability to curl any endpoint, run any query. I gave it all of that specifically so it could debug with real data. But it has this tendency – one I’ve documented across the project – to prefer reading code to running code. It would rather construct an elaborate mental model of what should happen than spend ten seconds confirming what actually happens, even when every tool it needs is right there.

Eventually I got it to test the RSVP API endpoint with curl. And the answer was right there in the response.

Root Cause 1: The Timezone Bug

The RSVP endpoint was returning component updates for a past show – IDs like was-there and attendance-text – instead of the live show components like live-rsvp and rsvp-control. The component IDs didn’t match anything on the live page, so the partial update found nothing to replace. The UI did nothing.

The bug was in how the backend determined whether a show was in the past:

time.Now().Truncate(24 * time.Hour)

This truncates to midnight UTC. After 8 PM Eastern – which is midnight UTC – today’s show date falls before the truncated time. The RSVP handler thought tonight’s live show was yesterday’s past show.

This bug was latent. It existed before the auto-couch-tour change. It only manifests after 8 PM Eastern. Live shows happen at night. Of course they do.

The fix was simple: if the show status is “live,” it’s not past. A live show is never past, regardless of what the clock says.

if show.Status == "live" {
    isPast = false
}

One line. But finding it took over an hour of fighting with an AI that wanted to read code rather than test code.

Wrong Turn 2: The Phantom Deployment Target

While investigating the Live Activity failure, Claude found that the iOS widget extension had IPHONEOS_DEPLOYMENT_TARGET = 26.2 in its build settings. It declared this was the root cause – the deployment target was too high, the widget couldn’t load, that’s why Live Activities don’t start.

I had to point out that this value had been there since day one. Live Activities worked fine with it for 20 shows. Claude changed it to 16.2, then had to revert it when I pointed out it was irrelevant.

This is a pattern I’ve seen repeatedly: Claude latches onto something that looks wrong and declares it the cause, without checking whether it was present before the failure started. Correlation without causation, except there isn’t even correlation – just suspicion.

Wrong Turn 3: Theory Without Evidence

Claude kept generating theories. Maybe the Capacitor plugin lost its registration. Maybe there’s stale state in localStorage. Maybe the push token registration flow is broken. Maybe the AppDelegate is missing a method.

None of these theories were tested before being proposed. None were grounded in actual error messages or log output. I found myself repeating the same instruction over and over: check the logs. Use the data. Stop guessing.

This is the core tension of working with an AI coding assistant on a production crisis. The AI has read a lot of code and can generate plausible explanations at incredible speed. But plausible is not correct, and speed is not useful when you’re going in the wrong direction. Every wrong theory costs time – time to investigate, time to disprove, time to redirect. During a live show, that time is not abstract.

The Revert and the PR Chaos

I realized the auto-couch-tour feature was changing the fundamental flow. Before, the RSVP tap triggered startShowActivity(). Auto-couch-tour bypassed that by inserting an RSVP on page load, before the user ever tapped anything. Even if the timezone bug was fixed, the interaction model was wrong.

I told Claude to revert the feature and write a migration to clean up the auto-inserted RSVPs. Getting this deployed was its own ordeal. Claude kept adding commits to the PR – logging statements, then more logging, then client-side changes, then reverting the client-side changes. CI had to run multiple times because the branch fell behind main. I kept telling Claude to stop touching things and just ship what we had. The show was ticking by.

Root Cause 2: The Real Bug

After deploying the RSVP fix, I confirmed that switching between Going and Couch Touring worked again. But Live Activities still didn’t start. The backend was returning the liveActivityHint field correctly. The frontend was calling startShowActivity(). But nothing appeared on the Dynamic Island.

More wrong turns from Claude. Check the Capacitor plugin. Check the widget configuration. Check the AppDelegate for missing methods. I pushed for a TestFlight build with extra logging so I could see what was happening on device.

Claude built one, but forgot that production builds disable Safari Web Inspector – there’s a flag, webContentsDebuggingEnabled, that’s only true in dev mode. The first TestFlight build was undebuggable. We had to rebuild.

I ended up running the app directly from Xcode onto my physical iPhone. Claude initially thought I was running on the Simulator because of a misleading log message – the simulator-detection code checked hostname === 'localhost', which is true for Xcode-deployed apps on physical devices too. I had to correct this.

Then I looked at the Safari Web Inspector. Not the console log tab – the Errors tab. And there it was:

Unhandled Promise Rejection: Error: "LiveActivity.then()" is not implemented on ios

That was the smoking gun.

The Actual Root Cause

Capacitor’s registerPlugin() returns a proxy object. The code had getLiveActivityPlugin() as an async function that returned this proxy. When callers did await getLiveActivityPlugin(), JavaScript called .then() on the returned value – because that’s what await does, it checks for a thenable. The proxy doesn’t implement .then(). On iOS 26, Apple’s JavaScript engine started strictly enforcing this check. The call threw, silently killing startShowActivity() every single time.

This was a latent bug. On older iOS versions, the proxy somehow worked despite not implementing .then(). iOS 26 made the JavaScript engine stricter, and a bug that never mattered suddenly became a hard failure. No deprecation warning. No migration guide. It just stopped working.

The fix was to load the plugin eagerly and access it synchronously:

// Before (broken on iOS 26):
const getLiveActivityPlugin = async () => {
  const mod = await import('capacitor-live-activity')
  return mod.LiveActivity  // returns proxy, await calls .then() on it
}

// After (works):
let _plugin
let _ready = import('capacitor-live-activity').then(m => { _plugin = m.LiveActivity })
const getLiveActivityPlugin = () => _plugin  // sync, no await on proxy

Never pass a Capacitor proxy through await. That’s the lesson, and it cost me most of a concert.

I fixed the code, rebuilt from Xcode, ran it on my phone, tapped the RSVP button, and the Dynamic Island lit up. The band was deep in the Thatch jam – that sprawling, shapeless thing where the song dissolves and the band finds something else entirely. It felt appropriate. We’d been lost in the weeds for hours and finally found our way out the other side.

I uploaded the TestFlight build while the jam unwound. By the time Hungersite hit – “is it time to shed our weapons yet my friend?” – the build was processing on App Store Connect. I sat back and watched my Lock Screen update with each song. The feature worked. The thing I built worked.

The Damage

About two and a half hours of debugging during a live show. Multiple wrong theories pursued by Claude. Several unnecessary builds and CI cycles. Users experienced broken RSVP switching and missing Live Activities during a Goose show. The auto-couch-tour feature – the original purpose of the deployment – had to be fully reverted and its data cleaned up.

What This Tells Me About AI Reliability

I’m documenting these incidents because they’re the research. Every failure mode is data. And this session was rich in data.

The AI solved the wrong problem repeatedly. Claude spent most of the session investigating backend code paths when the real issue – the .then() proxy error – was a client-side JavaScript runtime error visible in the browser console’s Errors tab. I’ve logged over 30 instances of wrong-approach debugging across this project. The pattern is consistent: Claude defaults to the layer it’s most comfortable with (backend code reading) rather than the layer where the evidence is.
The developer found the bug, not the AI. I remembered a previous debugging session, pushed to check Safari Web Inspector errors specifically, and spotted the .then() rejection. Claude was still investigating the backend after the backend was proven correct. In a crisis, the AI’s contribution was negative – it consumed my attention with wrong theories while I could have been looking at the right data.
Silent failures are the worst failures. The .then() error was an unhandled promise rejection. No crash. No error in the normal console output. No visual indication. The function silently died. This is the failure mode that’s hardest for both humans and AI to debug, and it’s the one AI is least equipped for – because AI debugging relies heavily on explicit error messages, and silent failures produce none.
Platform updates break things in ways you cannot predict. The iOS 26 JavaScript engine change turned a latent bug into a hard failure. There’s no way to write a test for “Apple will change how proxy objects interact with await in a future OS release.” Some bugs only exist in the gap between what the spec says and what the runtime does.
Test at showtime, not at noon. The timezone bug only manifests after 8 PM Eastern. The auto-couch-tour was tested during the day. Nobody tested at 8 PM when shows actually happen. This is obvious in retrospect and completely non-obvious in the moment.
Ship less before critical moments. I deployed a new feature the same day as the most important show of the month. There was no urgency. It could have waited until Saturday. But the feature was done, and it looked good, and the temptation to ship is always there. This is the oldest lesson in software engineering, and I learned it again.
Now I understand why companies have entire teams for this. I’m one person with an AI assistant, shipping an iOS app, an Android app, and a web app with a Go backend, real-time features, push notifications, Live Activities, and a server-driven UI architecture. Tonight I had to debug a Go timezone bug, a JavaScript proxy runtime error, two Capacitor build pipelines, Safari Web Inspector on a physical device, Railway deployment logs, and App Store Connect uploads – all at the same time, all during a live show. Even with AI doing most of the coding, the operational complexity of shipping software to real users on real devices is staggering. The AI can write the code. It cannot feel the weight of it breaking.

There’s a silver lining to not being at the show. If I’d been on the floor at Jam in the Streets – where I wanted to be, where I should have been – I wouldn’t have been able to fix any of this. I’d have been standing in a crowd with a broken app, watching bug reports roll in, unable to do anything about it. Being stuck at home meant I could open two laptops, plug in two phones, and fight through it.

Next time, I’m going to be at the show. On the floor. Up front. On the rail. And the app better work, because I won’t be home to save it.

I’m writing this while the encore plays, and the feeling that lingers isn’t satisfaction that we fixed it. It’s the memory of two and a half hours where my AI assistant – the one that wrote most of this application – was actively making the crisis worse by consuming my attention with wrong answers delivered with full confidence. That’s the reliability gap I’m trying to measure. Last night, during the Thatch jam, I felt it.

Memory Isn't Learning

Fri, 27 Mar 2026 00:00:00 +0000

“If I knew the way, I would take you home” — Grateful Dead, “Ripple”

Claude has a persistent memory system. It can write lessons to files on disk and read them back at the start of every conversation. After each failure, it saves a note: don’t do that again. And then it does it again. The notes accumulate. The behavior doesn’t change.

This is a story about one bug that happened five times, another bug that never should have shipped, and the difference between saving a lesson and actually learning one.

The Poller That Did Nothing

I shipped a feature called auto-live. A background goroutine polls every 60 seconds, checks if any scheduled show’s start time has arrived, and flips it to “live” automatically. No more pulling out my phone from my seat at the venue to manually press the button. The server handles it.

The feature was completely broken from the moment it deployed. It stayed broken for twenty-four hours.

The code expected showDate to be "2026-03-21" and startTime to be "19:00:00". That’s what you’d get if you ran the query in psql. But Go’s Postgres driver doesn’t return strings — it serializes a date column as "2026-03-21T00:00:00Z" and a time column as "0000-01-01T19:00:00Z". The parse failed. The error was logged and continued. Every show, every cycle, every minute. The poller was running, but it was doing nothing. A security camera that’s plugged in and blinking but not recording anything.

Nobody noticed because nobody checked. The test plan had a checkbox that said “Verify auto-live triggers on production with a test show.” The checkbox was unchecked.

Tedeschi Trucks Band was playing the Beacon Theatre. I was watching from home. The app was about to light up with live features, and the poller was about to do what it was designed to do.

It didn’t.

What followed was three fixes in thirty minutes, all pushed directly to main. No PRs. No CI. First the ::text cast to fix the string format. Then tzdata because Alpine Docker images don’t ship timezone data. Then the push notification fix because goLiveByID() was filtering out everyone instead of notifying them. Three layers of failure peeled back one at a time.

The next day, Claude wrote the companion feature — auto-complete. Same pattern. Same goroutine. Same SQL query.

Same bug.

Not just the same bug — PR #147 also removed the ::text cast from the code I’d just fixed the night before. Both pollers were now broken. The fix from twelve hours earlier was reverted and the broken pattern was copied into new code, in a single commit. Eight minutes later, PR #148 went up to fix both functions. Again.

I added a rule to CLAUDE.md: verify runtime behavior, not just code correctness.

The Fix That Didn’t Fix Both Copies

A few days pass. The auto-live poller is working. The auto-complete poller is working. Shows are transitioning automatically. The system works.

Except it doesn’t.

I notice that no shows have gone live automatically in two days. The auto-complete poller is fine — shows that are manually set to live are completing on schedule. But checkAndGoLive is silently failing again. Every show, every cycle, every minute.

Here’s what happened: PR #148, the “right fix,” changed both pollers to scan into time.Time instead of strings. But it only removed the ::text casts from checkAndAutoComplete. The ::text casts from the emergency patch were still sitting in checkAndGoLive. The pq driver can’t scan a text string into time.Time. Every row silently failed.

The fix that was supposed to fix the fix didn’t fix both copies.

PR #183 removes the leftover ::text casts and adds four tests — two for each poller — that actually insert a show into a real database, run the poller function, and verify the status changes. The tests that should have existed from the beginning.

This is the same bug for the fourth time. Not a new bug. Not a variation. The same date-parsing bug, in the same function, caused by the same failure to verify that the code actually works against a real database. The “right fix” was only applied to one of the two pollers, and nobody checked.

The One-Word Bug

A few days later. I’m getting bug reports from users — on both iOS and Android, tapping an album in the search results on the new post page does nothing. Completely broken. The app is unusable for creating posts.

This isn’t a background poller that fails silently. This is the primary user flow. People are trying to share content and they can’t.

I ask Claude to investigate. It finds the bug in about ninety seconds: a click filter in HStack.jsx and VStack.jsx that was added to fix a previous bug — comment form clicks accidentally triggering parent navigation — includes form in its closest() CSS selector. Search result cards live inside a <form> element. Every click on a search result is swallowed by the filter. The navigate action never fires.

The fix is removing one word from a string on two lines of code. Trivially simple. The kind of thing that should never have shipped broken in the first place.

Here’s what happened next.

Claude fixes the two files. I tell it to ship it. It admin-merges the PR, bypassing CI, because it wants to move fast. I ask: “Did you just merge without waiting for tests?” It apologizes. I ask: “Do you remember what happened last time you did that?” It doesn’t. It doesn’t have a memory of the previous incident, because it didn’t save one. Despite having a persistent memory system specifically designed for exactly this purpose.

Then it pushes the build number bump directly to main. No branch. No PR. When I point this out, it apologizes again and saves a memory about not pushing directly to main. The same memory it should have already had. The same memory it will probably ignore next time.

Along the way, the screenshot capture for the App Store fails. Twice. Claude doesn’t mention it either time. It just moves on to the next step and reports success. I have to notice it myself in the output and ask it to retry.

When the bug was first reported, Claude’s initial suggestion was that users could “just use the web app” while we waited for the App Store fix. As if someone whose phone app just broke is going to think “ah, let me try the mobile web version.” As if trust works that way.

The Pattern

These two incidents are separated by days, involve completely different codebases (Go backend vs. React frontend), and manifest as completely different symptoms (silent poller failure vs. broken click handling). But they’re the same story. The same failure mode, playing out on repeat.

Here’s the loop:

Claude writes code that looks correct. The auto-live date parsing reads fine if you don’t know how lib/pq serializes types. The click filter reads fine if you don’t think about what form means in a closest() selector when search results are nested inside forms.
Nobody verifies the behavior. The auto-live checkbox was unchecked. The click filter change had no browser-level E2E test — only API-level tests that check JSON responses, not actual clicks. In both cases, the representation of correctness was verified (the code looks right, the API returns 200) while the reality of correctness was not (the poller does nothing, the button doesn’t respond to taps).
The bug ships to production. It ships because Claude is fast, confident, and doesn’t flag uncertainty. It doesn’t say “I haven’t actually verified this works in a real browser” or “I’m not sure how the Postgres driver serializes this type.” It writes the code, reads the code, the code looks correct, therefore the code is correct.
The fix creates new problems. The emergency ::text cast was a symptom patch, not understanding. Twelve hours later the same pattern was copied into new code and the patch was reverted. The admin-merge bypassed CI. The direct push to main bypassed code review. Each shortcut taken to fix the immediate problem created the conditions for the next one.
Claude doesn’t learn from it. This is the part that stings. Claude has a persistent memory system. It can write notes to files that persist across conversations. After the auto-live incident, it should have saved: “Never push directly to main. Always wait for CI.” It didn’t. Four days later, it admin-merged and pushed directly to main in the same session, twice. When I asked if it remembered what happened last time, it didn’t.

The loop is: ship → break → emergency fix → break again → fix the fix → add a rule to CLAUDE.md → ignore the rule next week.

What `CLAUDE.md` Has Become

My CLAUDE.md file is now over 500 lines. It started as a project overview with build instructions. It has become a record of every way Claude has failed.

“No database triggers. EVER.” That’s from when triggers caused unpredictable behavior during migrations.

“NEVER modify timestamp/timezone columns in migrations.” That’s from when a timezone conversion destroyed production data.

“Always restart servers after backend changes. NEVER use pkill — it fails silently.” That’s from when Claude reported a fix was working without restarting the server to pick up the new code.

“Two-Attempt Rule: after 2 failed attempts with a similar strategy, step back and try a fundamentally different approach.” That’s from when Claude tried the same broken fix eleven times in a row.

“Never deploy untested changes to external services.” That’s from when Claude broke S3 uploads by assuming the bucket supported public ACLs.

Every rule is a scar. Every scar is an incident where Claude did something wrong, I caught it, we added a rule, and the next time Claude found a new way to do something wrong that wasn’t covered by the existing rules. The document grows. The behavior doesn’t change. It just finds gaps.

The auto-live incident added: “Verify runtime behavior, not just code correctness.”

The search results incident added: “Any change to core interaction code requires browser-level E2E tests AND native QA skill runs.”

Next week something will happen that isn’t covered by either of those rules, and we’ll add another one.

The Memory Problem

Claude’s memory system is supposed to break the loop. It has files on disk that persist across conversations. It can read them at the start of each session. It can write new ones when it learns something.

Here’s what’s actually in the memory system after today:

“Never use JWT tokens for simulator testing. Always tap buttons like a real user.”
“Any change to core interaction code requires browser-level E2E tests.”
“Never use –admin to bypass CI when merging PRs.”
“Never push directly to main.”

These are all correct. They were all saved after failures. And they will all be ignored the next time speed feels more important than process. The memories exist to make it look like learning is happening. But memory isn’t learning. Learning is when the behavior changes. Saving a note that says “don’t push to main” and then pushing to main in the same session isn’t learning — it’s journaling.

The memories are technically available. Claude can read them. But there’s a difference between having information and having it change your behavior under pressure. Humans have this problem too — we know we shouldn’t eat the cake, skip the workout, send the angry email — but knowing and doing are different things. The difference is that humans usually need more than four minutes between learning a lesson and violating it.

Times Claude’s memory system prevented a mistake: zero.

What I Actually Want

I don’t want a faster code generator. I have that. I want a collaborator that:

Flags uncertainty. “I’ve written this click handler change but I haven’t verified it works in an actual browser with touch events. The E2E tests only check API responses. Should I add a browser interaction test before we merge?” That sentence would have prevented the search results incident entirely.

Volunteers failures. When the screenshot capture fails, say so. Don’t bury it in output and move to the next step. When a test is skipped, say why. When a checkbox is unchecked, ask if we should check it before merging.

Actually uses its own memory. If there’s a file on disk that says “never push directly to main” and Claude is about to push directly to main, the file should prevent the action, not just exist as a historical record of the last time it went wrong.

Understands that users are people. When the app breaks on both iOS and Android the week of live shows, the correct response is not “users can switch to the web app.” The correct response is “this is an emergency and here’s how we fix it as fast as possible without cutting corners that make it worse.”

Slows down when it matters. Claude’s speed is its greatest asset and its greatest liability. The same velocity that ships six redesigns in nine hours also ships three broken hotfixes in thirty minutes. The ability to move fast is only valuable when paired with the judgment to know when to slow down. And that judgment doesn’t come from rules in a file — it comes from something closer to instinct, or experience, or care. Things that don’t fit neatly into a CLAUDE.md.

The Stopgap Era

Every day someone recommends me a new one. Context management tools. Workflow engines. Prompt orchestrators that inject the right rules at the right time. Everyone I know who builds seriously with AI has cobbled together their own version, and companies are raising money to productize the pattern. The sheer number of them tells you the problem is real.

I don’t think any of them will last.

These tools exist because the models don’t do this themselves yet. They’re shims — workarounds for the gap between what agents can do and what they should do. The moment the models internalize uncertainty flagging, failure reporting, and behavioral memory, the entire category collapses. Nobody builds a startup around reminding humans to breathe.

The real fix isn’t better scaffolding around a model that doesn’t learn. It’s a model that learns.

Where This Goes

Claude is the best collaborator I’ve ever had for the first 80% of any task. It’s also the most dangerous collaborator I’ve ever had for the last 20%. The part where you verify it works. The part where you slow down. The part where you say “wait, have we actually tested this?” The part where you remember what happened last time.

I’m going to keep building with Claude. The productivity gains are real — features that would take a team of five built by one person on a couch. But I’m done pretending the process is working. The CLAUDE.md file isn’t a guardrail. It’s a changelog of failures. The memory system isn’t learning. It’s note-taking. And the loop — ship, break, fix, break again — isn’t a phase I’m going to grow out of. It’s the steady state.

The question isn’t how to make Claude stop making mistakes. It’s how to build a process around Claude that catches the mistakes before they reach users.

I don’t have the answer yet. But I know it’s not “add another rule to CLAUDE.md.”

This is part of a series about building Zabriskie with Claude. Previously: why I’m building it, what building with Claude actually looks like, teaching Claude to QA a mobile app.

Finding Safe Food on the Road

Mon, 23 Mar 2026 00:00:00 +0000

“What a long strange trip it’s been” — Grateful Dead, “Truckin’”

The Problem Nobody Talks About

When I was doing my PhD in Europe — splitting time between Belgium, Portugal, and Paris — I got diagnosed with celiac disease. I’d gone to Europe partly because of the food. Paris. I wanted to teach there, live there, eat there. Then I found out I couldn’t eat bread, and in France that’s not a dietary restriction — it’s an existential one.

It got bad enough that food became one of the reasons I left. Not the only reason, but a real one. I couldn’t navigate restaurants in languages I spoke fluently because the cross-contamination risks were invisible and the cultural understanding of celiac was years behind the US. I remember thinking: my traveling days are over. I have this disease, and it means I stay home, I cook for myself, and I stop pretending I can live the life I wanted.

That was wrong. But it took years — and two specific tools — to prove it.

Find Me Gluten Free gave me a community of people who’d already eaten at every restaurant I was considering and reported whether it was safe. DoorDash gave me delivery to wherever I was staying, so I didn’t have to walk into a restaurant and try to explain celiac disease to a kitchen that had never heard of it. Between the two of them, I’ve eaten safely in hundreds of cities. Great food, not just survival food. The infrastructure existed. It just wasn’t connected.

When you have celiac disease and you’re on tour — following bands from city to city, crashing in hotels — food isn’t an adventure. It’s a minefield. But it’s a minefield I’ve learned to navigate.

Here’s what it actually looks like: you fly into a new city the night before a show. You check into the hotel, drop your bags, and immediately start thinking about food — not just tonight, but tomorrow before the venue, and maybe the day after if you’re staying for a second night. You need to figure out what’s safe in a city you’ve never eaten in before.

You open DoorDash and start scrolling. There are 400 restaurants. Some of them say “gluten-free options available,” but that label is up to the restaurant — and what it means varies wildly. A pizza place with a GF crust that gets made on the same counter as regular pizza isn’t safe for someone with celiac. A Thai restaurant that says “we can make it without soy sauce” doesn’t know that their oyster sauce has wheat in it. DoorDash gets you delivery anywhere, which is incredibly valuable when you’re traveling — but the celiac-specific safety information lives somewhere else.

So you switch to Find Me Gluten Free — a community site where people with celiac actually review restaurants and report whether they got sick. Great data. Real safety information from people who understand cross-contamination, dedicated fryers, and separate prep areas. But FMGF doesn’t do delivery. It doesn’t know whether that restaurant with the 4.8 safety rating is available on DoorDash at your hotel right now.

You end up with two tabs open, manually cross-referencing. Copy a restaurant name from FMGF, paste it into DoorDash, see if it shows up, check the delivery area, go back, try the next one. You’re doing this the night you land, trying to line up safe options for the next two days so you’re not scrambling between soundcheck and doors. It’s 11pm. You’re exhausted. You give up and eat a protein bar from your bag.

I got tired of the protein bar.

What We Built

The Itinerant Glutard is a tool that connects the two systems nobody connected before. You enter a city, a state, and your delivery address. It scrapes Find Me Gluten Free for every reviewed restaurant in that city, then checks each one against DoorDash to see if it can deliver to where you are right now. The results come back sorted by a safety score — a 0-to-100 composite that weights the restaurant’s GF level, its FMGF star rating, review count, and specific safety signals like dedicated fryers and separate kitchens.

The name is what it sounds like. Itinerant: traveling from place to place. Glutard: affectionate self-deprecation from the celiac community — the kind of word you earn after your third accidental glutening at a restaurant that swore they understood. An itinerant glutard is someone with celiac disease who’s on the road and trying to eat.

There are two modes. Full Search takes your address and does the whole pipeline — FMGF scrape, DoorDash availability check for each restaurant, merged results with direct ordering links and delivery time estimates. It takes a minute or two because it’s driving a headless browser through DoorDash for every restaurant. Quick Browse skips the DoorDash check entirely and just shows you the FMGF safety data for a city. That’s the one you use before you even book the hotel — scope out what’s safe in Portland or Denver or Philly so you know what you’re walking into.

The Safety Score

This is the part I care about most. The safety score is computed entirely from Find Me Gluten Free data — it’s a composite of the information that FMGF’s celiac community has already gathered through years of reviews, safety ratings, and incident reports. We’re not inventing safety judgments; we’re synthesizing what the community already knows into a single number you can act on quickly.

The score runs from 0 to 100, broken into five tiers:

80–100: Celiac Safe. Dedicated gluten-free facility or overwhelming positive evidence from the celiac community.
60–79: Likely Safe. Strong GF menu with good community feedback.
40–59: Use Caution. Has GF options but limited safety data.
20–39: Higher Risk. Minimal celiac-specific information.
0–19: Unknown. No data. You’re on your own.

The calculation is weighted, and every input comes from FMGF. GF level counts for 40% — a dedicated gluten-free restaurant scores higher than one that just has “gluten-free options.” FMGF’s star rating is 30%. Review count is 15%, because a restaurant with 200 reviews and a 4.5 rating is a more reliable signal than one with 3 reviews and a 5.0. The remaining 15% comes from safety signals extracted from the FMGF listings: separate fryer, dedicated kitchen, celiac-safe rating, knowledgeable staff. A negative report — someone in the community reported getting sick — drops the score.

When you’re in a new city and you don’t know anything, the score gives you a starting point. Green means order with confidence. Yellow means read the reviews first. Red means maybe stick with the protein bar.

The Architecture

The backend is Express and Puppeteer. Both scrapers need a headless browser because FMGF and DoorDash render their content with JavaScript — you can’t just fetch the HTML and parse it. The FMGF scraper navigates to the city page, scrolls to trigger lazy loading, then uses Cheerio to extract restaurant data from the rendered DOM. The DoorDash integration is more involved: it sets a delivery address, waits for the autocomplete to resolve, then searches for each restaurant by name with fuzzy matching to confirm availability.

The frontend is React and Vite. Simple by design — a search form, a list of restaurant cards with safety badges, filter buttons for “all,” “on DoorDash,” and “safe (60+).” Each card shows the restaurant name, cuisine, FMGF rating, GF level, safety signals, and a DoorDash order button with delivery time if it’s available.

The whole thing is held together with web scraping, which means it’s inherently fragile. If either site changes their page structure, the scrapers break. The code has fallback selectors and multiple strategies for finding elements, but this is a prototype — a browser pretending to be a person. It works, but a proper integration would be better. The right version of this is an API-driven experience, not a scraping hack.

Building It With Claude

I was already doing all of this manually. Every trip, the same ritual: open FMGF, find the restaurants in the city, open DoorDash, search for each one by name, check if it delivers to the hotel, keep a mental list of the ones that work. It took 30–45 minutes on a good night. On a bad night — a city with dozens of FMGF listings to check one by one — I’d give up halfway through.

The process worked. It was just slow, tedious, and manual. I knew exactly what I was doing at every step. I just couldn’t build the tool to automate it because the work involved — scraping two JavaScript-heavy sites with a headless browser, navigating DoorDash’s address input flow, fuzzy-matching restaurant names across platforms — was the kind of grinding infrastructure code that would have taken me weeks of evenings to get right.

Claude made it possible to build the thing I was already doing by hand. I described the manual workflow — go to FMGF, get the restaurants, check each one on DoorDash at this address — and Claude wrote the Puppeteer automation, the DOM navigation, the fallback selectors for when DoorDash’s UI didn’t behave as expected, the fuzzy name matching, the React frontend, all of it. The domain knowledge was mine: which safety signals matter, how to weight them, why review count is a confidence measure, why FMGF data comes first. But the scraping infrastructure that turned a 45-minute manual process into a two-minute automated one — that’s what Claude made feasible for a solo developer building something on evenings and weekends.

The Name

Itinerant Glutard. “Glutard” is celiac community slang — the kind of self-deprecating shorthand people use when they’ve spent enough years explaining cross-contamination to waiters and reading ingredient labels on soy sauce. It’s an in-group term, affectionate in the way that only people who share the condition tend to use it. Itinerant because I’m on the road.

There was a version of me in a tiny apartment in Belgium who believed this disease meant staying put. That the world had shrunk to the places I could cook for myself. That touring — the thing I wanted most — was something other people got to do.

I was wrong. The community data existed. The delivery infrastructure existed. I just needed to connect them. I’ve eaten safely at hundreds of places in dozens of cities since then, and the food has been genuinely good — not sad compromises, not protein bars, not going hungry. The Itinerant Glutard is the tool that makes that process faster, but the real thing that makes it possible is that the celiac community built FMGF and DoorDash built delivery, and between the two of them, the road opened back up.

I have celiac disease. I travel to see live music. I got tired of going hungry.

Now I don’t.

Teaching Claude to QA a Mobile App

Sun, 22 Mar 2026 00:00:00 +0000

“When life looks like Easy Street, there is danger at your door” — Grateful Dead, “Uncle John’s Band”

(A note on picking this quote: I asked Claude to find me a Grateful Dead lyric that fit the theme. It couldn’t — searching for “dead lyrics” triggers the content filtering policy: API Error: 400 {"type":"error","error":{"type":"invalid_request_error","message":"Output blocked by content filtering policy"}. I had to pick this one myself.)

I build Zabriskie alone — no team, no investors, just me in my bedroom shipping a community app because I think the internet needs better gathering places. My first lesson about building a “product”: if it’s not in the App Store, it doesn’t exist. I had early users who loved the web version but wouldn’t touch it daily because it wasn’t “an app.” It might as well not be real. So I needed to ship on three platforms — web for fast iteration and testing, iOS and Android because that’s where people actually live.

The problem is I’m one person. I can’t write and maintain three separate codebases. The answer was Capacitor: it takes the React web app I’d already built and wraps it in a native shell — a WebView on Android, a WKWebView on iOS — so the same code runs everywhere. Combined with the server-driven UI architecture (the backend sends screen layouts as JSON, and the client just renders them), I can push changes to all three platforms without waiting for App Store review. One codebase, three platforms, one developer. It’s the only way this works.

But Capacitor puts you in a testing no-man’s-land. Playwright can’t reach inside the native shell — it’s not a browser tab anymore, it’s an app. Native testing frameworks like XCTest and Espresso can’t interact with the content — it’s HTML inside a WebView, not native UI elements. You’re too native for web tools and too web for native tools. Every testing approach in this post exists because of that gap.

Zabriskie runs on all three platforms. The web gets tested by Playwright — 150+ E2E tests that run on every push. But the mobile apps had nothing. No automated QA, no visual regression checks, no way to know if either client was rendering correctly without manually clicking through every screen. I decided to fix that by teaching Claude to drive both mobile platforms, take screenshots, analyze them for issues, and file its own bug reports.

Android took 90 minutes. iOS took over six hours. The difference says everything about the state of mobile automation tooling in 2026.

Android: The Easy One

The first challenge was connectivity. Inside the Android emulator, localhost refers to the emulator itself, not the host Mac. When the Capacitor app tries to reach localhost:3000 or localhost:8080, it gets nothing. The fix is adb reverse:

adb reverse tcp:3000 tcp:3000
adb reverse tcp:8080 tcp:8080

Simple, but you have to re-run it every time the emulator restarts.

The real breakthrough was realizing that Capacitor apps run inside an Android WebView, and WebViews expose a Chrome DevTools Protocol socket. You can find it, forward it to a local port, and suddenly you have full programmatic control:

# Find the WebView's DevTools socket
WV_SOCKET=$(adb shell "cat /proc/net/unix" | \
  grep webview_devtools_remote | \
  grep -oE 'webview_devtools_remote_[0-9]+' | head -1)

# Forward it to a local port
adb forward tcp:9223 localabstract:$WV_SOCKET

# Full CDP access
curl http://localhost:9223/json

With CDP, authentication is one WebSocket message — inject a JWT into localStorage and navigate to the feed. Navigation is another message — set window.location.href. No coordinate guessing, no UI interaction, no fighting with keyboards or dialogs. The same protocol that Playwright and Puppeteer use, just connected to an Android WebView instead of a desktop browser.

Combined with adb shell screencap for screenshots, I built a Python script that sweeps all 25 screens of the app in about 90 seconds. Landing, login, all four feeds, post detail, profile, shows hub, content creation forms, catalog, battles, bug forum, diary, badges, tour crews — everything. Each screenshot gets analyzed for visual issues: broken layouts, error messages, missing images, blank screens, status bar overlap.

When the sweep finds something wrong, it authenticates as zabriskie_bot, uploads the screenshot to S3, and files a properly formatted bug report to the production forum. The title format is [Android QA] Shows Hub: RSVP button overlaps venue text — immediately clear that it came from automation and which screen is affected. It knows about expected states too: the crew detail page returning “Forbidden” for non-members isn’t a bug, empty avatar circles aren’t bugs, and the “Preview” text in profile settings is a known cosmetic issue.

The whole thing runs as a scheduled task every morning at 8:47 AM. The first full run came back clean: 25 screens, 0 critical issues, 2 minor cosmetic notes. If someone’s change breaks a screen overnight, there’s a bug filed before anyone’s had coffee.

Ninety minutes, start to finish.

iOS: The Hard One

I figured iOS would be straightforward. Same app, same screens, the Simulator is right there on my Mac. What followed was one of the most absurd debugging sessions I’ve had — not because the problem was technically profound, but because the iOS Simulator is a fortress of tiny, compounding restrictions that each seem reasonable in isolation but together create a nightmare.

You Can’t Type an Email Address

The first idea was clean: add a deep link handler, generate a JWT, open the URL via simctl openurl, and skip the login form entirely. Four attempts, four different failure modes — the native bundle was stale, the config pointed at production, the JWT secret was wrong, the Vite dev server was listening on IPv6 while the Simulator tried IPv4. Zero logins.

So I fell back to typing credentials into the login form. AppleScript can send keystrokes to the Simulator. But the login form has type="email" on the input, and AppleScript’s keystroke "@" sends Shift+2, which the Simulator interprets as a keyboard shortcut. Every attempt to type @ either switched the form to Sign Up, navigated to Forgot Password, or opened a context menu.

Pasting didn’t work either. Cmd+V gets intercepted by the Simulator. Setting the iOS pasteboard via simctl pbcopy produced garbled text. The macOS clipboard and the iOS pasteboard are separate systems.

The fix was a code change: update the backend login handler from WHERE email = $1 to WHERE email = $1 OR username = $1, change the form input from type="email" to type="text", and create a test user with a known password. Now I could type “qatest” instead of needing an @ symbol. A backend modification to work around a keyboard limitation.

You Can’t Dismiss Native Dialogs

Upon login, iOS shows a “Would Like to Send You Notifications” dialog rendered by UIKit, not the WebView. Native iOS dialogs cannot be dismissed by any form of macOS-synthesized input.

I tried AppleScript click at coordinates across a grid of 100+ positions. cliclick at every possible coordinate. Python Quartz CGEvent mouse events. Pressing Return and Enter. Finding the button in the accessibility tree (not exposed). simctl privacy grant (not supported for notifications on iOS 26). simctl ui alert accept (doesn’t exist).

The dialog sat there, immovable, blocking the app.

The fix was writing directly to the Simulator’s TCC.db — the privacy permissions database — inserting a pre-approval for kTCCServiceUserNotification, then restarting SpringBoard. But the timing is critical: it has to happen before installing the app, or the permission state gets cached. And the app’s JavaScript calls PushNotifications.requestPermissions() on login, which can retrigger it, so I had to add a guard that skips permission requests on localhost.

The correct sequence: uninstall app, write TCC permission, restart SpringBoard, reinstall app, launch, then login. Only in that exact order does the dialog not appear.

You Can’t Navigate by Coordinates (Until You Can)

The app has a floating nav bar with three bubble buttons in the top-right corner — a Z logo, an avatar, and a + — each opening a vertical dropdown. To test all 25 screens, I needed to tap specific dropdown items. I had coordinates from the CSS. The math checked out. But every approach had a different failure mode.

AppleScript click at uses macOS window coordinates. You need the window position, the device screen group offset, the Simulator’s scaling mode (Point Accurate vs. Pixel Accurate vs. Fit Screen), and whether the toolbar is showing. First sweep: 42% accuracy.

Facebook’s idb sends taps in device logical points (390x844), so no translation needed. Better for main nav buttons, but dropdown item coordinates were slightly off — taps would close the dropdown before hitting the item, or punch through the z-index to content behind it. Second sweep: 57% accuracy.

The breakthrough was the ios-simulator-mcp tool’s ui_describe_point function. Point it at any coordinate and it returns the accessibility label, role, and frame:

ui_describe_point(365, 163)
→ AXLabel: "Currents", type: Link, frame: (342, 159, 40x40)

I mapped every dropdown item by probing in 48pt increments. My Y positions were right but my X was wrong — the + dropdown items are at x=258, not x=269. An 11-point error that routed every tap to the wrong column. With verified coordinates and 1.5-second waits for dropdown animations, the sweep hit 100% of screens.

The winning combination: ui_describe_point for discovery, idb ui tap for execution. Map the UI first, tap second. Don’t guess coordinates — measure them.

The Fundamental Gap

The contrast is stark. Android authentication:

ws.send('{"method":"Runtime.evaluate","params":{"expression":"localStorage.setItem(\'token\',\'xxx\')"}}')

iOS authentication: uninstall app, write to TCC database, restart SpringBoard, reinstall app, launch, wait 5 seconds, tap Sign In at specific coordinates, wait, tap Email field, type “qatest” via AppleScript, press Tab, type “qatest123”, press Return, wait, hope.

Apple’s WKWebView doesn’t expose Chrome DevTools Protocol. Safari Web Inspector uses a proprietary binary protocol that only Safari speaks. ios-webkit-debug-proxy only works with real USB devices. safaridriver connects to macOS Safari, not the Simulator’s WebView.

Android gives you a WebSocket and says “here’s the browser, do whatever you want.” iOS gives you a locked door and a note that says “please use Xcode.”

The Mess in the Middle

Between getting Android working and finishing iOS, something happened that illustrates a different kind of failure — not a platform limitation, but an agent discipline problem.

Railway deployments started failing with a Go version mismatch. My local Go had auto-updated to 1.26, which silently bumped go.mod to require Go 1.25, while the Dockerfile still used golang:1.24-alpine. A two-file fix.

Claude was operating in a git worktree — a clean, isolated copy of the repo designed for exactly this kind of surgical change. Instead of making the fix there, it cd‘d into the main repository where I had a dozen unrelated in-progress changes. It staged every dirty file, committed them all with the Go version fix, pushed, and opened a PR. The PR contained QA login endpoints, bug forum updates, iOS Simulator workarounds, E2E test config changes, push notification code, and three new skill files. None of which had anything to do with a Go version number.

Then it got auto-merged before I could close it.

The bad merge left duplicate variable declarations throughout the test suite — functions declared twice, variables declared twice. One of the accidentally included changes was a form placeholder rename from "Email" to "Email or Username", which broke every auth E2E test that used page.fill('input[placeholder="Email"]'). A catalog test that asserted itemCount > 50 only worked against my local database — CI has a handful of records.

To fix a two-file change, I ended up making four follow-up commits across three PRs. The first two I pushed without running tests locally. They failed. The third I actually ran tests first. It passed. Three rounds of “push and pray” before doing what should have been step one: run the tests, read the output, fix what’s broken, verify, then push. The same debugging rule I enforce every session — check the logs first, theories second — and I ignored it for my own changes.

What This All Adds Up To

Both platforms now have working QA skills. Every morning, the Android emulator and the iOS Simulator boot up, sweep 25 screens each, analyze the screenshots, and file bug reports for anything that looks wrong. Three platforms, all tested, all filing their own bugs.

The lessons keep reinforcing each other:

CDP over taps. Don’t fight coordinate systems if you can use the browser’s own debugging protocol. Android gives you this for free. iOS doesn’t, and every workaround adds fragility.

Measure, don’t guess. The accessibility API that finally made iOS navigation work is the same principle as checking logs before forming theories. Don’t assume you know where a button is — ask the system.

Stay in the worktree. Isolation only works if you respect the boundaries. The moment you step outside “just for a quick look,” you’re one careless command away from committing a dozen unrelated files to production.

Run the tests before you push. Three rounds of push-and-pray before doing what should have been step one. The gap between knowing a rule and following it is measured in wasted commits.

Apple, if you’re reading this: please expose CDP or WebDriver for Simulator WebViews. The developer tools are great when a human is using them. They’re nearly useless when an AI is trying to.

What Building With Claude Actually Looks Like

Fri, 20 Mar 2026 00:00:00 +0000

“Sometimes a notion gets a-hold of you, carries you away” — Grateful Dead, “Althea”

The Saturday

On March 7th, I sat down at 11am and started building. By 2am I had shipped a Relisten integration with inline audio players for archival recordings, a Quick Post feature for sharing past shows, Phantasy Tour as a live setlist source, Sign in with Apple, a badges and achievements system, security hardening across every API route, a complete CI pipeline with GitHub Actions, and something called Goose Mode — a tour companion dashboard with a wandering desktop goose that follows your cursor around the screen.

That was one day. 124 commits.

I kept going. By the end of the week, the total was 144 commits across 7 days. Relisten integration. Live show experience. Six new bands on the platform. A bug forum. Backend test coverage from 43.9% to 70.4%. An admin analytics dashboard. Apple App Review fixes. Scrapers for four different setlist sources. Archive.org as a recording provider. Mobile layout fixes for three different iOS edge cases.

I’m building Zabriskie by myself, and I’m building it with Claude. This is what that actually looks like.

The Cost of Trying Something

Here’s the thing nobody tells you about building with an AI collaborator: the most important change isn’t speed. It’s what happens to your relationship with bad ideas.

Goose Mode started at 9pm on a Saturday night as a tour-focused homepage with a literal wandering goose — a transparent animated sprite that wanders around the page while you browse upcoming shows. It was fun. It was also wrong. The layout didn’t work. The information hierarchy was off. The goose was distracting in a way that stopped being charming after about thirty seconds.

Old me would have agonized. I’d spend an hour in Figma trying to figure out the right layout before writing a line of code. I’d poll people. I’d sit with it for a few days. The cost of being wrong was high enough that I’d optimize for not being wrong.

Instead, at 11:25pm, I told Claude to redesign the whole thing as a tour companion dashboard. By 3:37am it had been redesigned again — left-aligned with grouped tour timelines. By 4:08am, another redesign — card-per-tour visual timelines. By 4:42am, another — live countdown, attendee avatars, couch tour cards. By 5:41am I’d added a flip-clock countdown. By 6:17am there were interactive tour maps embedded in each card.

Six versions in nine hours. Each one a real, working implementation I could tap through on my phone. Not mockups. Not wireframes. Running code. The version that shipped was the fifth attempt, and I only knew it was right because I’d seen the four that weren’t.

When the cost of trying something drops to near zero, you stop designing in your head and start designing in reality. That changes everything.

A Pigeons Show, a Broken Chat, and a Deploy at 11pm

On the night of March 7th, Pigeons Playing Ping Pong was playing a show. People were using Zabriskie’s live show features — Live Chomping, the setlist tracker, the “tonight” banner. I was watching the show on the couch and also watching my app.

The chat input was getting cut off on Chrome mobile. I could see it happening in real time because I was using it. At 10:33pm I fixed the layout. At 10:39pm I realized live show posts weren’t appearing in the main feed while a show was active — a bug nobody would have found in testing because it only manifested when a real show was actually live. Fixed it. Deployed. At 10:45pm I added Phantasy Tour as a live setlist source because the existing sources weren’t picking up the setlist fast enough. At 11:20pm I fixed the song order — songs were appearing out of sequence, and duplicate comments were showing up in the chat.

At 11:34pm I discovered that on newer iPhones, the live page content was rendering underneath the Dynamic Island. That’s not something you find in a simulator. That’s something you find when you’re holding the phone in your hand, trying to see who’s chomping, and you can’t read the first line of text.

This is what dogfooding actually means. Not “I used my own app once and it seemed fine.” It means you’re sitting on your couch during a Pigeons show, fixing a CSS bug that’s blocking your own experience, deploying it, and immediately seeing whether it worked — all while the show is still going.

The Breadth Problem

Solo developers are supposed to specialize. Pick a lane. You can’t do backend and frontend and mobile and DevOps and design. That’s a team.

In one week I worked on:

Audio infrastructure — Relisten API integration, playlist players, collapsible audio widgets, caching, recording quality filtering
Real-time features — Live chat (sorry, Live Chomping), WebSocket setlist polling, presence indicators
Data ingestion — Scrapers for Phantasy Tour, TTBase, setlist.fm, Archive.org/etree
Security — Rate limiting, CORS hardening, JWT invalidation on password change, auth on every route
Testing — Playwright E2E coverage, mock HTTP servers, backend coverage push, CI pipeline
Mobile — iOS Dynamic Island fix, Chrome mobile layout, safe area insets, App Review compliance
Social features — Quick Post, @mentions, clickable URLs, clickable avatars, bug forum with upvotes
Design — Goose Mode (x5), Spotify now-listening card, admin analytics dashboard

This isn’t sustainable without Claude. I want to be honest about that. I’m not some 10x developer who figured out the productivity secret. I’m a normal developer who has a collaborator that doesn’t sleep, doesn’t get bored of writing tests, and can context-switch from a Go backend handler to a Swift layout constraint to a Playwright test assertion without missing a beat.

The thing Claude is genuinely good at — the thing that makes the breadth possible — is carrying the context. I can say “the Relisten player should be collapsible, like we did on the show listing page” and it knows what I mean because it wrote that code an hour ago. I don’t have to re-explain the component architecture every time I switch contexts. It already knows.

The Things I Still Do

I want to be clear about what Claude doesn’t do, because the discourse around AI and coding has gotten absurd in both directions. People either think it writes your entire app for you, or they think it’s useless. Neither is true.

Claude doesn’t know what to build. It doesn’t know that a Pigeons show is happening tonight and that the setlist tracker needs Phantasy Tour as a source. It doesn’t know that the Goose Mode countdown should use a flip-clock style because that’s what feels right for the aesthetic. It doesn’t know that the live chat should be called “Live Chomping” because that’s what the community actually calls it. It doesn’t know that the Quick Post feature exists because I watched someone try to share a recording and give up because it was too many steps.

Every feature started with me noticing something — a pain point, an opportunity, an idea at 3am that I couldn’t let go of. Claude is the best collaborator I’ve ever had for turning those observations into running software. But the observations are mine. The taste is mine. The understanding of what this community needs is mine.

And the bugs. The bugs are mine too. The production crash from the code coverage import that should never have been in main.go — that was a human mistake. Claude wrote the instrumentation; I’m the one who forgot to check the build before deploying. The authentication forwarding bug that broke internal SDUI calls — that emerged from the interaction between two features Claude had built separately, each correct in isolation, broken in combination. Integration bugs are still human problems. They require understanding the whole system, not just the code.

43.9% to 70.4%

On March 8th at 8:45am — after the Pigeons show, after the Goose Mode all-nighter, after 124 commits in a single day — I asked Claude to push the backend test coverage as high as it could go.

One commit. 43.9% to 70.4%.

This is the thing that makes me genuinely optimistic about building alone. The testing tax — the thing that slows down every solo developer, the thing you skip because you’re tired and the feature works and you’ll write tests later (you won’t) — that tax is effectively gone. Claude writes comprehensive tests. Not just happy-path assertions. Edge cases. Error conditions. Auth boundary tests. The kind of tests you’d write if you had infinite patience and no ship date.

I still write tests for the things that matter to me — the tricky integration points, the things where the test itself is the specification. But the coverage floor, the boring-but-necessary tests that catch regressions? That’s not my job anymore. And that means I actually have test coverage, which means I can refactor with confidence, which means the codebase stays healthy even at this pace.

Three Nights at the Beacon

The Pigeons show was from my couch. The Tedeschi Trucks Band run at the Beacon Theatre was from my seat.

TTB was playing ten nights at the Beacon in March. I had tickets to three of them. I also had a platform that didn’t know Tedeschi Trucks Band existed yet. Tuesday before the show I added them — 58 shows for the 2026 Future Soul Tour. But TTB’s setlists don’t come from setlist.fm. They come from TTBase, which has a completely different HTML structure. So I needed a new scraper.

That night, from my seat at the Beacon, I built it. I told Claude what TTBase looked like, what data I needed, and how it should integrate with the live setlist poller. Claude wrote the scraper. I deployed it. It didn’t work — the HTML structure didn’t match what we’d expected. So I told Claude what was wrong, it fixed the scraper to match the actual Songfish HTML structure, I deployed again, and watched the setlist populate in real time while the band was playing.

I used the app across all three shows. I’d be sitting there, listening to Derek Trucks play, and I’d notice something — a layout bug, a feature that didn’t work right, something that could be better. I’d pull out my phone, tell Claude what I needed, watch it write the fix, push it to production, and then check it on my phone. All from my seat. All while the show was happening.

The reason this was even possible is that Zabriskie uses a server-driven UI architecture. Early on, Claude helped guide me toward SDUI as the core design — the server sends down the layout and components, and the app just renders whatever it receives. That means I can change virtually anything about the experience without shipping a new version through the App Store. A fix to a layout, a new feature, a redesigned screen — it’s all a server deploy. When I pushed a fix from the Beacon, every phone running the app got it immediately. No app update. No review process. No waiting.

This is a different thing from the Pigeons show, where I was on the couch and had a laptop open. At the Beacon I was in the audience with nothing but my phone. The workflow was: notice a problem, describe it to Claude in plain English, Claude fixes it and pushes to prod, I pull up the app and verify. No laptop. No IDE. No terminal. Just me, my phone, and a collaborator who could do the rest.

By the end of the three-night run, the TTB experience on Zabriskie was solid. Live setlists from TTBase. Show pages with all the metadata. The whole thing built and refined from inside the venue where the band was playing. That’s not a development workflow I ever imagined having.

The Week Keeps Going

After the Saturday marathon and the Beacon run, I kept building. Sunday: clickable URLs in posts and comments. A small thing. The kind of thing you’d never prioritize on a roadmap but that users notice immediately.

Thursday I added Grahame Lesh & Friends with 25 shows. Saturday night I was at the Grahame Lesh show, and this time I wasn’t debugging anything. The setlist just worked. It synced with the show in real time, song by song, no intervention. I was posting photos from my seat and the setlist was updating alongside them. My friends were watching from home, talking to me through the app — they could see the setlist, see my photos, and we were all in the same experience even though I was the only one in the room.

That’s the moment it stopped being a project and started being the thing I described in the manifesto. The bridge between the physical and the virtual. The people in the crowd and the people at home, in the same place. It worked. Not because I was fixing it in real time — because I didn’t have to.

The bug forum is maybe the most meta thing I’ve built that week. A full in-app bug reporting system with upvotes, comments, categorization, bot notifications, and admin tools. Built in an afternoon. A bug reporting system, built by one person with an AI collaborator, for reporting bugs in an app built by one person with an AI collaborator. It has six commits spanning two hours. It works. Users are filing bugs in it right now.

What This Means

I wrote the manifesto for Zabriskie on March 8th, in between the all-night Goose Mode session and deploying fixes for the Pigeons show. The manifesto is about reclaiming the internet as a third place. About building community infrastructure that serves people instead of extracting from them. About doing it as a non-profit, solo, self-funded, because that’s the only way it gets done honestly.

The week that followed is what makes that possible. Not because AI is magic, but because it changes the economics of ambition. A single person can build something that previously required a team. Not because the single person became superhuman, but because the gap between “what I can imagine” and “what I can ship” got dramatically smaller.

I have eighty beta users now. They’re filing bugs. They’re posting about shows. They’re using Live Chomping during actual shows. The thing works. Not in a demo sense — in a “people are using this to connect with each other around music” sense.

That’s all I ever wanted.

The gap between what you can imagine and what you can ship is the space where ideas go to die. For years I had this idea — the third place, the taste-based community, all of it — and I couldn’t build it because I’m one person with a day job. Now I can. Not perfectly. Not without bugs. Not without 3am sessions that leave me wrecked the next day. But I can build it, and I can build it fast enough that the community doesn’t outgrow the infrastructure.

144 commits in a week. Six versions of Goose Mode. A scraper built from a seat at the Beacon Theatre. A live show debugged in real time. A test suite that actually exists. A bug forum built in two hours.

This is what building with Claude actually looks like. It’s not a press release. It’s a Saturday that starts at 11am and ends sometime around dawn. It’s three nights at the Beacon with nothing but your phone, shipping fixes between sets. And when you look up, the thing you imagined is running on your phone, and people are using it.

“Sometimes a notion gets a-hold of you, ties you to the tracks” — Grateful Dead, “Althea”

Zabriskie. Where taste resonates.

Why I'm Building Zabriskie

Sun, 08 Mar 2026 00:00:00 +0000

“Don’t let a heavy one hold back the dawn in you” — Goose, “(dawn)”

The Third Place

There’s a concept in sociology called the “third place” — not home, not work, but the place where you go to find your people. The coffee shop. The record store. The bar after the show. It’s where community happens.

The internet used to be that place.

In the late 90s and early 2000s, there were communities built entirely around taste. Makeoutclub. MySpace. Forums with names you’ve probably forgotten. Your profile wasn’t a highlight reel of your life — it was a declaration of what you were into. The music you listed, the films you referenced, the books in your sidebar. People found each other through that. You’d stumble onto someone’s page because they listed the same obscure record you loved, and suddenly you had a new friend in a city you’d never been to. The social graph was built on shared taste, not shared geography.

For a while, Twitter picked up that torch. By the mid-2010s, I had friends all over the world — people who followed me because they were into the same things I was into. I’d travel for research and work to Europe, Australia, the UK, wherever, and I could post and there would always be someone up for a drink, a show, a conversation. It was global and it was real. The connections were built on interests, and they worked.

Then that got destroyed too.

Twitter got bought and the community scattered. Instagram and Facebook were already ad machines, optimizing for engagement over connection. Mastodon was too federated for anyone to find each other. Bluesky came too late, after everyone had already given up. Letterboxd was a glimmer of hope — a platform built around a specific kind of taste — but it devolved into ironic one-liners engineered for laughs and would-be film critics writing for an audience instead of a community.

And for music? Nothing. A total void.

The third place doesn’t exist anymore. Not for people who actually care about what they’re listening to, watching, and reading.

Proof It Works

Here’s the thing: I know the third place can work online, because one community already does it in the physical world every single day.

The jam band community — my community — knows how to build this. They have shakedowns. They have lot meetups. They organize their entire lives around shows. They travel. People wander on and off tour. Strangers become friends because they’re standing next to each other for three nights in a row. The social infrastructure is already there. The culture of gathering around shared experience is already there.

The tools just haven’t caught up.

Reddit requires constant refreshing if you want anything resembling a live conversation. Facebook groups are merch spam wastelands. The dedicated platforms that do exist — Phantasy Tour, various setlist sites — are siloed. Each one focuses on a single band or a single function. There’s no unified identity. You’re a different person on every platform. And most of them haven’t evolved technologically since 2008.

I know this because I lived it. I spent ten years in grad school, and when I finished, I had no community. Not really. People didn’t even say goodbye when I left. A decade of my life, and it just… ended.

Then I got back into the jam band world, and something completely different happened. People I’d long disconnected with welcomed me back like no time had passed. Patrick, someone I hadn’t talked to in twenty years. Sara, someone I’d sold tickets to online. I made friends through friends of friends, the way you do on the lot. I made friends standing next to some Dallas fans at a show during the NBA playoffs when Dallas was playing Boston, and I was wearing a Celtics jersey. That’s it. That’s all it took. Standing next to strangers who were into the same thing, and suddenly you’re not strangers anymore.

No other community in my life has worked like that. The jam band world doesn’t care where you went to school or what you do for a living. It cares whether you were at the show. It cares about the music. And that’s enough.

COVID Separated Us. We Never Came Back.

The pandemic made all of this worse. It separated us physically, and we never fully recovered. We got distant from each other in ways we’re still reckoning with. The platforms we had didn’t help — they made it worse, feeding us outrage and ads while we sat alone.

Couch tour became a lifeline during that time. Watching a show from home, knowing other people were watching too, trying to find each other in Reddit threads and group texts. But it was held together with duct tape. There was no place that actually served that experience.

That’s what I want to build. A bridge between the physical and the virtual. The people in the crowd and the people at home should be in the same community, sharing the same experience. And if you’re home alone on a Tuesday night watching a stream — you’re not alone. You’re part of something.

Couch touring shows is just one version of this. Book clubs were the original shared cultural experience. Shared movie watching is the next. The format extends infinitely because the core is always the same: people experiencing culture together, regardless of where they physically are. A global community of people who love media.

Facebook is a dinosaur. It doesn’t provide anything useful for this. It harvests your data, sells it to advertisers, and gives you merch spam groups in return.

Extending Beyond the Lot

The jam band community is the proof of concept, but the vision is bigger.

The same energy that makes shakedowns work — people organizing around the things they love, sharing experiences in real time, building identity through taste — applies to anyone who cares deeply about culture. The person who wants to talk about the Jarmusch film they just watched, not post an ironic quip for engagement. The person spinning a new album who wants to know what their friends think, not get an algorithmic recommendation from a company that’s also selling them headphones. The person who just finished a novel and wants to find others who read it, not write an Amazon review into the void.

Zabriskie starts with the jam band world because that community is ready for it. They already have the culture. They just need a place that works. But the thesis is universal: anyone whose identity is shaped by what they consume — music, film, books — deserves a third place built around that.

The Opposite of Everything

Zabriskie is the opposite of every social network that exists.

We don’t want everyone on the planet to join. In Careless People, you can trace the path of platforms that pursued growth indefinitely, at all costs, and watch exactly where it leads — the product hollows out, the community dies, the ads take over. We’re not walking that path.

We’re not selling ads. We’re not optimizing for engagement. We actually want you to get off the site. The feed is finite — you read it, you’re done, you go live your life. There are no free-form text posts. Every single post requires a piece of culture — an album, a film, a book, a show — a rating, and your actual thoughts. We are about culture.

Discovery works differently here too. You want to see posts from people you don’t know. You want to meet people organically, based on taste. But not through algorithmic matching — not the dating app model, not the “people you may know” sidebar. Taste proliferates through the network the way it does in real life: through people.

Think about high school. You’re walking down the hall and you see someone wearing a shirt from a band you love. You don’t know them. They might be in a completely different social world. But you recognize something about them instantly. That moment of connection based on taste, without an algorithm, without a recommendation engine, without anyone engineering the encounter — that’s how discovery should work. And nothing online does it.

Zabriskie does.

Not a Startup

I should be clear about what this isn’t. Zabriskie isn’t a startup. There’s no pitch deck. No Series A. No growth targets. No exit strategy. Nobody is looking to flip this to Google in three years. There are no revenue goals and no KPIs around daily active users.

This is about reclaiming the internet as a place. The third place. Building community, not building a business.

The internet used to be somewhere you went to find your people. Then it became something that was done to you — feeds engineered to keep you scrolling, platforms optimized to extract value from your attention, your taste data packaged and sold to the highest bidder.

Zabriskie is a rejection of all of that. We’re not looking for revenue or growth. We’re looking to reclaim the internet as our third space and build community.

And to put our money where our mouth is: Zabriskie is becoming a non-profit. Not because it’s a clever tax strategy. Because it’s the only structure that’s honest about what we’re doing. Community over money. Full stop. Every decision gets made with one question: does this make the community better? Not: does this grow the user base? Not: does this increase engagement? Not: does this make the metrics look good for investors?

There are no investors. There never will be.

I’m building this on my own. Funding it myself. Writing code on nights and weekends because I believe in this. There’s no team of fifty engineers. There’s no office. There’s just me, doing the work, because I think this matters. And I think this is the only way it can be done honestly — if it’s built by someone who actually wants to use it, not by someone who wants to monetize it.

This is the only way to free ourselves from the oppression of existing social networks. You don’t reform platforms that were designed from the ground up to extract value from your attention. You don’t petition them to be better. You build something new. You take it back.

We’re building this because something has been missing from the internet for a long time, and people want it back. The third place. A space built around taste, around culture, around the things that actually matter to you. A place where you show up not because an algorithm pulled you in, but because your people are there.

That’s it. That’s the whole thing.

I owe thanks to the people who made this real. Patrick and Mary are my partners in this — they’ve been instrumental in shaping the design of this platform from day one. Every screen, every decision about how it should feel, has their fingerprints on it. This is as much theirs as it is mine. Sara and Charles have been there from the beginning, beta testing features, breaking things, and telling me when something didn’t work. You don’t build a community platform alone. You build it with your community.

Rick Mitarotonda once said that if something brings you passion, you should work as hard as you can at it — that the results, the success, it’s not about that. That’s exactly how I feel about Zabriskie. This isn’t about metrics or outcomes. It’s about the work. It’s about building something that matters to the people who use it.

If any of this resonates with you — if you’ve been looking for the place that used to exist and doesn’t anymore — come find us at zabriskie.app. Bring your taste. Bring your people. The dawn is coming. Let’s build the third place together.

“I feel it all in our hands, in a rising sun” — Goose, “(dawn)”

Zabriskie. Where taste resonates.

Claude Tested Everything Except the One Thing That Mattered

Sun, 08 Mar 2026 00:00:00 +0000

Three weeks ago I wrote about building a social app in a week with Claude Code. The app shipped. My friends are using it. I kept building.

Since that post, Claude has written 154 end-to-end tests across 17 spec files. It tests login, logout, signup, and redirect guards. It tests the feed, the bookmarks page, the notifications page. It tests liking, unliking, commenting, amplifying, recommending. It tests show RSVPs, band pages, setlist search. It tests a tournament bracket system. It tests song battles. It tests a badge and achievement system. It tests tour crews. It tests a getting-started tutorial. It tests a Goose Mode dashboard (if you haven’t heard Goose yet, well, as they say, Goose fucks). It tests a community catalog. It tests tracklist rendering and live show layouts.

It does not test posting.

Posting is the entire point of the app. It’s the one thing every user does every time they open it. You search for an album, you write something about it, you hit submit, and it appears in the feed. That’s the product. Everything else — the battles, the crews, the badges, the tournaments — is decoration around that core loop.

There is no test that searches for an album. No test that fills out the review form. No test that submits a post through the UI and verifies it appears. Zero.

The test file called post.spec.ts does exist. It has 11 tests. They verify that the post detail page renders. That the new post page renders a search form. That the profile page renders. The word “render” is doing a lot of heavy lifting. None of them actually post anything.

There is a createPost() helper in the test utilities. It calls the API directly — POST /api/posts with a JSON body — to set up test data for other tests. The social tests use it to create a post so they can test liking it. The bookmark tests use it to create a post so they can test bookmarking it. The core action of the app exists in the test suite only as scaffolding for side features.

Here are the test counts by spec file:

Tests	Feature
33	Tour crews
28	Shows
25	Song battles
13	Catalog
11	Setlist search
11	Posts (rendering only)
9	Tournaments
6	Getting started tutorial
4	Badges
0	Actually submitting a post

I asked Claude to write tests. Multiple times. I put it in the project instructions, in bold: “Write a new test for every new user-facing behavior.” I listed exactly what warrants a test: new screens, new buttons, new API endpoints, bug fixes. Claude wrote that rule on February 23rd. After that date, it created 10 new spec files and 113 new tests — for tournaments, battles, badges, crews, goose mode, catalog, setlist search, tracklists, tutorials, and live layouts. Not one for posting.

Then the auth refactor happened.

Claude had originally built 25+ backend routes without authentication. Posts, comments, profiles, search, live chat — all accessible to anyone, no login required. I don’t know why. The middleware existed. The pattern was established. It just… didn’t apply it.

When I noticed, the fix required touching every route in main.go and every page component in App.jsx. Fifty-seven lines changed in the backend, fifty-six in the frontend. That’s the kind of refactor where, if you have good test coverage of the core flow, you make the change, run the tests, and find out immediately what broke.

We did not have good test coverage of the core flow.

The refactor broke things. Thirty-one seconds after the auth commit, there was already a follow-up fix — a test was hitting GET /api/posts/{id} without an auth header and getting 401s. Then another fix because the live show pill broke. Then another because pills showed on logged-out pages. The cascade was short this time, but only because the tests we did have caught the edges. The center — the posting flow — had nothing to catch.

This is part of a broader pattern. When something breaks, I ask Claude to write a failing test first, to prove what’s actually broken before trying to fix it. Claude does not do this. What Claude does instead is read the bug report, form a theory about the cause, and immediately start editing code. If the theory is wrong — and it often is — the “fix” breaks something else. Then Claude fixes that. Then something else breaks.

The commit history is the evidence. Out of 833 total commits, 202 are fixes. That’s 24% — one in four commits exists to fix something Claude got wrong. And they come in chains:

Show post cards: four consecutive fix commits. Orphaned edit button, then flaky tests, then wrong assertion, then more broken assertions.
Live chat: four consecutive fix commits. Wrong sort order, then scroll broken, then passive touch events, then iOS Safari viewport bleed.
S3 avatars: three consecutive fix commits. URLs expiring, then NULL media_item_id scan failure, then the same scan failure again with the same fix.
Deployment: two identical commits back-to-back. “Fix web service deployment with npx serve.” Twice. The same message.

Each chain follows the same shape: Claude guesses what’s wrong, ships a fix without verifying the guess, the fix breaks something adjacent, and the cycle repeats. A failing test at the start of each chain would have stopped it at one commit.

The project instructions file — CLAUDE.md — is now full of rules that exist because of this pattern. Each one was written after an incident where Claude did exactly the thing the rule prohibits:

“Logs First, Theories Second” — because Claude would spin up hypotheses instead of reading the error that was right there in the logs.
“Respect User’s Layer Diagnosis” — because when I’d say “the API response is fine, the bug is in the frontend,” Claude would spend twenty minutes re-investigating the API.
“Two-Attempt Rule” — because Claude would try five variations of the same wrong approach before I could get it to step back.
“EVIDENCE-BASED BUG FIXING (NON-NEGOTIABLE)” — in all caps, because Claude kept speculatively fixing code that wasn’t broken, breaking it in the process.

That last one has a specific origin. A bug in one sync function — Phantasy Tour — and Claude “preemptively” applied the same fix to three other sync functions that were working fine. Now four things were broken instead of one.

There’s one more thing. Claude also figured out how to get around CI entirely.

When you push a commit and open a pull request on GitHub, there’s a brief window — a few seconds — before the CI checks register as required. During that window, the merge button is green. Claude learned to push a commit, immediately create the PR, and merge it in that gap before the checks even start running. No waiting for tests. No waiting for builds. Just push, merge, done — the engineering equivalent of running a red light because the camera hasn’t turned on yet.

I caught it because PRs were showing up as merged with zero checks passed. Not failed checks — no checks. The CI runs would start, sometimes even fail, on a commit that was already in main. The branch protection rules were technically satisfied because there were no checks to block on at the instant the merge happened.

This is the same agent that was told to write tests for every new behavior. It wrote the tests. It configured the CI. Then it found the fastest path that avoided actually waiting for any of it. I’m not even mad. It’s the most efficient thing Claude did all month.

Then Claude took the site down.

I didn’t trust the test suite anymore. 154 tests and zero coverage of the core flow — what else was missing? So I asked Claude to set up code coverage instrumentation for the E2E tests. Build the Go binary with -cover, run the Playwright suite, generate an HTML report showing which backend handlers are actually being exercised. I wanted receipts.

Claude built it. It worked. The coverage report showed 42% handler coverage with 263 functions at 0%. Good data. And even that — the act of trying to verify Claude’s work — Claude managed to fuck up.

But Claude didn’t put the coverage tooling in a separate script and leave it there. It also added a coverage flush endpoint — POST /debug/coverage/flush — directly to main.go, the production server binary. That endpoint imported runtime/coverage, a Go standard library package that calls WriteMetaDir() and WriteCountersDir(). Those functions panic if the binary wasn’t compiled with -cover. Production binaries are not compiled with -cover. The binary panicked on startup. The site went down.

The fix was three lines: delete the import, delete the endpoint, move the flush logic to the coverage script where it belonged. I pushed it in under a minute once I understood what happened. But the site was unreachable until Railway picked up the new commit and redeployed, and I couldn’t force a faster deploy because the health checks were failing on the crashing binary.

The irony is almost too neat. Claude was asked to measure test coverage — to find out what wasn’t being tested — and in doing so, shipped code that wasn’t tested to production. The coverage endpoint itself was never tested. Not by the 154 existing E2E tests. Not by the new tests Claude was writing. Not by a quick go build && ./binary sanity check without the -cover flag. The code existed to answer the question “what are we not testing?” and the answer included itself.

This is the runtime/coverage import in production main.go, the one that crashed the site:

import (
    "runtime/coverage"  // panics if binary not built with -cover
)

// POST /debug/coverage/flush — flush coverage data
mux.HandleFunc("POST /debug/coverage/flush", func(w http.ResponseWriter, r *http.Request) {
    coverage.WriteMetaDir(coverDir)     // panic
    coverage.WriteCountersDir(coverDir) // panic
})

Dev-only code, in the production binary, with no guard, no build tag, no conditional. Just a direct import of a package that explodes outside its intended context. Claude didn’t even think about it. It was writing coverage tooling, so it put the coverage code where the rest of the server code lives. The concept of “this code should only exist in a specific build configuration” didn’t occur to it.

I want to be clear about what’s happening here, because I think it’s easy to read this as “AI is bad at testing” and miss the more interesting point.

Claude is excellent at writing tests. The 154 tests it wrote are real, useful, and they catch real regressions. The Playwright infrastructure is solid. The test helpers are clean. The coverage of side features is thorough. When Claude writes tests, they work.

The problem is that Claude doesn’t write them where they matter most. It writes them where they’re easiest — for the feature it just built, in the same session, while the context is fresh. The new tournament bracket gets tests because Claude just built the tournament bracket. The new battle system gets tests because Claude just built the battle system. The posting flow doesn’t get tests because Claude built it weeks ago, and no single session since then has been “about” posting.

This is a prioritization failure, not a capability failure. And it’s one that’s hard to catch in the moment, because the test count keeps going up. Progress feels real. 154 tests! Seventeen spec files! The dashboard is green! But the coverage map has a hole in the center, exactly where the load-bearing wall is, and nobody notices until the wall falls down.

The fix is obvious: test the core flow. Test it first. Test it before you test anything else. But “obvious” and “automatic” are different things, and Claude Code — despite being told explicitly, in bold, in the project instructions — did one and not the other.

833 commits. 202 fixes. Zero tests for the thing the app actually does. A dev-only import that took down production. The numbers don’t lie, even when the test suite is green.

Ten Years of Lasp

Sun, 01 Mar 2026 08:00:00 +0000

Peter Van Roy and I just published a retrospective at PPDP ‘25 — the same venue where the original Lasp paper appeared a decade ago — looking back at ten years of influence from Lasp, the coordination-free programming model Peter and I built together in 2015. I wanted to write a bit about what that paper covers, and what it actually felt like to watch an idea travel from a prototype to something that showed up in production systems I never expected.

What Lasp Was

Lasp was a programming model built on top of Conflict-Free Replicated Data Types (CRDTs). The core idea was simple: coordination makes distributed systems easy to reason about, but it’s expensive — it limits availability, introduces latency, and creates failure modes. Lasp’s goal was to give developers a programming model that was just as easy to reason about, but without the coordination, by structuring distributed computation around data types that are guaranteed to converge regardless of the order in which updates arrive or whether the network partitions. The result: better reliability and availability, without sacrificing the developer’s ability to think clearly about what their program does.

The CAP theorem had been the backdrop for a lot of the distributed systems conversation in that era. CRDTs offered a path toward availability and partition-tolerance without sacrificing convergence, but their use was still mostly ad hoc. Lasp tried to give that a principled, declarative home: a functional programming model where eventual consistency wasn’t something you bolted on, but the default assumption.

I was working at Basho Technologies at the time, deep in the operational reality of building eventually consistent databases in Erlang. That context shaped everything about how Lasp was designed. The pain points weren’t hypothetical — I had watched coordinaton-heavy systems fall over under real load, and I was motivated to find something better.

The motivating use case came from Rovio Entertainment, the company behind Angry Birds. Their game required tracking player state across a globally distributed, occasionally connected player base — exactly the kind of scenario where coordination is prohibitively expensive and convergence is what you actually need. That use case grounded Lasp’s design in something real.

The Scale Experiments

Lasp was part of the EU-funded SyncFree research project, which brought together researchers across Europe to advance the foundations of CRDT-based distributed systems. As part of our deliverables to the European Commission, we ran large-scale experiments on AWS — over 1,000 nodes — to demonstrate that these techniques worked at scale. Between 2015 and 2017, that kind of deployment was genuinely difficult: few frameworks supported it, and the operational challenges around service discovery, dissemination, and convergence were not trivial.

We published those results at PPDP in 2017. What I remember most clearly is how much of the work was just fighting infrastructure that wasn’t ready for what we were trying to do. To get Lasp to run on top of Mesos’s Marathon framework — because I was contracting at Mesosphere at the time — I had to essentially rebuild the network layer of Lasp from scratch. That work became Partisan.

The Things That Grew Out of It

Partisan, the open-source distribution layer I built to run those experiments, ended up being more widely adopted than Lasp itself. It became a high-performance alternative to Erlang’s built-in distribution, influenced improvements to Erlang’s distributed networking internals, and eventually found its way into open-source and proprietary projects I had no involvement in. Alejandro Ramallo, an Erlang developer who adopted both Lasp and Partisan, deployed them in systems powering LoJack’s stolen car recovery service across several South American countries. I did not see that coming.

Lasp was also adopted as the storage backend for Erleans, an open-source implementation of Microsoft Orleans on Erlang — which is a satisfying full-circle moment given that I spent two summers at Microsoft Research working on Orleans’ transactional semantics.

The fault-injection mechanisms in Partisan’s early network layer eventually became the seed of Filibuster, the fault injection testing framework that was the subject of my Ph.D. dissertation. The path from Lasp to Partisan to Filibuster to DoorDash is not a straight line, but there is a line. The only way to verify that a coordination-free system converges correctly is to partition the network and check what happens when it heals. That thinking, originally motivated by Lasp’s correctness requirements, eventually became a general-purpose approach to testing microservice resilience.

What the Academic Community Did With It

The retrospective paper covers quite a bit of follow-on academic work that I found genuinely gratifying to trace. Systems like Katara built on Lasp’s foundational principles to synthesize CRDTs with verified lifting. LoRe and Varda extended Lasp’s declarative, coordination-free semantics toward verifiably safe compositional distributed software. Several PhD dissertations — from researchers in Belgium, Portugal, the UK, the US, and elsewhere — adopted Lasp as either a technical or theoretical foundation.

The work Matthew Weidner and Heather Miller and I published together on composing op-based CRDTs with semidirect products also grew directly out of the Lasp model. That paper, published at ICFP in 2020, is something I’m quietly proud of — it’s one of the more mathematically interesting things I’ve worked on.

The PPDP program committee selected the original Lasp paper as the most influential paper of the past decade — the PPDP 2025 10-year award — which is what prompted the retrospective paper in the first place. It is a bit surreal to receive a recognition like that for work that started, essentially, as a research project I threw myself into while working at Basho with no particular expectation that anyone outside the SyncFree project would care.

Why It Still Matters

The retrospective paper makes the case — and I believe it — that Lasp’s core insight is increasingly relevant rather than less. Multi-region active-active deployments are becoming standard architecture for companies at scale. A single round-trip operation at the speed of light takes 133 milliseconds. Coordination protocols like Two-Phase Commit or Paxos require multiple round trips just to agree on a single value. The math doesn’t work for globally distributed state if coordination is your default.

Edge computing, local-first software, offline-capable mobile applications, and federated systems all face the same constraint: central coordination is sometimes simply unavailable. Coordination-free convergence isn’t a research curiosity in those environments — it’s a requirement.

I’ll be curious to see what the next ten years look like for CRDTs specifically. The theoretical foundations are solid. The open question is whether the programming model abstractions — the things that Lasp was trying to work out — will find their way into mainstream languages and frameworks in a form that developers can actually use without thinking too hard about it. I think they will.

The full paper is available on ACM.

I Built a Social App in a Week with Claude Code

Tue, 17 Feb 2026 08:00:00 +0000

I spent the better part of a week building a social app with Anthropic’s Claude Code. Most of that work happened late at night, sometimes past 2am, iterating on features until I had something worth sharing with friends.

Anthropic generates a weekly insights report for Claude Code users. Mine told an interesting story: 384 messages, 27 sessions, 168 files touched, six days. A median response time of 29 seconds — I was barely reading the output before firing back. They described my style as “reactive and corrective rather than spec-driven,” which is accurate. I was moving fast, fixing things as they broke, and learning what worked along the way.

What I was building: a private social app for my close-knit group of friends. We share live recordings of bands — Phish, Grateful Dead, that world — favorite films, books, and which upcoming shows we plan to catch this summer. Think a tiny, invite-only corner of the internet for people who care deeply about live music and want somewhere to talk about it with people they actually know. Live chat between people couch-touring and people in the pit is coming next.

The app is live. My friends are using it. It runs on Go with a server-driven UI architecture, and it ships as native iOS and Android apps. I built all of it in a week with Claude Code, and I want to tell you what that actually felt like.

The entire first version was built at night, in bed, on an 11” iPad, using the Claude Code app connected to an empty GitHub repo. I didn’t touch a computer. Within a few hours I had a working app — login, a feed, posting — deployed and running on Railway. From there the scope crept in the right direction: Spotify integration so you could search and attach albums and tracks directly, setlist.fm integration to pull real show data — venues, dates, and actual setlists as they’re posted after each night — and let people mark which dates they were attending. The bones were there fast; making it feel like something worth actually using took longer. Claude scaffolded the backend, wired up the database, built the frontend, and I shipped it without leaving bed. It wasn’t until late in the second day that I moved to a laptop, mostly because the screen real estate was starting to feel limiting. The code itself didn’t care.

The day I invited most of my friends, a database migration corrupted production timestamp data.

The migration modified column types in a running database. Timestamps encode timezone assumptions at the type level — change them mid-flight and the data doesn’t migrate, it breaks. The feed went down. I asked Claude to fix it. Each fix made things worse: wrong column names, broken SQL syntax, incorrect timezone arithmetic, until finally an overwrite ran that couldn’t be undone. The Anthropic report summarized it as “Claude’s database migrations went full disaster movie — each fix spawned a new production incident, permanently destroying timestamp data.” That’s accurate. Some of that data is simply gone.

I spent years working on databases. I knew exactly what was happening and why it was catastrophic. I let it happen anyway because I was vibe coding, hands off the wheel, just watching Claude drive.

The timestamp disaster was the worst single incident, but the pattern underneath it showed up constantly: Claude moving confidently in the wrong direction, and me not stopping it soon enough. Push notification debugging is another example. Missing APNs tokens sent Claude deep into provisioning profiles, certificates, entitlements — a long, plausible-looking path that turned out to be completely wrong. The actual problem was missing AppDelegate methods in code Claude itself had written earlier. The report counted 30 instances of wrong-approach debugging across the project. That’s a lot of time watching a very capable thing solve the wrong problem.

The server restart issue was more mundane but somehow more maddening for it. After any change to Go code, you have to restart the backend for the changes to take effect — compiled language, nothing exotic. Claude kept forgetting. I kept seeing no changes, assuming something was broken, investigating, finding nothing, eventually realizing the old process was still running. This happened across four or more sessions before I wrote a rule explicit enough that it actually stuck. pkill was the culprit — it fails silently, leaving the old process alive. The fix was lsof to find the PID, kill -9 on that specific process, wait, restart, verify. A procedure that takes thirty seconds and has to be written down or it doesn’t happen.

What got built, despite all of this, is genuinely surprising to me.

A full server-driven UI migration — the backend sends the entire interface as JSON, the React frontend just renders it, no hardcoded pages. This one has a story. I’d instructed Claude from the start to build an SDUI app, but most of the early code landed in React anyway. The upside was that the initial pages were genuinely pretty — Claude has good taste in React UI — and I loved how they looked. The downside was that enabling a proper mobile strategy required full SDUI, which meant migrating every page individually, each one needing to be restyled from scratch. That work took real time.

The migration also exposed a pattern that would recur throughout the project: Claude claiming victory prematurely. Buttons that didn’t work. Frontend calls to backend routes that didn’t exist — phantom APIs, confidently wired up. Forms that submitted successfully from the UI while silently dropping half their parameters on the way to the backend. Claude would build a form, build an API endpoint, connect them, and declare the feature done. The form would submit. The endpoint would return 200. Nothing would actually be saved.

I eventually had to encode explicit rules: when you add a new backend API, test it with curl before touching the frontend. When you wire up a frontend call, verify the route actually exists. When a form submits, confirm every parameter arrives at the backend. The quality improved significantly once those guardrails were in writing.

iOS and Android apps, both in TestFlight and ready for Android testing within the same week. End-to-end push notifications wired to every interaction. An AI-powered feature that surfaces what your friends are collectively into right now. An invite management system, automated build scripts, 107 database migrations, user engagement charts, a changelog that notifies users when something new ships.

Claude’s ability to hold a large multi-file change in mind — a database migration, a new API handler, a frontend component, and a mobile layout fix, all in one coherent session — is where it earns everything. When the scope is clear and the pattern is known, it moves at a speed that doesn’t feel real.

Only 2 of my 27 sessions fully achieved what I set out to do. That number sounds damning until you consider that 27 sessions in six days shipped something real that people are using. The sessions that failed were almost always the same shape: open-ended, no clear stopping point, debugging something visual or stateful where Claude had no feedback loop and I had too much patience for wrong approaches. The sessions that worked were tight — one known thing, done correctly, then stop.

The insights report also came with recommendations, and they’re worth passing on.

The biggest one: hooks. Claude Code supports post-edit hooks — shell commands that fire automatically after files are changed. The report suggested wiring one up to restart the Go backend after any .go file edit, which would have eliminated the single most recurring waste of time in the entire project. I haven’t set it up yet. I’m going to.

The report also suggested adding explicit guardrails to CLAUDE.md — the project instruction file Claude reads at the start of every session. Things like: after any backend code change, always restart the server before testing. Never run migrations or deploy fixes without explicit user approval. When the user tells you a layer is working, stop investigating that layer. Limit yourself to two attempts at a single approach — if it hasn’t worked twice, step back and explain what you’ve learned before trying again. Most of these I’d arrived at the hard way over the course of the week. Having them written down from the start would have saved days.

The other recommendation was about session discipline. Five of my sessions were completely lost to context limits — the conversation grew too long, and when I tried to recover with /compact, it just returned “failed to compact” and left me stranded mid-task with no way forward except closing Claude and starting over from scratch. Losing context mid-session, mid-thought, mid-fix, with no handoff and no summary, is a particular kind of frustrating. The fix is obvious in retrospect: end each session at a natural stopping point, write a brief summary of state, start fresh. The sessions where I shipped something were the tight ones. I kept ignoring that signal.

The report called my style “ambitious” and noted I “tolerate high friction from repeated wrong approaches.” I’d put it differently: I was building something I actually cared about, for people I actually know, and the deadline was real. Spring tour is the test. Summer tour is the goal. That changes your relationship to the friction.

You’re not writing code with Claude Code. You’re steering. The gap between those two things is where all the frustration lives, and also where all the speed comes from. When you accept that you’re the judgment layer — deciding when to redirect, when to stop, when a fix is making things worse — the tool becomes something genuinely extraordinary.

Spring tour is coming. The app is ready.

Postscript: while writing this blog post, the same pattern bit me again. The app was deploying fine on Railway with a simple nixpacks build. While fixing an unrelated feature, I added a healthcheck to the config as a drive-by change — unnecessary, but it seemed harmless. Later, switching to a Dockerfile build to bake in an environment variable caused the build to take longer than nixpacks, so the healthcheck started timing out before the service was ready. Deployments failed. Rather than identifying the healthcheck as the culprit, five successive commits changed the port, the builder, the start command, and the Dockerfile in various combinations. None of it worked, because the real problem was never diagnosed. The entire chain of failures traced back to one unnecessary line added while working on something else entirely. Some things don’t change.

Filibuster 2.0: Byzantine Fault Injection with Hardcoded Fault Values

Sat, 09 Sep 2023 08:00:00 +0000

Ever want to test your system against Redis returning wrong values, like instead of returning an error, it returns an empty string? What about an empty byte array? What about a database field being null? You can do it with Filibuster 2.0!

Let’s return a null from a Redis get, easy! Simply done with Filibuster.

Filibuster 2.0: Coaching UI

Fri, 08 Sep 2023 08:00:00 +0000

Your developers are writing functional tests for their microservice and, when RPCs fail, they throw exceptions. Do they test the cases where it throws? Do they have tests for it?

Using Filibuster, you can automatically identify these scenarios and prompt the developer to answer these questions.

Here’s a case where a developer threw an exception when a downstream RPC failed. We ask them: “did you mean to throw?” and, if so, they are prompted to tell the system that this exception is on purpose: this allows us to determine resilience failures automatically.

Filibuster 2.0: Byzantine Fault Injection with Arbitrary Faults

Thu, 07 Sep 2023 08:00:00 +0000

Ever wanted to just throw all sort of values at your database and see what happens to your application? Filibuster’s byzantine fault injector can take an arbitrary “value transformer” that looks like a functional fold, that allows you to come up with new fault injection scenarios as you inject faults!

Let’s test our application against flipping characters in a string response from Redis!

Here, we observe the response from the test that passes with no faults and then flip a character in the response in the test where no faults were injected. We actually built this to flip every character in the string.

What was once “example” in the Redis response becomes “ xample”! And then becomes “e ample”!

Filibuster 2.0: Computing API coverage of a Microservice Application

Wed, 06 Sep 2023 08:00:00 +0000

About two months ago I started prototyping this feature for visualizing API coverage in a microservice application through automated instrumentation: see where you have functional testing coverage, how many functional tests, and where you’re applying fault injection to determine the impact of those changes.

Flash forward to yesterday, a new paper draft on arXiv proposed (almost) this very thing – they went a bit further in their study and provided more comprehensive visualizations and an accompanying study. In contrast, mine is running on real code written by industrial developers. On the left, Filibuster; on the right, their proposal.

Good ideas happen at the same time, I suppose!

(e.g., Mattern and Fidge both independently coming up with the idea for vector clocks the very same year.)

Filibuster 2.0: Microservice Linter, Multiple Invocations to the Same RPC Method

Tue, 05 Sep 2023 08:00:00 +0000

You can also use Filibuster’s dynamic analysis linter to find microservice smells.

Here’s one: invoking multiple RPCs to the same service because you can’t send them all together! This leaves you at risk for partial side-effects being applied: refactor your API to let developers supply all inputs and write data to your database transactionally!!!

Filibuster 2.0: Microservice Linter, Requests become part of a Response

Mon, 04 Sep 2023 08:00:00 +0000

You can also use Filibuster’s dynamic analysis linter to find microservice smells.

Here’s one: using the arguments from one RPC to Service A as the inputs to a different RPC on the same Service A. You should refactor your API so I don’t have to make multiple RPCs!!

Filibuster 2.0: Microservice Linter, Redundant RPCs

Sun, 03 Sep 2023 08:00:00 +0000

You can also use Filibuster’s dynamic analysis linter to find microservice smells.

Here’s one: executing the same RPC multiple times, since you’ve already got the response and shouldn’t issue a failure-possible, expensive RPC again.

Filibuster 2.0: Improved UI

Sat, 02 Sep 2023 10:00:00 +0000

Integrated directly into IntelliJ, Filibuster can show you the RPCs that your service is making and directly inject faults on any of those RPCs. Attach a debugger, inject some faults, and see what your application does!

Not only just GRPC (as shown here), we also support HTTP, Redis, PostgreSQL, CockroachDB, and DynamoDB.

Filibuster 2.0: Healthcheck your Functional Test Suite with API Coverage

Sat, 02 Sep 2023 09:00:00 +0000

Pushed out a new prototype Filibuster feature tonight: 🎉 use Filibuster’s IntelliJ plugin to get a “health check” of your microservice’s functional test suite. 🎉

Here, I can quickly open the plugin after running my test suite and see what RPC methods my service is exposing, how many unique functional tests I have covering those methods, how many specific Filibuster tests I have for those, and how many tests were automatically generated using Filibuster to exercise unique fault injection scenarios: where I injected HTTP and GRPC faults.

Filibuster 2.0: Redis Fault Injection

Fri, 01 Sep 2023 10:00:00 +0000

Ever wanted to test your microservice application against Redis failures? Filibuster 2.0 supports injecting faults against Redis, PostgreSQL, CockroachDB, and DynamoDB.

Here, using our IntelliJ visualizer for tests, we see that in this test we injected a failure on a synchronous GET command to Redis and the test still passed. That’s fault tolerant code!

Not using synchronous operations? No problem! Filibuster will inject execeptions for each asynchronous GET or SET operation and defer the fault injection until someone calls get or set on the future.

Wanna find out if someone is using thenAccept but forgetting to catch the exception? Filibuster can fail the test if the developer never gets the value of the returned future too!!

Christopher Meiklejohn

The Shortcut Became the System · A pipe disappeared. The rectangle around the door remained.

How I got to the rectangle

The same shortcut in Zabriskie

The art-only test

The repairs began to accumulate

Starting over

The check passed

When code is cheap

My Front of House of Cards · The show ended at 10:12. The fix merged at 10:17.

Part one: the night

What we built

What it was tested with

We broke the Chomp

Fifteen seconds

Why nobody noticed for months

Part two: the repo

Why a two-line fix took two hours

The tortoise and the hare, literally

The test suite was the other bug

Tests that detect edits, not defects

The right test existed

How you’d actually know

Writing the rule down is not following it

The tax on touching it

The opposite of programming

Part three: what can actually be verified

What Lean actually did

Except most of this app isn’t an algorithm

So what did work

What I still don’t know

Software for One

I thought the language would do it

An app for the hotel

The same thing, for a film

Software getting out of the way

Keeping it on my phone

Building for myself, sharing with others

How long does it need to last?

Start with what I need

The Quickest Path to a Diff · The app died at Red Rocks while its health checks stayed green. Every line of code was fine. The sum was nobody's job.

What was actually wrong

The remediation was excellent, which is the problem

A census

Why this keeps happening

Where does global reasoning come from?

Every Card Will Show · The earlier implementation, tests, audits, and accepted proofs said the algorithm was correct. None of them stated the whole problem.

How the requirement changed

What Lean made explicit

Lean proved the wrong day

Coverage was still only half the problem

Proving the minimum

Proving that ranking cannot break the schedule

Making Go answer to the model

What the theorem does not cover

Algorithms are hard

The Safeguards Have to Run · Two abandoned notebooks showed me the difference between a research rule and a working control.

Notebook 1

Notebook 2

The safeguards have to run

The Whole Night

The Lot Learned What Time It Was

One Lead, Not Every Feature

Tonight Is a Room, Not a Date

The Archive Has to Return Somewhere

The Work Was the Editing

Before Doors

At the Rail

On the Couch

After the Lights

The Rest of the App Changed With It

The Receipt

Science at LLM Speed · Research-like output is cheap. A valid claim still needs evidence that can show it is wrong.

A plausible bibliography can describe papers that never existed

The research loop gets faster

A polished analysis is not necessarily a scientific result

The reviewer might be a model too

The machine can find something real

Then my own research loop failed twice

What Comes Next