I wanted to build a tool that could tell Goose fans what song they were hearing while a show was still happening. I also wanted an autonomous LLM system to do the research needed to build it. The system would propose experiments, write and run code, evaluate the results, and decide what to try next. The tool became SetScope.
By my estimate, the agent compressed weeks of implementation into hours. It also helped produce two polished research notebooks that I later discarded. One leaked recordings between training and test. The other analyzed the wrong portions of the songs because it inherited a 90-second segmentation rule that contradicted the method I had written. Both came with clean code, persuasive charts, and conclusions that sounded finished.
I have a PhD and years of systems-research experience. That background did not prevent me from accepting either result. The project began with automatic song and set detection. The recordings pulled it into harder questions about improvisation. After both notebooks failed, I returned to the original goal: build the live song guesser.
On August 13, we ran SetScope during a Goose show whose audio had not existed during development. The saved record shows that SetScope produced the right song at least once for 10 of the 12 performances. It also missed two performances and switched songs three times when it should not have. The record is incomplete, so those numbers do not amount to a whole-show accuracy score.
This series will follow that story from beginning to end. It starts with what the autonomous researcher made possible and how it convinced me twice that invalid work was finished. It then explains how a review system in which every check passed still accepted the wrong interpretation of the audio. After the eleven-show August 13-28 run, the final post will report which versions actually ran, how quickly correct guesses appeared, where the system failed, and what the surviving records can establish about the viewer-facing system.
This is not a prompting guide. It is for people using LLMs to run experiments or build the systems around them. The question throughout is simple: when an agent can produce the next result by itself, what evidence gets to say that the result is wrong?