I wanted to build a tool that could tell Goose fans what song they were hearing while a show was still happening. I also wanted an autonomous LLM system to do the research needed to build it. The system would propose experiments, write and run code, evaluate the results, and decide what to try next. The tool became SetScope.
By my estimate, the agent compressed weeks of implementation into hours. It also helped produce two polished research notebooks that I later discarded. The first expanded across song recognition and improvisation research until its central public claim failed held-out validation and the notebook no longer had independent evidence for the path it had taken. The second explicitly rejected a 90-second segmentation rule, then reused that rule in its first implementation. Both came with clean code, persuasive charts, and conclusions that sounded finished.
I have a PhD and years of systems-research experience. The problem was not that I had never heard of a holdout or did not understand that a measurement must match its method. I worked from the reports because an autonomous researcher that must be manually reconstructed after every run is not autonomous. The project began with automatic song and set detection. The recordings pulled it into harder questions about improvisation. After both notebooks failed, I returned to the original goal: build the live song guesser.
On August 13, we ran SetScope during a Goose show whose audio had not existed during development. The saved record shows that SetScope produced the right song at least once for 10 of the 12 performances. It also missed two performances and switched songs three times when it should not have. The record is incomplete, so those numbers do not amount to a whole-show accuracy score.
This series will follow that story from beginning to end. It starts with what the autonomous researcher made possible and how it convinced me twice that invalid work was finished. It then explains how a review system in which every check passed still accepted the wrong interpretation of the audio. After the eleven-show August 13-28 run, the final post will report which versions actually ran, how quickly correct guesses appeared, where the system failed, and what the surviving records can establish about the viewer-facing system.
This is not a prompting guide. It is for people using LLMs to run experiments or build the systems around them. The question throughout is simple: when an agent can produce the next result by itself, what evidence gets to say that the result is wrong?