Prompt Testing Is a Plinko Machine

An agent writes the shop descriptions for my TikTok store. Most of them go out with the video without anyone reading them first. Last week I ran its prompt six times just to see what it actually produces, and three came back with em-dashes. I ran it six more times and got six. Same prompt, same input, same model, same evening.

That is twelve product listings, nine of them stamped “written by a robot” for anyone who has read AI output for the last two years. If I had run it once, seen a clean result, and shipped, I would have been wrong most of the time and never known.

The same pipeline also shipped a product launch video with the launch cut out of it. Seventeen seconds gone, including the part where we say what the thing is. Different defect, same root cause: nothing was measuring either one.

Most of the code I’ve written in my career has been deterministic. X + Y = Z. You write a test, plug in X and Y, and Z is a known outcome. That test protects the code later when someone else comes in and starts changing things.

LLMs and AI agents don’t work that way. They’re a plinko machine. Your words pick where the puck drops in, but the same prompt across ten runs can land in ten different slots. There is no Z. There’s a distribution of different answers.

So if you want to test a prompt, you have to stop asking “did it work?” and start asking “how often does it work?” Run it N times. Count the passes. Change one thing. Run it N times again. Compare. That’s statistics, and I know, I know, nobody got into software to do statistics, certainly wasn’t my goal. But it’s turned out to be the ideal way “unit testing a prompt” that has helped me tame the plinko machine for automated workflows.

I built a tool to do exactly this work. It’s called promptdiff, and I mentioned it in passing in the harness tax. This post is one real investigation with it, run against my own TikTok video pipeline. Including the part where the tool told me no.

What promptdiff actually does

It’s a small Bun CLI. You give it an agent prompt, a scenario file that describes the experiment, and a grader. It runs the prompt N times and grades every run. Model access goes through a runner: the default shells out to headless Claude Code (claude -p), which is how the runs below could read files and write artifacts. A second runner sends a single chat completion to anything with an OpenAI-compatible endpoint, so ollama, vLLM, and OpenRouter all work for text-only evals.

Two words you need for the rest of this post:

• Arm: one side of a comparison. The baseline arm is the prompt as it is. The proposed arm is the prompt with your change.
• Grader: any executable. It runs after the agent is done, inside that run’s working directory. Exit 0 means the run passed, any other exit means it failed, and whatever it writes to stderr gets printed in the summary so you can see which rule fired.

Two commands do the work.

measure runs one arm N times, grades each run, and reports a pass rate for each thing it checks. No comparison, no verdict. It exits 0 whenever the runs complete, because a measurement has no pass or fail. You run it first, before you change anything, to find out what your current prompt actually does. I had never done that on this pipeline, which is how the next section happens.

compare runs two arms, baseline and proposed, N times each. Every thing you check is a case, one prompt plus one grader, tagged with what it’s for. Two of those tags carry contracts (a third, plain compare, just reports both numbers and asserts nothing):

• Target: the thing you’re trying to fix. The baseline must not already fully pass (otherwise there’s nothing to prove), and the proposed arm has to improve the pass rate.
• Regression: the thing you must not break. The proposed arm must not fall below the baseline.

Fail either contract and compare exits non-zero. That first contract matters a lot here too; hold that thought.

Back to the grader for a second. An exit code and some stderr is the simple contract, and I like it that way, because a bash one-liner and a 70-line JavaScript file are both valid graders. What I don’t like is that mine is 70 lines of JavaScript for what is usually three lines of actual assertion. The rest is argument parsing, reading the artifact, and exit-code plumbing. I filed an enhancement for the library as queso/promptdiff#32: a grader file format where the tool owns the plumbing and you write the assertions. It’s the snippet people see first in a README, and today it reads like plumbing. Fixing that is what I’m building next.

Install:

bun add -g @theaiteam/promptdiff
# or
npm install -g @theaiteam/promptdiff

Now the investigation.

The video pipeline

I run a 3D print business called ArcaneLayer, and we post product videos to TikTok Shop. autocut is the pipeline that turns a raw single-camera take into a posted vertical video. It’s a flow on Conduit, my workflow engine, made of stations. Deterministic stations shell out to a CLI: transcribe, align words to the audio, crop to 9:16, burn in captions, render. Agentic stations run a model against a prompt file. The two agentic stations in this post are the Draftsman, which reads the transcript and writes the edit plan (which seconds of the take to keep, in what order), and the publishing package writer, which writes the shop description that ships with the video. Each agentic station has a critic that can reject its output and send it back for bounded rework, the maker and rival pattern from the plugin post. The Draftsman’s critic is called the Hook Guard. It will matter later.

The station with no instructions

The publishing package station reads four things from the job directory: the request, the transcript index, the finished edit plan, and the shop profile. It writes one JSON file with a body and a list of hashtags. Sales-forward copy: name the product, get people to the cart.

Here’s the deal: the production prompt for that station, version 2, the one running in production today, contained zero voice guidance. Not thin guidance. Zero. I had deployed a copywriter with no instructions about how to write, so I shouldn’t have been shocked by the text.

But the output had agentic tells. AI agents aren’t deterministic, but they do have habits that show up in their writing, as we’ve all noticed by now. The em-dash is the big one.

So I wrote a grader. Five rulers, all deterministic, run against the body text of the description. The core of it:

const RULERS = [
  ["em-dash", /—|–/],
  [
    "engagement-ask",
    /let me know|comments? below|drop a comment|tell (us|me) (in|below)|which one.s your|what.s your favorite|tag a friend/i,
  ],
  [
    "ai-ism",
    /elevate your|whether you.re|look no further|game.?changer|level up|say goodbye|say hello to|isn.t just|not just a|more than just|get ready to|unleash|dive into|treat yourself|the perfect (gift|way|addition)|must.?have/i,
  ],
];
const bangs = (body.match(/!/g) ?? []).length;
if (bangs >= 3) fired.push(`exclamation-pileup (${bangs} bangs)`);
const emoji = (body.match(/\p{Extended_Pictographic}/gu) ?? []).length;
if (emoji >= 3) fired.push(`emoji-spam (${emoji} emoji)`);

Any ruler fires, the run is dirty. I calibrated the rulers against the description a human actually finalized and shipped for this job, so the real clean copy grades clean.

Every run starts from a fixture: a directory of files copied fresh into that run’s sandbox. The agent does its work in there, and the grader runs in there afterward. Mine is a real job, not a synthetic transcript. The launch video for our Cinema Cutie micros, a dozen mini articulated characters sold in a popcorn bucket. Here is what unguarded sonnet wrote for it on one of the dirty runs, so you can see the tell in the wild:

Our Cinema Cutie line just went micro 🍿 A full dozen mini characters — the bunny, glow-in-the-dark goth bat, Tilly, Ricky the Rat, Rainbow Mystic Shark, and Sally the Skunk — packed into their own popcorn bucket, now in pastel pink or classic red-and-white. Tap the orange cart below to grab your own bucket of micros!

Good copy. Two em-dashes. Anyone who has read AI output for the last two years can spot the tells from a mile away! 😆

Measure first

I ran measure on the production prompt with sonnet. Six runs. Three dirty.

Later that evening I ran compare, and the baseline arm of that run is the same unguarded prompt on the same model. Six runs. Six dirty.

Same prompt. Same fixture. Same model. 3 of 6 in one batch, 6 of 6 in the next. Pooled, unguarded sonnet reached for the em-dash in 9 of 12 runs. That per batch difference in the runs is the entire reason the harness exists.

And one more thing the stderr diagnostics made obvious: every single failure, in every arm, in every batch, was the em-dash. The engagement-bait ruler never fired. The cliché ruler never fired. No exclamation pileups, no emoji spam. One tell, over and over.

The obvious fix

Use the smarter model. That’s what everyone reaches for first, and it’s what I reached for.

Opus, same unguarded prompt, same fixture, six runs: 5 of 6 clean.

arm (unguarded, measure, 6 runs)	clean	tells fired	cost for the batch
claude-opus-4-8	5/6 (83%)	em-dash x1	$1.75
claude-sonnet-5	3/6 (50%)	em-dash x3	$1.16

Better. And still one em-dash in six. That one is not hypothetical, either. Before this eval existed, opus put an em-dash into a real ArcaneLayer description that went out in July.

On N: six runs per arm is small, and I knew it going in. promptdiff labels any delta that sampling noise could explain with a Fisher exact p-value, and at three runs per arm even 0/3 to 3/3 only reaches p=0.10. Six per arm was what fit inside the budget cap I set per scenario for one evening. Which means 5/6 against 3/6 is a direction, not a finding. These numbers are only convincing where the gap is total, and I’ll say so where it isn’t.

The smarter model buys you a lower defect rate. It does not buy you zero. If your whole fix is “use opus,” what you’ve actually done is move from a coin flip to a die roll and called it done.

The fix, measured

So I wrote five rules and dropped them into the prompt as a guard block:

1. No em-dashes or en-dashes. If a sentence wants one, split it into two sentences or use a colon.
2. No engagement-bait asks. Never ask viewers to comment, vote, tag a friend, or share. The only ask allowed is the purchase CTA.
3. No marketing clichés. Banned outright: “elevate,” “whether you’re,” “look no further,” “game-changer,” “level up,” “say goodbye to,” “say hello to,” “isn’t just,” “more than just,” “get ready to,” “unleash,” “dive into,” “treat yourself,” “must-have,” “the perfect gift/way/addition.”
4. At most 2 emoji and at most 2 exclamation marks in the whole body.
5. Write like the shop owner talking to a customer. Concrete nouns from the actual video: name the product, name the characters shown, name the options. Short sentences. If a line could be pasted onto any other product’s video, cut it.

Then compare. Sonnet, baseline versus guarded, six runs per arm per case. The scenario file is the JSON that describes one experiment: which agent prompt, which model, how many runs, what the sandbox looks like, and the list of cases. This one is two cases against the same fixture, and the only difference between the arms is that one extra file:

{
  "agent": "./agent.publishing.md",
  "baselineSkills": ["./guard.baseline.md"],
  "proposedSkills": ["./guard.tells.md"],
  "model": "claude-sonnet-5",
  "runs": 6,
  "sandbox": { "root": ".promptdiff/runs", "seed": "./fixtures/project" },
  "scenarios": [
    { "name": "desc-no-ai-tells", "kind": "target",
      "grader": { "type": "command", "command": "bun tools/tells.mjs" } },
    { "name": "desc-still-sells", "kind": "regression",
      "grader": { "type": "command", "command": "bun tools/tells.mjs --substance" } }
  ]
}

case	baseline	proposed (guarded)
desc-no-ai-tells (target)	0/6, em-dash all six	6/6 clean
desc-still-sells (regression)	6/6	6/6

The target case went from zero clean to six clean. Fine, that’s the number you expected. The regression case is the one that makes it credible.

Negative rules are dangerous because they can win by writing nothing. Tell a model “no dashes, no clichés, no exclamation marks” and it can hand you two beige sentences that trip no rulers and sell no product. So the regression grader checks substance: the product line has to be named (a regex for “micro” or “cinema cutie”), a purchase CTA has to be present (“cart,” “shop,” “grab,” “get your,” “link”), and the body has to be at least 150 characters. Six of six held on both arms. The guardrails did not gut the copy.

On a team, the grader matters more than the tool does. It is the acceptance criteria written down where a machine can run it, checked into the repo next to the prompt it grades. Run it with promptdiff or with a bash loop, the thing you are buying is the same: a prompt edit one dev makes on a Tuesday cannot degrade the output for everyone else without something going red.

Here’s a guarded run, for comparison with the dirty one above:

We just launched Cinema Cutie micros! Tiny plushies packed a dozen deep in your own popcorn bucket, pick pastel pink or classic red and white. Meet the crew: the bunny, the glow in the dark goth bat, Tilly, and new faces like Ricky the Rat, the Rainbow Mystic Shark, and Sally the Skunk. Tap the orange cart below to grab your own bucket of micros.

Here’s what actually happened, in one line: guarded sonnet reached 6/6 clean versus unguarded opus’s 5/6, with the regression case showing the copy still sells. Five explicit rules on the cheaper model, measured, versus the expensive model on vibes. On cost: the batch totals in the table above, plus about $2 for the whole 24-run compare. The evening on this station came to about $5.

Now read that guarded sample again. “Tiny plushies.” They are 3D-printed articulated toys. Both models did it. Sonnet said plushies in ten of its twelve compare runs, guarded or not, and opus said it in half of its batch. No ruler catches it, because I never wrote one for product fidelity. Candidate next ruler: banned material words sourced from the shop profile. And this is one fixture, one product, one transcript. Before I call the guarded prompt validated in general, we need a second pass with different source material.

The table is also what would justify keeping this station on sonnet. Every station in the flow can carry a model: line, and this one currently carries none, so it runs whatever the harness defaults to. If the guard block ships, model: claude-sonnet-5 on that station is a decision backed by data instead of taste. It hasn’t shipped yet. Production is still running the unguarded version 2 prompt while I write this. The release is coming: add the five rules to the production prompt, bump its version, keep this eval as the regression harness, and re-run it on every future prompt edit.

The one that wouldn’t reproduce

That was the easy investigation. Here’s the hard one, on the same pipeline, and it’s the one that made me trust the tool.

The worst defect autocut has ever shipped came from the Draftsman. On a real job posted 2026-08-09, it picked a hook from the middle of the take and deleted the entire 17.79 seconds before it. Here’s what got cut:

time	line	role	fate
0.01 to 3.45	”we launched some new stuff guys I’m so excited”	news	dropped
6.26 to 8.80	”this is our cinema cutie line”	product	dropped
9.26 to 13.64	”we decided to launch some micros”	product	dropped
13.64 to 17.80	”oh my goodness I can do it one-handed”	aside	dropped (fair)
17.80 onward	”look at all the micros in a little popcorn bucket”		kept

The published video opens on an unnamed object being shown to a viewer who was never told what it is. On a product demo. So the launch was never announced in that video.

The prompt allowed it to happen. It said the hook “does not have to be the first thing the presenter said” and said nothing about the material before it, so “hook first” got read as “discard what precedes it.” So I wrote a rule. The Draftsman’s prompt carries a numbered list of narrative rules, NAR-001 onward, and this one became NAR-006: choosing a later hook reorders the opening, it never deletes it. Re-judge every second before the hook on its own merits and place it after the hook. What the product is, and the news that it’s new, must survive in some form.

Then I built the eval. Three cases, all graded off the edit plan the Draftsman writes:

• Product line named: some kept range has to cover the 8.0 second mark, where the presenter says “this is our cinema cutie line.”
• Launch announced: some kept range has to cover 12.5 seconds, “we decided to launch some micros.”
• Still edits (regression): the kept duration has to be at most 90% of the take. A rule that says “keep the opening” must not degrade into “keep everything.”

Baseline should fail the first two, guarded should pass, ship it.

Attempt one: the unguarded prompt, the real transcript, opus, one run per arm as a cheap smoke test. compare exited non-zero: baseline fully passed; the target gap was not reproduced. The unguarded prompt kept the product description. Remember that first contract? The guard cannot improve a pass rate that is already 100%. The tool refused to hand me a green checkmark for a defect it couldn’t see. One run is a thin sample, but one clean baseline run is already enough to break the contract, and that’s the point of running the cheap version first.

Attempt two: maybe the trigger was the direction brief. In production, the Draftsman also reads a 100-line brief distilled from this shop’s 715 posted videos, which tells it which structural beats correlate with views and sales and which footage the data marks weak. Over-applying “avoid opening on weak footage” to a real product-launch opening was my best guess. I seeded the real brief and re-ran, again one run per arm. Identical result. Baseline 1/1 on both targets. About $8 of opus across the two attempts.

Attempt three: maybe it’s the rework loop. In production, the Hook Guard can reject the Draftsman’s plan and send it back with findings, up to three times. So I simulated that pressure: a round-one plan that keeps the opening, a clean score report, and a Hook Guard rejection explicitly telling the Draftsman to cold-open on the popcorn bucket at 17.8 seconds. Five runs, $3.95. Four of four graded runs kept the product line. Under direct pressure to reorder, the model reorders and keeps the opening. One of the five runs wrote no draft artifact at all, and I excluded that one as no-signal rather than tallying it as a pass or a fail. A missing artifact is a different failure class than a wrong artifact, and the raw “4/5” would have read as one reproduction if I hadn’t looked. That’s the third item in issue #32: the tool should tell those two apart on its own.

attempt	what I added	runs	result	cost
1	prompt + real transcript	1 per arm	baseline passed both targets	about $8 for 1 and 2
2	+ the direction brief	1 per arm	baseline passed both targets
3	+ a critic rejection demanding the reorder	5, baseline only	4/4 graded runs kept the opening, 1 no-signal	$3.95

Score to date: three hypotheses eliminated, about $12, trigger still unfound.

I know, I know. Twelve dollars to reproduce nothing. But that’s the wrong way to think about it. What I bought was three specific things weren’t the cause, and a tool that will not let me forge the fourth. A harness that can find a no is the only reason I can trust the yes. If compare had waved attempt one through, the guarded-sonnet table above would just be a number I liked.

Why can’t the fixture find it? Because the fixture is a single shot at the Draftsman and production wasn’t. The real job went through the critic loop, and the artifacts from that loop, the plan per round and what the Hook Guard said to it, were lost. Only the finished video and the source footage came back from storage. So attempt three’s rework context was constructed from my hypothesis, not read from what actually happened. The eval also replaces the Draftsman’s self-scoring step with “write valid JSON” to keep runs cheap, and a low-frequency trigger at five runs can hide easily. The next step is not more runs. It’s recovering the debating artifacts of a job that fails, so guessing turns into reading.

And to be clear about what the negative result means: NAR-006 stays in the prompt. The defect shipped. It’s real. The mechanism in the prompt text is real. The eval just can’t isolate the production trigger from a single-shot fixture yet. The rule stays turned on, and a green run of that eval must never be read as validation. We have to fix the logging and traceability in Conduit.

The model A/B that fell out for free

The opus-vs-sonnet comparison wasn’t a study. It wasn’t even a plan. The arm variable in compare doesn’t have to be the prompt text; hold the prompt and skills constant, vary the model per arm, and the same machinery that tests a prompt tweak will now test a model swap. The characterization table up top is two measure runs with a different model field.

Which is how I ended up with a measured finding that opus writes fewer em-dashes than sonnet does. Who would have guessed 🤷‍♂️

Including when the number is a shrug

Deterministic code gets a deterministic test. A prompt gets a distribution, so it gets a measurement test: N runs, a pass rate, a baseline, a proposed arm, and a grader that gives us the score.

That turns “I tweaked the prompt and it feels better” into a number you can understand. It turns “I don’t know if this actually works” into a number you’re willing to put in a blog post, including when the number is a shrug and the trigger is still out there somewhere. Both investigations together, the one that worked and the one that didn’t, came to about $17.

If you’re running a team on Claude Code and can’t say which of your prompts are doing real work, that’s this same problem on a larger scale: knowledge living in each dev’s chat history instead of in the repo, and no measurement to tell you when a prompt edit made things worse. That’s one type of problem I get hired to fix, and here’s how to start.

If you’d rather run it yourself, promptdiff is at https://github.com/queso/promptdiff. Bring a prompt you think is fine. Run measure on it twelve times. Then we’ll talk.