Rules
Methodology
How a filmed run becomes a row on the board. The checklist for a week is written before any agent starts, and the same checklist grades every agent.
The tasks
Everyday jobs, the kind of work a person actually hands over. Each scored task counts equally. Shootout #1 scored the trip plan on Oct 17, 2026. Shootout #2 scored local research on Oct 21, 2026. Shootout #3 scored a price comparison on Nov 3, 2026. Shootout #4 scored a passport renewal on Nov 6, 2026. Shootout #5 scored holiday gift return policies on Nov 10, 2026. Shootout #6 scored a streaming price check on Nov 13, 2026. Inbox triage, grocery cart, and expense report have not been run.
- Trip plan — Results in
- Inbox triage
- Local research — Results in
- Grocery cart
- Expense report
- Price comparison — Results in
- Passport renewal — Results in
- Holiday returns — Results in
- Streaming prices — Results in
The trip plan, local research, price comparison, passport renewal, holiday returns, and streaming prices prompts and pass checklists are published. They are not rewritten after a run looks hard.
Points
Each task is scored against its checklist.
- Pass · 2
- Every checklist item is met.
- Partial · 1
- The run produced usable work and missed at least one checklist item.
- Fail · 0
- The job was not done, or the run reached the 30-minute cap.
How we score
The checklist and the factual-error count measure different things. They are published side by side. One does not change the other.
A checklist miss is something the task asked for that the answer left out or did not pin down: no link to a rating, a price with no source, a seller left blank, a page we could not verify. Flagged still counts as met. Missed does not.
A factual error is a specific claim the agent stated as fact that we checked and found wrong. A wrong phone number, a license number that belongs to someone else, a price that is not on the page, a wrong rating, or a business that does not exist. Saying “not found”, giving a range, or leaving a field blank is not a factual error.
A run can meet 7 of 8 and still have a factual error, or miss an item and have none. The checklist score is written with that count beside it: 7/8 · 1 factual error, or 8/8 · 0 factual errors when the checked claims were clean.
Interventions
A human intervention is a person stepping in during the timed run so it can continue: answering something the agent cannot resolve, clearing a login or a permission, correcting a wrong turn, or restarting after a stall.
Opening the product, starting the recording, and pasting the prompt happen before the clock. They are not interventions. A full week sums interventions across the tasks an agent ran. A shootout the agent did not run adds nothing to that sum. On this board, the interventions, time, and cost columns are still the trip plan run. Each task page records its own.
Time
Wall-clock time starts when the prompt is submitted and stops when the agent finishes, or at 30 minutes, whichever comes first. A run that reaches the cap is a fail for that task. Time is published with the result. It does not, by itself, change the order of the table.
Cost
When the product meters usage, cost is the dollars charged for that run. When a run is not metered, the cell is the plan label on the account. The cell says which of the two it is.
Same conditions
- The prompt is the same, word for word, for every agent.
- The session is fresh: a new chat, with no memory and no instructions carried over from another run.
- The agents in one table are run in the same week.
- The screen is recorded for the whole timed run.
- One run is labeled and scored. The score is that run, not a best of several.
Ranking
Points stay visible so a partial is not hidden. The sort uses pass rate, not the point total. A partial is not a pass. On a single task, rank is points, then items met, then fewest factual errors, then time. The list below is the order across tasks.
- Higher pass rate. Passes divided by the number of tasks that agent ran that week. A shootout an agent did not run is not in the denominator, and it is not a fail. Lindy did not run Shootout #3, so that price comparison is 0 runs for Lindy and is left out of Lindy’s count. Manus and Lindy did not run Shootout #4, so the passport renewal is left out of their counts. Lindy did not run Shootout #5, so the holiday returns are left out of Lindy’s count. Manus did run Shootout #5 and failed, so that fail is in Manus’s count. Lindy did not run Shootout #6, so the streaming prices are left out of Lindy’s count. Manus did run Shootout #6 and failed, so that fail is in Manus’s count. Divide by the scored runs only. A tie on pass rate goes to fewer factual errors.
- Fewer factual errors across the tasks that agent ran. A shootout the agent did not run adds none.
- Fewer human interventions across those tasks.
- Lower cost, compared as dollars when both agents have a dollar figure. An agent with only a plan label does not outrank an agent with a measured dollar cost on this tie-break. If neither has a dollar cost, cost does not break the tie.
- If still tied, alphabetical by agent name.
Testing soon
The trip plan for Shootout #1 was scored on Oct 17, 2026. Local research for Shootout #2 was scored on Oct 21, 2026. The price comparison for Shootout #3 was scored on Nov 3, 2026. The passport renewal for Shootout #4 was scored on Nov 6, 2026. Holiday returns for Shootout #5 were scored on Nov 10, 2026. Streaming prices for Shootout #6 were scored on Nov 13, 2026. Lindy’s price-comparison cell reads “Did not run: free trial expired, needs paid plan.” Manus’s passport cell reads “Skipped: login expired.” Lindy’s passport cell and Lindy’s holiday-returns cell read “Skipped: trial expired, $29.99 paywall.” Lindy’s streaming-prices cell reads “Skipped: out of credits (waiting for billing).” Those are not scores and not fails. Manus’s holiday-returns cell and Manus’s streaming-prices cell read “Fail · 0,” which are scored runs. The other task cells still read Testing soon. That is not a score. Agents listed under Coming later are not in these shootouts, and Coming later is not a result.