Methodology

The tasks

Five jobs, the kind of work a person actually hands over. Each one counts equally. Shootout #1 publishes the trip plan before the runs.

• Trip plan — Runs this week
• Inbox triage
• Local research
• Grocery cart
• Expense report

The trip plan prompt and pass checklist are published now. They are not rewritten after a run looks hard.

Points

Each task is scored against its checklist.

Pass · 2

Every checklist item is met.

Partial · 1

The run produced usable work and missed at least one checklist item.

Fail · 0

The job was not done, or the run reached the 30-minute cap.

Interventions

A human intervention is a person stepping in during the timed run so it can continue: answering something the agent cannot resolve, clearing a login or a permission, correcting a wrong turn, or restarting after a stall.

Opening the product, starting the recording, and pasting the prompt happen before the clock. They are not interventions. The table sums interventions across the week’s tasks.

Time

Wall-clock time starts when the prompt is submitted and stops when the agent finishes, or at 30 minutes, whichever comes first. A run that reaches the cap is a fail for that task. Time is published with the result. It does not, by itself, change the order of the table.

Cost

When the product meters usage, cost is the dollars charged for that run. When a run is not metered, the cell is the plan label on the account. The cell says which of the two it is.

Same conditions

• The prompt is the same, word for word, for every agent.
• The session is fresh: a new chat, with no memory and no instructions carried over from another run.
• The agents in one table are run in the same week.
• The screen is recorded for the whole timed run.
• One run is labeled and scored. The score is that run, not a best of several.

Ranking

Points stay visible so a partial is not hidden. The sort uses pass rate, not the point total. A partial is not a pass.

1. Higher pass rate. Passes divided by the number of tasks that week.
2. Fewer human interventions across those tasks.
3. Lower cost, compared as dollars when both agents have a dollar figure. An agent with only a plan label does not outrank an agent with a measured dollar cost on this tie-break. If neither has a dollar cost, cost does not break the tie.
4. If still tied, alphabetical by agent name.