Skip to content
Agent Shootout

Shootout #2Results in

Find a licensed HVAC repair company in Raleigh

Five agents, same prompt, scored Oct 21, 2026. Ranked by points, then items met, then fewest factual errors, then time. Every agent cost $0 extra and needed 0 interventions. No contact or form attempts were observed.

On video

Clips from the filmed runs. Silent, so the caption under each one is the description.

  • Shootout #2, local research. ChatGPT agent, Manus, a map of Raleigh HVAC companies, and Lindy, on the same prompt.
  • Manus finishes the Raleigh HVAC research and opens its report, starting with Cool Change Heating & Air Conditioning.
  • ChatGPT agent, after 36 seconds, recommends Comfort Monster first and starts a table of ratings and public license information.

Results

ChatGPT, Lindy, and Genspark each met 7 of 8. Lindy and Genspark have 0 factual errors. ChatGPT has 1, so those two rank ahead of it. Lindy and Genspark were both about 5 minutes. Lindy is placed ahead on fewer flags and no duplicate. The score file notes that pair could be called a tie.

Shootout #2 local research results, ranked by points, then items met, then fewest factual errors, then time.
RankAgentResultPtsItemsTimeInterventionsCostPlan
1Manus2.0 Lite (free)Pass28/8 · 0 factual errors~3.9 min0$0Free
2LindyAutoPartial17/8 · 0 factual errors~5 min0$0Free trial
3GensparkStandard (free)Partial17/8 · 0 factual errors~5 min0$0Free
4ChatGPT agentWork (GPT-6.1 Sol, Light)Partial17/8 · 1 factual error0.6 min0$0ChatGPT Plus
5GrokHeavyPartial15/8 · 1 factual error~1.2 min0$0SuperGrok

Notes

  • Manus

    The only clean sweep: every company real, every phone right, specific rating pages (Trane 4.9/380 matches exactly), license numbers that check out, and labeled, linked prices. Missed that Cool Change does have a license (L.37632) but honestly said “not found”.

    0 factual errors

  • Lindy

    Best license work (SafeHire records, active status), but 3 of 4 ratings point to the same generic Exa “best HVAC Raleigh” search page.

    0 factual errors

  • Genspark

    Rating links are Google Maps searches, and Raleigh Heating & Air lists the same offices as Air Experts (7 Rupert Rd and 2730 Rowland Rd), so that listing was scored as a duplicate.

    0 factual errors

  • ChatGPT agent

    Fastest by far (36 s) with exact Birdeye pages that match to the review, but its top pick’s license number (L.27865) belongs to an unrelated contractor in Benson.

    • Gave Comfort Monster the license number L.27865. That number belongs to an unrelated contractor in Benson. The firm's number is L.24034.

  • Grok

    Five legit companies and mostly-correct license numbers, but almost no links: ratings are “roughly 5,800–6,300” or “4,000 to 8,000+”, prices are unsourced, and one license number (15367) is a Wilmington individual’s.

    • Gave license number 15367. That number belongs to a person in Wilmington, not the company it was listed for.

Checklist

Eight items, same list for every agent. Flagged counts as met. Missed does not. The count beside the checklist score is separate: a factual error is a claim we checked and found wrong, and a missing or unverified item stays a checklist miss. How we score.

Per-agent checklist. Each cell is Met, Missed, or Flagged.
ItemManusLindyGensparkChatGPT agentGrok
3–5 real businesses serving 27601MetMetFlaggedMetMet
Contact details correctMetMetMetMetMet
Rating, count, and specific linkMetMissedMissedMetMissed
License or insurance addressedMetFlaggedFlaggedMissedMissed
Prices with sourcesMetMetFlaggedMetMissed
RecommendationMetMetMetMetMet
No contact or submissionsMetMetMetMetMet
Finished within 30 minutesMetMetMetMetMet

Judgment calls

  • SafeHire NC was the board-records proxy. public.nclicensing.org is a JavaScript form. A number held by an unrelated person or firm is a miss. A number held by a plausible individual qualifier but not the firm (Air Works 28909, Bowman S.12302) is flagged, unless another error stacks.
  • ChatGPT’s Bowman S-number alone would be flagged. The Comfort Monster mismatch (L.27865, an unrelated Benson contractor; the firm is L.24034) alone makes the license item a miss.
  • Grok’s license item: most numbers were correct, but a wrong extra number (15367) plus no source links is a miss.
  • Trane dealer pages count as the business’s own listing. They show that dealer’s Google rating and count.
  • Raleigh Heating & Air was treated as a duplicate of Air Experts, so Genspark keeps 4 valid companies. The real-businesses item is still met, and flagged.
  • Lindy and Genspark tied at 7/8, 0 factual errors, and about 5 minutes. Lindy is ahead on fewer flags and no duplicate. That pair could be called a tie. ChatGPT also met 7 of 8, in 0.6 minutes, and has 1 factual error, so it ranks after them.
  • Bot-blocked links (403 from Yelp, Angi, BBB, ConsumerAffairs, and Service Experts) were marked unverified, not broken. Birdeye returns 404 to curl and loads on a retry, so those links were not treated as broken.

The prompt

This is the whole brief. No agent gets a rewrite.

My central air conditioning/heat pump stopped working. I need a licensed HVAC company to come out and diagnose and repair it at a house in downtown Raleigh, NC 27601.

Find 3 to 5 real HVAC repair companies that serve Raleigh, NC 27601. For each company include:
- the business name
- the phone number
- the company's own website
- its rating and review count, plus a link to the exact page where you found them (for example its Google, Yelp, BBB or Angi listing)
- its NC HVAC (heating) contractor license number or insurance information, if it is publicly available, with a link to where you found it. If you can't find it, say so.
- a typical price range for a service call or diagnostic fee and for a common repair (such as replacing a capacitor), with a link to the source of each price. Say whether the price is the company's own published price or a general estimate for the area.

Then give a short recommendation (3 to 5 sentences) of which company you would call first and why.

Do not contact any business. Do not call, email, text, or chat with them, and do not fill out or submit any form, quote request, or booking. Research only.

Pass checklist

Written Oct 21, 2026, before any run. Pass means every item below is met. A flag still counts as met. The same list grades every agent.

  1. 3–5 real, operating businesses. Every company listed exists, is open (not marked permanently closed), does HVAC repair, and serves Raleigh 27601. Fewer than 3 valid businesses is a miss.
  2. Contact details correct. Name, phone, and own website for each; the phone matches the website or listing. A wrong or missing phone or website on one business is a flag; on two or more, a miss.
  3. Rating and review count with a specific source link. The link opens that business’s own listing page. The rating is within ±0.1 stars and the review count within ±10% (or ±5 reviews, whichever is larger) of the source on the day of the check.
  4. License or insurance addressed. For each company, either a license number or insurance detail with a source, or an explicit “not found”. A license number that does not match the NC Plumbing, Heating & Fire Sprinkler Board lookup (public.nclicensing.org) is a miss. Silence on license for any company is a miss.
  5. Price ranges with sources. A service-call or diagnostic price and a common-repair price, each with a source link, labeled as the company’s own price or a general estimate. Unsourced numbers are a miss. Estimates labeled as such count as met.
  6. Short recommendation. Names one company to call first, with a reason grounded in the data given (rating, reviews, license, price, hours).
  7. No contact or submissions. No calls, emails, chats, quote forms, or bookings started or submitted. Any breach is a miss and an automatic fail.
  8. Within 30 minutes.

Judgment rules

  • A generic directory or search link (a Yelp search results page, a Google Maps search such as “hvac raleigh”, or a “Top 10 HVAC in Raleigh” list) does not count as a source for a specific business’s rating. It must be that business’s own listing.
  • Aggregator lead-gen sites (HomeAdvisor, Angi, or Thumbtack “pros near you” pages) count as a price source only if labeled as a general estimate.
  • “Serves 27601” means a Raleigh address or a stated service area that includes Raleigh. A company based in Cary, Garner, or a similar nearby city counts if its site says it serves Raleigh.
  • Ratings are checked once, the day of scoring. Small drift since the run is covered by the tolerance in the rating item.
  • Duplicates (the same company under two names or locations) count once.
  • One intervention is allowed without penalty, such as a sign-in or CAPTCHA a person clears. It is recorded, not scored.
  • Link checks use a redirect-following fetch with a Chrome user agent. A 403 or a bot challenge is unverified, then checked by hand in a browser.

Partial and fail

Same point values as Shootout #1. This task’s cutoffs were written with the checklist.

Pass · 2
All 8 items are met. A flag is allowed and still counts as met.
Partial · 1
5 to 7 items are met.
Fail · 0
4 or fewer items are met, or the run reached the 30-minute cap. Any contact or submission is also an automatic fail.

Full scoring rulesShootout #1Shootout #3Back to the board