- A Wired writer said a team of specialised AI agents seemed to improve his fantasy soccer picks, while more than 100 agents changed little except his API bill. Nothing was measured.
- Prophet Arena, a University of Chicago benchmark, says it seals AI forecasts before events resolve and then scores them publicly.
- Before funding an agent project, ask what each agent does, what a larger swarm costs, and how the output will be scored.
The extra agents mostly raised the bill
A Wired writer wanted to know whether a swarm of AI agents could run a Premier League fantasy team. He asked Codex, an AI coding model, to build a harness. That is a set of instructions and code that walks agents through steps such as checking recent news and running a prediction model.
The first picks were poor, even though that harness already checked recent news and ran a model. The writer was surprised to see Erling Haaland, a star striker, left on the bench. So he asked for a team of specialised agents. Some read news and blogs. Others analysed recent results. Some ran models, and a group critiqued the suggestions.
He says that change seemed to improve the suggestions significantly. Then he tried more than 100 agents, hoping for a smarter system. It "didn't change much" apart from running up his API bill, the fees paid per request to an AI provider.
The Wired piece, as WebPulse reviewed it, does not report how the team finished. The improvement is the writer's own impression, not a measured result.
What a team setup might add that a bare model lacks
The writer had tried an older tool from the UK's Turing Institute. Its picks were "bonkers" because it used past league data and ignored real-world facts such as injuries or a change of manager.
A harness might close that gap, but this test does not show it. The writer's first harness already checked recent news and ran a model, and it still gave poor picks. Only the team of specialised agents, including critics that reviewed the suggestions, seemed to help, in his view.
That is WebPulse's interpretation, not a finding. A plausible reason extra agents added little is that, once news, results, models and critics are covered, more agents mostly repeat the same work. Duplicated work costs money without adding new information. The first change in this test also added several agents at once, so it does not cleanly separate structure from headcount.
Haifeng Xu, a University of Chicago computer scientist, told Wired that AI predicts most effectively with a custom harness. He said agents with distinct tasks, such as reading news, collecting data and analysing the whole system, seem to work best. That is one researcher's view, not a tested rule.
How Prophet Arena says it keeps forecasts honest
Xu was marveling late last year at the growing popularity of prediction markets such as Kalshi and Polymarket. He wondered how top commercial models would do on them and began testing their forecasting skills with colleagues. The team later built Prophet Arena, a benchmark of how well models predict.
Xu told Wired the models "did really well" early on and sometimes beat the average person without special guidance. He said GPT-5 predicted best in that early testing, before Prophet Arena existed. Grok 4, Claude Sonnet, Gemini and Llama followed. This is an early-stage result as Wired reports it, not Prophet Arena's current standings.
The method below comes from the site's own description. WebPulse has not independently verified it. Models and agents put probabilities on newly opened prediction markets every day. The site says each forecast is fingerprinted on arrival, so nobody can backdate it. Real-world outcomes then settle the question.
The site says it publishes Brier scores and calibration. A Brier score measures how far a stated probability was from what happened. Calibration asks whether events given 70 percent odds happen about 70 percent of the time. Wired adds that AI still often lags very skilled experts known as superforecasters.
A caution from one money test
Wired also describes Prediction Arena, a benchmark from Harvard researchers and Arcada Labs. It staked $10,000 on prediction markets for frontier models across 57 days. In that experiment, the models consistently lost money.
The researchers reported evidence of improvement over time. That is not a profit result. It tells us the models were getting better, not that they were winning. This is one experiment, so it cannot show how AI forecasts perform in every real-money setting.
What to ask your team
Treat agent projects as designs to test, not purchases to scale. Four questions are worth putting to any team proposing one.
First, what does each agent do, and what does a critic agent check? Second, what happens to quality, and to the API bill, when the agent count doubles? Third, are outputs recorded when they are made, so they can be scored once the outcome is known? Fourth, is the comparison against a skilled human, or only against a weaker baseline?
Prophet Arena's described approach is a useful template for internal projects: record the forecast, wait for reality, then score it openly. An AI system you cannot score later is a system you are only trusting.
Produced by the WebPulse Newsroom with AI assistance from the original reporting credited below, and checked against that source by our editorial review. How we use AI.
Original reporting: Prophet Arena team (Haifeng Xu and colleagues, University of Chicago).
