Agent Arena adds six indexes for shortlisting agents

The new breakdown covers work, coding and chat, alongside outcome, steering and tool use. Teams can use those signals to shortlist candidates and measure delivery and revision effort in their own workflow.

Article published: 2026-11-20Mokyun Editorial5 min read

Public developments and original analysis · Announcement date: · Sources

Imagine a procurement analyst receiving a comparison sheet and noticing that one supplier quoted before tax. The analyst asks the assistant to normalize the prices. Two results can be observed separately: the completeness of the initial delivery and whether the revision changes only the affected fields while keeping the evidence traceable. When two assistants make equally usable first drafts, revision behavior may decide which one gets the review pack out that day.

AI-generated conceptual illustration. Three prints represent different ways to examine the same evaluation material.
AI-generated conceptual illustration. Three prints represent different ways to examine the same evaluation material.

On November 18, Arena introduced a headline Agent Arena Index and six focused indexes for work, coding, chat, task outcome, steering and tool use. Its Alignment Index remains alongside the headline score. Arena says the headline index spans more than 2.9 million sessions on its platform. That figure describes the publisher's dataset. Agent Mode sessions are shaped by Arena's own mix of tasks, tools and opportunities for feedback. A procurement team may work under different conditions.

What the breakdown measures

The Work, Coding and Chat indexes separate task domains. They use signals for task completion, praise or complaint, steerability and Bash reliability. The Outcome index combines completion with feedback, while Steering focuses on what happens when a user corrects the assistant. Tool Use currently uses Bash reliability. A team interested in how an ERP connector handles a failed request can use that last index to guide a shortlist, and then check the actual connector with its own test cases.

Two details matter when reading these scores. Task Completion counts only tasks on which users explicitly answered the completion prompt. Steering rewards turns that need no correction and counts repeated correction as greater effort. A strong Work score can justify shortlisting a tool, while a trial should deliberately include a required revision. Otherwise, infrequent corrections on routine work can hide the effort of revising a more demanding deliverable.

Arena's earlier methodology describes random component assignment to estimate how choices such as the orchestrating model affect results. The design provides evidence about differences inside Arena's environment. A company's retrieval system, tool interface and approval steps change the material the model sees, the actions it can take and the recovery available after an error. Buyers should ask where a supplier operates the agent and what that environment includes, alongside the model name.

The update also rescaled scores. An overlapping set of well sampled models provides an anchor between consecutive releases. Arena says the rescaling itself preserves the ordering. Changes to a score are easier to interpret with a dated record of the methodology and sample. This update also retired a signal for attempts to call nonexistent tools. A purchasing team that records the index, model version and scope of its pilot can later distinguish changes in its workflow from a new candidate version or a change in the public benchmark.

Trialling the supplier comparison

Return to the hypothetical procurement team. Give each candidate the same redacted historical quotes, exchange and tax rates, and field definitions. Check whether the first sheet preserves currency and tax status, then apply a revision written in advance. Include a quote whose tax rate is missing. It should stay marked for confirmation as the assistant fills other parts of the table. Check that revised values still point to the original pages and that the summary sent to reviewers reflects the changes.

Teams with stable templates can run a simpler comparison. Checked formulas can handle arithmetic and amount checks, while the assistant finds prices, units and the relevant terms. If an agent revises explanations effectively but overwrites confirmed formulas, keep that section under spreadsheet controls and measure how easily people can apply its suggestions. Paying more for repeated steering becomes easier to justify when work spans several sources, requirements change often or coordination crosses systems.

Record usable first deliveries, usable revisions and cases that still require a person to redo the work. Weigh review time against the consequences of errors. For a team buying a small number of critical items each month, one serious price error may consume the time saved on several tasks. For large volumes of standard quotes, routine completion rates and the cost of each usable delivery may decide how far to deploy. The ordering can therefore differ across jobs, and more than one candidate may make sense when review costs vary.

Public indexes are useful before trials, when there are many candidates to compare. A budget decision calls for evidence that an assistant can make the first delivery, apply a traceable revision and leave an acceptable amount of work for staff under the team's own constraints. A small set of real artifacts can answer those questions once the index has helped narrow the field.

Sources

  1. Arena, Introducing the Agent Arena Index

    Source published: · Accessed:

  2. Arena, Agent Arena methodology

    Source published: · Accessed:

  3. Arena, Agent Arena signals

    Accessed:

Mokyun Editorial · neo@mokyun.com