What is an AI agent, and can it finish the job yet?

Start by refusing the demo's definition

An AI agent is a model given a goal, a set of tools and permission to loop: it plans, acts through software such as browsers, terminals and APIs, checks its own results and tries again, without a human prompting every step. That is the whole definition. A chatbot answers you. An agent acts for you. If the product needs you to click approve at each step, it is a workflow with good branding, and if it collapses the first time reality deviates from the script, it is a demo.

NVIDIA's engineers, writing about agent training data on Hugging Face in July, put the bar plainly: "An agent that can't recover from a broken API call, or a workflow it has never seen, is not really an agent."

The only independent yardstick worth your time

The marketing question is whether agents are magic. The measurable question is how long one can work on its own before it breaks, and the best independent answer comes from METR, a non-profit evaluator whose "time horizon" metric estimates the length of task, measured in expert-human working time, that a model completes at a given reliability.

METR's Time Horizon 1.1 update, published on 29 January 2026 across a suite of 228 tasks, put the longest measured horizon at Claude Opus 4.5: about 320 minutes at 50% reliability, with a wide error range of 170 to 729 minutes. GPT-5 measured around 214 minutes. More important than any single number is the curve: the horizon has doubled roughly every seven months across six years of models, and roughly every three months for models released since 2024. METR is candid about the limits, noting that "these confidence intervals are still very wide" and that the trend is "somewhat sensitive to task composition". The measured models also predate this autumn's GPT-6 and Opus 5.5 wave, so read the finding as a curve, not a snapshot.

Half the time is the operative phrase. A model that completes five-hour expert tasks at a coin-flip rate is a genuine research milestone and an operations nightmare in the same breath. Nobody staffs a role with someone who silently fails every second assignment; you staff around them with review, and that is exactly what sensible agent deployments do.

What the vendors' own charts admit

Read the current launch posts closely and the honest numbers are hiding in plain sight. OpenAI's GPT-6 material shows its best configuration completing 33.2% of AutomationBench's end-to-end business workflows at $0.27 per task, and 56.4% on the Agents' Last Exam professional suite. Those are vendor-run evaluations chosen to flatter, and they still describe a system that does not finish most business workflows it starts. The pitch, stripped of styling, is that it fails cheaper than the competition.

Anthropic's launch material for Opus 5.5 leans on testimonials, including a developer at legal-software firm Clio reporting a run that "stayed on task for over 18 hours... required minimal reworking". Real, attributed, and vendor-selected: the best run makes the launch page, the median run does not.

The economics point the same direction. OpenAI disclosed that, valued at API prices, its median researcher burns more than $600 of tokens a day, with the 90th percentile past $7,000. Agents are token furnaces, which is why September's API price cuts are aimed squarely at them.

Where agents earn their keep today

The pattern behind every agent success story is the same: the work was agent-shaped. Output that can be verified mechanically, by tests, compilers or a submitted form. Actions that can be undone. Rich tool access. Tolerance for retries. Coding leads the field not because models love code but because verification is built into the job, which is why every lab demos a coding agent first.

The failure pattern is equally consistent: irreversible, externally visible actions taken without a checkpoint, and tasks where checking the agent's work costs as much as doing it yourself. If you are evaluating a vendor, ask for the completion rate on your tasks rather than a benchmark, ask precisely what happens on failure, insist on human gates wherever an action cannot be undone, and put a quarterly re-test in the calendar, because the underlying curve moves in months, not years.

Our take

Agents are the rare AI story where the honest numbers are impressive without the marketing. An independent, methodologically transparent measure says autonomous working time doubles every few months, and the vendors' own charts, read plainly, admit today's systems abandon most long workflows they begin. Both facts are true and neither cancels the other. Deploy narrow, verify always, re-test quarterly, and ignore anyone selling a digital employee. What is on offer today is a fast, tireless assistant with a coin-flip memory for finishing, on the steepest improvement curve in software.