Introducing Horizon

Every Horizon task has a long historical trace that the agent must learn from to complete the task correctly. Each trace is real and months-long, pulled from one of our products, Acadia Learning.

For example: months ago in this trace, the agent sent a worksheet as a .docx attachment, the student could not open it on a school-issued Chromebook, and part of the session was lost before a PDF link worked. Nothing in that exchange is marked as a preference or a rule; it is one failed handoff inside months of routine activity. When a new request to send practice materials arrives, the task tests whether the agent sends a PDF link on the first try. Lessons can be anywhere in the trace, occur multiple times, or require multiple data points to extract the required pattern.

Horizon contains 195 tasks, but is private to prevent overfitting and keep user data secure^*. We have included a few example eval cases in our public repo, including a public HuggingFace dataset of traces, to show how the benchmark is structured.

Each task runs in an environment with real tools (email and SMS inboxes, and more) and is graded on completion, cost, and speed, judged from the final environment state by an LLM plus deterministic checks.

We categorize tasks based on three dimensions: predictability, burial depth, and number of learnings required. Predictability is how easy it is to predict what the agent will need to learn, manually categorized. Burial depth is how far back from the present task the required fact sits, as a percentage of the trace. Number of learnings required is how many separate facts from the trace must be learned and combined to pass the task.

All harnesses show similar patterns where predictability and number of learnings required are the clearest detractors. This makes sense, as Horizon's hardest tasks tend to be the least predictable and require more learnings. The data also suggests that agents are better at learning from their early and late experiences than their middle ones, similar to in-context rot.

Future work will prioritize tasks that are less predictable and require more learnings, like testing if the agent can recognize implicit but unexpected patterns in realistic traces.

Models are slowly getting better at Horizon, suggesting that scaling pretraining and reinforcement learning improves a model's ability to learn from long-horizon traces. However, the improvement rate of models is much slower on hard tasks, suggesting that intelligence alone may not be enough to learn effectively from long-horizon traces.

Scaling test-time compute is weakly correlated with pass rate, but correlation varies widely between harnesses. Harnesses like OpenClaw and Hermes primarily rely on accumulating learnings over time for fast access at test time, while harnesses like RLM and RAG spend more tokens on searching the trace during the task. Our sample is small, but harnesses that accumulate do not seem to benefit from additional test-time scaling while harnesses that search do improve.

Importantly, none of the tasks in Horizon are challenging to reason through. When we ran an oracle with perfect context, it only used a few thousand tokens to successfully complete the task. This suggests that test-time scaling may not be necessary with the right harnesses.

The clearest takeaway from Horizon is that when learnings are not predictable, both search and accumulation strategies fail. None of Horizon's tasks are cognitively challenging, and while models are getting better at searching traces, we are also excited about representation learning and harness research as potential solutions.

Future versions of Horizon will focus on low predictability pattern matching tasks, as we believe this is the most important remaining capability for agents to operate autonomously in the messy real world. If you're interested in working on this with us, we're hiring.

To ensure that each task is fair, we built four test agents.

1. Oracle: a script to deterministically solve each task. We made sure that this reliably scored 100% with low variance, showing that our rubrics are consistent.
2. Anti-Oracle: a script that does nothing. We made sure that this reliably scored 0% with low variance, showing that our rubrics are consistent.
3. PerfectContext: for each task, we manually fed the agent the important lines from the trace. We made sure that this reliably scored 100% with low variance, showing that each task is easily solvable with the right context.
4. EnvironmentOnly: an agent that has no way to access the trace, and can only interact with the task's environment. Since each environment is stateful (email inboxes, sms inboxes, etc), ensuring a 0% score with low variance verifies that the solution cannot be derived from the environment.

All 195 tasks passed these tests with low variance, showing that they are solvable, the judges are fair, and the tasks do not leak information.

Testing a human baseline is impossible (even reading the traces is equivalent to 1,300 Harry Potter books), but each task has been reviewed by a human and deemed reasonable. Agent implementations are available in our public repo.

We did not get a chance to benchmark Anthropic's Fable 5 before it was removed.

Thank you to Dr. Furong Huang, Mehul Arora, Sean McLeish, Hamidah Oderinwale and others for reviewing this post. We are also grateful to the teams at Daytona, Harbor, OpenAI, and Anthropic for their support.