Benchmark results

Terminal-Bench 2.1

Canopy solved 79.3% of Terminal-Bench 2.1 on DeepSeek-V4.1-Flash, averaged over low and high reasoning effort. That is level with the leading harnesses on the public board for this model; the difference is inside rerun noise.

79.3%

Mean of low and high effort

Complete. Runs 10–12 Nov 2026.

Results

EffortScore95% intervalTrials passedTasks passed 3 of 3Tasks passed at least onceRun (UTC)
Low77.4%67.6–84.9202 of 26154 of 8778 of 8710 Nov 19:25 – 11 Nov 20:58
High81.2%71.8–88.0212 of 26161 of 8778 of 8711 Nov 21:02 – 12 Nov 12:49
Mean79.3%

Setup

Benchmark
Terminal-Bench 2.1, 89 tasks, 87 scored (the board excludes qemu-startup and qemu-alpine-ssh)
Model
DeepSeek-V4.1-Flash, served by Azure AI Foundry
Attempts
3 per task. Concurrency: 3. Runner: Harbor 0.25.0
Memory
Off for every trial, so no trial could recall an earlier one
Harness version
Not pinned; production deploys continued during both runs

Comparison with the public board

  • Canopyour run79.3%
  • Kimi Code77.8%
  • OpenCode75.3%
  • Codex75.1%
  • mini-swe-agent74.3%
  • Claude Code74.0%
  • DeepSeek Harness73.4%
  • Pi73.0%
  • Qwen Code72.6%
  • Hermes Agent65.9%
  • OpenClaw56.5%
Public Harness Leaderboard, v0.1.6, DeepSeek-V4.1-Flash, Terminal-Bench 2.1, mean of low and high effort, read 12 Nov 2026. Canopy’s row is our own run.

Where trials failed

High effort, 49 failed trials
ReasonTrials
Wrong answer after the agent reported done25
Ran out of time13
Ran out of time with a server restart during the trial6
Lost to a crash loop in our background workerCanopy fault4
Our CLI exited at a server restartCanopy fault1
Low effort, 59 failed trials
ReasonTrials
Wrong answer after the agent reported done23
Ran out of time14
Ran out of time with a server restart during the trial19
Our step limit ended the turnCanopy fault2
Our CLI exited at a server restartCanopy fault1

The last two rows in each table are faults in Canopy, not the model. They count as failures in the scores above and have since been fixed.

Tasks never solved in three attempts (high effort)

  • extract-moves-from-video
  • filter-js-from-html
  • gpt2-codegolf
  • make-doom-for-mips
  • model-extraction-relu-logits
  • mteb-retrieve
  • pytorch-model-cli
  • query-optimize
  • train-fasttext

Read this before comparing

Limits of this comparison

  • Our route to the model (Azure) differs from the board’s providers.
  • On our route, low effort was not shown to reason less than high effort.
  • The board reports that re-running the same harness flips about 21% of Terminal-Bench tasks, so differences of a few points are noise.
  • Per-trial transcripts are not published yet.
  • Cost: $33.59 recorded for the low run over the 221 trials that reported one; not recorded for the high run.

Your next customer is about to call.

Start free, or book a demo.