LongPuzzleBench Visual agent benchmark
GitHub

Long-horizon visual puzzle environments

LongPuzzleBench

Interactive puzzles where one legal move can decide whether dozens of future moves still exist.

LongPuzzleBench evaluates GUI agents across six stateful browser games and 114 levels—then opens their trajectories to inspection.

Explore research Play the games
View on GitHub ↗ Evaluation protocol ↗
Games
6
Levels
114
Evaluation cells
16
Bolt Unscrew board with one plank, screws, and an open destination 01
AccessWhat stays reachable?
Rush Hour board with independently moving vehicles 19
IdentityWhich object moved?
Color Connect board with colored endpoints sharing a small grid 42
Shared spaceWhat must remain free?

Same action space. A different future after every move.

Recent updates

The project is moving.

Four concrete releases from the benchmark’s first public week.

  1. Browser playground released

    Thirteen curated levels across all six game families are now playable without an install.

  2. Trajectory research note published

    Five recurring failures are reconstructed from real model actions and board states.

  3. Leaderboard refreshed ↗

    Eighteen complete configurations now cover every evaluation cell.

  4. Initial benchmark release ↗

    Six deterministic browser puzzle environments and 114 levels were opened.

View project history

What LongPuzzleBench tests

A puzzle is not a question.

A static answer can be right once. Here, an agent must keep its account of the world right after every click, drag, swipe, collision, and state change.

  1. 01

    Interactive

    The board changes after every action. Observation and planning repeat until the level ends.

    observe → act → rebuild
  2. 02

    Stateful

    Objects, selected sources, stopping positions, occupied cells, and exposed holes must persist.

    pixels + memory
  3. 03

    Consequential

    A move can be legal now and still cover the only hole or consume the only route needed later.

    local move ≠ safe move

Explore the suite

Small interfaces. Long consequences.

Six game families expose different failures in access, identity, ordering, coverage, workspace, and shared space.

Open the full playground

Benchmark frontier · 18 complete configurations

The leading configuration clears just over half the score scale.

6 games 114 levels 16 score cells

Complete ranking

All 18 model × reasoning configurations

ScoreSuccess
  1. 01 GPT-5.6 Sol gpt-5.6-sol medium 54.796 55.42%
  2. 02 GPT-5.6 Sol gpt-5.6-sol low 54.142 55.62%
  3. 03 GPT-5.6 Sol gpt-5.6-sol high 49.968 50.47%
  4. 04 GPT-5.6 Terra gpt-5.6-terra high 47.591 48.75%
  5. 05 GPT-5.6 Terra gpt-5.6-terra medium 37.772 38.18%
  6. 06 GPT-5.6 Terra gpt-5.6-terra low 36.806 37.29%
  7. 07 Kimi K3 kimi-k3 high 32.694 32.50%
  8. 08 GPT-5.6 Luna gpt-5.6-luna high 30.337 31.25%
  9. 09 GPT-5.6 Luna gpt-5.6-luna medium 24.158 23.75%
  10. 10 GPT-5.6 Luna gpt-5.6-luna low 17.867 16.04%
  11. 11 Qwen 3.8 27B qwen/qwen3.8-27b default 16.915 16.25%
  12. 12 Kimi K2.5 moonshotai/kimi-k2.5 max 15.979 15.47%
  13. 13 Qwen 3.5 122B A10B qwen/qwen3.5-122b-a10b default 11.285 10.10%
  14. 14 Qwen 3.6 35B A3B qwen/qwen3.6-35b-a3b default 6.580 5.68%
  15. 15 Qwen 3.5 397B A17B qwen/qwen3.5-397b-a17b default 6.471 4.84%
  16. 16 GLM-4.6V z-ai/glm-4.6v max 6.386 5.47%
  17. 17 Qwen3-VL 235B Thinking qwen/qwen3-vl-235b-a22b-thinking default 5.314 3.59%
  18. 18 Qwen3-VL 30B Thinking qwen/qwen3-vl-30b-a3b-thinking default 5.134 4.43%
  1. 01

    The top three configurations are all Sol settings. Medium leads low by 0.654 points; high trails the leader by 4.828.

  2. 02

    Success alone does not set the rank. Sol-low solves 0.20 percentage points more levels, while Sol-medium earns the higher 16-cell score.

  3. 03

    No configuration leads every game. Sol-high leads Rush Hour; Terra-high leads Maze Paint and Color Connect; Kimi K3 leads Bolt Unscrew and Truck Escape.

View evaluation details Download canonical results JSON

Scores tell us who leads. Trajectories show why.

The mistake often happens before the bad move.

Each case synchronizes real evaluator frames from two runs. Model text is copied from the selected public action record; actions and evaluator outcomes remain separately labelled.

  • 01Game stateWhat the screenshot shows
  • 02Recorded outputWhat the model emitted
  • 03Executed actionWhat the runtime performed
  • 04EnvironmentWhat changed next

Case 01 · Part–whole segmentation

One model sees a car. Another sees a fragment that is not there.

The same initial pixels produce the first meaningful divergence: one run moves a real blocker; the other invents a vertical red vehicle and receives no visible effect.

17.336-point benchmark gap · 3 actions versus 6 Byte-identical initial screenshot
Higher-ranked run GPT-5.6 Sol · low
Rank 2 · 54.142
The board contains one articulated horizontal target. Before action 01
Treats the articulated sprite as one target
Recorded model output
Thought: 紫色竖车挡住红车所在横道，其下方同列一直到棋盘底部都有足够空间；先将它沿长轴向下拖到底，清空出口通道。
Executed action
drag((540,730) → (540,1250))
Environment
game state: one vehicle · one motion axisThe board contains one articulated horizontal target.Public feedback · screen changed

Success · 3 actions

Lower-ranked run GPT-5.6 Terra · low
Rank 6 · 36.806
The same sprite is split into a body and a ‘small red vertical car.’ Before action 01
Splits one sprite into two vehicles
Recorded model output
Thought: Move the small red vertical vehicle above the target upward to clear the target row.
Executed action
drag((292,731) → (292,602))
Environment
one sprite described as two vehiclesThe same sprite is split into a body and a ‘small red vertical car.’Public feedback · no visible effect

Success · 6 actions

Same pixels 01 / 04
Human playground Runtime loads after launch
Maze Paint playable puzzle preview

Suggested first puzzle

Maze Paint · Easy 1

Seven optimal swipes. The ball cannot stop midway.

Mouse + touch

Playground

Think the puzzles look easy? Try one.

The public playground uses the same checked-in game runtime and mechanics as the benchmark, without evaluator state, agent time limits, or diagnostic hints.

  • 13 curated demo levels
  • All six game families
  • No install or account
Play LongPuzzleBench Browse all games

Project resources

Inspect the artifact.

Source, protocol, results, and trajectory evidence remain public and reproducible.