Project log · started April 2026 · still running

A reasoning loop for games a model has never played

I'm building the scaffolding that lets a local vision-language model perceive an unfamiliar interactive puzzle, guess what it's for, act, and correct itself — the ARC-AGI-3 benchmark, where humans score ~100% and the best public systems are still near 1%.

↓ watch it reason ↗ code ✉ email
BUILD MANIFEST 4 SYSTEMS · SOLO
  1. 01
    Evaluation harness

    Reimplemented the official RHAE scorer — matches it 91% across 340 human replays.

  2. 02
    The reasoning loop

    Perceive → understand → hypothesize → plan → act → reflect, frame by frame.

  3. 03
    Instruments

    Replay viewers, CoT diagrams, and forensics that grade the agent's frames — not its story.

  4. 04
    Generalization gating

    A fix only counts if it holds on games it was never tuned on.

01

What ARC-AGI-3 is

ARC — the Abstraction and Reasoning Corpus, from François Chollet — tests fluid intelligence: solving problems you've never seen before, not recalling facts you memorised. ARC-AGI-3 is the interactive version — an agent is dropped into an unfamiliar grid mini-game with no instructions and has to work out the rules by acting.

People figure it out almost every time. Today's best AI models solve under 1%.

you're given this
you act — it changes
infer the rule
No tutorial, no manual — act, observe, infer.
02

How it's scored

You're graded on efficiency vs a real person — not just whether you finish.

score = min( human actionsagent actions , 1)2

Finish a level in as few moves as a human and you approach 100. Take more, you fall off fast — it's squared. Don't finish the level at all, and you score nothing for it.

I've completed 0 levels so far — so my honest score is 0. The point of this project isn't the number. It's the diagnosis of why small models fail.
WHERE EVERYONE ACTUALLY IS0–100 SCALE
Humans
~100
Best public AI · Tufa Labs
~1.21
This project
0
FRONTIER ZOOM · ×50 · 0–2 slice
Tufa Labs
1.21
This project
0

The scale is brutal: the best public AI (~1.21) and a human (~100) differ by ~80×. Drawn linearly so the gap stays honest — the tiny bars are the point. This is an unsolved frontier, and everyone is near the start line.

03

My solution

LLMs today interpret MRIs better than doctors, crack proofs that stump experts, and surface zero-day bugs that went unnoticed for years. Drop one into a simple game with novel-but-intuitive mechanics, though, and it struggles with — well — intuition. That gap is what ARC-AGI-3 measures, and I've been fascinated by it since February.

I think the key is reasoning like a human. The model can already reason — the bottleneck is the flow of information. Hand it clean, well-worded facts about a scene and it does fine; make it gather those facts itself across a hundred noisy frames and it drowns.

So I sat down and solved the games myself, and tried to pin down how I actually did it — how do I know what I control? how do I read walls or place-specific mechanics? how do I derive a goal, and adapt when things don't match what I expected? The loop is my attempt to give the model those same steps, over and over.

Hand-drawn diagram of the reasoning loop: perceive, understand, hypothesize, plan, act, reflect — connected in a cycle with the questions the agent asks at each stage.
The loop, in my own hand — the questions I ask myself at each stage are the questions I'm trying to route to the model.
MODELS a 32B vision-language model does the perceiving; a 27B-class model judges progress — all local, no internet at evaluation.
04

Replay theater

Real episodes from the latest run. Click through the highlighted moments — I've annotated what matters at each one so you don't have to read every frame. The grid is re-derived ground truth (the real engine frames); the reasoning shown is the loop's own words.

STEP 0

the agent's raw reasoning at this exact frame ▾
GOAL

REASONING
JUDGE

05

Where the flow still breaks

Each failure mode is a fact that didn't route cleanly — and it shows up as a shape. Below, the controllable object's actual path across the board (re-derived from the saved frames, not the model's claims), tinted by the judge's live verdict. Three episodes, three unmistakable signatures.

judge: toward goal away no change start end
wa30 — overshoot & oscillate

The loop closes the gap and the judge confirms it's moving the right object the right way — then it overshoots and swings back and forth (28 TOWARD / 24 AWAY), never landing the exact alignment. The trace buzzes across the target instead of settling.

tu93 — direction tunnelling

Clean step-by-step navigation reasoning — but the executor presses RIGHT twenty-one times and re-probes the same tiny corridor, never taking the untried downward path toward the goal. The path collapses into a knot: the frontier fact never reaches the planner.

re86 — a fact mangled in transit the thesis, live

Correct goal. Correct target. But the plan says moving up with button 4 will advance the group — and button 4 goes right. The reasoning is sound; a fact got corrupted between perception and language, and the whole plan follows it off a cliff. Not a reasoning failure — a routing failure.

→ Next: tighten the perception-to-language handoff so direction facts survive the trip, and give the planner the untried-frontier fact it's currently blind to.