Courtyard
Jev and OpenAI Decisions each played 50 attempts from the same starting checkpoint. Both received structured game state without screenshots.
Completion rate
Jev finished more attempts, with an average API cost per attempt about 78% lower. OpenAI responded faster at the median, and its successful runs earned a higher average score.
| Metric | Jev | OpenAI Decisions |
|---|---|---|
| Mean API cost / attempt | $0.0093 | $0.0414 |
| Mean API cost / call | $0.000169 | $0.000378 |
| API latency · median / p95 | 226 / 317 ms | 190 / 375 ms |
| Deaths / time-limit failures | 25 / 2 | 29 / 9 |
| Mean completion time · wins only | 21.6 s | 28.9 s |
| Mean score · wins only | 417 | 533 |
| Mean hearts · wins only, max. 3 | 2.65 | 2.75 |
| Gap falls / fatal goon contact | 25 / 0 | 28 / 1 |
Latency measures the full HTTP response. Each call asks three choice questions. Cost includes all attempts; time, score and hearts describe each model's successful subset. API costs exclude emulator and host compute.
Two successful runs
19.1 seconds · 600 points · 3 hearts
27.6 seconds · 700 points · 2 hearts
Illustrative examples, selected nearest each model's median completion time among winners. Both videos match their recorded native frames and game states. They do not show the failures in the full sample.
Different styles of play
OpenAI's play looked more cautious: it repeatedly selected no horizontal movement, while Jev usually kept moving. Averaged across the 50 attempts each, the share of gameplay with no left/right input was 48% for OpenAI and 11% for Jev.
The controller keeps the last chosen controls during API calls, so this pause pattern reflects the models' control choices. The pause durations also include time waiting for the next response. This doesn't establish why they chose to pause or whether the pauses caused the difference in completion rates.
This is an exploratory comparison on one level and starting checkpoint. I extended the sample from 20 to 30, 40 and then 50 attempts per model after reviewing earlier results. These numbers describe this controller and workload; they are not a general ranking of the providers.
Courtyard methods and limitations
- Both adapters use the same canonical JSON text format, prompts, choice descriptions, geometry and approximate forecasts. Their observed states diverge as their choices change the game.
- Observations include selected emulator state within 250 world pixels on either side of Kittie, overlapping terrain, internal motion timers, three-action history and shot feedback. Some information is offscreen; forecasts are estimates, not observed future states.
- The models choose movement, A and B controls. The game continues at native speed while API calls run, holding the previous controls until a response arrives.
- The movement comparison averages each attempt's fraction of recorded frames with neither Left nor Right held, excluding the initial wait before the first applied choice. It includes all 50 attempts per model. No horizontal input can occur during jumps or other actions; it does not necessarily mean Kittie is physically stationary.
- The frame-age controller estimates response delay from earlier applied observation ages. Both providers use the same initial estimate and update rule.
- All starts use the same ROM and checkpoint. Provider order is seeded and interleaved. The time limit is 7,200 native frames, about 120.5 seconds, with a six-frame minimum action interval.
- No retries, fallback controls or replacement failures. This 100-start sample has no service-error or refusal aborts. Older runs with a different delay-estimation rule are excluded.
- Costs use the recorded input-token rates: $0.042 per million for Jev and $0.10 per million for Decisions. Longer attempts and more calls affect cost per attempt.
Completion-rate 95% Wilson intervals are 33.0% to 59.6% for Jev and 14.3% to 37.4% for OpenAI. They are unadjusted for the repeated sample reviews.
Traversal was the main failure mode: 53 of the 54 deaths were gap falls, and both models died at the first gap 21 times.
Kittie is my wife's plushie, with his own Instagram account: @kittiecutiepie. The game was an anniversary gift; the Chromatic came home with me from OpenAI DevDay.
Results recorded October 9, 2026.