Jev
14.2 seconds · 2 hearts · jev-02
Head Goon
Does seeing the game help? I compared Jev, OpenAI Decisions with structured state, and OpenAI with the same state plus a screenshot.
Completion rate
Jev
jev-1.13.0
36.7%11 of 30 completed
OpenAI text
gpt-6-luna
3.3%1 of 30 completed
OpenAI + image
gpt-6-luna
20.0%6 of 30 completed
OpenAI completed more attempts with screenshots, rising from 1/30 to 6/30. Jev had the most wins in this sample at 11/30.
| Metric | Jev | OpenAI text | OpenAI + image |
|---|---|---|---|
| Mean API cost / attempt | $0.0070 | $0.0123 | $0.0164 |
| Mean boss hits / 3 | 2.17 | 1.07 | 1.40 |
| API latency · median / p95 | 226 / 339 ms | 224 / 789 ms | 225 / 346 ms |
| Mean completion time · wins only | 15.3 s | 14.1 s | 19.4 s |
| 95% Wilson interval | 21.9% to 54.5% | 0.6% to 16.7% | 9.5% to 37.3% |
Costs include all selected attempts and exclude emulator and host compute. Winning-time averages describe different successful subsets; OpenAI text had only one win.
Three successful runs
OpenAI text
14.1 seconds · 2 hearts · openai-text-24
OpenAI + image
20.0 seconds · 1 heart · openai-image-12
Each example is nearest its setup's median winning time. These clips illustrate successful play; most attempts failed. Watch all 90 attempts.
What changed with images
The extra input was one screenshot from the same frame as the structured observation, enlarged from 160×144 to 640×576 without smoothing. The model still received the boss phase and timers, ball state and approximate physics forecasts.
Adding screenshots improved OpenAI's results in this sample, even alongside structured game state. Jev still completed more attempts. The sample does not establish a reliable image advantage.
Head Goon methods and limitations
- All 90 selected attempts used the same contact-rule ROM and arena checkpoint. The first five image-assisted pilot runs are excluded.
- The goal was to hit Head Goon three times during tray-down recovery, then reach the exit alive. One physical wool ball had to be retrieved.
- Each run had its own delay estimate, starting at ten frames and updating with 0.65 old / 0.35 new weights. Previous controls remained held during API calls; the minimum interval after applying an answer was six frames.
- The three setups were collected in successive batches. Some batches were interleaved, but the complete comparison was not a single contemporaneous randomized experiment. Network conditions differed.
- More attempts were requested after earlier results were inspected. Confidence intervals do not adjust for those repeated looks. The exploratory two-sided Fisher comparison of OpenAI text and image gives p = 0.103 before multiple-comparison adjustment.
- No retries, fallback controls or replacement failures. All selected replays passed framebuffer and game-state verification.
Results recorded October 9, 2026.