Notes on May 14, 2026

ProgramBench¹. A new benchmark that evaluates whether language models can rebuild black-box software systems from scratch.

It was created by Meta, and co-first author John Yang is also the creator of SWE-bench.

Unlike previous benchmarks such as SWE-bench, which work with existing codebases, each ProgramBench task gives the model only a compiled binary and its documentation. The model must then implement a complete codebase that reproduces the original program’s behavior. During this process, the model has no internet access, and decompilation tools are banned.

This is a difficult benchmark. As of today (May 14), only one task had been fully solved by GPT-5.5², giving it a score of 0.5% across the 200 tasks.

ProgramBench leaderboard