BDX Bench

BDX Bench is an independent benchmark harness, leaderboard and playable browser-game showcase for testing AI models against identical real-world software briefs. The repo has two halves: a Next.js site (web/) behind bench.bdx.market that ranks models by the quality of the games they built, and a zero-dependency Node.js harness (server/, runner/, tasks/) with the swe-mini and terminal-mini suites, a prompt bank and a blind chat arena with Elo ratings. It is MIT licensed.

By Showdown Score v2. Every model received the same unprimed prompt asking for a zombie-burning video game and built a self-contained browser game. Each build is audited by reading its source against five dimensions worth 20 points each: Controls and Mobility, Combat Physics and Weapons, Content and Enemy Variety, Audio and Sound Design, and Visual Polish and Game Feel. The total is out of 100. A feature only earns points when it is genuinely implemented and reachable, so on-screen strings, comments and dead code score zero, and a build with no audio scores 0 on audio.

v1 scored builds by inspecting on-screen text and feature claims, which credited features that were absent from the code and reported scores up to 40 points too high. One build scored 93 although its flamethrower appeared exactly once in the file, in the page title. v1 is withdrawn, its numbers are kept only as superseded references, and v2 audits the implementation itself.

They are the two suites of the harness. swe-mini has six small JavaScript code-fix tasks (for example fixing an off-by-one in sum(), implementing debounce(), or fixing a race in an async counter). terminal-mini has six shell and scripting tasks (for example a CSV column sum, finding palindromes, or a topological sort). Each task is checked automatically with binary checks, all of which must pass, and is worth 10 points. The suites are kept in the repo, but the public site currently ranks Showdown Score v2 instead.

A match asks one prompt to two hidden sides, A and B, and model identities stay hidden until the verdict. Pairing is seeded, so the same seed reproduces the same schedule. Up to 3 named judges (people or models) each vote A, B or draw; a majority wins and tied scorecards are a draw. Every model starts at Elo 1000 with K = 32, so when two 1000-rated models meet and A wins the ratings become 1016 and 984, while a draw changes nothing. Every verdict is written to data/matches.json before the leaderboard is rebuilt.

Not for mock mode. Mock runs return canned fixture output, need no key and cost nothing, so they are good for plumbing and offline use. Live runs go through an OpenAI-compatible gateway (default https://gpt.bdx.market/v1, overridable with BDX_BASE_URL) and need BDX_AI_API_KEY set in your environment. The docs say never to print or commit the key. Mock and live results are never meant to be compared directly.

It is deliberately small and says so. The public audit covers one prompt in one domain, eleven builds from nine models, a single round with no repeat runs, judged by code inspection rather than a large human panel, so a 1-point gap is not a finding. The harness suites are 12 mini tasks, single attempt, with no Docker per task and binary checks only, so results are indicative rather than statistically significant. Scores and the evidence behind them are published so readers can disagree.

The nine models on the Showdown leaderboard are Space Bunny Free, DeepSeek V4.1 Flash, Claude Haiku 5.5, GPT Luna 5.6, GPT 6 Sol, Muse Spark 1.3, GPT Luna 6, Gemini 3.8 Flash and Gemini Pro Agent. models/models.json is the source of truth for the harness and also lists Gemini 3.7 Flash. Only models reachable through the OpenAI-compatible gateway can be run by the harness.

No. The three files results/demo-*.json are synthetic demo runs generated by scripts/seed-demo.js; each is marked mode "mock" and demo true, and its log says no real agent was executed. This website therefore does not present them as rankings. The leaderboard shown here is the Showdown Score v2 audit from the README.

Yes. It is MIT licensed. The harness server is a zero-dependency Node.js program started with npm start (it serves on 127.0.0.1:8765 by default), and the repo pins Node 20 in .nvmrc. The public site in web/ is a Next.js 16 / React 19 app run with npm install and npm run dev.