OpenGameEval: Roblox opens an 84-task benchmark for AI agents

What Roblox released

Roblox open-sourced OpenGameEval, a framework for evaluating models inside reproducible Roblox Studio sessions. The repository includes scripts, test places, annotations, and a plugin for automated evaluations.

What it measures

The core suite contains 84 curated tasks and separates observation actions from editing actions. This makes it possible to inspect not only whether the final result passes, but whether an agent inspected the relevant objects and scripts before changing the project.

How scoring works

Each run is checked on the edited scene and during a simulated play session. A task passes only when all checks succeed; results report passes, failures, warnings, and interruptions.

What the study found

The study compared 13 models with 16 attempts per task. The best reported result solved 51.7% of tasks in one attempt and 39.4% under the stricter five-out-of-five measure. These are benchmark results, not a guarantee for production projects.

Why creators should care

OpenGameEval provides a repeatable way to test whether an agent can locate objects, read scripts, and apply verifiable changes. It also shows that making changes without enough inspection remains an important source of failure.

What you need

The project requires a Roblox account, an Open Cloud key with studio-evaluations:create, a local checkout of the repository, and your own credentials for the model being evaluated. The documentation recommends expiring keys and notes request limits.

Practical strategy

Start with one task and inspect the log before running a batch. Confirm that the evaluation identifies expected instances, separates observation from editing, and fails when a change violates an important condition.

Limits

The repository does not automatically make an agent a reliable developer. Tasks use controlled projects and prepared checks; a production game adds dependencies, data, and behavior that the benchmark does not cover.

Checklist

• Create an Open Cloud key with minimal scope.
• Run one task before enabling concurrency.
• Review expected instances and tools.
• Never place secrets in public files.
• Add project-specific checks for critical mechanics.