SWE-bench

Free

SWE-bench is a benchmark that tests whether AI models can fix real GitHub issues. It is a developer tool for people who need to know if a model can handle actual software bugs, not toy problems.

SWE-bench runs models against 2,294 real Python issues from open-source repos like pandas, Django, and scikit-learn. The task is simple to state: take an issue description and produce a patch that makes failing tests pass.

You can try variants built for different needs. SWE-bench Lite has 300 instances for cheaper runs. SWE-bench Verified has 500 human-filtered tasks with cleaner problem statements. There are also multilingual and multimodal versions if you need to test beyond Python.

The whole thing is Docker-based. You get a container with the full repo state, dependencies installed, and tests ready to run. The model or agent does the rest.

Open it if you need to compare how well models fix real bugs. Skip it if you want something you can run casually on a laptop.

It helps most when your job is to know what a model can actually do.

• AI researchers benchmarking code-fixing models: You get a standardized way to test patches against real test suites. The community uses it, so your numbers mean something to others.
• Teams evaluating which LLM to use for automated code repair: You see clear gaps between models. Claude Sonnet 3.7 sits around 70% resolved; GPT-4o around 38% on the same Verified set.
• Organizations needing validated, defensible benchmarks: You can point to human-reviewed tasks and pass-fail validation against real tests.
• Teams evaluating coding agents for production deployment: You test the full loop: explore the repo, edit files, run tests, iterate. This matches how agents work in practice.
• Researchers measuring true software engineering generalization: SWE-bench Pro adds proprietary codebases so you know if a model memorized its training data.
• Engineering teams selecting between frontier models: The leaderboard shows current results from Anthropic, OpenAI, Google, and others tested with the same reference agent.
• Teams building AI coding tools who need rigorous model comparison: You isolate model capability from your own scaffolding because everyone uses the same harness.
• Anyone trying to understand why benchmark improvements matter in practice: You can trace what "70%" means in terms of actual bugs that would have needed human time.

These are the parts you notice after running a few evaluations.

• Tests on real GitHub issues: You are not trusting synthetic coding puzzles. The model has to read actual issue descriptions, understand existing code, and fix something that matters to real users.
• Pass-fail evaluation against real test suites: There is no room for generous scoring. The patch either makes the tests pass or it does not.
• Multiple dataset variants: You can start with Lite for fast iteration, move to Verified for publishable results, or run Multilingual if you need to test across nine languages.
• Docker-based reproducibility: You get the same environment every time. Dependencies, file states, and test configurations are locked down.
• Cloud evaluation options: You can use sb-cli or Modal if you do not want to manage the 120GB storage and 8-core minimums yourself.

The setup is involved, but the path is clear.

1. Install Docker and ensure you have 120GB free storage, 16GB RAM, and 8 CPU cores.
2. Clone the repository and configure the v5 CLI with your model or agent.
3. Select your dataset: full, Lite, Verified, or another variant.
4. Run the evaluation harness, which builds containers and runs your agent against each issue.
5. Check the results: resolved percentage, logs, and the actual patches generated.

Skip it when the fit is wrong.

• Skip it if you have limited hardware: You need serious resources. This is not a weekend project on a MacBook Air.
• Skip it if you want simple scores to trust blindly: OpenAI stopped reporting Verified scores in early 2026 after audit concerns about test quality. About 30% of SWE-bench Pro public tasks may be flawed.
• Skip it if you need to isolate model from scaffolding: The harness, prompts, agent loops, and compute budget all change results. The same model can swing 20 points with different setups.
• Skip it if your code is not Python: The original dataset is Python-only. Other languages need the Multilingual variant, which has fewer tasks.
• Skip it if you need proprietary codebase coverage: Most tasks are open-source. The hidden Pro set exists but access is limited.
• Skip it if ARM64 is your only option: Support for Apple Silicon is experimental and requires workarounds.