58Evidence
not yet measuredWhat it measures
Repo-level bug fixing across nine programming languages.
What it does not measure
Neuentwicklung, Architektur, Performance-Optimierung.
Deckt die Python-Schlagseite von SWE-bench Verified ab.
How the score is composed
Currency50
Discrimination50
Headroom50
Contamination safety33
Gaming resistance100
Methodological rigor80
Reproducibility60
Details
- Benchmaxxing risk
- low
- Difficulty
- high
- Public test set
- yes
- Human-verified
- yes
- LLM-judged
- no
- Sample size
- 300
- Skala
- 0–100 % solved (höher besser)
Hand-maintained — no automatic adapter yet.
Best measured models
No results ingested yet.