Existing benchmarks measure solve rate and token output/cost, but don't punish excess code, which makes a codebase harder to work with in the long term. I wanted to measure this since it matters most for long-term projects.Method: it takes the patches that benchmarks already publish (DeepSWE, SWE-bench Verified, SWE-bench Live) and counts "coding units": syntax elements like statements, calls, names and literals. I didn't use lines of code, so cramming code isn't rewarded. Deleted code is subtracted, so it measures net units.GPT-6 and Claude 5.5 have no public patches yet, so I ran them myself on 3–10 tasks on my own subscriptions (shown as faded circles, not ranked).Any feedback or model contributions are welcome, especially public patch exports for newer models.GitHub: https://github.com/La5u/ParsimonyNot affiliated with any of the benchmarks.
Show HN: Parsimony – an open-source benchmark for how much code AI models add
An open-source benchmark of how much code AI models add.
HN Hacker News 
Key takeaways
Parsimony is an open-source benchmark that measures how much code AI models add, focusing on code efficiency.
- It counts coding units like statements and calls, not lines of code.
- Models are ranked by the net units of code they add.
- Users can input their own code-size numbers to generate charts and tables.
Summarised automatically by AI from the original article by Hacker News. AI can make mistakes, so check the original for details.
Read the full story on Hacker News That's the opening of the story. The full piece is published by Hacker News.
Continue reading ↗ Story details
- Published
- By
- lasu
- Format
- Article
- Original
- parsimony.lasu.dev ↗
Related reporting
6 stories on this topic from 5 sources, oldest first.
- Hacker News Show HN: Parsimony – an open-source benchmark for how much code AI models add (this story)
