Opus 4.5 scores 80.6% on SWE-bench Verified. Opus 4 scored 72.5%. So Opus 4.5 is better at programming than Opus 4, right?
Well… maybe . But that’s not what SWE-bench Verified tells you. What it tells you is a model’s ability to fix small bugs in 12 popular open source Python repositories, all of which are likely part of its training data. SWE-bench Verified doesn’t test a model’s ability to navigate your TypeScript monorepo, or your Spring Boot application, or the custom ORM your previous CTO insisted on building.
I got the itch to write this post because I kept seeing the same set of benchmarks appearing in new model releases. I had no idea what they meant. So I went and read the papers, read the code, and read a bunch of critiques. The result is a summary of 14 benchmarks: what they test, how they were created, what criticisms have been levied against them, and my own thoughts.
Click the tabs below to learn more about each benchmark.
I see a growing negative sentiment around AI benchmarking, and after writing this post I understand why. The pace of modern LLM development has left benchmark creators in a difficult spot. To make a benchmark in time to be useful without getting instantly saturated you need to move quickly. It’s a pace that the industry is not used to, and trade-offs need to be made.
It’s important to remember that we’re still in the early stages of benchmarking this new technology. No-one knows for sure where these models will be in 12 months’ time. It’s one of the fastest moving targets I’ve seen in tech since I started my career in 2012.
If you take one thing away from this post, please let it be that understanding the numbers you’re seeing is crucial. Benchmark scores are difficult to connect back to reality, and if you need to measure how good a model is at something you care about I don’t think there’s a good substitute to creating your own tests. When time comes to run those tests, I hope you’ll consider ngrok.ai to route your requests to many models and providers using a single SDK client.
I plan to create my own set of benchmarks to test how good models are at doing ngrok-related tasks in the near future, and I’ll share them on this blog when they’re ready!
A complete guide to what quantization is, how it works, and how it's used to compress large language models
A far more detailed explanation of prompt caching than anyone asked for: how tokens, embeddings, and attention make cached LLM tokens 10x cheaper and faster.