Sunsetting the Techempower Framework Benchmarks

3/24/2026 at 9:15:11 AM

I've contributed a few optimisations to some implementations in these benchmarks, but as I read the code of many other implementations (and some frameworks) I lost most of the trust I had in these benchmarks.

I knew that once a benchmark is famous, people start optimising for it or even gaming it, but I didn't realise how much it made the benchmarks meaningless. Some frameworks were just not production ready, or had shortcuts made just for a benchmark case. Some implementations were supposed to use a framework, but the code was skewed in an unrealistic way. And sometimes the algorithm was different (IIRC, some implementation converted the "multiple sql updates" requirements into a single complex update using CASE).

I would ignore the results for most cases, especially the emerging software, but at least the benchmarks suggested orders of magnitudes in a few cases. I.e. the speed of JSON serialization in different languages, or that PHP Laravel was more or less twice slower than PHP Symfony which could be twice slower than Rails.

by idoubtit

3/24/2026 at 11:24:35 AM

This was also my experience.

by brightball

3/24/2026 at 4:09:55 AM

I really liked these benchmarks, and would check in with them from time to time.

No benchmark is perfect, but these ones cover such a wide variety of different languages and frameworks, it's a good resource for getting a rough idea of the kind of performance that a given stack is capable of.

I don't know much about TechEmpower the company, it seems to be a small consultancy, maintaining this project probably takes non insignificant resources from them.

The end of the project seems kind of unceremonious, but they don't owe anything to anyone.

Hopefully an active fork emerges.

by WatchDog

3/24/2026 at 5:42:25 AM

It's cool in a 'how much can you tune it' kind of way, but has little practical value. Most sites would be tickled with a 4 digit requests per second number, so does it matter if your chosen framework does 50k/sec or 3 million/sec? Not really.

I think the biggest problem was it just had too many entries, most of which seem tuned to cheating benchmarks. Would probably be more valuable just choosing the top 3 by popularity from the top 15 languages or so.

by silisili

3/24/2026 at 9:04:08 AM

> too many entries, most of which seem tuned to cheating benchmarks

Even for entries that didn't cheat, the code was sometimes unidiomatic in the sense that "real programmers can write Fortran in any language".

This[0] article articulates the issue with by highlighting an ASP.NET implementation that was faster than more 'honest' Java/Go implementations primarily by not using ASP.NET features, skirting some philosophical line of what it means to use something.

For me, the more interesting discussion of whether a language/library is faster/leaner than another exists in actual idiomatic use. In some languages you are actively sweating over individual allocations; in some you're encouraged to allocate collections and immediately throw them away. Being highly concerned with memory and performance in the latter type of language happens, but is seldom the dominant approach in the larger ecosystem.

[0] https://dusted.codes/how-fast-is-really-aspnet-core

by fredrikholm

3/24/2026 at 10:45:14 AM

> most of which seem tuned to cheating benchmarks

The fix would have been requiring tests to catch the cheating. There were suggestions but it didn't happen.

It was definitely possible to catch not having sent date headers (or caching them) etc.

by re-thc

3/24/2026 at 3:32:15 AM

This text lacks information about why it is being sunset.

by mseepgood

3/24/2026 at 9:27:22 AM

Maintaining something like this is probably a little bit stressful.

We all know some of us take our language and framework choices as seriously as religion. I wouldn't be surprised if there was a lawsuit involved.

by bob1029

3/24/2026 at 5:58:11 AM

Engineering has kind of moved on in a weird way from web frameworks. Now AI just writes document.getElementById('longVariableName') javascript and straight SQL without complaining at all. The abstraction isn't as important as it used to be because AI doesn't mind typing.

by narrator

3/24/2026 at 10:40:36 AM

> Now AI just writes document.getElementById('longVariableName') javascript and straight SQL without complaining at all

I got a newer model that bypasses all that. It takes out Wireshark and send bytes straight.

by re-thc

3/24/2026 at 6:31:13 AM

Would you know any alternative?

by GrooveSAN

3/24/2026 at 1:54:03 AM

[dead]

by nbrady

3/24/2026 at 4:11:24 AM

[dead]

by andrewmcwatters