alt.hn

6/16/2026 at 9:51:57 PM

Data Compression Explained (2012)

https://mattmahoney.net/dc/dce.html

by mtdewcmu

6/20/2026 at 4:56:23 AM

The leader boards are from the pre Fabrice Bellard days, btw. Neural network modeling helped finding better patterns in text.

Also, you could say the same for the related data search problem. How to prepare data, so that it can most efficiently searched. Smallest encoding vs fastest search. Databases are mostly very, very stupid compared to more data-specific tuned algorithms. Like factor 1000 slower and bigger.

by rurban

6/20/2026 at 6:06:26 AM

Isn’t the idea of AI precisely to find universal compression from arbitrary input data, at least with LLMs?

by usernametaken29

6/20/2026 at 9:21:50 AM

Which is why in this book the title of a paragraph is "Compression is an artificial intelligence problem".

However, I believe that is less useful to think that AI "finds an universal compression", than to think that the training of an AI model has the purpose to find a specific lossy data compression method, which is close to optimal for the input data that constitutes the training data set.

One could consider the training algorithm as a universal lossy data compression method, but this view is not useful in practice, because unlike a traditional lossy data compression algorithm, used e.g. for movie or picture compression, which you can use every day to compress many data sets, even in real time on an incoming data stream, the training of an AI model is a very long and expensive operation, which can be done only infrequently and which makes sense only for special important datasets, from which data will be frequently extracted by querying (i.e. AI inference) for a long time, to make worthwhile the compression, i.e. training, cost.

Moreover, for the best compression results the training of a new improved AI model does not consist only in determining the values of parameters (weights) of a fixed inference algorithm, but the structure of the inference algorithms is also tweaked for each new generation of models.

This is an additional reason that makes impractical to think about training as a universal compression algorithm (instead of a method for searching specific compression algorithms, which work for a given training set), because it is not a fixed algorithm, but a family of algorithms that evolves continuously, at least for now.

by adrian_b

6/20/2026 at 7:49:58 AM

I think so, specifically lossy compression though.

A modern version of the book would include an extra section in the 'Lossy compression' chapter - 'Text' (alongside Images/Video/Audio) that would discuss LLM's.

by briansm

6/20/2026 at 8:39:11 AM

No, it's not for lossy compression only.

An LLM can give you a probability distribution for the next token. You can pair that with arithmetic coding to get a lossless compression/decompression algorithm. See https://en.wikipedia.org/wiki/Arithmetic_coding

by eru

6/20/2026 at 8:13:09 AM

I say transformers are the best compression systems

by brownpoints

6/20/2026 at 5:42:07 AM

This is the guy who created Zpaq btw. Super interesting but niche backup/archive software.

by wps

6/20/2026 at 5:38:20 AM

does anyone have any sources to read about ai-based compression?

I remember hearing a lot about "compression is a lot about prediction", but I don't remember reading any practical result

by NooneAtAll3

6/20/2026 at 6:29:38 AM

It can and has been done just not very practical. Having a dozen GB language model just to squeeze out few more percent on plaintext compression which already compresses well and is tiny in comparison of images or video is not worth it outside benchmarks. Even superior traditional conpression algorithms are often not used due to insufficient software support. Multigabyte decompressor as big as rest of your OS installation is not practical to distribute or standardize. It would also take a lot of memory at runtime for decompressing thus shadowing the efficiency gains in everyday use. Only if you have huge archival scale of data it might be worth the gains. But for long term archival fragile formats which depend on huge arbitrary extra knowledge isnt a good idea. I am not quite sure if ai based compression would make it more robust by allowing to fix corruption based on context or make it worse by having single bitflip produce completely opposite but still plausible looking text. At least with traditional compression its usually obvious when corruption causes gibberish. And then you have problem of versioning, you need to have exactly the same version of dozen GB model for decompression as was used for compression. Just one of them is questionable now imagine having to store few dozen of them. Most computers have code for supporting at least half a dozen compression formats, and many of those are parametrized allowing single algorithm to handle multiple varations of the compression scheme, and then many apps bundle their own copies of compression library.

by Karliss

6/20/2026 at 8:40:59 AM

I mostly agree, however:

> But for long term archival fragile formats which depend on huge arbitrary extra knowledge isnt a good idea.

This doesn't need to be a problem: you can and should layer an error correcting code on top.

by eru

6/20/2026 at 5:00:57 AM

Matt is a great guy to explain this kind of stuff. He's very helpful.

by blobbers