Zstandard, pronounced “Zee standard”, often abbreviated as Zstd, is a real-time data compression algorithm and format developed by Facebook. It was first introduced in 2015 and has since gained significant attention and traction due to its impressive balance of high compression ratios and fast decompression speeds. Soon it can become a standard on the Web. But why?
Technical Overview
At its core, Zstandard is based on a variant of the LZ77 algorithm, a widely used method in data compression. It also uses a range of techniques like Huffman coding and Finite State Entropy (FSE), a form of entropy encoding. This combination allows Zstandard to adapt dynamically to the content being compressed. This is the key to possible Zstd success, as we’ll lean soon.
Comparison with Other Algorithms
Zstandard stands out when compared to other well-known compression algorithms like gzip and Brotli. While gzip is known for its simplicity and widespread support, Zstandard often surpasses it in both compression ratio and speed. Brotli, on the other hand, is closer to Zstandard in terms of compression efficiency, especially for web content, but Zstandard typically offers faster decompression speeds, making it more suitable for real-time applications.
That’s why it’s already a popular compression method used in many applications, from databases to real-time data transfer. But is it really needed on the web?
Zstd’s secret weapon - training mode
Almost all compression algorithms perform worse as the compressed data becomes smaller. It’s because most of them predict how to compress the data based on the already processed file contents. Zstandard mitigated this by providing the user with a “training mode”, that creates a dictionary for the data it trained on.
This dictionary is a tuning map for the algorithm. This increases all basic metrics dramatically. Zstd trained in compression of a certain type of data performs 3x better on compression ratios, nearly 4x better on compression speed, and over 4x better on decompression (according to Facebook’s benchmarks).
This makes it even better for dynamic content compression, as it’s possible to train it on sample data and effectively compress small payloads as well. And there’s a lot of dynamic and relatively small payloads that need to be compressed on the web.
Zstandard’s Journey to the Web
Initially, Zstandard gained traction in areas outside the web technology, particularly in database compression and internal data processing at large tech companies. Linux adopted it as a compression method for filesystems like btrfs or squashfs.
For some time Zstandard had its own RFC, to introduce it to the web, both as a compression algorithm (via Accept-Encoding
and Content-Encoding
headers) and new Media Type.
Chrome supports zstd as of Chrome 123. Zstd is supported by other Chromium-based browsers like Edge or Opera, but also by Firefox. Safari should follow suit soon, as they realized that all other major players already implemented it.
So now, that we know how zstd works and that is should be more common soon, let’s look at the numbers.
Zstandard Benchmarks
In the GitHub repository, zstd provides us with some benchmarks against leading compression algorithms:
Compressor name | Ratio | Compression | Decompress. |
---|---|---|---|
zstd 1.4.5 -1 | 2.884 | 500 MB/s | 1660 MB/s |
zlib 1.2.11 -1 | 2.743 | 90 MB/s | 400 MB/s |
brotli 1.0.7 -0 | 2.703 | 400 MB/s | 450 MB/s |
zstd 1.4.5 —fast=1 | 2.434 | 570 MB/s | 2200 MB/s |
zstd 1.4.5 —fast=3 | 2.312 | 640 MB/s | 2300 MB/s |
quicklz 1.5.0 -1 | 2.238 | 560 MB/s | 710 MB/s |
zstd 1.4.5 —fast=5 | 2.178 | 700 MB/s | 2420 MB/s |
lzo1x 2.10 -1 | 2.106 | 690 MB/s | 820 MB/s |
lz4 1.9.2 | 2.101 | 740 MB/s | 4530 MB/s |
lzf 3.6 -1 | 2.077 | 410 MB/s | 860 MB/s |
snappy 1.1.8 | 2.073 | 560 MB/s | 1790 MB/s |
As we can see zstd is capable of delivering rapid decompression speeds and great compression. Also if the high compression is not needed Zstandard can deliver fast compression times as well.
Chrome developers found that in terms of performance for static web content is similar to brotli. In such a scenario compression speed is not that important (as it happens once) and decompression happens on the client. Both offer comparable compression ratios.
For dynamic content using faster compression makes a lot of sense. It enables fast on-the-fly compression. This decreases CPU usage on servers and CDNs, another benefit is really fast decompression on the client (but really decompressing is not an big task a modern hardware)
Final Thoughts: The Rising Star of Web Compression
Zstd may not be a total game-changer, but it can be useful.
It’s a middle ground between brotli and gzip, offering a good compression ratio and fast compression and decompression. It really shines when it comes to the compression of small, dynamic payloads, as it can leverage its training ability.
For most websites, it won’t make a huge difference, but for heavy traffic ones, that serve a lot of content it can offload servers, and networks.