Zstandard, pronounced “Zee standard”, often abbreviated as Zstd, is a real-time data compression algorithm and format developed by Facebook. It was first introduced in 2015 and has since gained significant attention and traction due to its impressive balance of high compression ratios and fast decompression speeds. Soon it can become a standard on the Web.
Zstandard was created by Yann Collet at Facebook. The development focused on optimizing compression speed, decompression speed, and the compression ratio. Zstandard is part of a larger family of compression algorithms and is built upon a combination of tried-and-tested compression techniques along with new innovative approaches.
Technical Overview
At its core, Zstandard is based on a variant of the LZ77 algorithm, a widely used method in data compression. It also employs a range of techniques like Huffman coding and Finite State Entropy (FSE), a form of entropy encoding. This combination allows Zstandard to adapt dynamically to the content being compressed, achieving high compression ratios without sacrificing speed.
Zstandard offers multiple levels of compression, ranging from level 1 (fastest, but less compression) to level 22 (slowest, but maximum compression). This scalability makes it suitable for various applications, from quick real-time compression needs to scenarios where high compression ratio is needed.
Comparison with Other Algorithms
Zstandard stands out when compared to other well-known compression algorithms like gzip and Brotli. While gzip is known for its simplicity and widespread support, Zstandard often surpasses it in both compression ratio and speed. Brotli, on the other hand, is closer to Zstandard in terms of compression efficiency, especially for web content, but Zstandard typically offers faster decompression speeds, making it more suitable for real-time applications.
Zstandard’s efficient balance of speed and compression ratio has made it a popular choice in a wide range of applications, from database compression to real-time data transfer. With its growing adoption in web technologies, Zstandard is poised to become a key player in the field of data compression.
Zstd’s secret weapon - training mode
Almost all compression algorithms perform worse as the compressed data becomes smaller. It’s because most of them predict how to compress the data based on the already processed file contents. Zstandard mitigated this by providing the user with a “training mode”, that creates a dictionary for the data it trained on.
This dictionary is a tuning map for the algorithm. This increases all basic metrics dramatically. Zstd trained in compression of a certain type of data performs 3x better on compression ratios, nearly 4x better on compression speed, and over 4x better on decompression (according to Facebook’s benchmarks).
This makes it even better for dynamic content compression, as it’s possible to train it on sample data and effectively compress small payloads as well.
Zstandard’s Journey to the Web
Initially, Zstandard gained traction in areas outside the web technology, particularly in database compression and internal data processing at large tech companies. Its efficiency and speed quickly garnered attention, leading to its wider adoption in various software and systems. Linux adopted it as a compression method for filesystems like btrfs or squashfs.
For some time Zstandard had its own RFC, to introduce it to the web, both as a compression algorithm (via Accept-Encoding and Content-Encoding headers) and new Media Type.
Chrome supports zstd as of Chrome 123. From this point, the adoption of zstd should accelerate. Safari should follow suit soon, as the proposition to implement zstd gained support.
So now, that we know how zstd works and that is should be more common soon, let’s look at the numbers.
Zstandard Benchmarks
In the GitHub repository, zstd provides us with some benchmarks against leading compression algorithms:
| Compressor name | Ratio | Compression | Decompress. |
|---|---|---|---|
| zstd 1.4.5 -1 | 2.884 | 500 MB/s | 1660 MB/s |
| zlib 1.2.11 -1 | 2.743 | 90 MB/s | 400 MB/s |
| brotli 1.0.7 -0 | 2.703 | 400 MB/s | 450 MB/s |
| zstd 1.4.5 —fast=1 | 2.434 | 570 MB/s | 2200 MB/s |
| zstd 1.4.5 —fast=3 | 2.312 | 640 MB/s | 2300 MB/s |
| quicklz 1.5.0 -1 | 2.238 | 560 MB/s | 710 MB/s |
| zstd 1.4.5 —fast=5 | 2.178 | 700 MB/s | 2420 MB/s |
| lzo1x 2.10 -1 | 2.106 | 690 MB/s | 820 MB/s |
| lz4 1.9.2 | 2.101 | 740 MB/s | 4530 MB/s |
| lzf 3.6 -1 | 2.077 | 410 MB/s | 860 MB/s |
| snappy 1.1.8 | 2.073 | 560 MB/s | 1790 MB/s |
As we can see zstd is capable of delivering rapid decompression speeds and great compression. Also if the high compression is not needed Zstandard can deliver fast compression times as well.
Looking at benchmarks it’s clear Zstandard offers a wide range of compression levels, from very fast but less compressed to slower but highly compressed. This scalability allows developers to tailor the compression to their specific needs, whether they prioritize speed or size reduction.
This is crucial for zstd. Chrome developers found that in terms of performance for static web content is similar to brotli. In such a scenario compression speed is not that important (as it happens once) and decompression happens on the client. Both offer comparable compression ratios.
For dynamic content using faster compression makes a lot of sense. It enables fast on-the-fly compression. This decreases CPU usage on servers and CDNs, another benefit is really fast decompression on the client.
Final Thoughts: The Rising Star of Web Compression
Ok, zstd may not be a total game-changer. But it can be really useful, as its unique blend of speed, efficiency, and adaptability sets it apart.
It’s a middle ground between brotli and gzip, offering a good compression ratio and fast compression and decompression. It really shines when it comes to the compression of small, dynamic payloads, as it can leverage its training ability.
For most websites, it won’t make a huge difference, but for heavy traffic ones, that serve a lot of content it can offload servers, and networks and help users to get their favorite site even faster.