When Facebook announced a new lossless compression algorithm on August 31, it looked very much like a case of life imitating art.
Pop culture followers saw parallels between the release of Zstandard and the fictional Pied Piper compression algorithm, the McGuffin driving the action in the HBO series “Silicon Valley,” which explores the high-tech gold rush that erupts when scientists produce an algorithm that more effectively compresses data.
Facebook’s Zstandard is open source and available for anyone to use, so there likely won’t be a repeat of the show’s bidding wars among investors. Still, the coming of a hyper-powerful new compression algorithm could be big news — in the mobile enterprise world, data compression is all about the effective use of resources. Efficient compression can save storage capacity, speed file transfers and decrease expenses related to storage hardware and network bandwidth.
It’s also a relatively straightforward process. “Software helps with data processing through compression, which encodes information, like text, pictures, and other forms of digital data, using fewer bits than the original. These smaller files take up less space on hard drives and are transmitted faster to other systems,” Facebook notes. But there is a trade-off, as the activities of compression tend to slow data processing.
A New Standard for Data Compression
For two decades, the Deflate algorithm has set the standard, driving compression processes such as Zip, gzip and zlib and baked into practically every modern device. But Facebook engineers say their new standard compresses data to be smaller, and does so faster: According to Facebook, Zstandard 1.0 can squeeze data 10 to 15 percent smaller and three to five times faster than competing standards. Facebook says it’s already used Zstandard to compress databases, backup systems and other data.
In a related announcement, Facebook also released open source code for MyRocks, a next-gen MySQL storage engine that’s enabling Facebook to reduce the number of servers needed to host its MySQL databases by half.
In the TV version of events, the release of the Pied Piper compression algorithm lights a fire under investors who foresee a big impact on enterprise users struggling to contend with today’s avalanche of data. The coming months could show whether the arrival of such a capability in the real world will have a similarly galvanizing effect in the mobile enterprise space.
Open source code has the potential to benefit everyone, as illustrated by the White House’s new initiative to make open source code available to the public.
