Deduplication and Compression
How Data Deduplication Works
Data deduplication is a storage technique that finds identical pieces of data and keeps only one copy, replacing the others with references to it. The system splits data into blocks or chunks, calculates a fingerprint (a hash) for each one, and checks whether a block with the same fingerprint is already stored. If it is, the new copy is not written again.
Deduplication works best on data with a lot of repetition. Virtual machine images that share the same operating system files, nightly backups that change only slightly, and document libraries with many copies of one attachment often shrink substantially. Data that is already unique, such as encrypted files or compressed video, saves little.
How Compression Reduces Data Size
Compression encodes data more compactly by removing redundancy inside a piece of data, such as repeated patterns or predictable sequences. Storage systems use lossless compression, which restores the original bytes exactly when the data is read. Text, logs, database dumps and source code usually compress well. Photos in JPEG format and videos in common codecs are already compressed and barely shrink.
Compression and deduplication address different redundancy. Deduplication finds the same data across many files or blocks, while compression reduces repetition inside each block. Many storage systems use both. ZFS is one file system that offers both features.
The Costs: CPU, Memory and Complexity
Both techniques trade processing power for space. Compressing data and hashing blocks uses CPU time on writes and often on reads. Deduplication also needs an index of block fingerprints, and that index can need large amounts of memory, especially when many blocks are unique. An index that does not fit in memory can slow the whole system.
Deduplication also creates shared dependencies. Many files point to the same stored blocks, so damage to one shared block can affect all of them, and deleting data may not free space right away. Data that rarely changes is often placed on colder storage, as explained in hot vs cold storage tiers, and administrators usually enable deduplication only where the savings clearly justify these costs.
Frequently asked questions
Does deduplication always save space?
No. Savings depend on how much data repeats. Backups and virtual machine images often shrink a lot, while already compressed or encrypted files barely change in size. Before enabling deduplication on a production system, test with a copy of your real data to see the ratio you actually get.
Is compression safe for stored data?
Lossless compression is safe in the sense that the original data is restored exactly when read. Its real costs are CPU time and occasionally slower reads or writes. Storage systems usually let you choose a compression method and can skip compression for data that does not get smaller.