One of our customers backs up a 4 TB file server every night and keeps 30 days of history. On paper that is 120 TB. On disk it takes 6. This post explains where the other 114 went.

Fixed blocks break on inserts

The simple way to deduplicate is to cut every file into 4 MB blocks, hash each block and store each hash once. It works until someone adds a single byte at the start of a file. Every block boundary after that byte shifts, every hash changes, and the whole file is stored again.

Content-defined chunking

Stonebridge cuts files where the content says so instead. A rolling hash slides over the data, and wherever its low 20 bits are zero, a chunk ends. An insert only changes the chunk it lands in. The boundaries after it stay where they were, because they depend on the bytes around them and not on their offset.

Chunks average 1 MB, with a floor of 256 KB and a ceiling of 8 MB, so a run of zeros can’t produce a million tiny chunks.

The numbers

Logical size Stored
First full backup 4.0 TB 3.1 TB
30 nightly backups 120 TB 6.2 TB
Same data, fixed 4 MB blocks 120 TB 19.8 TB

The first backup already saves 0.9 TB, mostly from copies of the same installers and PDFs in different folders.

You can check yours

Run this on the Stonebridge server to see the ratio for each backup set:

sb stats --sets --sort ratio
SET            LOGICAL    STORED   RATIO
fileserver     120.4 TB   6.2 TB   19.4x
postgres-main   18.1 TB   2.7 TB    6.7x
mail            9.6 TB    3.9 TB    2.5x

Mail deduplicates poorly because most of it is already compressed attachments. That is expected, and it is why we show the ratio per set instead of one number for the whole server.