Add metadata checksum (#12017)

- Add 32-bit checksum (32 LSB part of xxhash64) of the serialized metadata.

This will ensure that we always reject corrupted metadata.

- Add automatic repair of inline data, so the data structure can be used.

If data was corrupted, we remove all unreadable entries to ensure that operations 
can succeed on the object. Since higher layers add bitrot checks this is not a big problem.

Cannot downgrade to v1.1 metadata, but since that isn't released, no need for a major bump.
This commit is contained in:
Klaus Post
2021-04-09 02:29:54 +02:00
committed by GitHub
parent 0e4794ea50
commit f0ca0b3ca9
2 changed files with 131 additions and 53 deletions

View File

@@ -137,8 +137,14 @@ func TestXLV2FormatData(t *testing.T) {
// Test trimmed
xl2 = xlMetaV2{}
failOnErr(xl2.Load(xlMetaV2TrimData(serialized)))
trimmed := xlMetaV2TrimData(serialized)
failOnErr(xl2.Load(trimmed))
if len(xl2.data) != 0 {
t.Fatal("data, was not trimmed, bytes left:", len(xl2.data))
}
// Corrupt metadata, last 5 bytes is the checksum, so go a bit further back.
trimmed[len(trimmed)-10] += 10
if err := xl2.Load(trimmed); err == nil {
t.Fatal("metadata corruption not detected")
}
}