mirror of
https://github.com/minio/minio.git
synced 2024-12-29 08:33:21 -05:00
77d5331e85
12 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Klaus Post
|
5f774951b1
|
Store object EC in metadata header (#19534)
Keep the EC in header, so it can be retrieved easily for dynamic quorum calculations.
To not force a full metadata decode on every read the value will be 0/0 for data written in previous versions.
Size is expected to increase by 2 bytes per version, since all valid values can be represented with 1 byte each.
Example:
```
λ xl-meta xl.meta
{
"Versions": [
{
"Header": {
"EcM": 4,
"EcN": 8,
"Flags": 6,
"ModTime": "2024-04-17T11:46:25.325613+02:00",
"Signature": "0a409875",
"Type": 1,
"VersionID": "8e03504e11234957b2727bc53eda0d55"
},
...
```
Not used for operations yet.
|
||
|
Klaus Post
|
a2f6252b2f
|
xl-meta: Add inline data bitrot check (#19240)
When using `-data` also perform a bitrot check on the data.
Example:
```
λ xl-meta -data net.zip
{
"minio-1.com:9000/data/minio1/p40/44b6/44b612e9a7294856bd2b5fe6f6cdeb0d.pdf/xl.meta": {"null":{"bitrot_valid":true,"bytes":11710}},
"minio-1.com:9000/data/minio2/p40/44b6/44b612e9a7294856bd2b5fe6f6cdeb0d.pdf/xl.meta": {"null":{"bitrot_valid":true,"bytes":11710}},
"minio-1.com:9000/data/minio3/p40/44b6/44b612e9a7294856bd2b5fe6f6cdeb0d.pdf/xl.meta": {"null":{"bitrot_valid":true,"bytes":11710}},
"minio-1.com:9000/data/minio4/p40/44b6/44b612e9a7294856bd2b5fe6f6cdeb0d.pdf/xl.meta": {"null":{"bitrot_valid":true,"bytes":11710}},
"minio-2.com:9000/data/minio1/p40/44b6/44b612e9a7294856bd2b5fe6f6cdeb0d.pdf/xl.meta": {"null":{"bitrot_valid":true,"bytes":11710}},
"minio-2.com:9000/data/minio2/p40/44b6/44b612e9a7294856bd2b5fe6f6cdeb0d.pdf/xl.meta": {"null":{"bitrot_valid":true,"bytes":11710}},
"minio-2.com:9000/data/minio3/p40/44b6/44b612e9a7294856bd2b5fe6f6cdeb0d.pdf/xl.meta": {"null":{"bitrot_valid":true,"bytes":11710}},
"minio-2.com:9000/data/minio4/p40/44b6/44b612e9a7294856bd2b5fe6f6cdeb0d.pdf/xl.meta": {"null":{"bitrot_valid":true,"bytes":11710}},
"minio-3.com:9000/data/minio1/p40/44b6/44b612e9a7294856bd2b5fe6f6cdeb0d.pdf/xl.meta": {"null":{"bitrot_valid":true,"bytes":11710}},
"minio-3.com:9000/data/minio2/p40/44b6/44b612e9a7294856bd2b5fe6f6cdeb0d.pdf/xl.meta": {"null":{"bitrot_valid":true,"bytes":11710}},
"minio-3.com:9000/data/minio3/p40/44b6/44b612e9a7294856bd2b5fe6f6cdeb0d.pdf/xl.meta": {"null":{"bitrot_valid":true,"bytes":11710}},
"minio-3.com:9000/data/minio4/p40/44b6/44b612e9a7294856bd2b5fe6f6cdeb0d.pdf/xl.meta": {"null":{"bitrot_valid":true,"bytes":11710}},
"minio-4.com:9000/data/minio1/p40/44b6/44b612e9a7294856bd2b5fe6f6cdeb0d.pdf/xl.meta": {"null":{"bitrot_valid":true,"bytes":11710}},
"minio-4.com:9000/data/minio2/p40/44b6/44b612e9a7294856bd2b5fe6f6cdeb0d.pdf/xl.meta": {"null":{"bitrot_valid":true,"bytes":11710}},
"minio-4.com:9000/data/minio3/p40/44b6/44b612e9a7294856bd2b5fe6f6cdeb0d.pdf/xl.meta": {"null":{"bitrot_valid":true,"bytes":11710}},
"minio-4.com:9000/data/minio4/p40/44b6/44b612e9a7294856bd2b5fe6f6cdeb0d.pdf/xl.meta": {"null":{"bitrot_valid":true,"bytes":11710}}
}
```
|
||
|
Klaus Post
|
ca258c04cb
|
xl-meta: Clean output (#18794)
* Sort files.
* Sort maps.
* Remove annoying stray ','
* Hide -ndjson - doesn't do what it claims.
Sample output:
```
{
"/tmp/xl1/testbucket/hosts/xl.meta": {"Versions":[{"Header":{"Flags":6,"ModTime":"2022-03-27T07:35:22.820280431+02:00","Signature":"ac150d34","Type":1,"VersionID":"12468eb5e6024ab581de7aa9a5220a4e"},"Idx":0,"Metadata":{"Type":1,"V2Obj":{"CSumAlgo":1,"DDir":"DvH10ieUSPmBLbKfCp3lVQ==","EcAlgo":1,"EcBSize":1048576,"EcDist":[5,6,7,8,9,10,11,12,13,14,15,16,1,2,3,4],"EcIndex":5,"EcM":12,"EcN":4,"ID":"EkaOteYCSrWB3nqppSIKTg==","MTime":1648359322820280300,"MetaSys":{"x-minio-internal-inline-data":"dHJ1ZQ=="},"MetaUsr":{"content-type":"application/octet-stream","etag":"ea210162eeb4adc31ff984cb744d6af6"},"PartASizes":[303],"PartETags":null,"PartNums":[1],"PartSizes":[303],"Size":303}}}]},
"/tmp/xl10/testbucket/hosts/xl.meta": {"Versions":[{"Header":{"Flags":0,"ModTime":"2022-03-27T07:40:36.271878663+02:00","Signature":"4ffe317b","Type":2,"VersionID":"afc0611a63e0403b8078bf21b9111a2b"},"Idx":0,"Metadata":{"DelObj":{"ID":"r8BhGmPgQDuAeL8huREaKw==","MTime":1648359636271878700,"MetaSys":{}},"Type":2}},{"Header":{"Flags":6,"ModTime":"2022-03-27T07:35:22.820280431+02:00","Signature":"ac150d34","Type":1,"VersionID":"12468eb5e6024ab581de7aa9a5220a4e"},"Idx":1,"Metadata":{"Type":1,"V2Obj":{"CSumAlgo":1,"DDir":"DvH10ieUSPmBLbKfCp3lVQ==","EcAlgo":1,"EcBSize":1048576,"EcDist":[5,6,7,8,9,10,11,12,13,14,15,16,1,2,3,4],"EcIndex":14,"EcM":12,"EcN":4,"ID":"EkaOteYCSrWB3nqppSIKTg==","MTime":1648359322820280300,"MetaSys":{"x-minio-internal-inline-data":"dHJ1ZQ=="},"MetaUsr":{"content-type":"application/octet-stream","etag":"ea210162eeb4adc31ff984cb744d6af6"},"PartASizes":[303],"PartETags":null,"PartNums":[1],"PartSizes":[303],"Size":303}}}]},
"/tmp/xl11/testbucket/hosts/xl.meta": {"Versions":[{"Header":{"Flags":6,"ModTime":"2022-03-27T07:35:22.820280431+02:00","Signature":"ac150d34","Type":1,"VersionID":"12468eb5e6024ab581de7aa9a5220a4e"},"Idx":0,"Metadata":{"Type":1,"V2Obj":{"CSumAlgo":1,"DDir":"DvH10ieUSPmBLbKfCp3lVQ==","EcAlgo":1,"EcBSize":1048576,"EcDist":[5,6,7,8,9,10,11,12,13,14,15,16,1,2,3,4],"EcIndex":15,"EcM":12,"EcN":4,"ID":"EkaOteYCSrWB3nqppSIKTg==","MTime":1648359322820280300,"MetaSys":{"x-minio-internal-inline-data":"dHJ1ZQ=="},"MetaUsr":{"content-type":"application/octet-stream","etag":"ea210162eeb4adc31ff984cb744d6af6"},"PartASizes":[303],"PartETags":null,"PartNums":[1],"PartSizes":[303],"Size":303}}}]},
"/tmp/xl12/testbucket/hosts/xl.meta": {"Versions":[{"Header":{"Flags":0,"ModTime":"2022-03-27T07:40:36.271878663+02:00","Signature":"4ffe317b","Type":2,"VersionID":"afc0611a63e0403b8078bf21b9111a2b"},"Idx":0,"Metadata":{"DelObj":{"ID":"r8BhGmPgQDuAeL8huREaKw==","MTime":1648359636271878700,"MetaSys":{}},"Type":2}},{"Header":{"Flags":6,"ModTime":"2022-03-27T07:35:22.820280431+02:00","Signature":"ac150d34","Type":1,"VersionID":"12468eb5e6024ab581de7aa9a5220a4e"},"Idx":1,"Metadata":{"Type":1,"V2Obj":{"CSumAlgo":1,"DDir":"DvH10ieUSPmBLbKfCp3lVQ==","EcAlgo":1,"EcBSize":1048576,"EcDist":[5,6,7,8,9,10,11,12,13,14,15,16,1,2,3,4],"EcIndex":16,"EcM":12,"EcN":4,"ID":"EkaOteYCSrWB3nqppSIKTg==","MTime":1648359322820280300,"MetaSys":{"x-minio-internal-inline-data":"dHJ1ZQ=="},"MetaUsr":{"content-type":"application/octet-stream","etag":"ea210162eeb4adc31ff984cb744d6af6"},"PartASizes":[303],"PartETags":null,"PartNums":[1],"PartSizes":[303],"Size":303}}}]},
"/tmp/xl13/testbucket/hosts/xl.meta": {"Versions":[{"Header":{"Flags":0,"ModTime":"2022-03-27T07:40:36.271878663+02:00","Signature":"4ffe317b","Type":2,"VersionID":"afc0611a63e0403b8078bf21b9111a2b"},"Idx":0,"Metadata":{"DelObj":{"ID":"r8BhGmPgQDuAeL8huREaKw==","MTime":1648359636271878700,"MetaSys":{}},"Type":2}},{"Header":{"Flags":6,"ModTime":"2022-03-27T07:35:22.820280431+02:00","Signature":"ac150d34","Type":1,"VersionID":"12468eb5e6024ab581de7aa9a5220a4e"},"Idx":1,"Metadata":{"Type":1,"V2Obj":{"CSumAlgo":1,"DDir":"DvH10ieUSPmBLbKfCp3lVQ==","EcAlgo":1,"EcBSize":1048576,"EcDist":[5,6,7,8,9,10,11,12,13,14,15,16,1,2,3,4],"EcIndex":1,"EcM":12,"EcN":4,"ID":"EkaOteYCSrWB3nqppSIKTg==","MTime":1648359322820280300,"MetaSys":{"x-minio-internal-inline-data":"dHJ1ZQ=="},"MetaUsr":{"content-type":"application/octet-stream","etag":"ea210162eeb4adc31ff984cb744d6af6"},"PartASizes":[303],"PartETags":null,"PartNums":[1],"PartSizes":[303],"Size":303}}}]},
"/tmp/xl14/testbucket/hosts/xl.meta": {"Versions":[{"Header":{"Flags":0,"ModTime":"2022-03-27T07:40:36.271878663+02:00","Signature":"4ffe317b","Type":2,"VersionID":"afc0611a63e0403b8078bf21b9111a2b"},"Idx":0,"Metadata":{"DelObj":{"ID":"r8BhGmPgQDuAeL8huREaKw==","MTime":1648359636271878700,"MetaSys":{}},"Type":2}},{"Header":{"Flags":6,"ModTime":"2022-03-27T07:35:22.820280431+02:00","Signature":"ac150d34","Type":1,"VersionID":"12468eb5e6024ab581de7aa9a5220a4e"},"Idx":1,"Metadata":{"Type":1,"V2Obj":{"CSumAlgo":1,"DDir":"DvH10ieUSPmBLbKfCp3lVQ==","EcAlgo":1,"EcBSize":1048576,"EcDist":[5,6,7,8,9,10,11,12,13,14,15,16,1,2,3,4],"EcIndex":2,"EcM":12,"EcN":4,"ID":"EkaOteYCSrWB3nqppSIKTg==","MTime":1648359322820280300,"MetaSys":{"x-minio-internal-inline-data":"dHJ1ZQ=="},"MetaUsr":{"content-type":"application/octet-stream","etag":"ea210162eeb4adc31ff984cb744d6af6"},"PartASizes":[303],"PartETags":null,"PartNums":[1],"PartSizes":[303],"Size":303}}}]},
"/tmp/xl15/testbucket/hosts/xl.meta": {"Versions":[{"Header":{"Flags":0,"ModTime":"2022-03-27T07:40:36.271878663+02:00","Signature":"4ffe317b","Type":2,"VersionID":"afc0611a63e0403b8078bf21b9111a2b"},"Idx":0,"Metadata":{"DelObj":{"ID":"r8BhGmPgQDuAeL8huREaKw==","MTime":1648359636271878700,"MetaSys":{}},"Type":2}},{"Header":{"Flags":6,"ModTime":"2022-03-27T07:35:22.820280431+02:00","Signature":"ac150d34","Type":1,"VersionID":"12468eb5e6024ab581de7aa9a5220a4e"},"Idx":1,"Metadata":{"Type":1,"V2Obj":{"CSumAlgo":1,"DDir":"DvH10ieUSPmBLbKfCp3lVQ==","EcAlgo":1,"EcBSize":1048576,"EcDist":[5,6,7,8,9,10,11,12,13,14,15,16,1,2,3,4],"EcIndex":3,"EcM":12,"EcN":4,"ID":"EkaOteYCSrWB3nqppSIKTg==","MTime":1648359322820280300,"MetaSys":{"x-minio-internal-inline-data":"dHJ1ZQ=="},"MetaUsr":{"content-type":"application/octet-stream","etag":"ea210162eeb4adc31ff984cb744d6af6"},"PartASizes":[303],"PartETags":null,"PartNums":[1],"PartSizes":[303],"Size":303}}}]},
"/tmp/xl16/testbucket/hosts/xl.meta": {"Versions":[{"Header":{"Flags":0,"ModTime":"2022-03-27T07:40:36.271878663+02:00","Signature":"4ffe317b","Type":2,"VersionID":"afc0611a63e0403b8078bf21b9111a2b"},"Idx":0,"Metadata":{"DelObj":{"ID":"r8BhGmPgQDuAeL8huREaKw==","MTime":1648359636271878700,"MetaSys":{}},"Type":2}},{"Header":{"Flags":6,"ModTime":"2022-03-27T07:35:22.820280431+02:00","Signature":"ac150d34","Type":1,"VersionID":"12468eb5e6024ab581de7aa9a5220a4e"},"Idx":1,"Metadata":{"Type":1,"V2Obj":{"CSumAlgo":1,"DDir":"DvH10ieUSPmBLbKfCp3lVQ==","EcAlgo":1,"EcBSize":1048576,"EcDist":[5,6,7,8,9,10,11,12,13,14,15,16,1,2,3,4],"EcIndex":4,"EcM":12,"EcN":4,"ID":"EkaOteYCSrWB3nqppSIKTg==","MTime":1648359322820280300,"MetaSys":{"x-minio-internal-inline-data":"dHJ1ZQ=="},"MetaUsr":{"content-type":"application/octet-stream","etag":"ea210162eeb4adc31ff984cb744d6af6"},"PartASizes":[303],"PartETags":null,"PartNums":[1],"PartSizes":[303],"Size":303}}}]},
"/tmp/xl2/testbucket/hosts/xl.meta": {"Versions":[{"Header":{"Flags":0,"ModTime":"2022-03-27T07:40:36.271878663+02:00","Signature":"4ffe317b","Type":2,"VersionID":"afc0611a63e0403b8078bf21b9111a2b"},"Idx":0,"Metadata":{"DelObj":{"ID":"r8BhGmPgQDuAeL8huREaKw==","MTime":1648359636271878700,"MetaSys":{}},"Type":2}},{"Header":{"Flags":6,"ModTime":"2022-03-27T07:35:22.820280431+02:00","Signature":"ac150d34","Type":1,"VersionID":"12468eb5e6024ab581de7aa9a5220a4e"},"Idx":1,"Metadata":{"Type":1,"V2Obj":{"CSumAlgo":1,"DDir":"DvH10ieUSPmBLbKfCp3lVQ==","EcAlgo":1,"EcBSize":1048576,"EcDist":[5,6,7,8,9,10,11,12,13,14,15,16,1,2,3,4],"EcIndex":6,"EcM":12,"EcN":4,"ID":"EkaOteYCSrWB3nqppSIKTg==","MTime":1648359322820280300,"MetaSys":{"x-minio-internal-inline-data":"dHJ1ZQ=="},"MetaUsr":{"content-type":"application/octet-stream","etag":"ea210162eeb4adc31ff984cb744d6af6"},"PartASizes":[303],"PartETags":null,"PartNums":[1],"PartSizes":[303],"Size":303}}}]},
"/tmp/xl3/testbucket/hosts/xl.meta": {"Versions":[{"Header":{"Flags":0,"ModTime":"2022-03-27T07:40:36.271878663+02:00","Signature":"4ffe317b","Type":2,"VersionID":"afc0611a63e0403b8078bf21b9111a2b"},"Idx":0,"Metadata":{"DelObj":{"ID":"r8BhGmPgQDuAeL8huREaKw==","MTime":1648359636271878700,"MetaSys":{}},"Type":2}},{"Header":{"Flags":6,"ModTime":"2022-03-27T07:35:22.820280431+02:00","Signature":"ac150d34","Type":1,"VersionID":"12468eb5e6024ab581de7aa9a5220a4e"},"Idx":1,"Metadata":{"Type":1,"V2Obj":{"CSumAlgo":1,"DDir":"DvH10ieUSPmBLbKfCp3lVQ==","EcAlgo":1,"EcBSize":1048576,"EcDist":[5,6,7,8,9,10,11,12,13,14,15,16,1,2,3,4],"EcIndex":7,"EcM":12,"EcN":4,"ID":"EkaOteYCSrWB3nqppSIKTg==","MTime":1648359322820280300,"MetaSys":{"x-minio-internal-inline-data":"dHJ1ZQ=="},"MetaUsr":{"content-type":"application/octet-stream","etag":"ea210162eeb4adc31ff984cb744d6af6"},"PartASizes":[303],"PartETags":null,"PartNums":[1],"PartSizes":[303],"Size":303}}}]},
"/tmp/xl4/testbucket/hosts/xl.meta": {"Versions":[{"Header":{"Flags":0,"ModTime":"2022-03-27T07:40:36.271878663+02:00","Signature":"4ffe317b","Type":2,"VersionID":"afc0611a63e0403b8078bf21b9111a2b"},"Idx":0,"Metadata":{"DelObj":{"ID":"r8BhGmPgQDuAeL8huREaKw==","MTime":1648359636271878700,"MetaSys":{}},"Type":2}},{"Header":{"Flags":6,"ModTime":"2022-03-27T07:35:22.820280431+02:00","Signature":"ac150d34","Type":1,"VersionID":"12468eb5e6024ab581de7aa9a5220a4e"},"Idx":1,"Metadata":{"Type":1,"V2Obj":{"CSumAlgo":1,"DDir":"DvH10ieUSPmBLbKfCp3lVQ==","EcAlgo":1,"EcBSize":1048576,"EcDist":[5,6,7,8,9,10,11,12,13,14,15,16,1,2,3,4],"EcIndex":8,"EcM":12,"EcN":4,"ID":"EkaOteYCSrWB3nqppSIKTg==","MTime":1648359322820280300,"MetaSys":{"x-minio-internal-inline-data":"dHJ1ZQ=="},"MetaUsr":{"content-type":"application/octet-stream","etag":"ea210162eeb4adc31ff984cb744d6af6"},"PartASizes":[303],"PartETags":null,"PartNums":[1],"PartSizes":[303],"Size":303}}}]},
"/tmp/xl5/testbucket/hosts/xl.meta": {"Versions":[{"Header":{"Flags":0,"ModTime":"2022-03-27T07:40:36.271878663+02:00","Signature":"4ffe317b","Type":2,"VersionID":"afc0611a63e0403b8078bf21b9111a2b"},"Idx":0,"Metadata":{"DelObj":{"ID":"r8BhGmPgQDuAeL8huREaKw==","MTime":1648359636271878700,"MetaSys":{}},"Type":2}},{"Header":{"Flags":6,"ModTime":"2022-03-27T07:35:22.820280431+02:00","Signature":"ac150d34","Type":1,"VersionID":"12468eb5e6024ab581de7aa9a5220a4e"},"Idx":1,"Metadata":{"Type":1,"V2Obj":{"CSumAlgo":1,"DDir":"DvH10ieUSPmBLbKfCp3lVQ==","EcAlgo":1,"EcBSize":1048576,"EcDist":[5,6,7,8,9,10,11,12,13,14,15,16,1,2,3,4],"EcIndex":9,"EcM":12,"EcN":4,"ID":"EkaOteYCSrWB3nqppSIKTg==","MTime":1648359322820280300,"MetaSys":{"x-minio-internal-inline-data":"dHJ1ZQ=="},"MetaUsr":{"content-type":"application/octet-stream","etag":"ea210162eeb4adc31ff984cb744d6af6"},"PartASizes":[303],"PartETags":null,"PartNums":[1],"PartSizes":[303],"Size":303}}}]},
"/tmp/xl6/testbucket/hosts/xl.meta": {"Versions":[{"Header":{"Flags":0,"ModTime":"2022-03-27T07:40:36.271878663+02:00","Signature":"4ffe317b","Type":2,"VersionID":"afc0611a63e0403b8078bf21b9111a2b"},"Idx":0,"Metadata":{"DelObj":{"ID":"r8BhGmPgQDuAeL8huREaKw==","MTime":1648359636271878700,"MetaSys":{}},"Type":2}},{"Header":{"Flags":6,"ModTime":"2022-03-27T07:35:22.820280431+02:00","Signature":"ac150d34","Type":1,"VersionID":"12468eb5e6024ab581de7aa9a5220a4e"},"Idx":1,"Metadata":{"Type":1,"V2Obj":{"CSumAlgo":1,"DDir":"DvH10ieUSPmBLbKfCp3lVQ==","EcAlgo":1,"EcBSize":1048576,"EcDist":[5,6,7,8,9,10,11,12,13,14,15,16,1,2,3,4],"EcIndex":10,"EcM":12,"EcN":4,"ID":"EkaOteYCSrWB3nqppSIKTg==","MTime":1648359322820280300,"MetaSys":{"x-minio-internal-inline-data":"dHJ1ZQ=="},"MetaUsr":{"content-type":"application/octet-stream","etag":"ea210162eeb4adc31ff984cb744d6af6"},"PartASizes":[303],"PartETags":null,"PartNums":[1],"PartSizes":[303],"Size":303}}}]},
"/tmp/xl7/testbucket/hosts/xl.meta": {"Versions":[{"Header":{"Flags":0,"ModTime":"2022-03-27T07:40:36.271878663+02:00","Signature":"4ffe317b","Type":2,"VersionID":"afc0611a63e0403b8078bf21b9111a2b"},"Idx":0,"Metadata":{"DelObj":{"ID":"r8BhGmPgQDuAeL8huREaKw==","MTime":1648359636271878700,"MetaSys":{}},"Type":2}},{"Header":{"Flags":6,"ModTime":"2022-03-27T07:35:22.820280431+02:00","Signature":"ac150d34","Type":1,"VersionID":"12468eb5e6024ab581de7aa9a5220a4e"},"Idx":1,"Metadata":{"Type":1,"V2Obj":{"CSumAlgo":1,"DDir":"DvH10ieUSPmBLbKfCp3lVQ==","EcAlgo":1,"EcBSize":1048576,"EcDist":[5,6,7,8,9,10,11,12,13,14,15,16,1,2,3,4],"EcIndex":11,"EcM":12,"EcN":4,"ID":"EkaOteYCSrWB3nqppSIKTg==","MTime":1648359322820280300,"MetaSys":{"x-minio-internal-inline-data":"dHJ1ZQ=="},"MetaUsr":{"content-type":"application/octet-stream","etag":"ea210162eeb4adc31ff984cb744d6af6"},"PartASizes":[303],"PartETags":null,"PartNums":[1],"PartSizes":[303],"Size":303}}}]},
"/tmp/xl8/testbucket/hosts/xl.meta": {"Versions":[{"Header":{"Flags":0,"ModTime":"2022-03-27T07:40:36.271878663+02:00","Signature":"4ffe317b","Type":2,"VersionID":"afc0611a63e0403b8078bf21b9111a2b"},"Idx":0,"Metadata":{"DelObj":{"ID":"r8BhGmPgQDuAeL8huREaKw==","MTime":1648359636271878700,"MetaSys":{}},"Type":2}},{"Header":{"Flags":6,"ModTime":"2022-03-27T07:35:22.820280431+02:00","Signature":"ac150d34","Type":1,"VersionID":"12468eb5e6024ab581de7aa9a5220a4e"},"Idx":1,"Metadata":{"Type":1,"V2Obj":{"CSumAlgo":1,"DDir":"DvH10ieUSPmBLbKfCp3lVQ==","EcAlgo":1,"EcBSize":1048576,"EcDist":[5,6,7,8,9,10,11,12,13,14,15,16,1,2,3,4],"EcIndex":12,"EcM":12,"EcN":4,"ID":"EkaOteYCSrWB3nqppSIKTg==","MTime":1648359322820280300,"MetaSys":{"x-minio-internal-inline-data":"dHJ1ZQ=="},"MetaUsr":{"content-type":"application/octet-stream","etag":"ea210162eeb4adc31ff984cb744d6af6"},"PartASizes":[303],"PartETags":null,"PartNums":[1],"PartSizes":[303],"Size":303}}}]},
"/tmp/xl9/testbucket/hosts/xl.meta": {"Versions":[{"Header":{"Flags":0,"ModTime":"2022-03-27T07:40:36.271878663+02:00","Signature":"4ffe317b","Type":2,"VersionID":"afc0611a63e0403b8078bf21b9111a2b"},"Idx":0,"Metadata":{"DelObj":{"ID":"r8BhGmPgQDuAeL8huREaKw==","MTime":1648359636271878700,"MetaSys":{}},"Type":2}},{"Header":{"Flags":6,"ModTime":"2022-03-27T07:35:22.820280431+02:00","Signature":"ac150d34","Type":1,"VersionID":"12468eb5e6024ab581de7aa9a5220a4e"},"Idx":1,"Metadata":{"Type":1,"V2Obj":{"CSumAlgo":1,"DDir":"DvH10ieUSPmBLbKfCp3lVQ==","EcAlgo":1,"EcBSize":1048576,"EcDist":[5,6,7,8,9,10,11,12,13,14,15,16,1,2,3,4],"EcIndex":13,"EcM":12,"EcN":4,"ID":"EkaOteYCSrWB3nqppSIKTg==","MTime":1648359322820280300,"MetaSys":{"x-minio-internal-inline-data":"dHJ1ZQ=="},"MetaUsr":{"content-type":"application/octet-stream","etag":"ea210162eeb4adc31ff984cb744d6af6"},"PartASizes":[303],"PartETags":null,"PartNums":[1],"PartSizes":[303],"Size":303}}}]}
}
```
|
||
|
Klaus Post
|
15daa2e74a
|
tooling: Add xlmeta --combine switch that will combine inline data (#17488)
Will combine or write partial data of each version found in the inspect data. Example: ``` > xl-meta -export -combine inspect-data.1228fb52.zip (... metadata json...) } Attempting to combine version "994f1113-da94-4be1-8551-9dbc54b204bc". Read shard 1 Data shards 9 Parity 4 (994f1113-da94-4be1-8551-9dbc54b204bc/shard-01-of-13.data) Read shard 2 Data shards 9 Parity 4 (994f1113-da94-4be1-8551-9dbc54b204bc/shard-02-of-13.data) Read shard 3 Data shards 9 Parity 4 (994f1113-da94-4be1-8551-9dbc54b204bc/shard-03-of-13.data) Read shard 4 Data shards 9 Parity 4 (994f1113-da94-4be1-8551-9dbc54b204bc/shard-04-of-13.data) Read shard 6 Data shards 9 Parity 4 (994f1113-da94-4be1-8551-9dbc54b204bc/shard-06-of-13.data) Read shard 7 Data shards 9 Parity 4 (994f1113-da94-4be1-8551-9dbc54b204bc/shard-07-of-13.data) Read shard 8 Data shards 8 Parity 5 (994f1113-da94-4be1-8551-9dbc54b204bc/shard-08-of-13.data) Read shard 9 Data shards 8 Parity 5 (994f1113-da94-4be1-8551-9dbc54b204bc/shard-09-of-13.data) Read shard 10 Data shards 8 Parity 5 (994f1113-da94-4be1-8551-9dbc54b204bc/shard-10-of-13.data) Read shard 11 Data shards 8 Parity 5 (994f1113-da94-4be1-8551-9dbc54b204bc/shard-11-of-13.data) Read shard 13 Data shards 8 Parity 5 (994f1113-da94-4be1-8551-9dbc54b204bc/shard-13-of-13.data) Attempting to reconstruct using parity sets: * Setup: Data shards: 9 - Parity blocks: 6 Have 6 complete remapped data shards and 6 complete parity shards. Could NOT reconstruct: too few shards given * Setup: Data shards: 8 - Parity blocks: 5 Have 5 complete remapped data shards and 5 complete parity shards. Could reconstruct completely 0 bytes missing. Truncating 0 from the end. Wrote output to 994f1113-da94-4be1-8551-9dbc54b204bc.complete ``` So far only inline data, but no real reason that external data can't also be included with some handling of blocks. Supports only unencrypted data. |
||
|
Klaus Post
|
3eb2d086b2
|
Replace filepathx with fork (#16192) | ||
Poorna
|
dc92bb4646
|
xl-meta: update metadata version (#15958) | ||
|
Klaus Post
|
ff12080ff5
|
Remove deprecated io/ioutil (#15707) | ||
Harshavardhana
|
a60ac7ca17
|
fix: audit log to support object names in multipleObjectNames() handler (#14017) | ||
|
Klaus Post
|
518612492c
|
xl-meta: Add header titles (#13880)
Add type for headers and create custom marshal to make
it easier to read. Group headers and metadata.
Restore functionality that will read `xl.meta` in the current dir with no params.
Before:
```
{
"Headers": [
[
"8M04bTiYRDmEMQGeAsk1yg==",
1639150471630100400,
"rLD1Rw==",
1,
6
],
],
"Versions": [
{
"Type": 1,
"V2Obj": {
"CSumAlgo": 1,
"DDir": "oC1Xpg4tRfW03g8o8w7Bzg==",
"EcAlgo": 1,
"EcBSize": 1048576,
"EcDist": [
7,
8,
1,
2,
3,
4,
5,
6
],
"EcIndex": 1,
"EcM": 4,
"EcN": 4,
"ID": "8M04bTiYRDmEMQGeAsk1yg==",
"MTime": 1639150471630100400,
"MetaSys": {
"x-minio-internal-inline-data": "dHJ1ZQ=="
},
"MetaUsr": {
"content-type": "application/octet-stream",
"etag": "b8252c86fad2d8937300aa92b467a3aa"
},
"PartASizes": [
1000
],
"PartETags": null,
"PartNums": [
1
],
"PartSizes": [
1000
],
"Size": 1000
}
}
]
}
```
After:
```
{
"Versions": [
{
"Header": {
"Flags": 6,
"ModTime": "2021-12-10T16:34:31.6301004+01:00",
"Signature": "acb0f547",
"Type": 1,
"VersionID": "f0cd386d389844398431019e02c935ca"
},
"Idx": 0,
"Metadata": {
"Type": 1,
"V2Obj": {
"CSumAlgo": 1,
"DDir": "oC1Xpg4tRfW03g8o8w7Bzg==",
"EcAlgo": 1,
"EcBSize": 1048576,
"EcDist": [
7,
8,
1,
2,
3,
4,
5,
6
],
"EcIndex": 1,
"EcM": 4,
"EcN": 4,
"ID": "8M04bTiYRDmEMQGeAsk1yg==",
"MTime": 1639150471630100400,
"MetaSys": {
"x-minio-internal-inline-data": "dHJ1ZQ=="
},
"MetaUsr": {
"content-type": "application/octet-stream",
"etag": "b8252c86fad2d8937300aa92b467a3aa"
},
"PartASizes": [
1000
],
"PartETags": null,
"PartNums": [
1
],
"PartSizes": [
1000
],
"Size": 1000
}
}
}
]
}
```
|
||
|
Klaus Post
|
faf013ec84
|
Improve performance on multiple versions (#13573)
Existing:
```go
type xlMetaV2 struct {
Versions []xlMetaV2Version `json:"Versions" msg:"Versions"`
}
```
Serialized as regular MessagePack.
```go
//msgp:tuple xlMetaV2VersionHeader
type xlMetaV2VersionHeader struct {
VersionID [16]byte
ModTime int64
Type VersionType
Flags xlFlags
}
```
Serialize as streaming MessagePack, format:
```
int(headerVersion)
int(xlmetaVersion)
int(nVersions)
for each version {
binary blob, xlMetaV2VersionHeader, serialized
binary blob, xlMetaV2Version, serialized.
}
```
xlMetaV2VersionHeader is <= 30 bytes serialized. Deserialized struct
can easily be reused and does not contain pointers, so efficient as a
slice (single allocation)
This allows quickly parsing everything as slices of bytes (no copy).
Versions are always *saved* sorted by modTime, newest *first*.
No more need to sort on load.
* Allows checking if a version exists.
* Allows reading single version without unmarshal all.
* Allows reading latest version of type without unmarshal all.
* Allows reading latest version without unmarshal of all.
* Allows checking if the latest is deleteMarker by reading first entry.
* Allows adding/updating/deleting a version with only header deserialization.
* Reduces allocations on conversion to FileInfo(s).
|
||
|
Harshavardhana
|
82d73f387d
|
add tool to read healing.bin for debugging (#13650) | ||
|
Klaus Post
|
1642867136
|
Add documentation for debugging tools (#13484)
Move `xl-meta` so it can be installed out-of-repo with a single command. |