Commit Graph

18 Commits

Author SHA1 Message Date
Klaus Post
3e6dc02f8f
Add actual inline data to JSON output in xl-meta (#19958)
Add the inlined data as base64 encoded field and try to add a string version if feasible.

Example:

```
λ xl-meta -data xl.meta
{
  "8e03504e-1123-4957-b272-7bc53eda0d55": {
    "bitrot_valid": true,
    "bytes": 58,
    "data_base64": "Z29sYW5nLm9yZy94L3N5cyB2MC4xNS4wIC8=",
    "data_string": "golang.org/x/sys v0.15.0 /"
}
```

The string will have quotes, newlines escaped to produce valid JSON.

If content isn't valid utf8 or the encoding otherwise fails, only the base64 data will be added.

`-export` can still be used separately to extract the data as files (including bitrot).
2024-06-20 07:46:44 -07:00
Klaus Post
d67bccf861
Add xl-meta partial shard reconstruction (#19841)
Add partial shard reconstruction

* Add partial shard reconstruction
* Fix padding causing the last shard to be rejected
* Add md5 checks on single parts
* Move md5 verified to `verified/filename.ext`
* Move complete (without md5) to `complete/filename.ext.partno`

It's not pretty, but at least now the md5 gives some confidence it works correctly.
2024-05-31 07:49:23 -07:00
Klaus Post
bd4eeb4522
Fix flipped EcM, EcN in metadata header (#19831)
Since this is a tuple encoded field we can just flip the struct members.
2024-05-29 12:14:09 -07:00
Klaus Post
b92ac55250
Add multipart combination to xl-meta (#19780)
Add combination of multiple parts.

Parts will be reconstructed and saved separately and can manually be combined to the complete object.

Parts will be named `(version_id)-(filename).(partnum).(in)complete`.
2024-05-23 09:37:31 -07:00
Klaus Post
2c7bcee53f
Add cross-version remapped merges to xl-meta (#19765)
Adds `-xver` which can be used with `-export` and `-combine` to attempt to combine files across versions if data is suspected to be the same. Overlapping data is compared.

Bonus: Make `inspect` accept wildcards.
2024-05-19 08:31:54 -07:00
Klaus Post
fbfeb59658
xl-meta: Allow combining multiple unversioned objects (#19604)
When inspecting files like `.minio.sys/pool.bin` that may be present on multiple sets, use signature to separate them.

Also fixes null versions to actually be useful with `-export -combine`.
2024-04-24 10:56:22 -07:00
Klaus Post
5f774951b1
Store object EC in metadata header (#19534)
Keep the EC in header, so it can be retrieved easily for dynamic quorum calculations.

To not force a full metadata decode on every read the value will be 0/0 for data written in previous versions.

Size is expected to increase by 2 bytes per version, since all valid values can be represented with 1 byte each.

Example:
```
λ xl-meta xl.meta
{
  "Versions": [
    {
      "Header": {
        "EcM": 4,
        "EcN": 8,
        "Flags": 6,
        "ModTime": "2024-04-17T11:46:25.325613+02:00",
        "Signature": "0a409875",
        "Type": 1,
        "VersionID": "8e03504e11234957b2727bc53eda0d55"
      },
...
```

Not used for operations yet.
2024-04-19 09:43:43 -07:00
Klaus Post
a2f6252b2f
xl-meta: Add inline data bitrot check (#19240)
When using `-data` also perform a bitrot check on the data.

Example:

```
λ xl-meta -data net.zip
{
        "minio-1.com:9000/data/minio1/p40/44b6/44b612e9a7294856bd2b5fe6f6cdeb0d.pdf/xl.meta": {"null":{"bitrot_valid":true,"bytes":11710}},
        "minio-1.com:9000/data/minio2/p40/44b6/44b612e9a7294856bd2b5fe6f6cdeb0d.pdf/xl.meta": {"null":{"bitrot_valid":true,"bytes":11710}},
        "minio-1.com:9000/data/minio3/p40/44b6/44b612e9a7294856bd2b5fe6f6cdeb0d.pdf/xl.meta": {"null":{"bitrot_valid":true,"bytes":11710}},
        "minio-1.com:9000/data/minio4/p40/44b6/44b612e9a7294856bd2b5fe6f6cdeb0d.pdf/xl.meta": {"null":{"bitrot_valid":true,"bytes":11710}},
        "minio-2.com:9000/data/minio1/p40/44b6/44b612e9a7294856bd2b5fe6f6cdeb0d.pdf/xl.meta": {"null":{"bitrot_valid":true,"bytes":11710}},
        "minio-2.com:9000/data/minio2/p40/44b6/44b612e9a7294856bd2b5fe6f6cdeb0d.pdf/xl.meta": {"null":{"bitrot_valid":true,"bytes":11710}},
        "minio-2.com:9000/data/minio3/p40/44b6/44b612e9a7294856bd2b5fe6f6cdeb0d.pdf/xl.meta": {"null":{"bitrot_valid":true,"bytes":11710}},
        "minio-2.com:9000/data/minio4/p40/44b6/44b612e9a7294856bd2b5fe6f6cdeb0d.pdf/xl.meta": {"null":{"bitrot_valid":true,"bytes":11710}},
        "minio-3.com:9000/data/minio1/p40/44b6/44b612e9a7294856bd2b5fe6f6cdeb0d.pdf/xl.meta": {"null":{"bitrot_valid":true,"bytes":11710}},
        "minio-3.com:9000/data/minio2/p40/44b6/44b612e9a7294856bd2b5fe6f6cdeb0d.pdf/xl.meta": {"null":{"bitrot_valid":true,"bytes":11710}},
        "minio-3.com:9000/data/minio3/p40/44b6/44b612e9a7294856bd2b5fe6f6cdeb0d.pdf/xl.meta": {"null":{"bitrot_valid":true,"bytes":11710}},
        "minio-3.com:9000/data/minio4/p40/44b6/44b612e9a7294856bd2b5fe6f6cdeb0d.pdf/xl.meta": {"null":{"bitrot_valid":true,"bytes":11710}},
        "minio-4.com:9000/data/minio1/p40/44b6/44b612e9a7294856bd2b5fe6f6cdeb0d.pdf/xl.meta": {"null":{"bitrot_valid":true,"bytes":11710}},
        "minio-4.com:9000/data/minio2/p40/44b6/44b612e9a7294856bd2b5fe6f6cdeb0d.pdf/xl.meta": {"null":{"bitrot_valid":true,"bytes":11710}},
        "minio-4.com:9000/data/minio3/p40/44b6/44b612e9a7294856bd2b5fe6f6cdeb0d.pdf/xl.meta": {"null":{"bitrot_valid":true,"bytes":11710}},
        "minio-4.com:9000/data/minio4/p40/44b6/44b612e9a7294856bd2b5fe6f6cdeb0d.pdf/xl.meta": {"null":{"bitrot_valid":true,"bytes":11710}}
}
```
2024-03-11 10:57:11 -07:00
Klaus Post
ca258c04cb
xl-meta: Clean output (#18794)
* Sort files.
* Sort maps.
* Remove annoying stray ','
* Hide -ndjson - doesn't do what it claims.

Sample output:

```
{
	"/tmp/xl1/testbucket/hosts/xl.meta": {"Versions":[{"Header":{"Flags":6,"ModTime":"2022-03-27T07:35:22.820280431+02:00","Signature":"ac150d34","Type":1,"VersionID":"12468eb5e6024ab581de7aa9a5220a4e"},"Idx":0,"Metadata":{"Type":1,"V2Obj":{"CSumAlgo":1,"DDir":"DvH10ieUSPmBLbKfCp3lVQ==","EcAlgo":1,"EcBSize":1048576,"EcDist":[5,6,7,8,9,10,11,12,13,14,15,16,1,2,3,4],"EcIndex":5,"EcM":12,"EcN":4,"ID":"EkaOteYCSrWB3nqppSIKTg==","MTime":1648359322820280300,"MetaSys":{"x-minio-internal-inline-data":"dHJ1ZQ=="},"MetaUsr":{"content-type":"application/octet-stream","etag":"ea210162eeb4adc31ff984cb744d6af6"},"PartASizes":[303],"PartETags":null,"PartNums":[1],"PartSizes":[303],"Size":303}}}]},
	"/tmp/xl10/testbucket/hosts/xl.meta": {"Versions":[{"Header":{"Flags":0,"ModTime":"2022-03-27T07:40:36.271878663+02:00","Signature":"4ffe317b","Type":2,"VersionID":"afc0611a63e0403b8078bf21b9111a2b"},"Idx":0,"Metadata":{"DelObj":{"ID":"r8BhGmPgQDuAeL8huREaKw==","MTime":1648359636271878700,"MetaSys":{}},"Type":2}},{"Header":{"Flags":6,"ModTime":"2022-03-27T07:35:22.820280431+02:00","Signature":"ac150d34","Type":1,"VersionID":"12468eb5e6024ab581de7aa9a5220a4e"},"Idx":1,"Metadata":{"Type":1,"V2Obj":{"CSumAlgo":1,"DDir":"DvH10ieUSPmBLbKfCp3lVQ==","EcAlgo":1,"EcBSize":1048576,"EcDist":[5,6,7,8,9,10,11,12,13,14,15,16,1,2,3,4],"EcIndex":14,"EcM":12,"EcN":4,"ID":"EkaOteYCSrWB3nqppSIKTg==","MTime":1648359322820280300,"MetaSys":{"x-minio-internal-inline-data":"dHJ1ZQ=="},"MetaUsr":{"content-type":"application/octet-stream","etag":"ea210162eeb4adc31ff984cb744d6af6"},"PartASizes":[303],"PartETags":null,"PartNums":[1],"PartSizes":[303],"Size":303}}}]},
	"/tmp/xl11/testbucket/hosts/xl.meta": {"Versions":[{"Header":{"Flags":6,"ModTime":"2022-03-27T07:35:22.820280431+02:00","Signature":"ac150d34","Type":1,"VersionID":"12468eb5e6024ab581de7aa9a5220a4e"},"Idx":0,"Metadata":{"Type":1,"V2Obj":{"CSumAlgo":1,"DDir":"DvH10ieUSPmBLbKfCp3lVQ==","EcAlgo":1,"EcBSize":1048576,"EcDist":[5,6,7,8,9,10,11,12,13,14,15,16,1,2,3,4],"EcIndex":15,"EcM":12,"EcN":4,"ID":"EkaOteYCSrWB3nqppSIKTg==","MTime":1648359322820280300,"MetaSys":{"x-minio-internal-inline-data":"dHJ1ZQ=="},"MetaUsr":{"content-type":"application/octet-stream","etag":"ea210162eeb4adc31ff984cb744d6af6"},"PartASizes":[303],"PartETags":null,"PartNums":[1],"PartSizes":[303],"Size":303}}}]},
	"/tmp/xl12/testbucket/hosts/xl.meta": {"Versions":[{"Header":{"Flags":0,"ModTime":"2022-03-27T07:40:36.271878663+02:00","Signature":"4ffe317b","Type":2,"VersionID":"afc0611a63e0403b8078bf21b9111a2b"},"Idx":0,"Metadata":{"DelObj":{"ID":"r8BhGmPgQDuAeL8huREaKw==","MTime":1648359636271878700,"MetaSys":{}},"Type":2}},{"Header":{"Flags":6,"ModTime":"2022-03-27T07:35:22.820280431+02:00","Signature":"ac150d34","Type":1,"VersionID":"12468eb5e6024ab581de7aa9a5220a4e"},"Idx":1,"Metadata":{"Type":1,"V2Obj":{"CSumAlgo":1,"DDir":"DvH10ieUSPmBLbKfCp3lVQ==","EcAlgo":1,"EcBSize":1048576,"EcDist":[5,6,7,8,9,10,11,12,13,14,15,16,1,2,3,4],"EcIndex":16,"EcM":12,"EcN":4,"ID":"EkaOteYCSrWB3nqppSIKTg==","MTime":1648359322820280300,"MetaSys":{"x-minio-internal-inline-data":"dHJ1ZQ=="},"MetaUsr":{"content-type":"application/octet-stream","etag":"ea210162eeb4adc31ff984cb744d6af6"},"PartASizes":[303],"PartETags":null,"PartNums":[1],"PartSizes":[303],"Size":303}}}]},
	"/tmp/xl13/testbucket/hosts/xl.meta": {"Versions":[{"Header":{"Flags":0,"ModTime":"2022-03-27T07:40:36.271878663+02:00","Signature":"4ffe317b","Type":2,"VersionID":"afc0611a63e0403b8078bf21b9111a2b"},"Idx":0,"Metadata":{"DelObj":{"ID":"r8BhGmPgQDuAeL8huREaKw==","MTime":1648359636271878700,"MetaSys":{}},"Type":2}},{"Header":{"Flags":6,"ModTime":"2022-03-27T07:35:22.820280431+02:00","Signature":"ac150d34","Type":1,"VersionID":"12468eb5e6024ab581de7aa9a5220a4e"},"Idx":1,"Metadata":{"Type":1,"V2Obj":{"CSumAlgo":1,"DDir":"DvH10ieUSPmBLbKfCp3lVQ==","EcAlgo":1,"EcBSize":1048576,"EcDist":[5,6,7,8,9,10,11,12,13,14,15,16,1,2,3,4],"EcIndex":1,"EcM":12,"EcN":4,"ID":"EkaOteYCSrWB3nqppSIKTg==","MTime":1648359322820280300,"MetaSys":{"x-minio-internal-inline-data":"dHJ1ZQ=="},"MetaUsr":{"content-type":"application/octet-stream","etag":"ea210162eeb4adc31ff984cb744d6af6"},"PartASizes":[303],"PartETags":null,"PartNums":[1],"PartSizes":[303],"Size":303}}}]},
	"/tmp/xl14/testbucket/hosts/xl.meta": {"Versions":[{"Header":{"Flags":0,"ModTime":"2022-03-27T07:40:36.271878663+02:00","Signature":"4ffe317b","Type":2,"VersionID":"afc0611a63e0403b8078bf21b9111a2b"},"Idx":0,"Metadata":{"DelObj":{"ID":"r8BhGmPgQDuAeL8huREaKw==","MTime":1648359636271878700,"MetaSys":{}},"Type":2}},{"Header":{"Flags":6,"ModTime":"2022-03-27T07:35:22.820280431+02:00","Signature":"ac150d34","Type":1,"VersionID":"12468eb5e6024ab581de7aa9a5220a4e"},"Idx":1,"Metadata":{"Type":1,"V2Obj":{"CSumAlgo":1,"DDir":"DvH10ieUSPmBLbKfCp3lVQ==","EcAlgo":1,"EcBSize":1048576,"EcDist":[5,6,7,8,9,10,11,12,13,14,15,16,1,2,3,4],"EcIndex":2,"EcM":12,"EcN":4,"ID":"EkaOteYCSrWB3nqppSIKTg==","MTime":1648359322820280300,"MetaSys":{"x-minio-internal-inline-data":"dHJ1ZQ=="},"MetaUsr":{"content-type":"application/octet-stream","etag":"ea210162eeb4adc31ff984cb744d6af6"},"PartASizes":[303],"PartETags":null,"PartNums":[1],"PartSizes":[303],"Size":303}}}]},
	"/tmp/xl15/testbucket/hosts/xl.meta": {"Versions":[{"Header":{"Flags":0,"ModTime":"2022-03-27T07:40:36.271878663+02:00","Signature":"4ffe317b","Type":2,"VersionID":"afc0611a63e0403b8078bf21b9111a2b"},"Idx":0,"Metadata":{"DelObj":{"ID":"r8BhGmPgQDuAeL8huREaKw==","MTime":1648359636271878700,"MetaSys":{}},"Type":2}},{"Header":{"Flags":6,"ModTime":"2022-03-27T07:35:22.820280431+02:00","Signature":"ac150d34","Type":1,"VersionID":"12468eb5e6024ab581de7aa9a5220a4e"},"Idx":1,"Metadata":{"Type":1,"V2Obj":{"CSumAlgo":1,"DDir":"DvH10ieUSPmBLbKfCp3lVQ==","EcAlgo":1,"EcBSize":1048576,"EcDist":[5,6,7,8,9,10,11,12,13,14,15,16,1,2,3,4],"EcIndex":3,"EcM":12,"EcN":4,"ID":"EkaOteYCSrWB3nqppSIKTg==","MTime":1648359322820280300,"MetaSys":{"x-minio-internal-inline-data":"dHJ1ZQ=="},"MetaUsr":{"content-type":"application/octet-stream","etag":"ea210162eeb4adc31ff984cb744d6af6"},"PartASizes":[303],"PartETags":null,"PartNums":[1],"PartSizes":[303],"Size":303}}}]},
	"/tmp/xl16/testbucket/hosts/xl.meta": {"Versions":[{"Header":{"Flags":0,"ModTime":"2022-03-27T07:40:36.271878663+02:00","Signature":"4ffe317b","Type":2,"VersionID":"afc0611a63e0403b8078bf21b9111a2b"},"Idx":0,"Metadata":{"DelObj":{"ID":"r8BhGmPgQDuAeL8huREaKw==","MTime":1648359636271878700,"MetaSys":{}},"Type":2}},{"Header":{"Flags":6,"ModTime":"2022-03-27T07:35:22.820280431+02:00","Signature":"ac150d34","Type":1,"VersionID":"12468eb5e6024ab581de7aa9a5220a4e"},"Idx":1,"Metadata":{"Type":1,"V2Obj":{"CSumAlgo":1,"DDir":"DvH10ieUSPmBLbKfCp3lVQ==","EcAlgo":1,"EcBSize":1048576,"EcDist":[5,6,7,8,9,10,11,12,13,14,15,16,1,2,3,4],"EcIndex":4,"EcM":12,"EcN":4,"ID":"EkaOteYCSrWB3nqppSIKTg==","MTime":1648359322820280300,"MetaSys":{"x-minio-internal-inline-data":"dHJ1ZQ=="},"MetaUsr":{"content-type":"application/octet-stream","etag":"ea210162eeb4adc31ff984cb744d6af6"},"PartASizes":[303],"PartETags":null,"PartNums":[1],"PartSizes":[303],"Size":303}}}]},
	"/tmp/xl2/testbucket/hosts/xl.meta": {"Versions":[{"Header":{"Flags":0,"ModTime":"2022-03-27T07:40:36.271878663+02:00","Signature":"4ffe317b","Type":2,"VersionID":"afc0611a63e0403b8078bf21b9111a2b"},"Idx":0,"Metadata":{"DelObj":{"ID":"r8BhGmPgQDuAeL8huREaKw==","MTime":1648359636271878700,"MetaSys":{}},"Type":2}},{"Header":{"Flags":6,"ModTime":"2022-03-27T07:35:22.820280431+02:00","Signature":"ac150d34","Type":1,"VersionID":"12468eb5e6024ab581de7aa9a5220a4e"},"Idx":1,"Metadata":{"Type":1,"V2Obj":{"CSumAlgo":1,"DDir":"DvH10ieUSPmBLbKfCp3lVQ==","EcAlgo":1,"EcBSize":1048576,"EcDist":[5,6,7,8,9,10,11,12,13,14,15,16,1,2,3,4],"EcIndex":6,"EcM":12,"EcN":4,"ID":"EkaOteYCSrWB3nqppSIKTg==","MTime":1648359322820280300,"MetaSys":{"x-minio-internal-inline-data":"dHJ1ZQ=="},"MetaUsr":{"content-type":"application/octet-stream","etag":"ea210162eeb4adc31ff984cb744d6af6"},"PartASizes":[303],"PartETags":null,"PartNums":[1],"PartSizes":[303],"Size":303}}}]},
	"/tmp/xl3/testbucket/hosts/xl.meta": {"Versions":[{"Header":{"Flags":0,"ModTime":"2022-03-27T07:40:36.271878663+02:00","Signature":"4ffe317b","Type":2,"VersionID":"afc0611a63e0403b8078bf21b9111a2b"},"Idx":0,"Metadata":{"DelObj":{"ID":"r8BhGmPgQDuAeL8huREaKw==","MTime":1648359636271878700,"MetaSys":{}},"Type":2}},{"Header":{"Flags":6,"ModTime":"2022-03-27T07:35:22.820280431+02:00","Signature":"ac150d34","Type":1,"VersionID":"12468eb5e6024ab581de7aa9a5220a4e"},"Idx":1,"Metadata":{"Type":1,"V2Obj":{"CSumAlgo":1,"DDir":"DvH10ieUSPmBLbKfCp3lVQ==","EcAlgo":1,"EcBSize":1048576,"EcDist":[5,6,7,8,9,10,11,12,13,14,15,16,1,2,3,4],"EcIndex":7,"EcM":12,"EcN":4,"ID":"EkaOteYCSrWB3nqppSIKTg==","MTime":1648359322820280300,"MetaSys":{"x-minio-internal-inline-data":"dHJ1ZQ=="},"MetaUsr":{"content-type":"application/octet-stream","etag":"ea210162eeb4adc31ff984cb744d6af6"},"PartASizes":[303],"PartETags":null,"PartNums":[1],"PartSizes":[303],"Size":303}}}]},
	"/tmp/xl4/testbucket/hosts/xl.meta": {"Versions":[{"Header":{"Flags":0,"ModTime":"2022-03-27T07:40:36.271878663+02:00","Signature":"4ffe317b","Type":2,"VersionID":"afc0611a63e0403b8078bf21b9111a2b"},"Idx":0,"Metadata":{"DelObj":{"ID":"r8BhGmPgQDuAeL8huREaKw==","MTime":1648359636271878700,"MetaSys":{}},"Type":2}},{"Header":{"Flags":6,"ModTime":"2022-03-27T07:35:22.820280431+02:00","Signature":"ac150d34","Type":1,"VersionID":"12468eb5e6024ab581de7aa9a5220a4e"},"Idx":1,"Metadata":{"Type":1,"V2Obj":{"CSumAlgo":1,"DDir":"DvH10ieUSPmBLbKfCp3lVQ==","EcAlgo":1,"EcBSize":1048576,"EcDist":[5,6,7,8,9,10,11,12,13,14,15,16,1,2,3,4],"EcIndex":8,"EcM":12,"EcN":4,"ID":"EkaOteYCSrWB3nqppSIKTg==","MTime":1648359322820280300,"MetaSys":{"x-minio-internal-inline-data":"dHJ1ZQ=="},"MetaUsr":{"content-type":"application/octet-stream","etag":"ea210162eeb4adc31ff984cb744d6af6"},"PartASizes":[303],"PartETags":null,"PartNums":[1],"PartSizes":[303],"Size":303}}}]},
	"/tmp/xl5/testbucket/hosts/xl.meta": {"Versions":[{"Header":{"Flags":0,"ModTime":"2022-03-27T07:40:36.271878663+02:00","Signature":"4ffe317b","Type":2,"VersionID":"afc0611a63e0403b8078bf21b9111a2b"},"Idx":0,"Metadata":{"DelObj":{"ID":"r8BhGmPgQDuAeL8huREaKw==","MTime":1648359636271878700,"MetaSys":{}},"Type":2}},{"Header":{"Flags":6,"ModTime":"2022-03-27T07:35:22.820280431+02:00","Signature":"ac150d34","Type":1,"VersionID":"12468eb5e6024ab581de7aa9a5220a4e"},"Idx":1,"Metadata":{"Type":1,"V2Obj":{"CSumAlgo":1,"DDir":"DvH10ieUSPmBLbKfCp3lVQ==","EcAlgo":1,"EcBSize":1048576,"EcDist":[5,6,7,8,9,10,11,12,13,14,15,16,1,2,3,4],"EcIndex":9,"EcM":12,"EcN":4,"ID":"EkaOteYCSrWB3nqppSIKTg==","MTime":1648359322820280300,"MetaSys":{"x-minio-internal-inline-data":"dHJ1ZQ=="},"MetaUsr":{"content-type":"application/octet-stream","etag":"ea210162eeb4adc31ff984cb744d6af6"},"PartASizes":[303],"PartETags":null,"PartNums":[1],"PartSizes":[303],"Size":303}}}]},
	"/tmp/xl6/testbucket/hosts/xl.meta": {"Versions":[{"Header":{"Flags":0,"ModTime":"2022-03-27T07:40:36.271878663+02:00","Signature":"4ffe317b","Type":2,"VersionID":"afc0611a63e0403b8078bf21b9111a2b"},"Idx":0,"Metadata":{"DelObj":{"ID":"r8BhGmPgQDuAeL8huREaKw==","MTime":1648359636271878700,"MetaSys":{}},"Type":2}},{"Header":{"Flags":6,"ModTime":"2022-03-27T07:35:22.820280431+02:00","Signature":"ac150d34","Type":1,"VersionID":"12468eb5e6024ab581de7aa9a5220a4e"},"Idx":1,"Metadata":{"Type":1,"V2Obj":{"CSumAlgo":1,"DDir":"DvH10ieUSPmBLbKfCp3lVQ==","EcAlgo":1,"EcBSize":1048576,"EcDist":[5,6,7,8,9,10,11,12,13,14,15,16,1,2,3,4],"EcIndex":10,"EcM":12,"EcN":4,"ID":"EkaOteYCSrWB3nqppSIKTg==","MTime":1648359322820280300,"MetaSys":{"x-minio-internal-inline-data":"dHJ1ZQ=="},"MetaUsr":{"content-type":"application/octet-stream","etag":"ea210162eeb4adc31ff984cb744d6af6"},"PartASizes":[303],"PartETags":null,"PartNums":[1],"PartSizes":[303],"Size":303}}}]},
	"/tmp/xl7/testbucket/hosts/xl.meta": {"Versions":[{"Header":{"Flags":0,"ModTime":"2022-03-27T07:40:36.271878663+02:00","Signature":"4ffe317b","Type":2,"VersionID":"afc0611a63e0403b8078bf21b9111a2b"},"Idx":0,"Metadata":{"DelObj":{"ID":"r8BhGmPgQDuAeL8huREaKw==","MTime":1648359636271878700,"MetaSys":{}},"Type":2}},{"Header":{"Flags":6,"ModTime":"2022-03-27T07:35:22.820280431+02:00","Signature":"ac150d34","Type":1,"VersionID":"12468eb5e6024ab581de7aa9a5220a4e"},"Idx":1,"Metadata":{"Type":1,"V2Obj":{"CSumAlgo":1,"DDir":"DvH10ieUSPmBLbKfCp3lVQ==","EcAlgo":1,"EcBSize":1048576,"EcDist":[5,6,7,8,9,10,11,12,13,14,15,16,1,2,3,4],"EcIndex":11,"EcM":12,"EcN":4,"ID":"EkaOteYCSrWB3nqppSIKTg==","MTime":1648359322820280300,"MetaSys":{"x-minio-internal-inline-data":"dHJ1ZQ=="},"MetaUsr":{"content-type":"application/octet-stream","etag":"ea210162eeb4adc31ff984cb744d6af6"},"PartASizes":[303],"PartETags":null,"PartNums":[1],"PartSizes":[303],"Size":303}}}]},
	"/tmp/xl8/testbucket/hosts/xl.meta": {"Versions":[{"Header":{"Flags":0,"ModTime":"2022-03-27T07:40:36.271878663+02:00","Signature":"4ffe317b","Type":2,"VersionID":"afc0611a63e0403b8078bf21b9111a2b"},"Idx":0,"Metadata":{"DelObj":{"ID":"r8BhGmPgQDuAeL8huREaKw==","MTime":1648359636271878700,"MetaSys":{}},"Type":2}},{"Header":{"Flags":6,"ModTime":"2022-03-27T07:35:22.820280431+02:00","Signature":"ac150d34","Type":1,"VersionID":"12468eb5e6024ab581de7aa9a5220a4e"},"Idx":1,"Metadata":{"Type":1,"V2Obj":{"CSumAlgo":1,"DDir":"DvH10ieUSPmBLbKfCp3lVQ==","EcAlgo":1,"EcBSize":1048576,"EcDist":[5,6,7,8,9,10,11,12,13,14,15,16,1,2,3,4],"EcIndex":12,"EcM":12,"EcN":4,"ID":"EkaOteYCSrWB3nqppSIKTg==","MTime":1648359322820280300,"MetaSys":{"x-minio-internal-inline-data":"dHJ1ZQ=="},"MetaUsr":{"content-type":"application/octet-stream","etag":"ea210162eeb4adc31ff984cb744d6af6"},"PartASizes":[303],"PartETags":null,"PartNums":[1],"PartSizes":[303],"Size":303}}}]},
	"/tmp/xl9/testbucket/hosts/xl.meta": {"Versions":[{"Header":{"Flags":0,"ModTime":"2022-03-27T07:40:36.271878663+02:00","Signature":"4ffe317b","Type":2,"VersionID":"afc0611a63e0403b8078bf21b9111a2b"},"Idx":0,"Metadata":{"DelObj":{"ID":"r8BhGmPgQDuAeL8huREaKw==","MTime":1648359636271878700,"MetaSys":{}},"Type":2}},{"Header":{"Flags":6,"ModTime":"2022-03-27T07:35:22.820280431+02:00","Signature":"ac150d34","Type":1,"VersionID":"12468eb5e6024ab581de7aa9a5220a4e"},"Idx":1,"Metadata":{"Type":1,"V2Obj":{"CSumAlgo":1,"DDir":"DvH10ieUSPmBLbKfCp3lVQ==","EcAlgo":1,"EcBSize":1048576,"EcDist":[5,6,7,8,9,10,11,12,13,14,15,16,1,2,3,4],"EcIndex":13,"EcM":12,"EcN":4,"ID":"EkaOteYCSrWB3nqppSIKTg==","MTime":1648359322820280300,"MetaSys":{"x-minio-internal-inline-data":"dHJ1ZQ=="},"MetaUsr":{"content-type":"application/octet-stream","etag":"ea210162eeb4adc31ff984cb744d6af6"},"PartASizes":[303],"PartETags":null,"PartNums":[1],"PartSizes":[303],"Size":303}}}]}
}
```
2024-01-16 08:07:38 -08:00
Klaus Post
15daa2e74a
tooling: Add xlmeta --combine switch that will combine inline data (#17488)
Will combine or write partial data of each version found in the inspect data.

Example:

```
> xl-meta -export -combine inspect-data.1228fb52.zip

(... metadata json...)
}
Attempting to combine version "994f1113-da94-4be1-8551-9dbc54b204bc".
Read shard 1 Data shards 9 Parity 4 (994f1113-da94-4be1-8551-9dbc54b204bc/shard-01-of-13.data)
Read shard 2 Data shards 9 Parity 4 (994f1113-da94-4be1-8551-9dbc54b204bc/shard-02-of-13.data)
Read shard 3 Data shards 9 Parity 4 (994f1113-da94-4be1-8551-9dbc54b204bc/shard-03-of-13.data)
Read shard 4 Data shards 9 Parity 4 (994f1113-da94-4be1-8551-9dbc54b204bc/shard-04-of-13.data)
Read shard 6 Data shards 9 Parity 4 (994f1113-da94-4be1-8551-9dbc54b204bc/shard-06-of-13.data)
Read shard 7 Data shards 9 Parity 4 (994f1113-da94-4be1-8551-9dbc54b204bc/shard-07-of-13.data)
Read shard 8 Data shards 8 Parity 5 (994f1113-da94-4be1-8551-9dbc54b204bc/shard-08-of-13.data)
Read shard 9 Data shards 8 Parity 5 (994f1113-da94-4be1-8551-9dbc54b204bc/shard-09-of-13.data)
Read shard 10 Data shards 8 Parity 5 (994f1113-da94-4be1-8551-9dbc54b204bc/shard-10-of-13.data)
Read shard 11 Data shards 8 Parity 5 (994f1113-da94-4be1-8551-9dbc54b204bc/shard-11-of-13.data)
Read shard 13 Data shards 8 Parity 5 (994f1113-da94-4be1-8551-9dbc54b204bc/shard-13-of-13.data)
Attempting to reconstruct using parity sets:
* Setup: Data shards: 9 - Parity blocks: 6
Have 6 complete remapped data shards and 6 complete parity shards. Could NOT reconstruct: too few shards given
* Setup: Data shards: 8 - Parity blocks: 5
Have 5 complete remapped data shards and 5 complete parity shards. Could reconstruct completely
0 bytes missing. Truncating 0 from the end.
Wrote output to 994f1113-da94-4be1-8551-9dbc54b204bc.complete
```

So far only inline data, but no real reason that external data can't also be included with some handling of blocks.

Supports only unencrypted data.
2023-06-22 12:41:24 -07:00
Klaus Post
3eb2d086b2
Replace filepathx with fork (#16192) 2022-12-08 10:42:44 -08:00
Poorna
dc92bb4646
xl-meta: update metadata version (#15958) 2022-10-28 02:48:43 -07:00
Klaus Post
ff12080ff5
Remove deprecated io/ioutil (#15707) 2022-09-19 11:05:16 -07:00
Harshavardhana
a60ac7ca17
fix: audit log to support object names in multipleObjectNames() handler (#14017) 2022-01-03 01:28:52 -08:00
Klaus Post
518612492c
xl-meta: Add header titles (#13880)
Add type for headers and create custom marshal to make 
it easier to read. Group headers and metadata.

Restore functionality that will read `xl.meta` in the current dir with no params.

Before:
```
{
  "Headers": [
    [
      "8M04bTiYRDmEMQGeAsk1yg==",
      1639150471630100400,
      "rLD1Rw==",
      1,
      6
    ],
  ],
    "Versions": [
    {
      "Type": 1,
      "V2Obj": {
        "CSumAlgo": 1,
        "DDir": "oC1Xpg4tRfW03g8o8w7Bzg==",
        "EcAlgo": 1,
        "EcBSize": 1048576,
        "EcDist": [
          7,
          8,
          1,
          2,
          3,
          4,
          5,
          6
        ],
        "EcIndex": 1,
        "EcM": 4,
        "EcN": 4,
        "ID": "8M04bTiYRDmEMQGeAsk1yg==",
        "MTime": 1639150471630100400,
        "MetaSys": {
          "x-minio-internal-inline-data": "dHJ1ZQ=="
        },
        "MetaUsr": {
          "content-type": "application/octet-stream",
          "etag": "b8252c86fad2d8937300aa92b467a3aa"
        },
        "PartASizes": [
          1000
        ],
        "PartETags": null,
        "PartNums": [
          1
        ],
        "PartSizes": [
          1000
        ],
        "Size": 1000
      }
    }
  ]
}
```

After:
```
{
  "Versions": [
    {
      "Header": {
        "Flags": 6,
        "ModTime": "2021-12-10T16:34:31.6301004+01:00",
        "Signature": "acb0f547",
        "Type": 1,
        "VersionID": "f0cd386d389844398431019e02c935ca"
      },
      "Idx": 0,
      "Metadata": {
        "Type": 1,
        "V2Obj": {
          "CSumAlgo": 1,
          "DDir": "oC1Xpg4tRfW03g8o8w7Bzg==",
          "EcAlgo": 1,
          "EcBSize": 1048576,
          "EcDist": [
            7,
            8,
            1,
            2,
            3,
            4,
            5,
            6
          ],
          "EcIndex": 1,
          "EcM": 4,
          "EcN": 4,
          "ID": "8M04bTiYRDmEMQGeAsk1yg==",
          "MTime": 1639150471630100400,
          "MetaSys": {
            "x-minio-internal-inline-data": "dHJ1ZQ=="
          },
          "MetaUsr": {
            "content-type": "application/octet-stream",
            "etag": "b8252c86fad2d8937300aa92b467a3aa"
          },
          "PartASizes": [
            1000
          ],
          "PartETags": null,
          "PartNums": [
            1
          ],
          "PartSizes": [
            1000
          ],
          "Size": 1000
        }
      }
    }
  ]
}
```
2021-12-10 15:03:25 -08:00
Klaus Post
faf013ec84
Improve performance on multiple versions (#13573)
Existing:

```go
type xlMetaV2 struct {
    Versions []xlMetaV2Version `json:"Versions" msg:"Versions"`
}
```

Serialized as regular MessagePack.

```go
//msgp:tuple xlMetaV2VersionHeader
type xlMetaV2VersionHeader struct {
	VersionID [16]byte
	ModTime   int64
	Type      VersionType
	Flags     xlFlags
}
```

Serialize as streaming MessagePack, format:

```
int(headerVersion)
int(xlmetaVersion)
int(nVersions)
for each version {
    binary blob, xlMetaV2VersionHeader, serialized
    binary blob, xlMetaV2Version, serialized.
}
```

xlMetaV2VersionHeader is <= 30 bytes serialized. Deserialized struct 
can easily be reused and does not contain pointers, so efficient as a 
slice (single allocation)

This allows quickly parsing everything as slices of bytes (no copy).

Versions are always *saved* sorted by modTime, newest *first*. 
No more need to sort on load.

* Allows checking if a version exists.
* Allows reading single version without unmarshal all.
* Allows reading latest version of type without unmarshal all.
* Allows reading latest version without unmarshal of all.
* Allows checking if the latest is deleteMarker by reading first entry.
* Allows adding/updating/deleting a version with only header deserialization.
* Reduces allocations on conversion to FileInfo(s).
2021-11-18 12:15:22 -08:00
Harshavardhana
82d73f387d
add tool to read healing.bin for debugging (#13650) 2021-11-12 16:18:53 -08:00
Klaus Post
1642867136
Add documentation for debugging tools (#13484)
Move `xl-meta` so it can be installed out-of-repo with a single command.
2021-10-20 10:12:46 -07:00