MyFavMirrors
/
minio
mirror of https://github.com/minio/minio.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
11,086 Commits 5 Branches 503 Tags 186 MiB
Commit Graph

9 Commits

Author SHA1 Message Date
Klaus Post 15daa2e74a
tooling: Add xlmeta --combine switch that will combine inline data (#17488) 
Will combine or write partial data of each version found in the inspect data.

Example:

```
> xl-meta -export -combine inspect-data.1228fb52.zip

(... metadata json...)
}
Attempting to combine version "994f1113-da94-4be1-8551-9dbc54b204bc".
Read shard 1 Data shards 9 Parity 4 (994f1113-da94-4be1-8551-9dbc54b204bc/shard-01-of-13.data)
Read shard 2 Data shards 9 Parity 4 (994f1113-da94-4be1-8551-9dbc54b204bc/shard-02-of-13.data)
Read shard 3 Data shards 9 Parity 4 (994f1113-da94-4be1-8551-9dbc54b204bc/shard-03-of-13.data)
Read shard 4 Data shards 9 Parity 4 (994f1113-da94-4be1-8551-9dbc54b204bc/shard-04-of-13.data)
Read shard 6 Data shards 9 Parity 4 (994f1113-da94-4be1-8551-9dbc54b204bc/shard-06-of-13.data)
Read shard 7 Data shards 9 Parity 4 (994f1113-da94-4be1-8551-9dbc54b204bc/shard-07-of-13.data)
Read shard 8 Data shards 8 Parity 5 (994f1113-da94-4be1-8551-9dbc54b204bc/shard-08-of-13.data)
Read shard 9 Data shards 8 Parity 5 (994f1113-da94-4be1-8551-9dbc54b204bc/shard-09-of-13.data)
Read shard 10 Data shards 8 Parity 5 (994f1113-da94-4be1-8551-9dbc54b204bc/shard-10-of-13.data)
Read shard 11 Data shards 8 Parity 5 (994f1113-da94-4be1-8551-9dbc54b204bc/shard-11-of-13.data)
Read shard 13 Data shards 8 Parity 5 (994f1113-da94-4be1-8551-9dbc54b204bc/shard-13-of-13.data)
Attempting to reconstruct using parity sets:
* Setup: Data shards: 9 - Parity blocks: 6
Have 6 complete remapped data shards and 6 complete parity shards. Could NOT reconstruct: too few shards given
* Setup: Data shards: 8 - Parity blocks: 5
Have 5 complete remapped data shards and 5 complete parity shards. Could reconstruct completely
0 bytes missing. Truncating 0 from the end.
Wrote output to 994f1113-da94-4be1-8551-9dbc54b204bc.complete
```

So far only inline data, but no real reason that external data can't also be included with some handling of blocks.

Supports only unencrypted data.
 2023-06-22 12:41:24 -07:00
Klaus Post 3eb2d086b2
Replace filepathx with fork (#16192) 2022-12-08 10:42:44 -08:00
Poorna dc92bb4646
xl-meta: update metadata version (#15958) 2022-10-28 02:48:43 -07:00
Klaus Post ff12080ff5
Remove deprecated io/ioutil (#15707) 2022-09-19 11:05:16 -07:00
Harshavardhana a60ac7ca17
fix: audit log to support object names in multipleObjectNames() handler (#14017) 2022-01-03 01:28:52 -08:00
Klaus Post 518612492c
xl-meta: Add header titles (#13880) 
Add type for headers and create custom marshal to make 
it easier to read. Group headers and metadata.

Restore functionality that will read `xl.meta` in the current dir with no params.

Before:
```
{
  "Headers": [
    [
      "8M04bTiYRDmEMQGeAsk1yg==",
      1639150471630100400,
      "rLD1Rw==",
      1,
      6
    ],
  ],
    "Versions": [
    {
      "Type": 1,
      "V2Obj": {
        "CSumAlgo": 1,
        "DDir": "oC1Xpg4tRfW03g8o8w7Bzg==",
        "EcAlgo": 1,
        "EcBSize": 1048576,
        "EcDist": [
          7,
          8,
          1,
          2,
          3,
          4,
          5,
          6
        ],
        "EcIndex": 1,
        "EcM": 4,
        "EcN": 4,
        "ID": "8M04bTiYRDmEMQGeAsk1yg==",
        "MTime": 1639150471630100400,
        "MetaSys": {
          "x-minio-internal-inline-data": "dHJ1ZQ=="
        },
        "MetaUsr": {
          "content-type": "application/octet-stream",
          "etag": "b8252c86fad2d8937300aa92b467a3aa"
        },
        "PartASizes": [
          1000
        ],
        "PartETags": null,
        "PartNums": [
          1
        ],
        "PartSizes": [
          1000
        ],
        "Size": 1000
      }
    }
  ]
}
```

After:
```
{
  "Versions": [
    {
      "Header": {
        "Flags": 6,
        "ModTime": "2021-12-10T16:34:31.6301004+01:00",
        "Signature": "acb0f547",
        "Type": 1,
        "VersionID": "f0cd386d389844398431019e02c935ca"
      },
      "Idx": 0,
      "Metadata": {
        "Type": 1,
        "V2Obj": {
          "CSumAlgo": 1,
          "DDir": "oC1Xpg4tRfW03g8o8w7Bzg==",
          "EcAlgo": 1,
          "EcBSize": 1048576,
          "EcDist": [
            7,
            8,
            1,
            2,
            3,
            4,
            5,
            6
          ],
          "EcIndex": 1,
          "EcM": 4,
          "EcN": 4,
          "ID": "8M04bTiYRDmEMQGeAsk1yg==",
          "MTime": 1639150471630100400,
          "MetaSys": {
            "x-minio-internal-inline-data": "dHJ1ZQ=="
          },
          "MetaUsr": {
            "content-type": "application/octet-stream",
            "etag": "b8252c86fad2d8937300aa92b467a3aa"
          },
          "PartASizes": [
            1000
          ],
          "PartETags": null,
          "PartNums": [
            1
          ],
          "PartSizes": [
            1000
          ],
          "Size": 1000
        }
      }
    }
  ]
}
```
 2021-12-10 15:03:25 -08:00
Klaus Post faf013ec84
Improve performance on multiple versions (#13573) 
Existing:

```go
type xlMetaV2 struct {
    Versions []xlMetaV2Version `json:"Versions" msg:"Versions"`
}
```

Serialized as regular MessagePack.

```go
//msgp:tuple xlMetaV2VersionHeader
type xlMetaV2VersionHeader struct {
	VersionID [16]byte
	ModTime   int64
	Type      VersionType
	Flags     xlFlags
}
```

Serialize as streaming MessagePack, format:

```
int(headerVersion)
int(xlmetaVersion)
int(nVersions)
for each version {
    binary blob, xlMetaV2VersionHeader, serialized
    binary blob, xlMetaV2Version, serialized.
}
```

xlMetaV2VersionHeader is <= 30 bytes serialized. Deserialized struct 
can easily be reused and does not contain pointers, so efficient as a 
slice (single allocation)

This allows quickly parsing everything as slices of bytes (no copy).

Versions are always *saved* sorted by modTime, newest *first*. 
No more need to sort on load.

* Allows checking if a version exists.
* Allows reading single version without unmarshal all.
* Allows reading latest version of type without unmarshal all.
* Allows reading latest version without unmarshal of all.
* Allows checking if the latest is deleteMarker by reading first entry.
* Allows adding/updating/deleting a version with only header deserialization.
* Reduces allocations on conversion to FileInfo(s).
 2021-11-18 12:15:22 -08:00
Harshavardhana 82d73f387d
add tool to read healing.bin for debugging (#13650) 2021-11-12 16:18:53 -08:00
Klaus Post 1642867136
Add documentation for debugging tools (#13484) 
Move `xl-meta` so it can be installed out-of-repo with a single command.
 2021-10-20 10:12:46 -07:00