MyFavMirrors
/
minio
mirror of https://github.com/minio/minio.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
11,572 Commits 5 Branches 503 Tags 186 MiB
Commit Graph

200 Commits

Author SHA1 Message Date
Klaus Post 1642867136
Add documentation for debugging tools (#13484) 
Move `xl-meta` so it can be installed out-of-repo with a single command.
 2021-10-20 10:12:46 -07:00
Klaus Post bc6067d195
Add admin inspect Glob support (#13328) 
* Add admin Glob support

Allow returning multiple files on inspect calls.

```
λ mc admin inspect --json local2/testbucket/nyc-taxi-data-10M.csv.zst/*

...

λ unzip -l inspect.5f0643b2.zip

Archive:  inspect.5f0643b2.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/
      802  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/xl.meta
        0  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/
      802  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/xl.meta
        0  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/
      802  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/xl.meta
        0  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/
      802  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/xl.meta
---------                     -------
     3208                     8 files
```

Using fully recursive:

```
λ  mc admin inspect local2/testbucket/nyc-taxi-data-10M.csv.zst/**

...

Archive:  inspect.79c261cb.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/
        0  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.1
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.10
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.11
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.12
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.13
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.14
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.15
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.16
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.17
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.18
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.19
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.2
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.20
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.21
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.22
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.23
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.24
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.25
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.26
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.27
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.28
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.29
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.3
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.30
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.31
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.32
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.33
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.34
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.35
  3439368  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.36
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.4
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.5
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.6
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.7
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.8
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.9
      802  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/xl.meta
        0  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/
        0  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.1
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.10
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.11
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.12
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.13
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.14
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.15
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.16
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.17
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.18
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.19
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.2
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.20
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.21
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.22
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.23
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.24
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.25
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.26
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.27
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.28
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.29
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.3
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.30
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.31
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.32
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.33
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.34
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.35
  3439368  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.36
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.4
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.5
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.6
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.7
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.8
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.9
      802  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/xl.meta
        0  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/
        0  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.1
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.10
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.11
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.12
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.13
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.14
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.15
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.16
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.17
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.18
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.19
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.2
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.20
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.21
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.22
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.23
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.24
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.25
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.26
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.27
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.28
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.29
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.3
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.30
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.31
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.32
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.33
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.34
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.35
  3439368  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.36
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.4
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.5
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.6
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.7
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.8
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.9
      802  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/xl.meta
        0  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/
        0  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.1
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.10
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.11
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.12
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.13
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.14
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.15
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.16
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.17
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.18
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.19
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.2
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.20
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.21
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.22
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.23
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.24
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.25
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.26
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.27
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.28
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.29
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.3
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.30
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.31
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.32
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.33
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.34
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.35
  3439368  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.36
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.4
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.5
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.6
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.7
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.8
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.9
      802  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/xl.meta
---------                     -------
601034920                     156 files

```

Furthermore allow `inspect` to do direct decode from `mc`, for example:

```
λ mc admin inspect --json local2/testbucket/nyc-taxi-data-10M.csv.zst/*|inspect -json
Output decrypted to inspect.5f0643b2.zip
```

- Correct error, forward non-EOF errors.
- Add some extra safety. Log FNF when no files.
- Add `xl-meta` zip support.
For `xl-meta` multiple inputs output object with names as key.
Automatically switches `xl-meta` to single-line output when multiple objects.
Add double-star wildcard support to xl-meta input.

Co-authored-by: Harshavardhana <harsha@minio.io>
 2021-10-01 11:50:00 -07:00
Poorna Krishnamoorthy 7f6ed35347
Allow null versions to be replicated (#13310) 
for pre-existing objects present in a bucket
prior to enabling existing object replication.

Co-authored-by: Poorna Krishnamoorthy <poorna@minio.io>
 2021-09-28 10:26:12 -07:00
Poorna Krishnamoorthy 0b55a0423e
fix: cache usage deserialization from v5 to v6 (#13258) 2021-09-21 09:01:51 -07:00
Harshavardhana f492f72154
add 3site replication script (#13256) 2021-09-20 18:24:24 -07:00
Poorna Krishnamoorthy c4373ef290
Add support for multi site replication (#12880) 2021-09-18 13:31:35 -07:00
Klaus Post 470553ff5d
Tweak readall allocation and renameData buffer reuse (#13108) 
Use a single allocation for reading the file, not the growing buffer of `io.ReadAll`.

Reuse the write buffer if we can when writing metadata in RenameData.
 2021-08-30 08:38:11 -07:00
Ricardo Katz a526ad2e80
Add headers into AMQP notifications (#12911) 
Signed-off-by: Ricardo Katz <rkatz@vmware.com>
 2021-08-11 22:24:19 -07:00
Harshavardhana c13cbc64d1 fix multiple typos in documentation 2021-08-04 14:15:45 -07:00
Harshavardhana 2ce60d772b
fix: update README.md for new release (#12637) 
rename all 'docker run' commands to 'podman run'

fixes #12633
 2021-07-07 18:07:29 -07:00
Poorna Krishnamoorthy 92e4c8be10
Update replication docs to fix mc reference (#12490) 
Signed-off-by: Poorna Krishnamoorthy <poorna@minio.io>
 2021-06-13 11:37:22 -07:00
Poorna Krishnamoorthy f2a3872301
Update design.md for replication (#12486) 
Fixes #12483
 2021-06-10 16:05:39 -07:00
Poorna Krishnamoorthy f199afcd6c
tiering: add aws role support for s3 (#12424) 
Signed-off-by: Poorna Krishnamoorthy <poorna@minio.io>
 2021-06-04 12:47:00 -07:00
Poorna Krishnamoorthy dbea8d2ee0
Add support for existing object replication. (#12109) 
Also adding an API to allow resyncing replication when
existing object replication is enabled and the remote target
is entirely lost. With the `mc replicate reset` command, the
objects that are eligible for replication as per the replication
config will be resynced to target if existing object replication
is enabled on the rule.
 2021-06-01 19:59:11 -07:00
Klaus Post 62b6615a2a
Add xl-meta data extraction (#12396) 
* Add `-export` flag to export data.
* Support wildcard inputs.
 2021-05-28 08:10:41 -07:00
Harshavardhana 41e9c6572f fix: docs links use non-raw links for markdown 2021-05-22 10:52:47 -07:00
Harshavardhana df4914b6f3 fix: update docs, fix wording and links 2021-05-21 12:36:03 -07:00
Poorna Krishnamoorthy a27339826c
Fix replication README.md docs (#12330) 
Signed-off-by: Poorna Krishnamoorthy <poorna@minio.io>
 2021-05-20 08:17:14 -07:00
Poorna Krishnamoorthy 951acf561c
Add support for syncing replica modifications (#11104) 
when bidirectional replication is set up.

If ReplicaModifications is enabled in the replication
configuration, sync metadata updates to source if
replication rules are met. By default, if this
configuration is unset, MinIO automatically sync's
metadata updates on replica back to the source.
 2021-05-13 19:20:45 -07:00
Krishnan Parthasarathi c829e3a13b Support for remote tier management (#12090) 
With this change, MinIO's ILM supports transitioning objects to a remote tier.
This change includes support for Azure Blob Storage, AWS S3 compatible object
storage incl. MinIO and Google Cloud Storage as remote tier storage backends.

Some new additions include:

 - Admin APIs remote tier configuration management

 - Simple journal to track remote objects to be 'collected'
   This is used by object API handlers which 'mutate' object versions by
   overwriting/replacing content (Put/CopyObject) or removing the version
   itself (e.g DeleteObjectVersion).

 - Rework of previous ILM transition to fit the new model
   In the new model, a storage class (a.k.a remote tier) is defined by the
   'remote' object storage type (one of s3, azure, GCS), bucket name and a
   prefix.

* Fixed bugs, review comments, and more unit-tests

- Leverage inline small object feature
- Migrate legacy objects to the latest object format before transitioning
- Fix restore to particular version if specified
- Extend SharedDataDirCount to handle transitioned and restored objects
- Restore-object should accept version-id for version-suspended bucket (#12091)
- Check if remote tier creds have sufficient permissions
- Bonus minor fixes to existing error messages

Co-authored-by: Poorna Krishnamoorthy <poorna@minio.io>
Co-authored-by: Krishna Srinivas <krishna@minio.io>
Signed-off-by: Harshavardhana <harsha@minio.io>
 2021-04-23 11:58:53 -07:00
Harshavardhana 069432566f update license change for MinIO 
Signed-off-by: Harshavardhana <harsha@minio.io>
 2021-04-23 11:58:53 -07:00
Poorna Krishnamoorthy 28f0ded1a4
Update replication design.md for sync mode (#12100) 2021-04-20 17:31:36 -07:00
Ravind Kumar ca9b48b3b4
Update Replication README to point at new docs (#12069) 
This is a minor change to call out the new documentation and warn 
users to change  their bookmarks. Once we are ready to set up 
some redirects, we can remove this page from Gluegun TOC.
 2021-04-15 16:32:44 -07:00
Klaus Post d50193d30f
xl-meta.go dump skip CRC check (#12031) 
Skip CRC of metadata.
 2021-04-09 17:38:23 -07:00
Poorna Krishnamoorthy 2899cc92b4
Update replication docs for required permission (#12010) 2021-04-07 15:56:02 -07:00
Klaus Post 2623338dc5
Inline small file data in xl.meta file (#11758) 2021-03-29 17:00:55 -07:00
Harshavardhana 9171d6ef65
rename all references from crawl -> scanner (#11621) 2021-02-26 15:11:42 -08:00
Poorna Krishnamoorthy 8e8a792d9d
Allow delete marker replication from replica (#11566) 
in the case of active-active replication.

This PR also has the following changes:

- add docs on replication design
- fix corner case of completing versioned delete on a delete marker
  when the target is down and `mc rm --vid` is performed repeatedly. Instead
  the version should still be retained in the `PENDING|FAILED` state until
  replication sync completes.
- remove `s3:Replication:OperationCompletedReplication` and
   `s3:Replication:OperationFailedReplication` from ObjectCreated 
  events type
 2021-02-18 00:33:51 -08:00
Harshavardhana 98f76008c7 fix: bucket lifecycle again to remove Days parameter 2021-01-17 01:50:56 -08:00
Harshavardhana 8da0b7cf03 fix: lifecycle documentation for DeleteMarker 2021-01-17 01:37:25 -08:00
Harshavardhana f903cae6ff
Support variable server pools (#11256) 
Current implementation requires server pools to have
same erasure stripe sizes, to facilitate same SLA
and expectations.

This PR allows server pools to be variadic, i.e they
do not have to be same erasure stripe sizes - instead
they should have SLA for parity ratio.

If the parity ratio cannot be guaranteed by the new
server pool, the deployment is rejected i.e server
pool expansion is not allowed.
 2021-01-16 12:08:02 -08:00
Poorna Krishnamoorthy 7090bcc8e0
fix: doc links and delete replication permissions enforcement (#11285) 2021-01-15 15:22:55 -08:00
Harshavardhana cc2d887e0e fix: whitespace and formatting in replication docs 2021-01-14 22:58:53 -08:00
Poorna Krishnamoorthy c1b4b24236
Update replication docs (#11279) 2021-01-15 10:22:57 +05:30
Poorna Krishnamoorthy feaf8dfb9a
Fix replication status reported on completion (#11273) 
Fixes: #11272
 2021-01-13 11:52:28 -08:00
Poorna Krishnamoorthy b97d53b29c
fix remote target healthcheck (#11267) 2021-01-12 20:48:04 -08:00
Poorna Krishnamoorthy 7824e19d20
Allow synchronous replication if enabled. (#11165) 
Synchronous replication can be enabled by setting the --sync
flag while adding a remote replication target.

This PR also adds proxying on GET/HEAD to another node in a
active-active replication setup in the event of a 404 on the current node.
 2021-01-11 22:36:51 -08:00
Klaus Post eb9172eecb
Allow Compression + encryption (#11103) 2021-01-05 20:08:35 -08:00
Ravind Kumar 6cf0008469
fix: docs typo in object lock docs (#11181) 2020-12-29 16:14:10 -08:00
Poorna Krishnamoorthy 7b8a456f68
Update lifecycle README docs (#11160) 
Removing reference to transition feature in docs as this
feature is being revamped to provide better extensibility
across different cloud targets.
 2020-12-23 19:56:55 -08:00
Harshavardhana b43906f6ee fix: docs typos and keywords 2020-12-23 11:59:20 -08:00
Harshavardhana 8c79f87f02
add dynamic config docs (#11048) 
Co-authored-by: Eco <41090896+eco-minio@users.noreply.github.com>
 2020-12-07 19:02:20 -08:00
Poorna Krishnamoorthy 934bed47fa
Add transition event notification (#11047) 
This is a MinIO specific extension to allow monitoring of transition events.
 2020-12-07 13:53:28 -08:00
Harshavardhana 790833f3b2 Revert "Support variable server sets (#10314)" 
This reverts commit aabf053d2f.
 2020-12-01 12:02:29 -08:00
Harshavardhana aabf053d2f
Support variable server sets (#10314) 2020-11-25 16:28:47 -08:00
Poorna Krishnamoorthy 39f3d5493b
Show Delete replication status header (#10946) 
X-Minio-Replication-Delete-Status header shows the
status of the replication of a permanent delete of a version.

All GETs are disallowed and return 405 on this object version.
In the case of replicating delete markers.

X-Minio-Replication-DeleteMarker-Status shows the status 
of replication, and would similarly return 405.

Additionally, this PR adds reporting of delete marker event completion
and updates documentation
 2020-11-21 23:48:50 -08:00
Harshavardhana 9263be8cca
docs: fix missing event types in notifications (#10944) 2020-11-20 11:27:27 -08:00
Poorna Krishnamoorthy 1ebf6f146a Add support for ILM transition (#10565) 
This PR adds transition support for ILM
to transition data to another MinIO target
represented by a storage class ARN. Subsequent
GET or HEAD for that object will be streamed from
the transition tier. If PostRestoreObject API is
invoked, the transitioned object can be restored for
duration specified to the source cluster.
 2020-11-19 18:47:17 -08:00
Harshavardhana 9a34fd5c4a Revert "Revert "Add delete marker replication support (#10396)"" 
This reverts commit 267d7bf0a9.
 2020-11-19 18:43:58 -08:00
Omar Alvarez 07859ef48b
docs: clarify notifications support for gateways (#10729) 
Co-authored-by: Harshavardhana <harsha@minio.io>
 2020-11-12 12:19:04 -08:00
Harshavardhana 267d7bf0a9 Revert "Add delete marker replication support (#10396)" 
This reverts commit 50c10a5087.

PR is moved to origin/dev branch
 2020-11-12 11:43:14 -08:00
Anton Melser 2c1e37197b
fix: bad example json for policy in replication docs (#10869) 2020-11-10 17:49:49 -08:00
Poorna Krishnamoorthy 50c10a5087
Add delete marker replication support (#10396) 
Delete marker replication is implemented for V2
configuration specified in AWS spec (though AWS
allows it only in the V1 configuration).

This PR also brings in a MinIO only extension of
replicating permanent deletes, i.e. deletes specifying
version id are replicated to target cluster.
 2020-11-10 15:24:14 -08:00
Anis Elleuch 6fd088f448
docs: Update ILM doc with versioning features (#10714) 2020-10-20 09:23:27 -07:00
Poorna Krishnamoorthy 01498a3e34
fix: add docs for new event types in notification (#10636) 2020-10-06 13:33:23 -07:00
Praveen raj Mani b880796aef
Set the maximum open connections limit in PG and MySQL target configs (#10558) 
As the bulk/recursive delete will require multiple connections to open at an instance,
The default open connections limit will be reached which results in the following error

```FATAL:  sorry, too many clients already```

By setting the open connections to a reasonable value - `2`, We ensure that the max open connections
will not be exhausted and lie under bounds.

The queries are simple inserts/updates/deletes which is operational and sufficient with the
the maximum open connection limit is 2.

Fixes #10553

Allow user configuration for MaxOpenConnections
 2020-09-24 22:20:30 -07:00
poornas e6ab4db6b8
Fix minimum replication workers started (#10560) 
This PR also fixes GetReplicationConfiguration permission
in web-handlers.go to use bucket as resource
 2020-09-24 12:25:41 -07:00
poornas a4006e23a0
Update replication docs to clarify permissions (#10536) 
Co-authored-by: Klaus Post <klauspost@gmail.com>
 2020-09-22 11:58:04 -07:00
Aleksey Pogibelev 6f45e303f5
fix tls flag (#10487) 2020-09-18 15:35:54 +05:30
poornas 73a6b4ea11
fix typo in replication docs (#10366) 2020-08-27 12:54:23 -07:00
Praveen raj Mani d0c910a6f3
Support https and basic-auth for elasticsearch notification target (#10332) 2020-08-23 09:43:48 -07:00
Ritesh H Shukla 8049184dcc
fix: documentation changes in replication docs (#10209) 2020-08-07 13:30:52 -07:00
poornas adcaa6f9de
fix: Change ListBucketTargets handler (#10217) 
to list all targets across a tenant.
Also fixing some validations.
 2020-08-06 17:10:21 -07:00
poornas 121164db56
fix: relax some replication validations (#10210) 
Also inherit storage class from source object
if replication configuration does not have a storage
class specified for destination bucket.
 2020-08-05 20:01:20 -07:00
poornas 3acc0ebb81
fix: Change service name in Arn for replication (#10205) 2020-08-05 00:43:18 -07:00
poornas a8dd7b3eda
Refactor replication target management. (#10154) 
Generalize replication target management so
that remote targets for a bucket can be
managed with ARNs. `mc admin bucket remote`
command will be used to manage targets.
 2020-07-30 19:55:22 -07:00
Harshavardhana 0b5d1bc91d
fix: bucket replication docs (#10104) 
* fix: bucket replication docs

* Update docs/bucket/replication/README.md

Co-authored-by: kannappanr <30541348+kannappanr@users.noreply.github.com>

Co-authored-by: kannappanr <30541348+kannappanr@users.noreply.github.com>
 2020-07-21 22:19:30 -07:00
poornas c43da3005a
Add support for server side bucket replication (#9882) 2020-07-21 17:49:56 -07:00
Harshavardhana 3520e946a2 fix: versioning docs add more examples 2020-07-11 00:57:46 -07:00
Harshavardhana d5ff1c8e3b fix docs image urls to be absolute path 2020-07-11 00:27:30 -07:00
Harshavardhana 60417950c7
fix: the versioning/object lock documentation appropriately (#9988) 
- Move the bucket level features into `docs/bucket` directory
- fix issue template and simplify some of them
 2020-07-08 08:44:43 -07:00
Harshavardhana 38eef5ce4c
fix: documentation fixes for docker ENV settings (#9975) 
- update CREDITS file
- fix markdown links
- talk a bit more about upgrades
 2020-07-06 06:42:34 -07:00
Harshavardhana 28a1a17187
docs update versioning images (#9925) 2020-06-27 10:27:23 -07:00
Eco a5e879fbe4
doc: typo fix in versioning README (#9926) 
requets -> requests
 2020-06-27 05:06:31 -07:00
Harshavardhana 21058c34d0
add some description of xl.meta (#9901) 2020-06-22 17:27:54 -07:00
Klaus Post 691dc04fac
Add neater metadata printer (#9898) 
* Just read files from args (more than 1 now supported)
* Pretty print by default. `-ndjson` will disable.
* Check header.
* Support stdin as '-'
* Don't just ignore errors.
 2020-06-22 13:20:22 -07:00
Harshavardhana 4915433bd2
Support bucket versioning (#9377) 
- Implement a new xl.json 2.0.0 format to support,
  this moves the entire marshaling logic to POSIX
  layer, top layer always consumes a common FileInfo
  construct which simplifies the metadata reads.
- Implement list object versions
- Migrate to siphash from crchash for new deployments
  for object placements.

Fixes #2111
 2020-06-12 20:04:01 -07:00
Praveen raj Mani 2ce2e88adf
Support mTLS Authentication in Webhooks (#9777) 2020-06-08 05:55:44 -07:00
Harshavardhana 4301a33371 update docs to mention quota not supported on FS 2020-05-29 07:57:45 -07:00
Harshavardhana 6e0575a53d
Revert "Disable crawler in FS/NAS gateway mode (#9695)" (#9702) 
This reverts commit eba423bb9d.

Additionally also address the FS crawler to properly
calculate the sizes for encrypted/compressed content.
 2020-05-25 11:32:53 -07:00
poornas 011a2c0b78
Add docs for bucket quota feature (#9503) 
This PR also adds a check to not enforce
bucket quota for server-side metadata copy
of an object onto itself.
 2020-05-16 19:27:33 -07:00
Praveen raj Mani c79358c67e
notification queue limit has no maxLimit (#9380) 
New value defaults to 100K events by default,
but users can tune this value upto any value
they seem necessary.

* increase the limit to maxint64 while validating
 2020-04-18 01:20:56 -07:00
Harshavardhana cc9b63eb51
add deprecation docs for PostgresSQL/MySQL targets (#9333) 2020-04-13 12:13:33 -07:00
Harshavardhana 3184205519
fix: config to support keys with special values (#9304) 
This PR adds context-based `k=v` splits based
on the sub-system which was obtained, if the
keys are not provided an error will be thrown
during parsing, if keys are provided with wrong
values an error will be thrown. Keys can now
have values which are of a much more complex
form such as `k="v=v"` or `k=" v = v"`
and other variations.

additionally, deprecate unnecessary postgres/mysql
configuration styles, support only

- connection_string for Postgres
- dsn_string for MySQL

All other parameters are removed.
 2020-04-09 21:45:17 -07:00
ebozduman fdda5f98c6
Makes mandatory dsn_string parameter optional (#8931) 2020-03-28 22:20:02 -07:00
Stephen N 1ffa983a9d
added support for SASL/SCRAM on Kafka bucket notifications. (#9168) 
fixes #9167
 2020-03-20 11:10:27 -07:00
kumy b809c84338
fix: notifications doc elaborate env values for targets (#9103) 2020-03-08 18:33:43 -07:00
Praveen raj Mani 1b427ddb69
Support for Kafka version in the config (#9001) 
Add a field for the Kafka version in the config. The user can explicitly 
set the version of the Kafka cluster.

Fixes #8768
 2020-02-17 07:56:33 +05:30
ebozduman a05af9bb98
Fixes several typos and clarifies some explanations in MySQL doc. (#8908) 2020-02-03 18:55:05 -08:00
Harshavardhana 86252ec7e1
fix: document _ENABLE for all notification targets (#8864) 
Fixes #8863
 2020-01-20 16:48:19 -08:00
ebozduman d74818b227 Typo in MySQL notification doc (#8807) 2020-01-13 13:06:42 -08:00
Etienne 5f34b5e6a5 Update README.md (#8788) 2020-01-10 21:08:09 -08:00
Harshavardhana 0f092c2be8 Add env values and description for config sub-systems (#8684) 
The output is directly borrowed from `mc` command output
 2019-12-21 22:52:16 +05:30
Harshavardhana 63c3114657 fix: doc notifications formatting issues (#8661) 2019-12-17 17:34:17 -08:00
Harshavardhana 476111968a Update help messages with new wording (#8616) 
Final update to all messages across sub-systems
after final review, the only change here is that
NATS now has TLS and TLSSkipVerify to be consistent
for all other notification targets.
 2019-12-06 13:53:51 -08:00
Aleksandr Petruhin d2dc964cb5 Support TLS auth for Kafka notification target (#8609) 2019-12-05 15:31:46 -08:00
Harshavardhana 0bfd20a8e3
Add client_id support for OpenID (#8579) 
- One click OpenID authorization on Login page
- Add client_id help, config keys etc

Thanks to @egorkaru @ihostage for the
original work and testing.
 2019-11-29 21:37:42 -08:00
Harshavardhana ee4a6a823d Migrate config to KV data format (#8392) 
- adding oauth support to MinIO browser (#8400) by @kanagaraj
- supports multi-line get/set/del for all config fields
- add support for comments, allow toggle
- add extensive validation of config before saving
- support MinIO browser to support proper claims, using STS tokens
- env support for all config parameters, legacy envs are also
  supported with all documentation now pointing to latest ENVs
- preserve accessKey/secretKey from FS mode setups
- add history support implements three APIs
  - ClearHistory
  - RestoreHistory
  - ListHistory
- add help command support for each config parameters
- all the bug fixes after migration to KV, and other bug
  fixes encountered during testing.
 2019-10-22 22:59:13 -07:00
k-tipp 208efb843b Update Kafka notification example doc (#8212) 2019-09-16 19:24:55 +05:30
Praveen raj Mani b0cea1c0f3 Enable event persistence in AMQP (#7565) 2019-07-25 11:20:24 -07:00