MyFavMirrors/minio
1
0
Fork 0
mirror of https://github.com/minio/minio.git synced 2024-12-27 15:45:55 -05:00
Code Issues Packages Projects Releases Wiki Activity
11,636 Commits 4 Branches 506 Tags 189 MiB
8ab3dac4f2
Commit Graph

1545 Commits

Author SHA1 Message Date
Krishnan Parthasarathi
3da9ee15d3
Add MaxNoncurrentVersions to NoncurrentExpiration action (#13580) 
This unit allows users to limit the maximum number of noncurrent 
versions of an object.

To enable this rule you need the following *ilm.json*
```
cat >> ilm.json <<EOF
{
    "Rules": [
        {
            "ID": "test-max-noncurrent",
            "Status": "Enabled",
            "Filter": {
                "Prefix": "user-uploads/"
            },
            "NoncurrentVersionExpiration": {
                "MaxNoncurrentVersions": 5
            }
        }
    ]
}
EOF
mc ilm import myminio/mybucket < ilm.json
```
 2021-11-19 17:54:10 -08:00
Mani
7b82411e6f
change the unit of measurement from TB to TiB (#13686) 2021-11-18 20:06:37 -08:00
Harshavardhana
79b3a1fe4e remove object torrent, AWS S3 removed support for torrent API 2021-11-18 12:21:48 -08:00
Klaus Post
faf013ec84
Improve performance on multiple versions (#13573) 
Existing:

```go
type xlMetaV2 struct {
    Versions []xlMetaV2Version `json:"Versions" msg:"Versions"`
}
```

Serialized as regular MessagePack.

```go
//msgp:tuple xlMetaV2VersionHeader
type xlMetaV2VersionHeader struct {
	VersionID [16]byte
	ModTime   int64
	Type      VersionType
	Flags     xlFlags
}
```

Serialize as streaming MessagePack, format:

```
int(headerVersion)
int(xlmetaVersion)
int(nVersions)
for each version {
    binary blob, xlMetaV2VersionHeader, serialized
    binary blob, xlMetaV2Version, serialized.
}
```

xlMetaV2VersionHeader is <= 30 bytes serialized. Deserialized struct 
can easily be reused and does not contain pointers, so efficient as a 
slice (single allocation)

This allows quickly parsing everything as slices of bytes (no copy).

Versions are always *saved* sorted by modTime, newest *first*. 
No more need to sort on load.

* Allows checking if a version exists.
* Allows reading single version without unmarshal all.
* Allows reading latest version of type without unmarshal all.
* Allows reading latest version without unmarshal of all.
* Allows checking if the latest is deleteMarker by reading first entry.
* Allows adding/updating/deleting a version with only header deserialization.
* Reduces allocations on conversion to FileInfo(s).
 2021-11-18 12:15:22 -08:00
Ashish Kumar Sinha
3d2bc15e9a
Add grafana json file for replication metrics (#13678) 2021-11-17 14:49:46 -08:00
Harshavardhana
661b263e77
add gocritic/ruleguard checks back again, cleanup code. (#13665) 
- remove some duplicated code
- reported a bug, separately fixed in #13664
- using strings.ReplaceAll() when needed
- using filepath.ToSlash() use when needed
- remove all non-Go style comments from the codebase

Co-authored-by: Aditya Manthramurthy <donatello@users.noreply.github.com>
 2021-11-16 09:28:29 -08:00
Harshavardhana
82d73f387d
add tool to read healing.bin for debugging (#13650) 2021-11-12 16:18:53 -08:00
Harshavardhana
5acc8c0134
add multi-site replication tests (#13631) 2021-11-10 18:18:09 -08:00
Poorna K
03725dc015
Default multipart caching to writethrough (#13613) 
when `MINIO_CACHE_COMMIT` is set.

- `writeback` caching applies only to single 
uploads. When cache commit mode is 
`writeback`, default multipart caching to be
synchronous.

- Add writethrough caching for single uploads
 2021-11-10 08:12:03 -08:00
Minio Trusted
edf1f4233b Update yaml files to latest version RELEASE.2021-11-09T03-21-45Z 2021-11-09 04:51:05 +00:00
Harshavardhana
5aef492b4c update disk-caching design guide 2021-11-07 01:21:34 -08:00
Harshavardhana
5d7ed8ff7d update S3 gateway limitation docs 2021-11-06 23:24:48 -07:00
Minio Trusted
cbb0828ab8 Update yaml files to latest version RELEASE.2021-11-05T09-16-26Z 2021-11-05 10:03:56 +00:00
Minio Trusted
9f3c151c3c Update yaml files to latest version RELEASE.2021-11-03T03-36-36Z 2021-11-03 06:48:34 +00:00
Harshavardhana
e106070640 update docs to mention the expected behavior for requests_max 
fixes #13561
 2021-11-02 14:10:21 -07:00
jandres - moscardo
1aa08f594d
Update README.md prometheus (#13514) 
Modify the doc to warn users about Prometheus sending `domain:port`
 2021-11-02 12:27:30 -07:00
Poorna K
15dcacc1fc
Add support for caching multipart in writethrough mode (#13507) 2021-11-01 08:11:58 -07:00
Harshavardhana
3c1220adca add tests for default governance replication 2021-10-30 08:57:59 -07:00
Harshavardhana
2af5445309 update 3-site replication tests 2021-10-29 22:09:55 -07:00
Harshavardhana
abb1916bda
update list objects limit to match S3 spec 2021-10-28 18:21:51 -07:00
Minio Trusted
8e29ae8c44 Update yaml files to latest version RELEASE.2021-10-27T16-29-42Z 2021-10-28 02:45:22 +00:00
Minio Trusted
0c76fb57f2 Update yaml files to latest version RELEASE.2021-10-23T03-28-24Z 2021-10-23 07:46:29 +00:00
Harshavardhana
52c5f6e152 remove extraneous whitespaces 2021-10-21 14:43:13 -07:00
Aditya Manthramurthy
26ca59859f
update LDAP doc (#13492) 
- clarify the login flow
- add some sections on configuration
- minor fixes to improve readability
 2021-10-21 14:41:32 -07:00
Klaus Post
1642867136
Add documentation for debugging tools (#13484) 
Move `xl-meta` so it can be installed out-of-repo with a single command.
 2021-10-20 10:12:46 -07:00
Harshavardhana
bedf739d16 update required IAM policies 2021-10-13 12:28:53 -07:00
Minio Trusted
6299e42aa9 Update yaml files to latest version RELEASE.2021-10-13T00-23-17Z 2021-10-13 01:14:23 +00:00
Harshavardhana
9ea45399ce
fix: enable AssumeRoleWithCertificate API only when asked (#13410) 
This is a breaking change but we need to do this to avoid
issues discussed in #13409 based on discussions from #13371

fixes #13371
fixes #13409
 2021-10-11 14:23:51 -07:00
Minio Trusted
e0db822a9b Update yaml files to latest version RELEASE.2021-10-10T16-53-30Z 2021-10-10 21:24:01 +00:00
Minio Trusted
c49ebaaf1a Update yaml files to latest version RELEASE.2021-10-08T23-58-24Z 2021-10-09 00:50:16 +00:00
Harshavardhana
d57b57bddc
feat: Add RX/TX to audit logging (#13382) 
add additional values for audit logging
 2021-10-07 19:03:46 -07:00
Aditya Manthramurthy
f81a188ef6
update site replication doc (#13377) 
make pre-requisites clear
 2021-10-07 15:21:01 -07:00
Minio Trusted
4ce6830a7b Update yaml files to latest version RELEASE.2021-10-06T23-36-31Z 2021-10-07 04:13:46 +00:00
Aditya Manthramurthy
3a7c79e2c7
Add new site replication feature (#13311) 
This change allows a set of MinIO sites (clusters) to be configured 
for mutual replication of all buckets (including bucket policies, tags, 
object-lock configuration and bucket encryption), IAM policies, 
LDAP service accounts and LDAP STS accounts.
 2021-10-06 16:36:31 -07:00
Minio Trusted
d6a3215fe2 Update yaml files to latest version RELEASE.2021-10-02T16-31-05Z 2021-10-03 21:43:13 +00:00
Klaus Post
bc6067d195
Add admin inspect Glob support (#13328) 
* Add admin Glob support

Allow returning multiple files on inspect calls.

```
λ mc admin inspect --json local2/testbucket/nyc-taxi-data-10M.csv.zst/*

...

λ unzip -l inspect.5f0643b2.zip

Archive:  inspect.5f0643b2.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/
      802  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/xl.meta
        0  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/
      802  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/xl.meta
        0  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/
      802  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/xl.meta
        0  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/
      802  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/xl.meta
---------                     -------
     3208                     8 files
```

Using fully recursive:

```
λ  mc admin inspect local2/testbucket/nyc-taxi-data-10M.csv.zst/**

...

Archive:  inspect.79c261cb.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/
        0  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.1
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.10
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.11
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.12
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.13
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.14
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.15
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.16
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.17
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.18
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.19
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.2
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.20
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.21
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.22
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.23
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.24
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.25
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.26
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.27
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.28
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.29
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.3
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.30
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.31
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.32
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.33
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.34
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.35
  3439368  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.36
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.4
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.5
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.6
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.7
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.8
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.9
      802  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/xl.meta
        0  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/
        0  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.1
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.10
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.11
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.12
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.13
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.14
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.15
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.16
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.17
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.18
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.19
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.2
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.20
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.21
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.22
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.23
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.24
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.25
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.26
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.27
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.28
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.29
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.3
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.30
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.31
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.32
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.33
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.34
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.35
  3439368  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.36
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.4
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.5
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.6
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.7
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.8
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.9
      802  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/xl.meta
        0  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/
        0  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.1
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.10
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.11
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.12
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.13
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.14
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.15
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.16
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.17
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.18
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.19
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.2
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.20
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.21
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.22
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.23
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.24
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.25
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.26
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.27
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.28
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.29
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.3
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.30
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.31
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.32
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.33
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.34
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.35
  3439368  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.36
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.4
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.5
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.6
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.7
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.8
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.9
      802  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/xl.meta
        0  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/
        0  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.1
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.10
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.11
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.12
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.13
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.14
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.15
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.16
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.17
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.18
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.19
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.2
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.20
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.21
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.22
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.23
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.24
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.25
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.26
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.27
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.28
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.29
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.3
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.30
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.31
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.32
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.33
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.34
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.35
  3439368  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.36
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.4
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.5
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.6
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.7
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.8
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.9
      802  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/xl.meta
---------                     -------
601034920                     156 files

```

Furthermore allow `inspect` to do direct decode from `mc`, for example:

```
λ mc admin inspect --json local2/testbucket/nyc-taxi-data-10M.csv.zst/*|inspect -json
Output decrypted to inspect.5f0643b2.zip
```

- Correct error, forward non-EOF errors.
- Add some extra safety. Log FNF when no files.
- Add `xl-meta` zip support.
For `xl-meta` multiple inputs output object with names as key.
Automatically switches `xl-meta` to single-line output when multiple objects.
Add double-star wildcard support to xl-meta input.

Co-authored-by: Harshavardhana <harsha@minio.io>
 2021-10-01 11:50:00 -07:00
Poorna Krishnamoorthy
7f6ed35347
Allow null versions to be replicated (#13310) 
for pre-existing objects present in a bucket
prior to enabling existing object replication.

Co-authored-by: Poorna Krishnamoorthy <poorna@minio.io>
 2021-09-28 10:26:12 -07:00
Minio Trusted
a5923a5d51 Update yaml files to latest version RELEASE.2021-09-24T00-24-24Z 2021-09-24 19:20:44 +00:00
Minio Trusted
f4d1b7c603 Update yaml files to latest version RELEASE.2021-09-23T04-46-24Z 2021-09-23 07:42:55 +00:00
Poorna Krishnamoorthy
0b55a0423e
fix: cache usage deserialization from v5 to v6 (#13258) 2021-09-21 09:01:51 -07:00
Harshavardhana
f492f72154
add 3site replication script (#13256) 2021-09-20 18:24:24 -07:00
Poorna Krishnamoorthy
c4373ef290
Add support for multi site replication (#12880) 2021-09-18 13:31:35 -07:00
Minio Trusted
0b8c5a6872 Update yaml files to latest version RELEASE.2021-09-18T18-09-59Z 2021-09-18 20:28:07 +00:00
Minio Trusted
bf409936e7 Update yaml files to latest version RELEASE.2021-09-15T04-54-25Z 2021-09-15 08:04:57 +00:00
Harshavardhana
af78c3925a
add userinfo support for OpenID (#12469) 
Some identity providers like GitLab do not provide
information about group membership as part of the
identity token claims. They only expose it via OIDC compatible
'/oauth/userinfo' endpoint, as described in the OpenID
Connect 1.0 sepcification.

But this of course requires application to make sure to add
additional accessToken, since idToken cannot be re-used to
perform the same 'userinfo' call. This is why this is specialized
requirement. Gitlab seems to be the only OpenID vendor that requires
this support for the time being.

fixes #12367
 2021-09-13 16:22:14 -07:00
Harshavardhana
90e505e58f calculate API requests/error as increase() intervals not as rate() 2021-09-12 11:28:28 -07:00
Minio Trusted
a216583d95 Update yaml files to latest version RELEASE.2021-09-09T21-37-07Z 2021-09-09 23:50:26 +00:00
soudhaf
950fe73c4f
fix: minio console nginx config (#13156) 2021-09-08 23:41:06 -07:00
Harshavardhana
aaa3fc3805
fix: remove deprecated LDAP username format support (#13165) 2021-09-08 13:31:51 -07:00
Andreas Auernhammer
e438dccf19
sts: add support for certificate-based authentication (#12748) 
This commit adds a new STS API for X.509 certificate
authentication.

A client can make an HTTP POST request over a TLS connection
and MinIO will verify the provided client certificate, map it to an 
S3 policy and return temp. S3 credentials to the client.

So, this STS API allows clients to authenticate with X.509
certificates over TLS and obtain temp. S3 credentials.

For more details and examples refer to the docs/sts/tls.md
documentation.

Signed-off-by: Andreas Auernhammer <hi@aead.dev>
 2021-09-07 19:03:48 -07:00
Klaus Post
b2c92cdaaa
select: Add more compression formats (#13142) 
Support Zstandard, LZ4, S2, and snappy as additional 
compression formats for S3 Select.
 2021-09-06 09:09:53 -07:00
Harshavardhana
1250312287
fail ready/liveness if etcd is unhealthy in gateway mode (#13146) 2021-09-03 17:05:41 -07:00
Minio Trusted
e2b6fb0a6a Update yaml files to latest version RELEASE.2021-09-03T03-56-13Z 2021-09-03 06:45:54 +00:00
Harshavardhana
f486cfae86 point all docs for containers to quay.io 2021-09-01 18:48:26 -07:00
Minio Trusted
5c6dc63577 Update yaml files to latest version RELEASE.2021-08-31T05-46-54Z 2021-08-31 07:27:43 +00:00
Klaus Post
470553ff5d
Tweak readall allocation and renameData buffer reuse (#13108) 
Use a single allocation for reading the file, not the growing buffer of `io.ReadAll`.

Reuse the write buffer if we can when writing metadata in RenameData.
 2021-08-30 08:38:11 -07:00
Minio Trusted
abf079135e Update yaml files to latest version RELEASE.2021-08-25T00-41-18Z 2021-08-25 02:01:05 +00:00
Nitish Tiwari
60394ddf83
Add support for changing job name in Grafana dashboard (#13050) 2021-08-24 09:51:09 -07:00
Minio Trusted
7802088e71 Update yaml files to latest version RELEASE.2021-08-20T18-32-01Z 2021-08-20 20:42:00 +00:00
Harshavardhana
202d0b64eb
fix: enable go1.17 github ci/cd (#12997) 2021-08-18 18:35:22 -07:00
Minio Trusted
16aeb68c28 Update yaml files to latest version RELEASE.2021-08-17T20-53-08Z 2021-08-17 22:34:33 +00:00
Krishnan Parthasarathi
30b77f59b1
doc: Add ilm prometheus metrics information (#12994) 2021-08-17 12:19:36 -07:00
Aditya Manthramurthy
9b7d593e28
Add helper script to call assume role (#12978) 2021-08-17 01:46:59 -07:00
Ricardo Katz
a526ad2e80
Add headers into AMQP notifications (#12911) 
Signed-off-by: Ricardo Katz <rkatz@vmware.com>
 2021-08-11 22:24:19 -07:00
Harshavardhana
39f81d2c5b update max_delay to max_sleep under healing docs 2021-08-10 08:52:42 -07:00
Minio Trusted
7b0b0f9101 Update yaml files to latest version RELEASE.2021-08-05T22-01-19Z 2021-08-06 02:19:42 +00:00
Harshavardhana
c13cbc64d1 fix multiple typos in documentation 2021-08-04 14:15:45 -07:00
Matt Sarrel
109c8acf4f
fixed typo in metrics README.md (#12874) 2021-08-04 12:48:57 -07:00
Harshavardhana
035882d292
fix: remove parentIsObject() check (#12851) 
we will allow situations such as

```
a/b/1.txt
a/b
```

and

```
a/b
a/b/1.txt
```

we are going to document that this usecase is
not supported and we will never support it, if
any application does this users have to delete
the top level parent to make sure namespace is
accessible at lower level.

rest of the situations where the prefixes get
created across sets are supported as is.
 2021-08-03 13:26:57 -07:00
Harshavardhana
ea64a9263c
add deprecation notice for LDAP username format (#12849) 2021-08-02 18:20:06 -07:00
Harshavardhana
7281e86d9e update docker-compose to RELEASE.2021-07-30T00-02-00Z 2021-07-29 18:07:06 -07:00
Harshavardhana
a78bc7bfdb rename all access token to id_tokens 2021-07-27 22:38:12 -07:00
Harshavardhana
3735450e7e
fix: allow audience claim to be an array (#12810) 
Some incorrect setups might have multiple audiences
where they are trying to use a single authentication
endpoint for multiple services.

Nevertheless OpenID spec allows it to make it
even more confusin for no good reason.

> It MUST contain the OAuth 2.0 client_id of the
> Relying Party as an audience value. It MAY also
> contain identifiers for other audiences. In the
> general case, the aud value is an array of case
> sensitive strings. In the common special case
> when there is one audience, the aud value MAY
> be a single case sensitive string.

fixes #12809
 2021-07-27 18:37:51 -07:00
Minio Trusted
471b4fd0c9 Update yaml files to latest version RELEASE.2021-07-27T02-40-15Z 2021-07-27 05:02:00 +00:00
Aditya Manthramurthy
7106e9394c
[LDAP] Document new LDAP synchronization behavior (#12786) 2021-07-25 10:45:53 -07:00
Aditya Manthramurthy
9a31030e74
DEPRECATION Warning: Avoid configuring default LDAP STS Expiry (#12781) 
- Show notice when `MINIO_IDENTITY_LDAP_STS_EXPIRY` or the 
  corresponding to the configuration option is used at server startup.
- Once support is removed, the default will be fixed at 1 hour.
- Users may specify expiry directly in the STS API.
- Update docs and help message
- Adds example in ldap.go to configure expiry in STS API.
 2021-07-22 16:43:57 -07:00
Aditya Manthramurthy
a3079a7de2
fix: Add support for DurationSeconds in LDAP STS API (#12778) 2021-07-22 12:13:21 -07:00
Minio Trusted
026a005b64 Update yaml files to latest version RELEASE.2021-07-22T05-23-32Z 2021-07-22 06:12:29 +00:00
Minio Trusted
dbd7f74bb9 Update yaml files to latest version RELEASE.2021-07-21T22-15-23Z 2021-07-21 23:46:28 +00:00
Ali Malek
c25ad94b7e
Refactor docker-compose, Use YAML variable for common minio configs (#12735) 2021-07-18 18:06:40 -07:00
Minio Trusted
ead8778305 Update yaml files to latest version RELEASE.2021-07-15T22-27-34Z 2021-07-16 04:27:54 +00:00
Aditya Manthramurthy
eae9c2f65b
Add changes to ensure session policy is enforced in LDAP STS (#12716) 
- Bonus: Fix bug in webidentity sts that doesnt parse session policy correctly.

- update ldap.go to support session policy argument
 2021-07-15 15:27:34 -07:00
Ravind Kumar
ce800ed347
DOCS: Remove http links in favor of HTTPS (#12725) 2021-07-15 13:31:59 -07:00
Harshavardhana
8d19efe7e0
feat: increase allowed maximum STS expiration timeout to 365 days (#12704) 2021-07-14 00:08:22 -07:00
Harshavardhana
e316873f84
feat: Add support for kakfa audit logger target (#12678) 2021-07-13 09:39:13 -07:00
Harshavardhana
e25ae1c01f add console-address flag for spoonfed users 2021-07-12 17:05:38 -07:00
Minio Trusted
bc8962102f Update yaml files to latest version RELEASE.2021-07-12T02-44-53Z 2021-07-12 03:45:14 +00:00
Harshavardhana
55b08403cb update docker docs with --console-address 
also update the orchestration docs for nginx
proxy for console.

fixes #12660
 2021-07-11 19:44:53 -07:00
Klaus Post
d6a2fe02d3
Add admin file inspector (#12635) 
Download files from *any* bucket/path as an encrypted zip file.

The key is included in the response but can be separated so zip 
and the key doesn't have to be sent on the same channel.

Requires https://github.com/minio/pkg/pull/6
 2021-07-09 11:29:16 -07:00
Harshavardhana
28adb29db3
feat: Add support to poll users on external SSO (#12592) 
Additional support for vendor-specific admin API
integrations for OpenID, to ensure validity of
credentials on MinIO.

Every 5minutes check for validity of credentials
on MinIO with vendor specific IDP.
 2021-07-09 11:17:21 -07:00
Minio Trusted
b6dd9b55a7 Update yaml files to latest version RELEASE.2021-07-08T19-43-25Z 2021-07-08 22:53:49 -07:00
Christian Weiske
95d29a6a53
fix: Mention GetBucketyPolicy in S3 gateway docs policy (#12642) 
That action is needed for the gateway to obtain the policy
setting for public access.

Resolves: https://github.com/minio/minio/issues/12638
 2021-07-08 10:34:56 -07:00
Minio Trusted
c14f965293 Update yaml files to latest version RELEASE.2021-07-08T01-15-01Z 2021-07-08 03:12:44 +00:00
Harshavardhana
2ce60d772b
fix: update README.md for new release (#12637) 
rename all 'docker run' commands to 'podman run'

fixes #12633
 2021-07-07 18:07:29 -07:00
Harshavardhana
3137dc2eb3
support implicit flow in web-identity.go example (#12600) 
when a client secret is not provided,
automatically assume implicit flow
for authentication and invoke
relevant code accordingly.
 2021-06-30 07:43:04 -07:00
Harshavardhana
b043e61e8e remove deprecated warning for 'mc encrypt' 2021-06-25 08:12:40 -07:00
iternity-dotcom
d40268d899
docs: add limits for bucket and object name length (#12564) 2021-06-23 12:51:31 -07:00
Harshavardhana
cdeccb5510
feat: Deprecate embedded browser and import console (#12460) 
This feature also changes the default port where
the browser is running, now the port has moved
to 9001 and it can be configured with

```
--console-address ":9001"
```
 2021-06-17 20:27:04 -07:00
Harshavardhana
e4fbc6a9ff update to RELEASE.2021-06-17T00-10-46Z 2021-06-16 17:50:32 -07:00
Klaus Post
0d1d26a4ea
Simplify s3zip example (#12500) 
There is no need to create a custom transport wrapper.

Just set the header in the options.
 2021-06-14 11:04:14 -07:00
Harshavardhana
264ee97219 update to RELEASE.2021-06-14T01-29-23Z release 2021-06-13 20:58:56 -07:00
Poorna Krishnamoorthy
92e4c8be10
Update replication docs to fix mc reference (#12490) 
Signed-off-by: Poorna Krishnamoorthy <poorna@minio.io>
 2021-06-13 11:37:22 -07:00
Poorna Krishnamoorthy
f2a3872301
Update design.md for replication (#12486) 
Fixes #12483
 2021-06-10 16:05:39 -07:00
Anis Elleuch
ba5fb2365c
feat: support of ZIP list/get/head as S3 extension (#12267) 
When enabled, it is possible to list/get files
inside a zip file without uncompressing it.

Signed-off-by: Anis Elleuch <anis@min.io>
 2021-06-10 08:17:03 -07:00
Minio Trusted
22c58be421 Update yaml files to latest version RELEASE.2021-06-09T18-51-39Z 2021-06-09 19:21:21 +00:00
iternity-dotcom
dd5e9493f1
docs: add missing admin actions (#12442) 2021-06-09 10:37:20 -07:00
Nitish Tiwari
32017454ee
fix typo in Grafana dashboard json (#12471) 2021-06-09 08:04:12 -07:00
Minio Trusted
1bf80a6eea Update yaml files to latest version RELEASE.2021-06-07T21-40-51Z 2021-06-07 22:13:41 +00:00
Nitish Tiwari
00c5d7e1b3
Add healing related metrics in official dashboard (#12456) 2021-06-07 12:46:54 -07:00
Poorna Krishnamoorthy
f199afcd6c
tiering: add aws role support for s3 (#12424) 
Signed-off-by: Poorna Krishnamoorthy <poorna@minio.io>
 2021-06-04 12:47:00 -07:00
Poorna Krishnamoorthy
dbea8d2ee0
Add support for existing object replication. (#12109) 
Also adding an API to allow resyncing replication when
existing object replication is enabled and the remote target
is entirely lost. With the `mc replicate reset` command, the
objects that are eligible for replication as per the replication
config will be resynced to target if existing object replication
is enabled on the rule.
 2021-06-01 19:59:11 -07:00
Harshavardhana
1f262daf6f
rename all remaining packages to internal/ (#12418) 
This is to ensure that there are no projects
that try to import `minio/minio/pkg` into
their own repo. Any such common packages should
go to `https://github.com/minio/pkg`
 2021-06-01 14:59:40 -07:00
Poorna Krishnamoorthy
3690de0c6b
Drop Pending size and count from replication metrics (#12378) 
Real-time metrics calculated in-memory rely on the initial
replication metrics saved with data usage. However, this can
lag behind the actual state of the cluster at the time of server 
restart leading to inaccurate Pending size/counts reported to
Prometheus. Dropping the Pending metrics as this can be more 
reliably monitored by applications with replication notifications.

Signed-off-by: Poorna Krishnamoorthy <poorna@minio.io>
 2021-05-31 20:26:52 -07:00
Harshavardhana
ab7410af11 docs: add hostname entry for docker stack deploy 2021-05-31 19:11:52 -07:00
Harshavardhana
5cd6253508 docs: mention docker stack deploy doesn't work with compose.yml 2021-05-31 18:59:32 -07:00
Harshavardhana
f27513ff28 remove all docker swarm references 2021-05-31 18:55:21 -07:00
Harshavardhana
4444ba13a4
support ldap:username for policy substitution (#12390) 
LDAPusername is the simpler form of LDAPUser (userDN),
using a simpler version is convenient from policy
conditions point of view, since these are unique id's
used for LDAP login.
 2021-05-28 10:33:07 -07:00
Klaus Post
62b6615a2a
Add xl-meta data extraction (#12396) 
* Add `-export` flag to export data.
* Support wildcard inputs.
 2021-05-28 08:10:41 -07:00
Minio Trusted
ba75281d5a Update yaml files to latest version RELEASE.2021-05-27T22-06-31Z 2021-05-27 22:35:50 +00:00
Klaus Post
acc452b7ce
Add more erasure codes on degraded systems. (#11852) 
In cases where a cluster is degraded, we do not uphold our consistency 
guarantee and we will write fewer erasure codes and rely on healing 
to recreate the missing shards.

In some cases replacing known bad disks in practice take days.
We want to change the behavior of a known degraded system to keep
the erasure code promise of the storage class for each object.

This will create the objects with the same confidence as a fully 
functional cluster. The tradeoff will be that objects created 
during a partial outage will take up slightly more space.

This means that when the storage class is EC:4, there should 
always be written 4 parity shards, even if some disks are unavailable.

When an object is created on a set, the disks are immediately 
checked. If any disks are unavailable additional parity shards 
will be made for each offline disk, up to 50% of the number of disks.

We add an internal metadata field with the actual and intended 
erasure code level, this can optionally be picked up later by 
the scanner if we decide that data like this should be re-sharded.
 2021-05-27 11:38:09 -07:00
Harshavardhana
9773b16f6f update TLS docs to use new certgen tool 2021-05-26 10:41:29 -07:00
Minio Trusted
7b06b754a6 Update yaml files to latest version RELEASE.2021-05-26T00-22-46Z 2021-05-26 00:52:38 +00:00
Harshavardhana
2baabd455b docs: fix per tenant limits docs formatting 2021-05-24 09:37:17 -07:00
Harshavardhana
41e9c6572f fix: docs links use non-raw links for markdown 2021-05-22 10:52:47 -07:00
Harshavardhana
0da84a0c72 Update yaml files to latest version RELEASE.2021-05-22T02-34-39Z 2021-05-21 20:32:14 -07:00
Harshavardhana
df4914b6f3 fix: update docs, fix wording and links 2021-05-21 12:36:03 -07:00
Minio Trusted
783ea5eb5c Update yaml files to latest version RELEASE.2021-05-20T22-31-44Z 2021-05-20 23:02:31 +00:00
Poorna Krishnamoorthy
a27339826c
Fix replication README.md docs (#12330) 
Signed-off-by: Poorna Krishnamoorthy <poorna@minio.io>
 2021-05-20 08:17:14 -07:00
Harshavardhana
bb7fbcdc09
fix: generating service accounts for group only LDAP accounts (#12318) 
fixes #12315
 2021-05-18 15:19:20 -07:00
Harshavardhana
267f12a2a1 Update to release RELEASE.2021-05-18T00-53-28Z 2021-05-17 18:24:38 -07:00
Harshavardhana
3d9873106d
feat: distributed setup can start now with default credentials (#12303) 
In lieu of new changes coming for server command line, this
change is to deprecate strict requirement for distributed setups
to provide root credentials.

Bonus: remove MINIO_WORM warning from April 2020, it is time to
remove this warning.
 2021-05-17 08:45:22 -07:00
Minio Trusted
ce3d9dc9fa Update yaml files to latest version RELEASE.2021-05-16T05-32-34Z 2021-05-16 06:03:09 +00:00
Harshavardhana
7334247c98 update docs about NFS consistency model 2021-05-14 11:34:56 -07:00
Poorna Krishnamoorthy
951acf561c
Add support for syncing replica modifications (#11104) 
when bidirectional replication is set up.

If ReplicaModifications is enabled in the replication
configuration, sync metadata updates to source if
replication rules are met. By default, if this
configuration is unset, MinIO automatically sync's
metadata updates on replica back to the source.
 2021-05-13 19:20:45 -07:00
Nitish Tiwari
a592d3be19
fix the dashboard to use $rate_interval (#12277) 
refer https://grafana.com/blog/2020/09/28/new-in-grafana-7.2-__rate_interval-for-prometheus-rate-queries-that-just-work/
for further information
 2021-05-12 08:06:47 -07:00
Minio Trusted
f63eedb2b4 Update yaml files to latest version RELEASE.2021-05-11T23-27-41Z 2021-05-12 00:16:44 +00:00
Harshavardhana
477cd85bef purge deprecate docker swarm documentation 
documentation license moved to CC-BY-4.0
 2021-05-10 09:50:06 -07:00
Harshavardhana
d09806c887 add docs LICENSE file 2021-05-07 09:32:51 -07:00
Harshavardhana
2fd9c13b50 rename minio-cluster to minio-job as per prometheus config 2021-05-06 12:39:58 -07:00
Nitish Tiwari
ddc1e4b5b3
Update Grafana dashboard to use the new v2 cluster metrics (#12220) 
Fixes #11543
 2021-05-06 14:44:03 +05:30
Harshavardhana
804a23a06d update docs to remove _OLD credential references 
also update the docs about config, IAM on encryption.
 2021-05-04 10:27:51 -07:00
Harshavardhana
f7a87b30bf Revert "deprecate embedded browser (#12163)" 
This reverts commit 736d8cbac4.

Bring contrib files for older contributions
 2021-04-30 08:50:39 -07:00
Andreas Auernhammer
e5ec1325fc
docs: add QuickStart section to KMS encryption of IAM data (#12190) 
This commit enhances the docs about IAM encryption.
It adds a quick-start section that explains how to
get started quickly with `MINIO_KMS_SECRET_KEY`
instead of setting up KES.

It also removes the startup message that gets printed
when the server migrates IAM data to plaintext.
We will point this out in the release notes.

Signed-off-by: Andreas Auernhammer <aead@mail.de>
 2021-04-29 14:20:28 -07:00
Harshavardhana
736d8cbac4
deprecate embedded browser (#12163) 
https://github.com/minio/console takes over the functionality for the
future object browser development

Signed-off-by: Harshavardhana <harsha@minio.io>
 2021-04-27 10:52:12 -07:00
Harshavardhana
f420996dfa
fix: allow parsing keys in both new and old format (#12144) 
Bonus fix fallback to decrypt previously
encrypted content as well using older master
key ciphertext format.

Signed-off-by: Harshavardhana <harsha@minio.io>
 2021-04-24 19:05:25 -07:00
Krishnan Parthasarathi
c829e3a13b Support for remote tier management (#12090) 
With this change, MinIO's ILM supports transitioning objects to a remote tier.
This change includes support for Azure Blob Storage, AWS S3 compatible object
storage incl. MinIO and Google Cloud Storage as remote tier storage backends.

Some new additions include:

 - Admin APIs remote tier configuration management

 - Simple journal to track remote objects to be 'collected'
   This is used by object API handlers which 'mutate' object versions by
   overwriting/replacing content (Put/CopyObject) or removing the version
   itself (e.g DeleteObjectVersion).

 - Rework of previous ILM transition to fit the new model
   In the new model, a storage class (a.k.a remote tier) is defined by the
   'remote' object storage type (one of s3, azure, GCS), bucket name and a
   prefix.

* Fixed bugs, review comments, and more unit-tests

- Leverage inline small object feature
- Migrate legacy objects to the latest object format before transitioning
- Fix restore to particular version if specified
- Extend SharedDataDirCount to handle transitioned and restored objects
- Restore-object should accept version-id for version-suspended bucket (#12091)
- Check if remote tier creds have sufficient permissions
- Bonus minor fixes to existing error messages

Co-authored-by: Poorna Krishnamoorthy <poorna@minio.io>
Co-authored-by: Krishna Srinivas <krishna@minio.io>
Signed-off-by: Harshavardhana <harsha@minio.io>
 2021-04-23 11:58:53 -07:00
Harshavardhana
069432566f update license change for MinIO 
Signed-off-by: Harshavardhana <harsha@minio.io>
 2021-04-23 11:58:53 -07:00
Minio Trusted
e05e14309c Update yaml files to latest version RELEASE.2021-04-22T15-44-28Z 2021-04-22 16:32:11 +00:00
Poorna Krishnamoorthy
28f0ded1a4
Update replication design.md for sync mode (#12100) 2021-04-20 17:31:36 -07:00
Minio Trusted
49b7923309 Update yaml files to latest version RELEASE.2021-04-18T19-26-29Z 2021-04-18 20:15:32 +00:00
Ravind Kumar
ca9b48b3b4
Update Replication README to point at new docs (#12069) 
This is a minor change to call out the new documentation and warn 
users to change  their bookmarks. Once we are ready to set up 
some redirects, we can remove this page from Gluegun TOC.
 2021-04-15 16:32:44 -07:00
Klaus Post
d50193d30f
xl-meta.go dump skip CRC check (#12031) 
Skip CRC of metadata.
 2021-04-09 17:38:23 -07:00
Aditya Manthramurthy
05a9108c24
Update etcd doc to clarify that any STS setup should work (#12022) 2021-04-08 17:30:17 -07:00
Harshavardhana
0e4794ea50
fix: allow S3 gateway passthrough for SSE-S3 header (#12020) 
only in case of S3 gateway we have a case where we
need to allow for SSE-S3 headers as passthrough,

If SSE-C headers are passed then they are rejected
if KMS is not configured.
 2021-04-08 16:40:38 -07:00
Poorna Krishnamoorthy
2899cc92b4
Update replication docs for required permission (#12010) 2021-04-07 15:56:02 -07:00
Minio Trusted
a772379dc5 Update yaml files to latest version RELEASE.2021-04-06T23-11-00Z 2021-04-06 17:59:45 -07:00
Harshavardhana
8a9d15ace2 update prometheus metrics with failed_count 2021-04-04 09:52:37 -07:00
Poorna Krishnamoorthy
47c09a1e6f
Various improvements in replication (#11949) 
- collect real time replication metrics for prometheus.
- add pending_count, failed_count metric for total pending/failed replication operations.

- add API to get replication metrics

- add MRF worker to handle spill-over replication operations

- multiple issues found with replication
- fixes an issue when client sends a bucket
 name with `/` at the end from SetRemoteTarget
 API call make sure to trim the bucket name to 
 avoid any extra `/`.

- hold write locks in GetObjectNInfo during replication
  to ensure that object version stack is not overwritten
  while reading the content.

- add additional protection during WriteMetadata() to
  ensure that we always write a valid FileInfo{} and avoid
  ever writing empty FileInfo{} to the lowest layers.

Co-authored-by: Poorna Krishnamoorthy <poorna@minio.io>
Co-authored-by: Harshavardhana <harsha@minio.io>
 2021-04-03 09:03:42 -07:00
Klaus Post
2623338dc5
Inline small file data in xl.meta file (#11758) 2021-03-29 17:00:55 -07:00
Minio Trusted
91eb1fe2ef Update yaml files to latest version RELEASE.2021-03-26T00-00-41Z 2021-03-26 00:23:59 +00:00
Ritesh H Shukla
23b03dadb8
Add process uptime metric (#11844) 2021-03-20 21:23:27 -07:00
Minio Trusted
b379ca3bb0 Update yaml files to latest version RELEASE.2021-03-17T02-33-02Z 2021-03-17 02:56:28 +00:00
Ravind Kumar
980311fdfd
Fix STANDARD defaults, point to new docs site. (#11800) 2021-03-16 12:04:28 -07:00
Minio Trusted
afd346417d Update yaml files to latest version RELEASE.2021-03-12T00-00-47Z 2021-03-12 00:23:57 +00:00
Minio Trusted
9d118b372e Update yaml files to latest version RELEASE.2021-03-10T05-11-33Z 2021-03-10 05:34:48 +00:00
Minio Trusted
464fa08f2e Update yaml files to latest version RELEASE.2021-03-04T00-53-13Z 2021-03-04 01:15:49 +00:00
Harshavardhana
2c198ae7b6
fix: prometheus metrics disks_online count when disks are down (#11689) 
prometheus metrics was using total disks instead
of online disk count, when disks were down, this
PR fixes this and also adds a new metric for
total_disk_count
 2021-03-03 11:18:41 -08:00
Nitish Tiwari
bbd1244a88
Add support for mTLS for Audit log target (#11645) 2021-03-01 09:19:13 -08:00
Minio Trusted
e3f8830ab7 Update yaml files to latest version RELEASE.2021-03-01T04-20-55Z 2021-03-01 04:43:28 +00:00
cbows
c67d1bf120
add unauthenticated lookup-bind mode to LDAP identity (#11655) 
Closes #11646
 2021-02-28 12:57:31 -08:00
Harshavardhana
9171d6ef65
rename all references from crawl -> scanner (#11621) 2021-02-26 15:11:42 -08:00
Minio Trusted
b912e9ab41 Update yaml files to latest version RELEASE.2021-02-24T18-44-45Z 2021-02-24 19:08:36 +00:00
Minio Trusted
1b63291ee2 Update yaml files to latest version RELEASE.2021-02-23T20-05-01Z 2021-02-23 20:28:30 +00:00
Aditya Manthramurthy
02e7de6367
LDAP config: fix substitution variables (#11586) 
- In username search filter and username format variables we support %s for
replacing with the username.

- In group search filter we support %s for username and %d for the full DN of
the username.
 2021-02-22 13:20:36 -08:00
Harshavardhana
8cad407e0b
fix: Bring support for symlink on regular files on NAS (#11383) 
fixes #11203
 2021-02-20 00:30:12 -08:00
WangYuMu
c70240b893
fix incorrect values in sizing guide (#11583) 2021-02-19 10:05:04 -08:00
Harshavardhana
8ba2136e06 Update yaml files to latest version RELEASE.2021-02-19T04-38-02Z 2021-02-18 21:02:25 -08:00
Poorna Krishnamoorthy
8e8a792d9d
Allow delete marker replication from replica (#11566) 
in the case of active-active replication.

This PR also has the following changes:

- add docs on replication design
- fix corner case of completing versioned delete on a delete marker
  when the target is down and `mc rm --vid` is performed repeatedly. Instead
  the version should still be retained in the `PENDING|FAILED` state until
  replication sync completes.
- remove `s3:Replication:OperationCompletedReplication` and
   `s3:Replication:OperationFailedReplication` from ObjectCreated 
  events type
 2021-02-18 00:33:51 -08:00
Harshavardhana
ffea6fcf09
fix: rename crawler as scanner in config (#11549) 2021-02-17 12:04:11 -08:00
Minio Trusted
cfc8b92dff Update yaml files to latest version RELEASE.2021-02-14T04-01-33Z 2021-02-14 04:25:52 +00:00
Minio Trusted
0ef3e359d8 Update yaml files to latest version RELEASE.2021-02-11T08-23-43Z 2021-02-11 08:47:10 +00:00
Aditya Manthramurthy
466e95bb59
Return group DN instead of group name in LDAP STS (#11501) 
- Additionally, check if the user or their groups has a policy attached during
the STS call.

- Remove the group name attribute configuration value.
 2021-02-10 16:52:49 -08:00
Krishna Srinivas
876b79b8d8
read-health check endpoint returns success if cluster can serve read requests (#11310) 2021-02-09 01:00:44 -08:00
Sarasa Kisaragi
152d7cd95b
HDFS support keytab (#11473) 2021-02-07 17:29:47 -08:00
Minio Trusted
647a209c73 Update yaml files to latest version RELEASE.2021-02-07T01-31-02Z 2021-02-07 01:53:27 +00:00
Harshavardhana
9497dfd804 docs: add deprecation notice for federation 2021-02-04 17:18:37 -08:00
Bahram Aghaei
f2d49ec21a
Update ldap.md: add a link to ldap.go (#11409) 2021-02-02 15:47:04 -08:00
Ritesh H Shukla
c4848f9b4f
Add process start time to cluster metrics. (#11405) 2021-02-01 23:02:18 -08:00
swartz-k
8c663f93f7
fix: typo in chinese docs (#11401) 2021-02-01 18:42:58 -08:00
Minio Trusted
b4cb7edf85 Update yaml files to latest version RELEASE.2021-02-01T22-56-52Z 2021-02-01 23:28:23 +00:00
Andreas Auernhammer
33554651e9
crypto: deprecate native Hashicorp Vault support (#11352) 
This commit deprecates the native Hashicorp Vault
support and removes the legacy Vault documentation.

The native Hashicorp Vault documentation is marked as
outdated and deprecated for over a year now. We give
another 6 months before we start removing Hashicorp Vault
support and show a deprecation warning when a MinIO server
starts with a native Vault configuration.
 2021-01-29 17:55:37 -08:00
Minio Trusted
451d9057f3 Update yaml files to latest version RELEASE.2021-01-30T00-20-58Z 2021-01-30 00:45:11 +00:00
Harshavardhana
6717295e18 fix: rename audit log docs and datastructure 2021-01-26 13:39:55 -08:00
Anis Elleuch
00cff1aac5
audit: per object send pool number, set number and servers per operation (#11233) 2021-01-26 13:21:51 -08:00
Aditya Manthramurthy
5f51ef0b40
Add LDAP Lookup-Bind mode (#11318) 
This change allows the MinIO server to be configured with a special (read-only)
LDAP account to perform user DN lookups.

The following configuration parameters are added (along with corresponding
environment variables) to LDAP identity configuration (under `identity_ldap`):

- lookup_bind_dn / MINIO_IDENTITY_LDAP_LOOKUP_BIND_DN
- lookup_bind_password / MINIO_IDENTITY_LDAP_LOOKUP_BIND_PASSWORD
- user_dn_search_base_dn / MINIO_IDENTITY_LDAP_USER_DN_SEARCH_BASE_DN
- user_dn_search_filter / MINIO_IDENTITY_LDAP_USER_DN_SEARCH_FILTER

This lookup-bind account is a service account that is used to lookup the user's
DN from their username provided in the STS API. When configured, searching for
the user DN is enabled and configuration of the base DN and filter for search is
required. In this "lookup-bind" mode, the username format is not checked and must
not be specified. This feature is to support Active Directory setups where the
DN cannot be simply derived from the username.

When the lookup-bind is not configured, the old behavior is enabled: the minio
server performs LDAP lookups as the LDAP user making the STS API request and the
username format is checked and configuring it is required.
 2021-01-25 14:26:10 -08:00
Ritesh H Shukla
0bf2d84f96
update new metrics url docs (#11342) 2021-01-25 01:03:07 -08:00
Ritesh H Shukla
7575c24037
Add open FD and FD limit to cluster metrics (#11328) 2021-01-22 18:30:16 -08:00
Harshavardhana
c080f04e66 fix: prometheus metrics link typo update to latest 2021-01-22 01:53:23 -08:00
Ritesh H Shukla
b4add82bb6
Updated Prometheus metrics (#11141) 
* Add metrics for nodes online and offline
* Add cluster capacity metrics
* Introduce v2 metrics
 2021-01-18 20:35:38 -08:00
Aditya Manthramurthy
3163a660aa
Fix support for multiple LDAP user formats (#11276) 
Fixes support for using multiple base DNs for user search in the LDAP directory
allowing users from different subtrees in the LDAP hierarchy to request
credentials.

- The username in the produced credentials is now the full DN of the LDAP user
to disambiguate users in different base DNs.
 2021-01-17 21:54:32 -08:00
Harshavardhana
98f76008c7 fix: bucket lifecycle again to remove Days parameter 2021-01-17 01:50:56 -08:00
Harshavardhana
8da0b7cf03 fix: lifecycle documentation for DeleteMarker 2021-01-17 01:37:25 -08:00
Harshavardhana
f903cae6ff
Support variable server pools (#11256) 
Current implementation requires server pools to have
same erasure stripe sizes, to facilitate same SLA
and expectations.

This PR allows server pools to be variadic, i.e they
do not have to be same erasure stripe sizes - instead
they should have SLA for parity ratio.

If the parity ratio cannot be guaranteed by the new
server pool, the deployment is rejected i.e server
pool expansion is not allowed.
 2021-01-16 12:08:02 -08:00
Minio Trusted
40d59c1961 Update yaml files to latest version RELEASE.2021-01-16T02-19-44Z 2021-01-16 02:43:53 +00:00
Poorna Krishnamoorthy
7090bcc8e0
fix: doc links and delete replication permissions enforcement (#11285) 2021-01-15 15:22:55 -08:00
Harshavardhana
cc2d887e0e fix: whitespace and formatting in replication docs 2021-01-14 22:58:53 -08:00
Poorna Krishnamoorthy
c1b4b24236
Update replication docs (#11279) 2021-01-15 10:22:57 +05:30
Poorna Krishnamoorthy
feaf8dfb9a
Fix replication status reported on completion (#11273) 
Fixes: #11272
 2021-01-13 11:52:28 -08:00
Poorna Krishnamoorthy
b97d53b29c
fix remote target healthcheck (#11267) 2021-01-12 20:48:04 -08:00
Aditya Manthramurthy
00af9881b0
LDAP doc fix: remove repeated paragraph and add emphasis (#11266) 2021-01-12 15:44:31 -08:00
Poorna Krishnamoorthy
7824e19d20
Allow synchronous replication if enabled. (#11165) 
Synchronous replication can be enabled by setting the --sync
flag while adding a remote replication target.

This PR also adds proxying on GET/HEAD to another node in a
active-active replication setup in the event of a 404 on the current node.
 2021-01-11 22:36:51 -08:00
George Tsatsis
e8176fe978
Use -new during OpenSSL certificate generation (#11199) 
As per https://stackoverflow.com/a/3758443/8156177, OpenSSL expects a certificate via STDIN. 
`-new` will allow a new certificate to be generated instead.
 2021-01-11 02:24:50 -08:00
Minio Trusted
d9224fbc65 Update yaml files to latest version RELEASE.2021-01-08T21-18-21Z 2021-01-08 21:37:35 +00:00
Harshavardhana
76e2713ffe
fix: use buffers only when necessary for io.Copy() (#11229) 
Use separate sync.Pool for writes/reads

Avoid passing buffers for io.CopyBuffer()
if the writer or reader implement io.WriteTo or io.ReadFrom
respectively then its useless for sync.Pool to allocate
buffers on its own since that will be completely ignored
by the io.CopyBuffer Go implementation.

Improve this wherever we see this to be optimal.

This allows us to be more efficient on memory usage.
```
   385  // copyBuffer is the actual implementation of Copy and CopyBuffer.
   386  // if buf is nil, one is allocated.
   387  func copyBuffer(dst Writer, src Reader, buf []byte) (written int64, err error) {
   388  	// If the reader has a WriteTo method, use it to do the copy.
   389  	// Avoids an allocation and a copy.
   390  	if wt, ok := src.(WriterTo); ok {
   391  		return wt.WriteTo(dst)
   392  	}
   393  	// Similarly, if the writer has a ReadFrom method, use it to do the copy.
   394  	if rt, ok := dst.(ReaderFrom); ok {
   395  		return rt.ReadFrom(src)
   396  	}
```

From readahead package
```
// WriteTo writes data to w until there's no more data to write or when an error occurs.
// The return value n is the number of bytes written.
// Any error encountered during the write is also returned.
func (a *reader) WriteTo(w io.Writer) (n int64, err error) {
	if a.err != nil {
		return 0, a.err
	}
	n = 0
	for {
		err = a.fill()
		if err != nil {
			return n, err
		}
		n2, err := w.Write(a.cur.buffer())
		a.cur.inc(n2)
		n += int64(n2)
		if err != nil {
			return n, err
		}
```
 2021-01-06 09:36:55 -08:00
Harshavardhana
b5d291ea88
fix: rename remaining zone -> pool (#11231) 2021-01-06 09:35:47 -08:00
Klaus Post
eb9172eecb
Allow Compression + encryption (#11103) 2021-01-05 20:08:35 -08:00
Harshavardhana
cb0eaeaad8
feat: migrate to ROOT_USER/PASSWORD from ACCESS/SECRET_KEY (#11185) 2021-01-05 10:22:57 -08:00
Minio Trusted
f3f0041ad0 Update yaml files to latest version RELEASE.2021-01-05T05-22-38Z 2021-01-05 05:42:54 +00:00
Baptiste Mille-Mathias
c1f6ca6697
Fix caddy project url (#11198) 2020-12-31 09:44:07 -08:00
0xflotus
cde801282d
chore: enabled syntax highlighting in docs (#11182) 2020-12-29 17:38:28 -08:00
Ravind Kumar
6cf0008469
fix: docs typo in object lock docs (#11181) 2020-12-29 16:14:10 -08:00
Minio Trusted
7b0330a98c Update yaml files to latest version RELEASE.2020-12-29T23-29-29Z 2020-12-29 23:46:24 +00:00
Minio Trusted
d3c853a3be Update yaml files to latest version RELEASE.2020-12-26T01-35-54Z 2020-12-26 01:53:30 +00:00
Poorna Krishnamoorthy
7b8a456f68
Update lifecycle README docs (#11160) 
Removing reference to transition feature in docs as this
feature is being revamped to provide better extensibility
across different cloud targets.
 2020-12-23 19:56:55 -08:00
Harshavardhana
b43906f6ee fix: docs typos and keywords 2020-12-23 11:59:20 -08:00
Minio Trusted
bfb92a27b7 Update yaml files to latest version RELEASE.2020-12-23T02-24-12Z 2020-12-23 02:43:25 +00:00
Minio Trusted
386dd56856 Update yaml files to latest version RELEASE.2020-12-18T03-27-42Z 2020-12-18 03:45:13 +00:00
Minio Trusted
26624552be Update yaml files to latest version RELEASE.2020-12-16T05-05-17Z 2020-12-16 05:23:19 +00:00
Minio Trusted
31bf6f0c25 Update yaml files to latest version RELEASE.2020-12-12T08-39-07Z 2020-12-12 08:56:30 +00:00
Harshavardhana
14792cdbc6
docs: fix the metrics formatting (#11081) 2020-12-10 18:15:47 -08:00
Harshavardhana
97856bfebf
fix: grafana double counting for bucket usage, histrogram and objects (#11070) 2020-12-09 20:30:37 -08:00
Minio Trusted
83a67a1d21 Update yaml files to latest version RELEASE.2020-12-10T01-54-29Z 2020-12-10 02:12:01 +00:00
Nitish Tiwari
54d243cd98
fix: grafana dashboard calculating online nodes (#11041) 
Also use a generic name instead of diff names per revision
 2020-12-09 00:26:42 -08:00
Ritesh H Shukla
04848dfa1c
Add documentation for bucket replication related metrics (#11055) 2020-12-08 12:48:10 -08:00
Harshavardhana
4a564336fe Revert "Add metrics for nodes online and offline (#11050)" 
This reverts commit f60bbdf86b.
 2020-12-08 09:23:35 -08:00
Ritesh H Shukla
f60bbdf86b
Add metrics for nodes online and offline (#11050) 2020-12-08 01:06:27 -08:00
Harshavardhana
8c79f87f02
add dynamic config docs (#11048) 
Co-authored-by: Eco <41090896+eco-minio@users.noreply.github.com>
 2020-12-07 19:02:20 -08:00
Poorna Krishnamoorthy
f3beb1236a
Add cache usage, total capacity to prometheus metrics (#11026) 2020-12-07 16:35:11 -08:00
Poorna Krishnamoorthy
934bed47fa
Add transition event notification (#11047) 
This is a MinIO specific extension to allow monitoring of transition events.
 2020-12-07 13:53:28 -08:00
Nitish Tiwari
6ff12f5f01
Add the dashboard json file (#11028) 
This will allow users to contribute to the dashboard as needed.
 2020-12-04 16:27:41 -08:00
Nitish Tiwari
de9b64834e
fix: update grafana dashboard docs (#11023) 
Refer to the official Grafana dashboard
 2020-12-03 15:56:15 -08:00
Minio Trusted
d54cf77356 Update yaml files to latest version RELEASE.2020-12-03T05-49-24Z 2020-12-03 06:05:56 +00:00
Minio Trusted
1ab4d6a6aa Update yaml files to latest version RELEASE.2020-12-03T00-03-10Z 2020-12-03 00:24:16 +00:00
Anis Elleuch
fe11e9047d
deprecate CommonName from TLS docs (#11017) 
CommonName is not supported anymore in Go 1.15

fix the TLS documentation to use subjAltNames
 2020-12-02 10:18:39 -08:00
Sudarshan (Sid)
3ca046b408
Added set keyword in the command set to enable encryption on buckets (#11010) 2020-12-01 16:00:49 -08:00
Harshavardhana
4ec45753e6 rename server sets to server pools 2020-12-01 13:50:33 -08:00
Harshavardhana
790833f3b2 Revert "Support variable server sets (#10314)" 
This reverts commit aabf053d2f.
 2020-12-01 12:02:29 -08:00
Harshavardhana
aabf053d2f
Support variable server sets (#10314) 2020-11-25 16:28:47 -08:00
Minio Trusted
f839bb5a0a Update yaml files to latest version RELEASE.2020-11-25T22-36-25Z 2020-11-25 22:53:02 +00:00
Anis Elleuch
75a8e81f8f
azure: Specify different Azure storage in the shell env (#10943) 
AZURE_STORAGE_ACCOUNT and AZURE_STORAGE_KEY are used in 
azure CLI to specify the azure blob storage access & secret keys. With this commit, 
it is possible to set them if you want the gateway's own credentials to be
different from the Azure blob credentials.

Co-authored-by: Harshavardhana <harsha@minio.io>
 2020-11-23 16:45:56 -08:00
Poorna Krishnamoorthy
39f3d5493b
Show Delete replication status header (#10946) 
X-Minio-Replication-Delete-Status header shows the
status of the replication of a permanent delete of a version.

All GETs are disallowed and return 405 on this object version.
In the case of replicating delete markers.

X-Minio-Replication-DeleteMarker-Status shows the status 
of replication, and would similarly return 405.

Additionally, this PR adds reporting of delete marker event completion
and updates documentation
 2020-11-21 23:48:50 -08:00
Shireesh Anjal
7bc47a14cc
Rename OBD to Health (#10842) 
Also, Remove thread stats and openfds from the health report 
as we already have process stats and numfds
 2020-11-20 12:52:53 -08:00
Harshavardhana
9263be8cca
docs: fix missing event types in notifications (#10944) 2020-11-20 11:27:27 -08:00
Poorna Krishnamoorthy
1ebf6f146a Add support for ILM transition (#10565) 
This PR adds transition support for ILM
to transition data to another MinIO target
represented by a storage class ARN. Subsequent
GET or HEAD for that object will be streamed from
the transition tier. If PostRestoreObject API is
invoked, the transitioned object can be restored for
duration specified to the source cluster.
 2020-11-19 18:47:17 -08:00
Harshavardhana
9a34fd5c4a Revert "Revert "Add delete marker replication support (#10396)"" 
This reverts commit 267d7bf0a9.
 2020-11-19 18:43:58 -08:00
Minio Trusted
b9e3a8b5ac Update yaml files to latest version RELEASE.2020-11-19T23-48-16Z 2020-11-20 00:06:03 +00:00
Minio Trusted
0784a0c33a Update yaml files to latest version RELEASE.2020-11-13T20-10-18Z 2020-11-13 20:27:01 +00:00
Minio Trusted
3595cb1267 Update yaml files to latest version RELEASE.2020-11-12T22-33-34Z 2020-11-12 22:51:11 +00:00
Omar Alvarez
07859ef48b
docs: clarify notifications support for gateways (#10729) 
Co-authored-by: Harshavardhana <harsha@minio.io>
 2020-11-12 12:19:04 -08:00
Harshavardhana
267d7bf0a9 Revert "Add delete marker replication support (#10396)" 
This reverts commit 50c10a5087.

PR is moved to origin/dev branch
 2020-11-12 11:43:14 -08:00
Anton Melser
2c1e37197b
fix: bad example json for policy in replication docs (#10869) 2020-11-10 17:49:49 -08:00
Poorna Krishnamoorthy
50c10a5087
Add delete marker replication support (#10396) 
Delete marker replication is implemented for V2
configuration specified in AWS spec (though AWS
allows it only in the V1 configuration).

This PR also brings in a MinIO only extension of
replicating permanent deletes, i.e. deletes specifying
version id are replicated to target cluster.
 2020-11-10 15:24:14 -08:00
Minio Trusted
9f4ad873bc Update yaml files to latest version RELEASE.2020-11-10T21-02-24Z 2020-11-10 21:18:35 +00:00
Anis Elleuch
8e8ddf7233
doc: Add definition of 1KB and 1MB in prometheus (#10857) 2020-11-09 10:05:01 -08:00
Minio Trusted
21016265e5 Update yaml files to latest version RELEASE.2020-11-06T23-17-07Z 2020-11-06 23:34:19 +00:00
Minio Trusted
51222cc664 Update yaml files to latest version RELEASE.2020-10-28T08-16-50Z 2020-10-28 08:33:22 +00:00
Minio Trusted
ca77ee1c0e Update yaml files to latest version RELEASE.2020-10-27T04-03-55Z 2020-10-27 04:20:10 +00:00
Harshavardhana
4442382c16 update STS examples to use latest v7 APIs 2020-10-24 11:16:59 -07:00
Anis Elleuch
6fd088f448
docs: Update ILM doc with versioning features (#10714) 2020-10-20 09:23:27 -07:00
Minio Trusted
ba5215561f Update yaml files to latest version RELEASE.2020-10-18T21-54-12Z 2020-10-18 22:15:30 +00:00
Márk Sági-Kazár
4eb45c9a0f
fix: dex getting started guide URL (#10701) 2020-10-17 16:47:28 -07:00
Harshavardhana
ad726b49b4
rename zones to serverSets to avoid terminology conflict (#10679) 
we are bringing in availability zones, we should avoid
zones as per server expansion concept.
 2020-10-15 14:28:50 -07:00
Harshavardhana
9c042a503b
remove deprecate readiness from healthcheck docs (#10659) 2020-10-12 18:56:03 -07:00
Minio Trusted
4a678ad70f Update yaml files to latest version RELEASE.2020-10-12T21-53-21Z 2020-10-12 22:15:06 +00:00
Minio Trusted
071c004f8b Update yaml files to latest version RELEASE.2020-10-09T22-55-05Z 2020-10-09 23:16:30 +00:00
miraculli
1738eb24b1
fix: caching doc README.md missing high watermark (#10646) 2020-10-09 08:32:51 -07:00
Poorna Krishnamoorthy
01498a3e34
fix: add docs for new event types in notification (#10636) 2020-10-06 13:33:23 -07:00
Ravind Kumar
57f0176759
Update KES table to include additional supported KMS providers (#10631) 2020-10-06 11:09:43 -07:00
Minio Trusted
017954e7ea Update yaml files to latest version RELEASE.2020-10-03T02-19-42Z 2020-10-03 02:36:54 +00:00
Minio Trusted
511424a287 Update yaml files to latest version RELEASE.2020-09-26T03-44-56Z 2020-09-26 04:02:42 +00:00
Praveen raj Mani
b880796aef
Set the maximum open connections limit in PG and MySQL target configs (#10558) 
As the bulk/recursive delete will require multiple connections to open at an instance,
The default open connections limit will be reached which results in the following error

```FATAL:  sorry, too many clients already```

By setting the open connections to a reasonable value - `2`, We ensure that the max open connections
will not be exhausted and lie under bounds.

The queries are simple inserts/updates/deletes which is operational and sufficient with the
the maximum open connection limit is 2.

Fixes #10553

Allow user configuration for MaxOpenConnections
 2020-09-24 22:20:30 -07:00
poornas
e6ab4db6b8
Fix minimum replication workers started (#10560) 
This PR also fixes GetReplicationConfiguration permission
in web-handlers.go to use bucket as resource
 2020-09-24 12:25:41 -07:00
Espen Finnesand
df08fd1f03
Update Kubernetes Helm Readme.md (#10559) 
The link to the Minio Helm Chart repository was wrong.
 2020-09-24 08:38:46 -07:00
Minio Trusted
ac82f416a4 Update yaml files to latest version RELEASE.2020-09-23T19-18-30Z 2020-09-23 19:38:14 +00:00
飞雪无情
5fdf47b118
Add Chinese documentation for distributed design (#10549) 2020-09-23 09:26:47 -07:00
Harshavardhana
8b74a72b21
fix: rename READY deadline to CLUSTER deadline ENV (#10535) 2020-09-23 09:14:33 -07:00
poornas
a4006e23a0
Update replication docs to clarify permissions (#10536) 
Co-authored-by: Klaus Post <klauspost@gmail.com>
 2020-09-22 11:58:04 -07:00
Minio Trusted
d73c4f09f3 Update yaml files to latest version RELEASE.2020-09-21T22-31-59Z 2020-09-21 22:58:40 +00:00
飞雪无情
dd4a2d7419
update distributed zh_CN document. (#10531) 2020-09-21 10:54:56 -07:00
Derek Bender
3168e93730
fix typo in healthcheck README.md (#10518) 2020-09-18 09:52:37 -07:00
Aleksey Pogibelev
6f45e303f5
fix tls flag (#10487) 2020-09-18 15:35:54 +05:30
飞雪无情
dff37aa33d
update disk caching zh_CN document. (#10472) 
add disk caching DESIGN zh_CN document.
 2020-09-18 00:07:48 -07:00
Minio Trusted
48d2c03250 Update yaml files to latest version RELEASE.2020-09-17T04-49-20Z 2020-09-17 05:07:58 +00:00
Minio Trusted
b2536476c9 Update yaml files to latest version RELEASE.2020-09-16T04-22-35Z 2020-09-16 04:41:00 +00:00
Harshavardhana
9d95937018 update KMS docs indicating deprecation of AUTO_ENCRYPTION env 2020-09-13 16:23:28 -07:00
Minio Trusted
eb3ded420e Update yaml files to latest version RELEASE.2020-09-10T22-02-45Z 2020-09-10 22:22:28 +00:00
飞雪无情
a966ccd17d
Support nginx LB with docker-compose example (#10434) 2020-09-10 09:19:17 -07:00
Li Yi
1dce6918c2
fix sysctl.sh quotes which are incompatible with sysctl (#10446) 2020-09-09 17:29:23 -07:00
Minio Trusted
6019628f7d Update yaml files to latest version RELEASE.2020-09-08T23-05-18Z 2020-09-08 23:25:40 +00:00
飞雪无情
a694ba93d9
update kernel tuning zh_CN document (#10433) 2020-09-07 19:21:52 -07:00
Minio Trusted
a9aaea0d67 Update yaml files to latest version RELEASE.2020-09-05T07-14-49Z 2020-09-05 07:32:10 +00:00
飞雪无情
746f1585eb
add debugging zh_CN document. (#10401) 2020-09-04 13:03:46 -07:00
Harshavardhana
b4e3956e69
update KES docs to talk about 'mc encrypt' command (#10400) 
add a deprecation notice for KMS_AUTO_ENCRYPTION
 2020-09-03 12:43:45 -07:00
Krishna Srinivas
c51229493b
azure docs remove default chunk size example (#10406) 2020-09-03 01:31:56 -07:00
Klaus Post
650dccfa9e
cache: Only start at high watermark (#10403) 
Currently, cache purges are triggered as soon as the low watermark is exceeded.
To reduce IO this should only be done when reaching the high watermark.
This simplifies checks and reduces all calls for a GC to go through
`dcache.diskSpaceAvailable(size)`. While a comment claims that 
`dcache.triggerGC <- struct{}{}` was non-blocking I don't see how 
that was possible. Instead, we add a 1 size to the queue channel 
and use channel  semantics to avoid blocking when a GC has 
already been requested.

`bytesToClear` now takes the high watermark into account to it will 
not request any bytes to be cleared until that is reached.
 2020-09-02 17:48:44 -07:00
Minio Trusted
d08b4b147d Update yaml files to latest version RELEASE.2020-09-02T18-19-50Z 2020-09-02 18:40:57 +00:00
Harshavardhana
4b6585d249
support 'ldap:user' variable replacement properly (#10391) 
also update `ldap.go` examples with latest
minio-go changes

Fixes #10367
 2020-09-01 12:26:22 +05:30
飞雪无情
cb61e50b51
since its an internal RPC API we do not need to document it. (#10382) 2020-08-31 10:20:07 -07:00
unlimitedbits
cd380251b3
Support custom paths for secret files in docker-entrypoint.sh (#10344) 2020-08-28 14:04:29 -07:00
Eco
92cd1eed45
Clarify zone example (#10374) 2020-08-28 14:03:29 -07:00
Harshavardhana
db32a24cb6 update kernel tuning docs to indicate sane values 2020-08-28 12:07:55 -07:00
poornas
73a6b4ea11
fix typo in replication docs (#10366) 2020-08-27 12:54:23 -07:00
Minio Trusted
c1b88c17cc Update yaml files to latest version RELEASE.2020-08-27T05-16-20Z 2020-08-27 05:36:01 +00:00
Minio Trusted
5a974fb10c Update yaml files to latest version RELEASE.2020-08-26T00-00-49Z 2020-08-26 00:19:30 +00:00
Minio Trusted
7d50a0cfea Update yaml files to latest version RELEASE.2020-08-25T00-21-20Z 2020-08-25 00:44:39 +00:00
飞雪无情
2a8e40f19f
add chroot zh_CN document (#10337) 2020-08-24 13:58:19 -07:00
飞雪无情
21e8440423
add compression zh_CN document (#10330) 2020-08-23 22:07:36 -07:00
Praveen raj Mani
d0c910a6f3
Support https and basic-auth for elasticsearch notification target (#10332) 2020-08-23 09:43:48 -07:00
飞雪无情
0ff246653b
CN bucket policy document that is no longer needed can be removed, (#10310) 
which is consistent with the English document.
 2020-08-22 11:05:21 -07:00
飞雪无情
113bcbdb78
update minio limit zh_CN document (#10320) 2020-08-22 11:04:31 -07:00
飞雪无情
331c517a5b
translate bucket versioning design document to zh_CN (#10300) 2020-08-20 13:18:51 -07:00
飞雪无情
2eb5f934d8
add bucket versioning zh_CN document (#10281) 2020-08-18 13:10:26 -07:00
Minio Trusted
b43d376a87 Update yaml files to latest version RELEASE.2020-08-18T19-41-00Z 2020-08-18 20:04:22 +00:00
Klaus Post
adca28801d
feat: disable Parquet by default (breaking change) (#9920) 
I have built a fuzz test and it crashes heavily in seconds and will OOM shortly after.
It seems like supporting Parquet is basically a completely open way to crash the 
server if you can upload a file and run s3 select on it.

Until Parquet is more hardened it is DISABLED by default since hostile 
crafted input can easily crash the server.

If you are in a controlled environment where it is safe to assume no hostile
content can be uploaded to your cluster you can safely enable Parquet.

To enable Parquet set the environment variable `MINIO_API_SELECT_PARQUET=on`
while starting the MinIO server.

Furthermore, we guard parquet by recover functions.
 2020-08-18 10:23:28 -07:00
Harshavardhana
ede86845e5
docs: Add policy variables for resource and conditions (#10278) 
Bonus fix adds LDAP policy variable and clarifies the
usage of policy variables for temporary credentials.

fixes #10197
 2020-08-17 17:39:55 -07:00
Minio Trusted
670724184c Update yaml files to latest version RELEASE.2020-08-16T18-39-38Z 2020-08-16 18:56:41 +00:00
飞雪无情
01a2ccc52f
add bucket retention zh_CN document (#10264) 2020-08-14 22:12:03 -07:00
Minio Trusted
1c865dd119 Update yaml files to latest version RELEASE.2020-08-13T02-39-50Z 2020-08-13 02:57:25 +00:00
飞雪无情
6914b2c99d
Add bucket replication zh_CN document (#10243) 2020-08-11 11:33:01 -07:00
飞雪无情
b4b7a18497
add bucket quota zh_CN document (#10232) 2020-08-10 10:55:34 -07:00
飞雪无情
a49e3647b6
add bucket lifecycle zh_CN document (#10231) 2020-08-09 02:29:25 -07:00
Minio Trusted
954e17c3d0 Update yaml files to latest version RELEASE.2020-08-08T04-50-06Z 2020-08-07 22:32:27 -07:00
Ritesh H Shukla
8049184dcc
fix: documentation changes in replication docs (#10209) 2020-08-07 13:30:52 -07:00
飞雪无情
19c4f3082b
update and improve bucket notifications zh_CN document (#10224) 2020-08-07 12:36:27 -07:00
Minio Trusted
6d64aab420 Update yaml files to latest version RELEASE.2020-08-07T01-23-07Z 2020-08-07 01:43:14 +00:00
poornas
adcaa6f9de
fix: Change ListBucketTargets handler (#10217) 
to list all targets across a tenant.
Also fixing some validations.
 2020-08-06 17:10:21 -07:00
飞雪无情
ce129efa09
update and improve config,throttle zh_CN document (#10214) 2020-08-06 11:43:30 -07:00
poornas
121164db56
fix: relax some replication validations (#10210) 
Also inherit storage class from source object
if replication configuration does not have a storage
class specified for destination bucket.
 2020-08-05 20:01:20 -07:00
Minio Trusted
195f95196e Update yaml files to latest version RELEASE.2020-08-05T21-34-13Z 2020-08-05 21:58:17 +00:00
poornas
3acc0ebb81
fix: Change service name in Arn for replication (#10205) 2020-08-05 00:43:18 -07:00
Minio Trusted
1cdaced8b6 Update yaml files to latest version RELEASE.2020-08-04T23-10-51Z 2020-08-04 23:46:06 +00:00
Harshavardhana
d90ab904e7
fix: multi-tenant deployment guide to use ellipses format (#10192) 2020-08-04 08:15:37 -07:00
飞雪无情
0cf4539fe8
fix: update and improve zh_CN documentation (#10174) 2020-08-01 11:52:46 -07:00
Minio Trusted
2174a22835 Update yaml files to latest version RELEASE.2020-07-31T03-39-05Z 2020-07-31 03:58:04 +00:00
poornas
a8dd7b3eda
Refactor replication target management. (#10154) 
Generalize replication target management so
that remote targets for a bucket can be
managed with ARNs. `mc admin bucket remote`
command will be used to manage targets.
 2020-07-30 19:55:22 -07:00
Harshavardhana
27266f8a54
fix: if OPA set do not enforce policy claim (#10149) 2020-07-28 11:47:57 -07:00
Minio Trusted
98691f75bc Update yaml files to latest version RELEASE.2020-07-27T18-37-02Z 2020-07-27 18:57:10 +00:00
Ritesh H Shukla
416ec316bd
Add clarification for minimum drives per server (#10130) 2020-07-24 22:09:01 -07:00
Minio Trusted
8a23988711 Update yaml files to latest version RELEASE.2020-07-24T22-43-05Z 2020-07-24 23:00:32 +00:00
Harshavardhana
0b5d1bc91d
fix: bucket replication docs (#10104) 
* fix: bucket replication docs

* Update docs/bucket/replication/README.md

Co-authored-by: kannappanr <30541348+kannappanr@users.noreply.github.com>

Co-authored-by: kannappanr <30541348+kannappanr@users.noreply.github.com>
 2020-07-21 22:19:30 -07:00
poornas
c43da3005a
Add support for server side bucket replication (#9882) 2020-07-21 17:49:56 -07:00
Minio Trusted
ca4c15bc63 Update yaml files to latest version RELEASE.2020-07-22T00-26-33Z 2020-07-22 00:44:03 +00:00
Harshavardhana
ec06089eda
fix: re-implement cluster healthcheck (#10101) 2020-07-20 18:31:22 -07:00
Minio Trusted
38f60b3c1d Update yaml files to latest version RELEASE.2020-07-20T02-25-16Z 2020-07-20 02:41:52 +00:00
Harshavardhana
30104cb12b docs: fix veeam document formatting 2020-07-18 18:38:12 -07:00
Minio Trusted
68aaa5bbc3 Update yaml files to latest version RELEASE.2020-07-18T18-48-16Z 2020-07-18 19:04:01 +00:00
Eco
5e8392c8ef
Update Veeam integration doc with immutability references (#10067) 2020-07-16 17:16:53 -07:00
Harshavardhana
07eb24b775
add absolute path for images (#10056) 2020-07-16 00:06:14 -07:00
Harshavardhana
cd849bc2ff
update STS docs with new values (#10055) 
Co-authored-by: Poorna <poornas@users.noreply.github.com>
 2020-07-15 14:36:14 -07:00
Harshavardhana
ec91fa55db
docs: Add more STS docs with dex and python example (#10047) 2020-07-15 17:25:55 +05:30
Minio Trusted
a97ce3c96e Update yaml files to latest version RELEASE.2020-07-14T19-14-30Z 2020-07-14 19:32:27 +00:00
Harshavardhana
369a876ebe
fix: handle array policies in JWT claim (#10041) 
PR #10014 was not complete as only handled
policy claims partially.
 2020-07-14 10:26:47 -07:00
Anis Elleuch
778e9c864f
Move dependency from minio-go v6 to v7 (#10042) 2020-07-14 09:38:05 -07:00
Minio Trusted
8d8f28eae4 Update yaml files to latest version RELEASE.2020-07-13T18-09-56Z 2020-07-13 18:29:35 +00:00
Minio Trusted
3939c6f6e7 Update yaml files to latest version RELEASE.2020-07-12T19-14-17Z 2020-07-12 19:48:42 +00:00
Minio Trusted
c2fdf73491 Update yaml files to latest version RELEASE.2020-07-11T21-14-23Z 2020-07-11 21:30:53 +00:00
Harshavardhana
3520e946a2 fix: versioning docs add more examples 2020-07-11 00:57:46 -07:00
Harshavardhana
f38adc1865 cleanup security overview guide 2020-07-11 00:34:56 -07:00
Harshavardhana
d5ff1c8e3b fix docs image urls to be absolute path 2020-07-11 00:27:30 -07:00
Minio Trusted
ad7417bc50 Update yaml files to latest version RELEASE.2020-07-11T06-07-16Z 2020-07-11 06:28:08 +00:00
Harshavardhana
ba756cf366
fix: extract array type for policy claim if present (#10014) 2020-07-10 14:48:44 -07:00
Harshavardhana
2e8fc6ebfe
cleanup STS docs (#10003) 2020-07-10 09:07:12 -07:00
Nitish Tiwari
30c251efd3
Add Grafana dashboard (#10000) 2020-07-09 12:01:58 -07:00
Ravind Kumar
ee20ebe07a Remove dead link related to DC/OS Deployment Guide (#9996) 2020-07-08 17:38:20 -07:00
Harshavardhana
60417950c7
fix: the versioning/object lock documentation appropriately (#9988) 
- Move the bucket level features into `docs/bucket` directory
- fix issue template and simplify some of them
 2020-07-08 08:44:43 -07:00
Harshavardhana
38eef5ce4c
fix: documentation fixes for docker ENV settings (#9975) 
- update CREDITS file
- fix markdown links
- talk a bit more about upgrades
 2020-07-06 06:42:34 -07:00
Minio Trusted
04de19c870 Update yaml files to latest version RELEASE.2020-07-02T00-15-09Z 2020-07-01 17:36:52 -07:00
Harshavardhana
c0ac25bfff
fix: readiness needs to be like liveness (#9941) 
Readiness as no reasoning to be cluster scope
because that is not how the k8s networking works
for pods, all the pods to a deployment are not
sharing the network in a singleton. Instead they
are run as local scopes to themselves, with
readiness failures the pod is potentially taken
out of the network to be resolvable - this
affects the distributed setup in myriad of
different ways.

Instead readiness should behave like liveness
with local scope alone, and should be a dummy
implementation.

This PR all the startup times and overal k8s
startup time dramatically improves.

Added another handler called as `/minio/health/cluster`
to understand the cluster scope health.
 2020-06-30 11:28:27 -07:00
poornas
55a3b071ea
Allow optionally to disable range caching. (#9908) 
The default behavior is to cache each range requested
to cache drive. Add an environment variable
`MINIO_RANGE_CACHE` - when set to off, it disables
range caching and instead downloads entire object
in the background.

Fixes #9870
 2020-06-29 13:25:29 -07:00
Harshavardhana
28a1a17187
docs update versioning images (#9925) 2020-06-27 10:27:23 -07:00
Eco
a5e879fbe4
doc: typo fix in versioning README (#9926) 
requets -> requests
 2020-06-27 05:06:31 -07:00
Praveen raj Mani
b1705599e1
Fix config leaks and deprecate file-based config setters in NAS gateway (#9884) 
This PR has the following changes

- Removing duplicate lookupConfigs() calls.
- Deprecate admin config APIs for NAS gateways. This will avoid repeated reloads of the config from the disk.
- WatchConfigNASDisk will be removed
- Migration guide for NAS gateways users to migrate to ENV settings.

NOTE: THIS PR HAS A BREAKING CHANGE

Fixes #9875

Co-authored-by: Harshavardhana <harsha@minio.io>
 2020-06-25 15:59:28 +05:30
Ivan Martinez-Ortiz
969b2d2110
Updates the usage documentation of OpenID custom scopes (#9902) 2020-06-24 07:49:09 -07:00
Harshavardhana
21058c34d0
add some description of xl.meta (#9901) 2020-06-22 17:27:54 -07:00
Klaus Post
691dc04fac
Add neater metadata printer (#9898) 
* Just read files from args (more than 1 now supported)
* Pretty print by default. `-ndjson` will disable.
* Check header.
* Support stdin as '-'
* Don't just ignore errors.
 2020-06-22 13:20:22 -07:00
Harshavardhana
e92434c2e7
fix: support client customized scopes for OpenID (#9880) 
Fixes #9238
 2020-06-22 12:08:50 -07:00
Minio Trusted
0fff9f9fa6 Update yaml files to latest version RELEASE.2020-06-22T03-12-50Z 2020-06-21 20:26:38 -07:00
Benjamin Sodenkamp
0becf7f03f
Added note to help differentiate MINIO creds from S3 rotating creds (#9887) 2020-06-21 12:39:03 -07:00
Minio Trusted
72743d1590 Update yaml files to latest version RELEASE.2020-06-18T02-23-35Z 2020-06-17 19:33:28 -07:00
Harshavardhana
f9aa239973
fix: export prometheus metrics for cache GC triggers (#9815) 
Bonus change to use channel to serialize triggers,
instead of using atomic variables. More efficient
mechanism for synchronization.

Co-authored-by: Nitish Tiwari <nitish@minio.io>
 2020-06-15 09:05:35 -07:00
Minio Trusted
1cd5d7942f Update yaml files to latest version RELEASE.2020-06-14T18-32-17Z 2020-06-14 11:41:18 -07:00
Harshavardhana
4915433bd2
Support bucket versioning (#9377) 
- Implement a new xl.json 2.0.0 format to support,
  this moves the entire marshaling logic to POSIX
  layer, top layer always consumes a common FileInfo
  construct which simplifies the metadata reads.
- Implement list object versions
- Migrate to siphash from crchash for new deployments
  for object placements.

Fixes #2111
 2020-06-12 20:04:01 -07:00
Dan Garrick
a53cb236bb
Add enable="on" to set compression command (#9811) 2020-06-11 18:14:13 -07:00
Minio Trusted
761992ca89 Update yaml files to latest version RELEASE.2020-06-12T00-06-19Z 2020-06-12 00:14:09 +00:00
Ritesh H Shukla
920b863955
Fix formatting and wording in distributed docs (#9784) 2020-06-08 18:03:15 -07:00
Harshavardhana
c1382c09d9
add reference to distributed doc for clarity (#9783) 2020-06-08 12:30:42 -07:00
Praveen raj Mani
2ce2e88adf
Support mTLS Authentication in Webhooks (#9777) 2020-06-08 05:55:44 -07:00
Harshavardhana
5e529a1c96
simplify context timeout for readiness (#9772) 
additionally also add CORS support to restrict
for specific origin, adds a new config and
updated the documentation as well
 2020-06-04 14:58:34 -07:00
Minio Trusted
7fee96e9de Update yaml files to latest version RELEASE.2020-06-03T22-13-49Z 2020-06-03 22:22:13 +00:00
Minio Trusted
4b68d69188 Update yaml files to latest version RELEASE.2020-06-01T17-28-03Z 2020-06-01 17:36:17 +00:00
Harshavardhana
4301a33371 update docs to mention quota not supported on FS 2020-05-29 07:57:45 -07:00
Minio Trusted
f1a03a4ee8 Update yaml files to latest version RELEASE.2020-05-29T14-08-49Z 2020-05-29 14:17:50 +00:00
Minio Trusted
3780ec699f Update yaml files to latest version RELEASE.2020-05-28T23-29-21Z 2020-05-28 23:38:19 +00:00
poornas
e5ecd20d44
add docs for caching drives with docker (#9712) 
fixes: #9688
 2020-05-27 09:49:09 -07:00
Harshavardhana
53aaa5d2a5
Export bucket usage counts as part of bucket metrics (#9710) 
Bonus fixes in quota enforcement to use the
new datastructure and use timedValue to cache
a value/reload automatically avoids one less
global variable.
 2020-05-27 06:45:43 -07:00
Justin Clift
5e15b0b844
fix misspelling of Certbot (#9705) 2020-05-26 08:56:50 -07:00
Harshavardhana
6e0575a53d
Revert "Disable crawler in FS/NAS gateway mode (#9695)" (#9702) 
This reverts commit eba423bb9d.

Additionally also address the FS crawler to properly
calculate the sizes for encrypted/compressed content.
 2020-05-25 11:32:53 -07:00
Harshavardhana
eba423bb9d
Disable crawler in FS/NAS gateway mode (#9695) 
No one really uses FS for large scale accounting
usage, neither we crawl in NAS gateway mode. It is
worthwhile to simply disable this feature as its
not useful for anyone.

Bonus disable bucket quota ops as well in, FS
and gateway mode
 2020-05-25 00:17:52 -07:00
Kianoosh Hooshmand
1868c7016d
remove additional back-ticks in compression docs (#9685) 2020-05-24 18:34:47 -07:00
谭九鼎
209a3ee7af
fix backtick markdown in chinese docs (#9691) 2020-05-24 09:27:47 -07:00
Harshavardhana
e45c90060f
remove references for deprecated dockerfiles and deployment styles (#9675) 2020-05-22 08:40:59 -07:00
Andreas Auernhammer
2164984d2b
fix KMS quickstart guide layout (#9658) 
This commit fixes a layout issue w.r.t. the KMS
Quickstart guide. The problem seems to be caused
by docs server not converting the markdown into html
as expected.

This commit fixes this by converting the ordered list
into subsections.
 2020-05-20 13:55:54 -07:00
Praveen raj Mani
0cc2ed04f5
humanize timeToFirstByte and timeToResponse upto nanoseconds (#9641) 2020-05-19 18:34:02 -07:00
Andreas Auernhammer
b11adfa5cd
simplify the KMS guide and remove unnecessary sections (#9629) 
This commit simplifies the KMS configuration guide by
adding a get started section that uses our KES play instance
at `https://play.min.io:7373`.

Further, it removes sections that we don't recommend for production
anyways (MASTER_KEY).
 2020-05-19 18:33:11 -07:00
poornas
011a2c0b78
Add docs for bucket quota feature (#9503) 
This PR also adds a check to not enforce
bucket quota for server-side metadata copy
of an object onto itself.
 2020-05-16 19:27:33 -07:00
Minio Trusted
0f8df3d340 Update yaml files to latest version RELEASE.2020-05-16T01-33-21Z 2020-05-16 01:40:31 +00:00
Andreas Auernhammer
c19ece6921
update KMS guide to reflect latest KES changes (#9591) 
This commit updates the two client env. variables:
```
KES_CLIENT_TLS_KEY_FILE
KES_CLIENT_TLS_CERT_FILE
```

The KES CLI client expects the client key and certificate
as `KES_CLIENT_KEY` resp. `KES_CLIENT_CERT`.
 2020-05-14 14:02:43 -07:00
Harshavardhana
2d735144b9 fix: distributed docs image path 2020-05-11 09:33:55 -07:00
Harshavardhana
0113035237
update distributed setup guide (#9566) 2020-05-11 09:19:10 -07:00
Minio Trusted
5bf3eeaa77 Update yaml files to latest version RELEASE.2020-05-08T02-40-49Z 2020-05-08 02:49:15 +00:00
Harshavardhana
9dda1fd624
Remove B2 gateway implementation (#9547) 
S3 is now natively supported by B2 cloud storage provider
there is no reason to use specialized gateway for B2 anymore,
our current S3 gateway with caching would work with B2.

Resolves #8584
 2020-05-07 19:00:30 -07:00
remche
0674c0075e
add LDAP StartTLS support (#9472) 2020-05-07 15:08:33 -07:00
Harshavardhana
53f4c0fdc0
fix: deprecate old settings and elaborate on tuning (#9542) 
fixes #9538
 2020-05-07 11:08:57 -07:00
Minio Trusted
24f20eb1bd Update yaml files to latest version RELEASE.2020-05-06T23-23-25Z 2020-05-06 23:32:41 +00:00
Minio Trusted
1242dd951a Update yaml files to latest version RELEASE.2020-05-01T22-19-14Z 2020-05-01 22:28:00 +00:00
Andreas Auernhammer
d1c8e9f31b
update KMS guide to work with latest KES changes (#9498) 
This commit updates the KMS guide to reflect the
latest changes in KES. Based on internal design
meetings we made some adjustments to the overall
KES configuration.
This commit ensures that the KMS guide contains
a working KES demo-setup with Vault.
 2020-05-01 12:36:30 -07:00
Arthur Lutz
da87188ff8
fix: tls doc markdown title (#9487) 2020-04-29 12:28:45 -07:00
Minio Trusted
b37a02cddf Update yaml files to latest version RELEASE.2020-04-28T23-56-56Z 2020-04-29 00:05:12 +00:00
Harshavardhana
60d415bb8a
deprecate/remove global WORM mode (#9436) 
global WORM mode is a complex piece for which
the time has passed, with the advent of S3 compatible
object locking and retention implementation global
WORM is sort of deprecated, this has been mentioned
in our documentation for some time, now the time
has come for this to go.
 2020-04-24 16:37:05 -07:00
Tim Hughes
e8160c9fae
fix: same endpoint for NewLDAPIdentity & NewWithCredentials (#9433) 
Also enables use of https endpoints

Fixes  #9431
 2020-04-24 10:44:45 +05:30
Boaz
ac5061df2c
fix: make azure gateway chunk size configurable (#9292) 2020-04-23 02:04:13 -07:00
Tim Hughes
cddb2714ef
documentation: fix group search filter (#9420) 2020-04-22 22:29:17 -07:00
Minio Trusted
d7d9cac20b Update yaml files to latest version RELEASE.2020-04-23T00-58-49Z 2020-04-23 01:07:52 +00:00
Minio Trusted
2d1ea86fc6 Update yaml files to latest version RELEASE.2020-04-22T00-11-12Z 2020-04-22 00:19:12 +00:00
Harshavardhana
69ee28a082
remove OSS gateway due to lack of licensing (#9390) 
OSS go sdk lacks licensing terms in their
repository, and there has been no activity

On the issue here https://github.com/aliyun/aliyun-oss-go-sdk/issues/245

This PR is to ensure we remove any dependency code which
lacks explicit license file in their repo.
 2020-04-18 22:12:51 -07:00
sreenivas alapati
d02deff3d7
fixed typo in KMS documentation (#9384) 2020-04-18 18:09:25 -07:00
Praveen raj Mani
c79358c67e
notification queue limit has no maxLimit (#9380) 
New value defaults to 100K events by default,
but users can tune this value upto any value
they seem necessary.

* increase the limit to maxint64 while validating
 2020-04-18 01:20:56 -07:00
Eco
7758524703
Add documentation for using MinIO with Veeam (#9355) 2020-04-16 17:36:14 -07:00
Minio Trusted
c50ae1fdbe Update yaml files to latest version RELEASE.2020-04-15T19-42-18Z 2020-04-15 20:00:16 +00:00
Minio Trusted
78f2183e70 Update yaml files to latest version RELEASE.2020-04-15T00-39-01Z 2020-04-15 00:46:50 +00:00
Anis Elleuch
8a94aebdb8
config: Add api requests max & deadline configs (#9273) 
Add two new configuration entries, api.requests-max and
api.requests-deadline which have the same role of
MINIO_API_REQUESTS_MAX and MINIO_API_REQUESTS_DEADLINE.
 2020-04-14 12:46:37 -07:00
Harshavardhana
a9d401ac10
fix: update docs to mention erasure guide (#9339) 2020-04-14 11:38:14 +05:30
Harshavardhana
cc9b63eb51
add deprecation docs for PostgresSQL/MySQL targets (#9333) 2020-04-13 12:13:33 -07:00
Harshavardhana
4314ee1670
fix: remove unusued PerfInfoHandler code (#9328) 
- Removes PerfInfo admin API as its not OBDInfo
- Keep the drive path without the metaBucket in OBD
  global latency map.
- Remove all the unused code related to PerfInfo API
- Do not redefined global mib,gib constants use
  humanize.MiByte and humanize.GiByte instead always
 2020-04-12 19:37:09 -07:00
Harshavardhana
3184205519
fix: config to support keys with special values (#9304) 
This PR adds context-based `k=v` splits based
on the sub-system which was obtained, if the
keys are not provided an error will be thrown
during parsing, if keys are provided with wrong
values an error will be thrown. Keys can now
have values which are of a much more complex
form such as `k="v=v"` or `k=" v = v"`
and other variations.

additionally, deprecate unnecessary postgres/mysql
configuration styles, support only

- connection_string for Postgres
- dsn_string for MySQL

All other parameters are removed.
 2020-04-09 21:45:17 -07:00
Minio Trusted
7c919329e8 Update yaml files to latest version RELEASE.2020-04-10T03-34-42Z 2020-04-10 03:47:00 +00:00
ebozduman
a78731a3ba
Adds info on policy for STS authentication using web-id (#9289) 2020-04-08 10:34:43 -07:00
kumy
f4e779c964
Fix typo in LDAP STS guide (#9294) 2020-04-08 08:58:03 -07:00
Minio Trusted
05cda35b14 Update yaml files to latest version RELEASE.2020-04-04T05-39-31Z 2020-04-04 05:48:22 +00:00
Minio Trusted
c6e62b9175 Update yaml files to latest version RELEASE.2020-04-02T21-34-49Z 2020-04-02 21:44:04 +00:00
poornas
336460f67e
fix: gateway_s3_bytes_sent metric for all API methods (#9242) 
Co-authored-by: Harshavardhana <harsha@minio.io>
 2020-04-01 12:52:31 -07:00
Harshavardhana
30707659b5
[feature] allow for an odd number of erasure packs (#9221) 
Too many deployments come up with an odd number
of hosts or drives, to facilitate even distribution
among those setups allow for odd and prime numbers
based packs.
 2020-03-31 09:32:16 -07:00
ebozduman
fdda5f98c6
Makes mandatory dsn_string parameter optional (#8931) 2020-03-28 22:20:02 -07:00
Ingmar Runge
fa4d627b57
B2 gateway S3 compat: return MD5 hash as ETag from PutObject (#9183) 
- B2 does actually return an MD5 hash for newly uploaded objects
  so we can use it to provide better compatibility with S3 client
  libraries that assume the ETag is the MD5 hash such as boto.
- depends on change in blazer library.
- new behaviour is only enabled if MinIO's --compat mode is active.
- behaviour for multipart uploads is unchanged (works fine as is).
 2020-03-28 13:59:55 -07:00
Sidhartha Mani
0c80bf45d0
Implement oboard diagnostics admin API (#9024) 
- Implement a graph algorithm to test network bandwidth from every 
  node to every other node
- Saturate any network bandwidth adaptively, accounting for slow 
  and fast network capacity
- Implement parallel drive OBD tests
- Implement a paging mechanism for OBD test to provide periodic updates to client
- Implement Sys, Process, Host, Mem OBD Infos
 2020-03-26 21:07:39 -07:00
Robert Thomas
2777956581
Improve YAML download links listed in K8s doc (#9213) 2020-03-26 11:17:00 -07:00
Minio Trusted
2196fd9cd5 Update yaml files to latest version RELEASE.2020-03-25T07-03-04Z 2020-03-25 07:11:33 +00:00
Nitish Tiwari
6b984410d5
Add support for self-healing related metrics in Prometheus (#9079) 
Fixes #8988

Co-authored-by: Anis Elleuch <vadmeste@users.noreply.github.com>
Co-authored-by: Harshavardhana <harsha@minio.io>
 2020-03-24 22:40:45 -07:00
Harshavardhana
38cf263409 fix: docs remove goreportcard, its deprecated 2020-03-24 14:51:06 -07:00
Harshavardhana
6f6a2214fc
Add rate limiter for S3 API layer (#9196) 
- total number of S3 API calls per server
- maximum wait duration for any S3 API call

This implementation is primarily meant for situations
where HDDs are not capable enough to handle the incoming
workload and there is no way to throttle the client.

This feature allows MinIO server to throttle itself
such that we do not overwhelm the HDDs.
 2020-03-24 12:43:40 -07:00
Harshavardhana
ea18e51f4d
Support multiple LDAP OU's, smAccountName support (#9139) 
Fixes #8532
 2020-03-21 22:47:26 -07:00
Stephen N
1ffa983a9d
added support for SASL/SCRAM on Kafka bucket notifications. (#9168) 
fixes #9167
 2020-03-20 11:10:27 -07:00
Nitish Tiwari
ecf1566266
Add an option to allow plaintext connection to LDAP/AD Server (#9151) 2020-03-19 19:20:51 -07:00
Minio Trusted
c5b87f93dd Update yaml files to latest version RELEASE.2020-03-19T21-49-00Z 2020-03-19 21:57:16 +00:00
Harshavardhana
b1a2169dcc
fix: data usage crawler env handling, usage-cache.bin location (#9163) 
canonicalize the ENVs such that we can bring these ENVs 
as part of the config values, as a subsequent change.

- fix location of per bucket usage to `.minio.sys/buckets/<bucket_name>/usage-cache.bin`
- fix location of the overall usage in `json` at `.minio.sys/buckets/.usage.json`
  (avoid conflicts with a bucket named `usage.json` )
- fix location of the overall usage in `msgp` at `.minio.sys/buckets/.usage.bin`
  (avoid conflicts with a bucket named `usage.bin`
 2020-03-19 09:47:47 -07:00
yeungc
7ec904d67b fix: wording and update content of chinese docs (#9140) 2020-03-16 10:04:16 -07:00
gzur
3fea1d5e35
Align STS web-identity code snippet to documentation (minio#9114) (#9130) 2020-03-13 22:58:53 -07:00
Harshavardhana
3ca9f5ffa3 Update yaml files to latest version RELEASE.2020-03-14T02-21-58Z 2020-03-13 20:05:27 -07:00
Andreas Auernhammer
ed4bd20a7c
change ca path env. var in KMS guide (#9125) 
This commit fixes the env. variable in the
KMS guide used to specify the CA certificates
for the KES server.

Before the env. variable `MINIO_KMS_KES_CAPATH` has
been used - which works in non-containerized environments
due to how MinIO merges the config file and environment
variables. In containerized environments (e.g. docker)
this does not work and trying to specify `MINIO_KMS_KES_CAPATH`
instead of `MINIO_KMS_KES_CA_PATH` eventually leads to MinIO not
trusting the certificate presented by the kes server.

See: cfd12914e1/cmd/crypto/config.go (L186)
 2020-03-12 07:47:40 -07:00
Minio Trusted
667f42515a Update yaml files to latest version RELEASE.2020-03-09T18-26-53Z 2020-03-09 18:38:07 +00:00
kumy
b809c84338
fix: notifications doc elaborate env values for targets (#9103) 2020-03-08 18:33:43 -07:00
Minio Trusted
52873ac3a3 Update yaml files to latest version RELEASE.2020-03-06T22-23-56Z 2020-03-06 22:32:45 +00:00
Minio Trusted
9ad1c2d07d Update yaml files to latest version RELEASE.2020-03-05T01-04-19Z 2020-03-05 01:10:15 +00:00
Minio Trusted
bb942c7311 Update yaml files to latest version RELEASE.2020-02-27T00-23-05Z 2020-02-27 00:32:47 +00:00
poornas
224b4f13b8
Add cache eviction low and high watermarks (#8958) 
To allow better control the cache eviction process.

Introduce MINIO_CACHE_WATERMARK_LOW and 
MINIO_CACHE_WATERMARK_HIGH env. variables to specify 
when to stop/start cache eviction process. 

Deprecate MINIO_CACHE_EXPIRY environment variable. Cache 
gc sweeps at 30 minute intervals whenever high watermark is
reached to clear least recently accessed entries in the cache
until sufficient space is cleared to reach the low watermark.

Garbage collection uses an adaptive file scoring approach based
on last access time, with greater weights assigned to larger
objects and those with more hits to find the candidates for eviction.

Thanks to @klauspost for this file scoring algorithm

Co-authored-by: Klaus Post <klauspost@minio.io>
 2020-02-23 19:03:39 +05:30
Harshavardhana
852fb320f7
Add all supported scopes from discovery doc (#9015) 
Fixes #9010
 2020-02-21 08:06:05 +05:30
Minio Trusted
8fb37a8417 Update yaml files to latest version RELEASE.2020-02-20T22-51-23Z 2020-02-20 23:00:50 +00:00
Praveen raj Mani
1b427ddb69
Support for Kafka version in the config (#9001) 
Add a field for the Kafka version in the config. The user can explicitly 
set the version of the Kafka cluster.

Fixes #8768
 2020-02-17 07:56:33 +05:30
Nitish Tiwari
63be4709b7
Add metrics support for Azure & GCS Gateway (#8954) 
We added support for caching and S3 related metrics in #8591. As
a continuation, it would be helpful to add support for Azure & GCS
gateway related metrics as well.
 2020-02-11 21:08:01 +05:30
Minio Trusted
f749a9bf0e Update yaml files to latest version RELEASE.2020-02-07T23-28-16Z 2020-02-07 23:35:04 +00:00
Minio Trusted
07edb7c7f8 Update yaml files to latest version RELEASE.2020-02-07T04-56-50Z 2020-02-07 05:03:28 +00:00
poornas
45d725c0a3
Fix docs link for vault legacy (#8948) 2020-02-06 10:15:58 +05:30
Andreas Auernhammer
f91c072f61
re-write the KMS get started guide (#8936) 
This commit updates the KMS getting started guide
and replaces the legacy MinIO<-->Vault setup with a
MinIO<-->KES<-->Vault setup.

Therefore, add some architecture ASCII diagrams and
provide a step-by-step guide to setup Vault, KES and
MinIO such that MinIO can encrypt objects with KES +
Vault.

The legacy Vault guide has been moved to `./vault-legacy.md`.

Co-authored-by: Harshavardhana <harsha@minio.io>
 2020-02-05 12:38:47 +05:30
poornas
278a165674
Allow caching based on a configurable number of hits. (#8891) 
Co-authored-by: Harshavardhana <harsha@minio.io>
 2020-02-04 09:10:01 +05:30
ebozduman
a05af9bb98
Fixes several typos and clarifies some explanations in MySQL doc. (#8908) 2020-02-03 18:55:05 -08:00
Li Yi
0414f01b6e doc: match code on sysctl -w vm.swappiness=1 (#8894) 2020-01-27 16:19:15 -08:00
poornas
2232e095d5 Make admin permissions more granular for admin handlers. (#8888) 2020-01-26 20:47:52 -06:00
Minio Trusted
aae337f5b5 Update yaml files to latest version RELEASE.2020-01-25T02-50-51Z 2020-01-25 02:57:23 +00:00
Anis Elleuch
3011692d93 doc: Add missing ``` in compression documentation (#8872) 2020-01-21 15:50:54 +05:30
Harshavardhana
86252ec7e1
fix: document _ENABLE for all notification targets (#8864) 
Fixes #8863
 2020-01-20 16:48:19 -08:00
bijialin
c260182412 zh_CN docs: add Expanding existing distributed setup (#8862) 2020-01-20 08:51:09 -08:00
Nitish Tiwari
61c17c8933 Add ObjectTagging Support (#8754) 
This PR adds support for AWS S3 ObjectTagging API as explained here
https://docs.aws.amazon.com/AmazonS3/latest/dev/object-tagging.html
 2020-01-20 08:45:59 -08:00
kannappanr
005ebbb9b2 Typo: Add missing '<' character (#8845) 
Fixes #8833
 2020-01-17 13:31:46 -08:00
poornas
60e60f68dd Add support for object locking with legal hold. (#8634) 2020-01-16 15:41:56 -08:00
Minio Trusted
ba758361b3 Update yaml files to latest version RELEASE.2020-01-16T22-40-29Z 2020-01-16 22:47:41 +00:00
Minio Trusted
080e0c2323 Update yaml files to latest version RELEASE.2020-01-16T03-05-44Z 2020-01-16 03:11:40 +00:00
ebozduman
d74818b227 Typo in MySQL notification doc (#8807) 2020-01-13 13:06:42 -08:00
Harshavardhana
3f20011862 docs: update how to query the keys and args (#8797) 
remove link to missing config.json.sample
 2020-01-11 15:13:13 -08:00
Etienne
5f34b5e6a5 Update README.md (#8788) 2020-01-10 21:08:09 -08:00
Kevin Humphreys
656146b699 doc: Prometheus metrics name fix (#8774) 
changed docs to reflect proper Prometheus metrics
 2020-01-09 18:36:58 -08:00