MyFavMirrors/minio
1
0
Fork 0
mirror of https://github.com/minio/minio.git synced 2024-12-27 15:45:55 -05:00
Code Issues Packages Projects Releases Wiki Activity
11,190 Commits 4 Branches 506 Tags 189 MiB
99c1a642a4
Commit Graph

1454 Commits

Author SHA1 Message Date
Poorna
ed3418c046
Refactor replication resync to be an active process (#14266) 
When resync is triggered, walk the bucket namespace and
resync objects that are unreplicated. This PR also adds
an API to report resync progress.
 2022-02-10 10:16:52 -08:00
Harshavardhana
a2230868e0 remove all stale old docs about 2018 releases 2022-02-10 09:54:27 -08:00
Krishnan Parthasarathi
0ee2933234
Export tier metrics via Prometheus (#13413) 
e.g
```
minio_cluster_ilm_transitioned_bytes{server="minio3:9000",tier="S3TIER-1"} 1.36317772e+08
minio_cluster_ilm_transitioned_bytes{server="minio3:9000",tier="S3TIER-2"} 2892
minio_cluster_ilm_transitioned_bytes{server="minio3:9000",tier="STANDARD"}
1.3631488e+08

minio_cluster_ilm_transitioned_objects{server="minio3:9000",tier="S3TIER-1"} 1
minio_cluster_ilm_transitioned_objects{server="minio3:9000",tier="S3TIER-2"} 0
minio_cluster_ilm_transitioned_objects{server="minio3:9000",tier="STANDARD"} 1

minio_cluster_ilm_transitioned_versions{server="minio3:9000",tier="S3TIER-1"} 3
minio_cluster_ilm_transitioned_versions{server="minio3:9000",tier="S3TIER-2"} 2
minio_cluster_ilm_transitioned_versions{server="minio3:9000",tier="STANDARD"} 1
```
 2022-02-08 12:45:28 -08:00
Minio Trusted
2480c66857 Update yaml files to latest version RELEASE.2022-02-07T08-17-33Z 2022-02-07 09:19:24 +00:00
Minio Trusted
570670be8c Update yaml files to latest version RELEASE.2022-02-05T04-40-59Z 2022-02-05 18:33:46 +00:00
Minio Trusted
9417fd933e Update yaml files to latest version RELEASE.2022-02-01T18-00-14Z 2022-02-01 23:37:07 +00:00
Harshavardhana
74faed166a
Add quota usage as part of prometheus metrics (#14222) 
Bonus: pass caller context when needed to all bucket metadata handling calls.
 2022-01-31 17:27:43 -08:00
Harshavardhana
dbd05d6e82
remove FIFO bucket quota, use ILM expiration instead (#14206) 2022-01-31 11:07:04 -08:00
Minio Trusted
c7df5fb119 Update yaml files to latest version RELEASE.2022-01-28T02-28-16Z 2022-01-28 05:12:35 +00:00
Minio Trusted
a66071099c Update yaml files to latest version RELEASE.2022-01-27T03-53-02Z 2022-01-27 04:47:45 +00:00
Aditya Manthramurthy
7dfa565d00
Identity LDAP: Allow multiple search base DNs (#14191) 
This change allows the MinIO server to lookup users in different directory
sub-trees by allowing specification of multiple search bases separated by
semicolons.
 2022-01-26 15:05:59 -08:00
Bingchang Chen
affe51cb19
docs: add casdoor as identity provider (#14185) 2022-01-26 09:49:10 -08:00
Minio Trusted
f556a72fe2 Update yaml files to latest version RELEASE.2022-01-25T19-56-04Z 2022-01-25 20:42:46 +00:00
Harshavardhana
cd7a5cab8a update docs for Decommission 2022-01-25 11:56:04 -08:00
Poorna
295730408b
Disallow delete replication for tag based rules (#14167) 2022-01-24 15:22:20 -08:00
Harshavardhana
f30afa4956
docs: add decommission docs about pool removal (#14159) 2022-01-24 09:47:06 -08:00
Harshavardhana
8fb4ae916c update decommission docs 2022-01-21 18:34:06 -08:00
Klaus Post
07df9eecda
Update xl.meta docs (#14150) 2022-01-21 08:47:46 -08:00
Harshavardhana
9d588319dd
support site replication to replicate IAM users,groups (#14128) 
- Site replication was missing replicating users,
  groups when an empty site was added.

- Add site replication for groups and users when they
  are disabled and enabled.

- Add support for replicating bucket quota config.
 2022-01-19 20:02:24 -08:00
Poorna
54a98773f8
fix: replication of tag removal (#14056) 
Currently tag removal leaves replication state as `PENDING` 
because the `HEAD` api returns just a tag count but not the 
actual tags, and this is treated as a no-op
 2022-01-10 19:06:10 -08:00
Harshavardhana
76b21de0c6
feat: decommission feature for pools (#14012) 
```
λ mc admin decommission start alias/ http://minio{1...2}/data{1...4}
```

```
λ mc admin decommission status alias/
┌─────┬─────────────────────────────────┬──────────────────────────────────┬────────┐
│ ID  │ Pools                           │ Capacity                         │ Status │
│ 1st │ http://minio{1...2}/data{1...4} │ 439 GiB (used) / 561 GiB (total) │ Active │
│ 2nd │ http://minio{3...4}/data{1...4} │ 329 GiB (used) / 421 GiB (total) │ Active │
└─────┴─────────────────────────────────┴──────────────────────────────────┴────────┘
```

```
λ mc admin decommission status alias/ http://minio{1...2}/data{1...4}
Progress: ===================> [1GiB/sec] [15%] [4TiB/50TiB]
Time Remaining: 4 hours (started 3 hours ago)
```

```
λ mc admin decommission status alias/ http://minio{1...2}/data{1...4}
ERROR: This pool is not scheduled for decommissioning currently.
```

```
λ mc admin decommission cancel alias/
┌─────┬─────────────────────────────────┬──────────────────────────────────┬──────────┐
│ ID  │ Pools                           │ Capacity                         │ Status   │
│ 1st │ http://minio{1...2}/data{1...4} │ 439 GiB (used) / 561 GiB (total) │ Draining │
└─────┴─────────────────────────────────┴──────────────────────────────────┴──────────┘
```

> NOTE: Canceled decommission will not make the pool active again, since we might have
> Potentially partial duplicate content on the other pools, to avoid this scenario be
> very sure to start decommissioning as a planned activity.

```
λ mc admin decommission cancel alias/ http://minio{1...2}/data{1...4}
┌─────┬─────────────────────────────────┬──────────────────────────────────┬────────────────────┐
│ ID  │ Pools                           │ Capacity                         │ Status             │
│ 1st │ http://minio{1...2}/data{1...4} │ 439 GiB (used) / 561 GiB (total) │ Draining(Canceled) │
└─────┴─────────────────────────────────┴──────────────────────────────────┴────────────────────┘
```
 2022-01-10 09:07:49 -08:00
Minio Trusted
f394313fee Update yaml files to latest version RELEASE.2022-01-08T03-11-54Z 2022-01-08 05:53:59 +00:00
Harshavardhana
b7c5e45fff
heal: isObjectDangling should return false when it cannot decide (#14053) 
In a multi-pool setup when disks are coming up, or in a single pool
setup let's say with 100's of erasure sets with a slow network.

It's possible when healing is attempted on `.minio.sys/config`
folder, it can lead to healing unexpectedly deleting some policy
files as dangling due to a mistake in understanding when `isObjectDangling`
is considered to be 'true'.

This issue happened in commit 30135eed86
when we assumed the validMeta with empty ErasureInfo is considered
to be fully dangling. This implementation issue gets exposed when
the server is starting up.

This is most easily seen with multiple-pool setups because of the
disconnected fashion pools that come up. The decision to purge the
object as dangling is taken incorrectly prior to the correct state
being achieved on each pool, when the corresponding drive let's say
returns 'errDiskNotFound', a 'delete' is triggered. At this point,
the 'drive' comes online because this is part of the startup sequence
as drives can come online lazily.

This kind of situation exists because we allow (totalDisks/2) number
of drives to be online when the server is being restarted.

Implementation made an incorrect assumption here leading to policies
getting deleted.

Added tests to capture the implementation requirements.
 2022-01-07 19:11:54 -08:00
Aditya Manthramurthy
0a224654c2
fix: progagation of service accounts for site replication (#14054) 
- Only non-root-owned service accounts are replicated for now.
- Add integration tests for OIDC with site replication
 2022-01-07 17:41:43 -08:00
Minio Trusted
e420a1de4d Update yaml files to latest version RELEASE.2022-01-07T01-53-23Z 2022-01-07 05:57:36 +00:00
Aditya Manthramurthy
62dc0f7698
Update site replication docs (#14044) 
- Now OpenID external IDP is also supported for site replication.

- IAM users/groups are also replicated
 2022-01-06 17:53:23 -08:00
Minio Trusted
f4c03e56b8 Update yaml files to latest version RELEASE.2022-01-04T07-41-07Z 2022-01-04 23:01:29 +00:00
Minio Trusted
5dd40b9377 Update yaml files to latest version RELEASE.2022-01-03T18-22-58Z 2022-01-04 00:12:07 +00:00
Harshavardhana
a60ac7ca17
fix: audit log to support object names in multipleObjectNames() handler (#14017) 2022-01-03 01:28:52 -08:00
Minio Trusted
c9e49f4366 Update yaml files to latest version RELEASE.2021-12-29T06-49-06Z 2021-12-29 07:38:12 +00:00
Yoann Guillerme
3e28af1723
docs: update TLS doc use -keyout instead of -key (#14001) 2021-12-28 12:51:38 -08:00
Minio Trusted
6aa0574a53 Update yaml files to latest version RELEASE.2021-12-27T07-23-18Z 2021-12-27 19:52:13 +00:00
Harshavardhana
7e3a7d7044
add healing for invalid shards by skipping the blocks (#13978) 
Built on top of #13945, now we need to simply skip the
shards and its automated.
 2021-12-23 23:01:46 -08:00
Harshavardhana
416977436e rename MINIO_CACHE_.._MASTER_KEY to MINIO_CACHE_.._SECRET_KEY 
fixes #13975
 2021-12-22 12:11:07 -08:00
Harshavardhana
0e3037631f
skip inconsistent shards if possible (#13945) 
data shards were wrong due to a healing bug
reported in #13803 mainly with unaligned object
sizes.

This PR is an attempt to automatically avoid
these shards, with available information about
the `xl.meta` and actually disk mtime.
 2021-12-21 10:08:26 -08:00
Minio Trusted
f3602d7d08 Update yaml files to latest version RELEASE.2021-12-20T22-07-16Z 2021-12-20 23:13:48 +00:00
Harshavardhana
0b21734571 update helm chart link 2021-12-20 13:20:59 -08:00
Minio Trusted
cfd66ab8c3 Update yaml files to latest version RELEASE.2021-12-18T04-42-33Z 2021-12-19 22:19:30 +00:00
Krishnan Parthasarathi
3ddb501190
Add docs for NewerNoncurrentVersions (#13944) 2021-12-17 19:32:29 -08:00
Harshavardhana
4fa250a6a1
add hash-set debugging currently supports SIPMOD (#13911) 
fixes a bug in s3-check-md5 for single part
uploaded multipart objects.
 2021-12-15 12:07:15 -08:00
Harshavardhana
88ad742da0
fix: error handling cases in site-replication (#13901) 
- Allow proper SRError to be propagated to
  handlers and converted appropriately.

- Make sure to enable object locking on buckets
  when requested in MakeBucketHook.

- When DNSConfig is enabled attempt to delete it
  first before deleting buckets locally.
 2021-12-14 14:09:57 -08:00
fpaupier
40dbe243d9
update: keycloak binding documentation (#13894) 2021-12-13 18:23:16 -08:00
Harshavardhana
109c927dad docs/debug: log any corruption and continue debug tool 2021-12-13 17:45:34 -08:00
Minio Trusted
878d368cea Update yaml files to latest version RELEASE.2021-12-10T23-03-39Z 2021-12-11 00:02:59 -08:00
Klaus Post
518612492c
xl-meta: Add header titles (#13880) 
Add type for headers and create custom marshal to make 
it easier to read. Group headers and metadata.

Restore functionality that will read `xl.meta` in the current dir with no params.

Before:
```
{
  "Headers": [
    [
      "8M04bTiYRDmEMQGeAsk1yg==",
      1639150471630100400,
      "rLD1Rw==",
      1,
      6
    ],
  ],
    "Versions": [
    {
      "Type": 1,
      "V2Obj": {
        "CSumAlgo": 1,
        "DDir": "oC1Xpg4tRfW03g8o8w7Bzg==",
        "EcAlgo": 1,
        "EcBSize": 1048576,
        "EcDist": [
          7,
          8,
          1,
          2,
          3,
          4,
          5,
          6
        ],
        "EcIndex": 1,
        "EcM": 4,
        "EcN": 4,
        "ID": "8M04bTiYRDmEMQGeAsk1yg==",
        "MTime": 1639150471630100400,
        "MetaSys": {
          "x-minio-internal-inline-data": "dHJ1ZQ=="
        },
        "MetaUsr": {
          "content-type": "application/octet-stream",
          "etag": "b8252c86fad2d8937300aa92b467a3aa"
        },
        "PartASizes": [
          1000
        ],
        "PartETags": null,
        "PartNums": [
          1
        ],
        "PartSizes": [
          1000
        ],
        "Size": 1000
      }
    }
  ]
}
```

After:
```
{
  "Versions": [
    {
      "Header": {
        "Flags": 6,
        "ModTime": "2021-12-10T16:34:31.6301004+01:00",
        "Signature": "acb0f547",
        "Type": 1,
        "VersionID": "f0cd386d389844398431019e02c935ca"
      },
      "Idx": 0,
      "Metadata": {
        "Type": 1,
        "V2Obj": {
          "CSumAlgo": 1,
          "DDir": "oC1Xpg4tRfW03g8o8w7Bzg==",
          "EcAlgo": 1,
          "EcBSize": 1048576,
          "EcDist": [
            7,
            8,
            1,
            2,
            3,
            4,
            5,
            6
          ],
          "EcIndex": 1,
          "EcM": 4,
          "EcN": 4,
          "ID": "8M04bTiYRDmEMQGeAsk1yg==",
          "MTime": 1639150471630100400,
          "MetaSys": {
            "x-minio-internal-inline-data": "dHJ1ZQ=="
          },
          "MetaUsr": {
            "content-type": "application/octet-stream",
            "etag": "b8252c86fad2d8937300aa92b467a3aa"
          },
          "PartASizes": [
            1000
          ],
          "PartETags": null,
          "PartNums": [
            1
          ],
          "PartSizes": [
            1000
          ],
          "Size": 1000
        }
      }
    }
  ]
}
```
 2021-12-10 15:03:25 -08:00
Ravind Kumar
c76f86fdbd
Clarify example for Standalone Docker instructions (#13879) 
Closes #13868 

Also points users to the web documentation for docker installation.
 2021-12-10 09:43:02 -08:00
Minio Trusted
4be8023408 Update yaml files to latest version RELEASE.2021-12-09T06-19-41Z 2021-12-09 08:40:46 +00:00
Harshavardhana
dcff6c996d
fix: do not list delete-marked objects (#13864) 
delete marked objects should not be considered
for listing when listing is delimited, this issue
as introduced in PR #13804 which was mainly to
address listing of directories in listing when
delimited.

This PR fixes this properly and adds tests to
ensure that we behave in accordance with how
an S3 API behaves for ListObjects() without
versions.
 2021-12-08 17:34:52 -08:00
Harshavardhana
e82a5c5c54
fix: site replication issues and add tests (#13861) 
- deleting policies was deleting all LDAP
  user mapping, this was a regression introduced
  in #13567

- deleting of policies is properly sent across
  all sites.

- remove unexpected errors instead embed the real
  errors as part of the 500 error response.
 2021-12-08 11:50:15 -08:00
Harshavardhana
92fdcafb66
add verification tests for ETag on replicated content (#13857) 2021-12-07 10:08:26 -08:00
Harshavardhana
be34fc9134
fix: kms-id header should have arn:aws:kms: prefix (#13833) 
arn:aws:kms: is a must for KMS keyID.
 2021-12-06 00:39:32 -08:00
Harshavardhana
20c60aae68 Update hotfix documentation and container building 2021-12-02 17:52:46 -08:00
Harshavardhana
f840080e5b
cleanup site-replication docs (#13812) 2021-12-02 13:27:01 -08:00
Harshavardhana
ffe9acfe4a docs: Add a markdown documentation on hotfix branches and process 2021-12-01 01:00:18 -08:00
Poorna K
9ec197f2e8
Add support for adding new site(s) to site replication (#13696) 
Currently, the new site is expected to be empty
 2021-11-30 13:16:37 -08:00
Aditya Manthramurthy
4c0f48c548
Add role ARN support for OIDC identity provider (#13651) 
- Allows setting a role policy parameter when configuring OIDC provider

- When role policy is set, the server prints a role ARN usable in STS API requests

- The given role policy is applied to STS API requests when the roleARN parameter is provided.

- Service accounts for role policy are also possible and work as expected.
 2021-11-26 19:22:40 -08:00
Aditya Manthramurthy
4ce6d35e30
Add new site config sub-system intended to replace region (#13672) 
- New sub-system has "region" and "name" fields.

- `region` subsystem is marked as deprecated, however still works, unless the
new region parameter under `site` is set - in this case, the region subsystem is
ignored. `region` subsystem is hidden from top-level help (i.e. from `mc admin
config set myminio`), but appears when specifically requested (i.e. with `mc
admin config set myminio region`).

- MINIO_REGION, MINIO_REGION_NAME are supported as legacy environment variables for server region.

- Adds MINIO_SITE_REGION as the current environment variable to configure the
server region and MINIO_SITE_NAME for the site name.
 2021-11-25 13:06:25 -08:00
Minio Trusted
a5db4ca092 Update yaml files to latest version RELEASE.2021-11-24T23-19-33Z 2021-11-25 07:39:00 +00:00
chrisbecke
ef0b8367b5
Update minio-overview.json data source panel (#13730) 
Add missing datasource in `Healing` panel.
 2021-11-23 09:01:07 -08:00
Krishnan Parthasarathi
3da9ee15d3
Add MaxNoncurrentVersions to NoncurrentExpiration action (#13580) 
This unit allows users to limit the maximum number of noncurrent 
versions of an object.

To enable this rule you need the following *ilm.json*
```
cat >> ilm.json <<EOF
{
    "Rules": [
        {
            "ID": "test-max-noncurrent",
            "Status": "Enabled",
            "Filter": {
                "Prefix": "user-uploads/"
            },
            "NoncurrentVersionExpiration": {
                "MaxNoncurrentVersions": 5
            }
        }
    ]
}
EOF
mc ilm import myminio/mybucket < ilm.json
```
 2021-11-19 17:54:10 -08:00
Mani
7b82411e6f
change the unit of measurement from TB to TiB (#13686) 2021-11-18 20:06:37 -08:00
Harshavardhana
79b3a1fe4e remove object torrent, AWS S3 removed support for torrent API 2021-11-18 12:21:48 -08:00
Klaus Post
faf013ec84
Improve performance on multiple versions (#13573) 
Existing:

```go
type xlMetaV2 struct {
    Versions []xlMetaV2Version `json:"Versions" msg:"Versions"`
}
```

Serialized as regular MessagePack.

```go
//msgp:tuple xlMetaV2VersionHeader
type xlMetaV2VersionHeader struct {
	VersionID [16]byte
	ModTime   int64
	Type      VersionType
	Flags     xlFlags
}
```

Serialize as streaming MessagePack, format:

```
int(headerVersion)
int(xlmetaVersion)
int(nVersions)
for each version {
    binary blob, xlMetaV2VersionHeader, serialized
    binary blob, xlMetaV2Version, serialized.
}
```

xlMetaV2VersionHeader is <= 30 bytes serialized. Deserialized struct 
can easily be reused and does not contain pointers, so efficient as a 
slice (single allocation)

This allows quickly parsing everything as slices of bytes (no copy).

Versions are always *saved* sorted by modTime, newest *first*. 
No more need to sort on load.

* Allows checking if a version exists.
* Allows reading single version without unmarshal all.
* Allows reading latest version of type without unmarshal all.
* Allows reading latest version without unmarshal of all.
* Allows checking if the latest is deleteMarker by reading first entry.
* Allows adding/updating/deleting a version with only header deserialization.
* Reduces allocations on conversion to FileInfo(s).
 2021-11-18 12:15:22 -08:00
Ashish Kumar Sinha
3d2bc15e9a
Add grafana json file for replication metrics (#13678) 2021-11-17 14:49:46 -08:00
Harshavardhana
661b263e77
add gocritic/ruleguard checks back again, cleanup code. (#13665) 
- remove some duplicated code
- reported a bug, separately fixed in #13664
- using strings.ReplaceAll() when needed
- using filepath.ToSlash() use when needed
- remove all non-Go style comments from the codebase

Co-authored-by: Aditya Manthramurthy <donatello@users.noreply.github.com>
 2021-11-16 09:28:29 -08:00
Harshavardhana
82d73f387d
add tool to read healing.bin for debugging (#13650) 2021-11-12 16:18:53 -08:00
Harshavardhana
5acc8c0134
add multi-site replication tests (#13631) 2021-11-10 18:18:09 -08:00
Poorna K
03725dc015
Default multipart caching to writethrough (#13613) 
when `MINIO_CACHE_COMMIT` is set.

- `writeback` caching applies only to single 
uploads. When cache commit mode is 
`writeback`, default multipart caching to be
synchronous.

- Add writethrough caching for single uploads
 2021-11-10 08:12:03 -08:00
Minio Trusted
edf1f4233b Update yaml files to latest version RELEASE.2021-11-09T03-21-45Z 2021-11-09 04:51:05 +00:00
Harshavardhana
5aef492b4c update disk-caching design guide 2021-11-07 01:21:34 -08:00
Harshavardhana
5d7ed8ff7d update S3 gateway limitation docs 2021-11-06 23:24:48 -07:00
Minio Trusted
cbb0828ab8 Update yaml files to latest version RELEASE.2021-11-05T09-16-26Z 2021-11-05 10:03:56 +00:00
Minio Trusted
9f3c151c3c Update yaml files to latest version RELEASE.2021-11-03T03-36-36Z 2021-11-03 06:48:34 +00:00
Harshavardhana
e106070640 update docs to mention the expected behavior for requests_max 
fixes #13561
 2021-11-02 14:10:21 -07:00
jandres - moscardo
1aa08f594d
Update README.md prometheus (#13514) 
Modify the doc to warn users about Prometheus sending `domain:port`
 2021-11-02 12:27:30 -07:00
Poorna K
15dcacc1fc
Add support for caching multipart in writethrough mode (#13507) 2021-11-01 08:11:58 -07:00
Harshavardhana
3c1220adca add tests for default governance replication 2021-10-30 08:57:59 -07:00
Harshavardhana
2af5445309 update 3-site replication tests 2021-10-29 22:09:55 -07:00
Harshavardhana
abb1916bda
update list objects limit to match S3 spec 2021-10-28 18:21:51 -07:00
Minio Trusted
8e29ae8c44 Update yaml files to latest version RELEASE.2021-10-27T16-29-42Z 2021-10-28 02:45:22 +00:00
Minio Trusted
0c76fb57f2 Update yaml files to latest version RELEASE.2021-10-23T03-28-24Z 2021-10-23 07:46:29 +00:00
Harshavardhana
52c5f6e152 remove extraneous whitespaces 2021-10-21 14:43:13 -07:00
Aditya Manthramurthy
26ca59859f
update LDAP doc (#13492) 
- clarify the login flow
- add some sections on configuration
- minor fixes to improve readability
 2021-10-21 14:41:32 -07:00
Klaus Post
1642867136
Add documentation for debugging tools (#13484) 
Move `xl-meta` so it can be installed out-of-repo with a single command.
 2021-10-20 10:12:46 -07:00
Harshavardhana
bedf739d16 update required IAM policies 2021-10-13 12:28:53 -07:00
Minio Trusted
6299e42aa9 Update yaml files to latest version RELEASE.2021-10-13T00-23-17Z 2021-10-13 01:14:23 +00:00
Harshavardhana
9ea45399ce
fix: enable AssumeRoleWithCertificate API only when asked (#13410) 
This is a breaking change but we need to do this to avoid
issues discussed in #13409 based on discussions from #13371

fixes #13371
fixes #13409
 2021-10-11 14:23:51 -07:00
Minio Trusted
e0db822a9b Update yaml files to latest version RELEASE.2021-10-10T16-53-30Z 2021-10-10 21:24:01 +00:00
Minio Trusted
c49ebaaf1a Update yaml files to latest version RELEASE.2021-10-08T23-58-24Z 2021-10-09 00:50:16 +00:00
Harshavardhana
d57b57bddc
feat: Add RX/TX to audit logging (#13382) 
add additional values for audit logging
 2021-10-07 19:03:46 -07:00
Aditya Manthramurthy
f81a188ef6
update site replication doc (#13377) 
make pre-requisites clear
 2021-10-07 15:21:01 -07:00
Minio Trusted
4ce6830a7b Update yaml files to latest version RELEASE.2021-10-06T23-36-31Z 2021-10-07 04:13:46 +00:00
Aditya Manthramurthy
3a7c79e2c7
Add new site replication feature (#13311) 
This change allows a set of MinIO sites (clusters) to be configured 
for mutual replication of all buckets (including bucket policies, tags, 
object-lock configuration and bucket encryption), IAM policies, 
LDAP service accounts and LDAP STS accounts.
 2021-10-06 16:36:31 -07:00
Minio Trusted
d6a3215fe2 Update yaml files to latest version RELEASE.2021-10-02T16-31-05Z 2021-10-03 21:43:13 +00:00
Klaus Post
bc6067d195
Add admin inspect Glob support (#13328) 
* Add admin Glob support

Allow returning multiple files on inspect calls.

```
λ mc admin inspect --json local2/testbucket/nyc-taxi-data-10M.csv.zst/*

...

λ unzip -l inspect.5f0643b2.zip

Archive:  inspect.5f0643b2.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/
      802  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/xl.meta
        0  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/
      802  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/xl.meta
        0  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/
      802  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/xl.meta
        0  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/
      802  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/xl.meta
---------                     -------
     3208                     8 files
```

Using fully recursive:

```
λ  mc admin inspect local2/testbucket/nyc-taxi-data-10M.csv.zst/**

...

Archive:  inspect.79c261cb.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/
        0  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.1
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.10
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.11
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.12
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.13
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.14
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.15
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.16
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.17
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.18
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.19
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.2
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.20
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.21
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.22
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.23
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.24
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.25
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.26
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.27
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.28
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.29
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.3
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.30
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.31
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.32
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.33
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.34
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.35
  3439368  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.36
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.4
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.5
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.6
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.7
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.8
  4194816  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.9
      802  2021-09-03 12:50   192.168.1.78:9001/a221edde-48fe-45f5-ad32-3bc7131c7659/testbucket/nyc-taxi-data-10M.csv.zst/xl.meta
        0  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/
        0  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.1
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.10
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.11
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.12
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.13
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.14
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.15
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.16
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.17
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.18
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.19
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.2
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.20
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.21
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.22
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.23
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.24
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.25
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.26
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.27
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.28
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.29
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.3
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.30
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.31
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.32
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.33
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.34
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.35
  3439368  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.36
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.4
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.5
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.6
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.7
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.8
  4194816  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.9
      802  2021-09-03 12:50   192.168.1.78:9001/cb7440ef-f0d9-42a8-b137-f00f519276ca/testbucket/nyc-taxi-data-10M.csv.zst/xl.meta
        0  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/
        0  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.1
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.10
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.11
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.12
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.13
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.14
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.15
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.16
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.17
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.18
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.19
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.2
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.20
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.21
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.22
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.23
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.24
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.25
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.26
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.27
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.28
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.29
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.3
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.30
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.31
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.32
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.33
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.34
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.35
  3439368  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.36
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.4
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.5
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.6
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.7
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.8
  4194816  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.9
      802  2021-09-03 12:50   192.168.1.78:9001/759cd5ac-7860-4cf3-acad-a375fcbae338/testbucket/nyc-taxi-data-10M.csv.zst/xl.meta
        0  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/
        0  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.1
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.10
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.11
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.12
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.13
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.14
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.15
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.16
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.17
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.18
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.19
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.2
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.20
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.21
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.22
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.23
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.24
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.25
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.26
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.27
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.28
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.29
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.3
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.30
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.31
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.32
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.33
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.34
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.35
  3439368  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.36
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.4
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.5
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.6
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.7
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.8
  4194816  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/18a50b3e-3c56-418e-a045-ad5c58c1d44b/part.9
      802  2021-09-09 15:56   192.168.1.78:9001/2b48619c-c2fa-4e69-839e-58fc82c1b43e/testbucket/nyc-taxi-data-10M.csv.zst/xl.meta
---------                     -------
601034920                     156 files

```

Furthermore allow `inspect` to do direct decode from `mc`, for example:

```
λ mc admin inspect --json local2/testbucket/nyc-taxi-data-10M.csv.zst/*|inspect -json
Output decrypted to inspect.5f0643b2.zip
```

- Correct error, forward non-EOF errors.
- Add some extra safety. Log FNF when no files.
- Add `xl-meta` zip support.
For `xl-meta` multiple inputs output object with names as key.
Automatically switches `xl-meta` to single-line output when multiple objects.
Add double-star wildcard support to xl-meta input.

Co-authored-by: Harshavardhana <harsha@minio.io>
 2021-10-01 11:50:00 -07:00
Poorna Krishnamoorthy
7f6ed35347
Allow null versions to be replicated (#13310) 
for pre-existing objects present in a bucket
prior to enabling existing object replication.

Co-authored-by: Poorna Krishnamoorthy <poorna@minio.io>
 2021-09-28 10:26:12 -07:00
Minio Trusted
a5923a5d51 Update yaml files to latest version RELEASE.2021-09-24T00-24-24Z 2021-09-24 19:20:44 +00:00
Minio Trusted
f4d1b7c603 Update yaml files to latest version RELEASE.2021-09-23T04-46-24Z 2021-09-23 07:42:55 +00:00
Poorna Krishnamoorthy
0b55a0423e
fix: cache usage deserialization from v5 to v6 (#13258) 2021-09-21 09:01:51 -07:00
Harshavardhana
f492f72154
add 3site replication script (#13256) 2021-09-20 18:24:24 -07:00
Poorna Krishnamoorthy
c4373ef290
Add support for multi site replication (#12880) 2021-09-18 13:31:35 -07:00
Minio Trusted
0b8c5a6872 Update yaml files to latest version RELEASE.2021-09-18T18-09-59Z 2021-09-18 20:28:07 +00:00
Minio Trusted
bf409936e7 Update yaml files to latest version RELEASE.2021-09-15T04-54-25Z 2021-09-15 08:04:57 +00:00
Harshavardhana
af78c3925a
add userinfo support for OpenID (#12469) 
Some identity providers like GitLab do not provide
information about group membership as part of the
identity token claims. They only expose it via OIDC compatible
'/oauth/userinfo' endpoint, as described in the OpenID
Connect 1.0 sepcification.

But this of course requires application to make sure to add
additional accessToken, since idToken cannot be re-used to
perform the same 'userinfo' call. This is why this is specialized
requirement. Gitlab seems to be the only OpenID vendor that requires
this support for the time being.

fixes #12367
 2021-09-13 16:22:14 -07:00
Harshavardhana
90e505e58f calculate API requests/error as increase() intervals not as rate() 2021-09-12 11:28:28 -07:00
Minio Trusted
a216583d95 Update yaml files to latest version RELEASE.2021-09-09T21-37-07Z 2021-09-09 23:50:26 +00:00
soudhaf
950fe73c4f
fix: minio console nginx config (#13156) 2021-09-08 23:41:06 -07:00
Harshavardhana
aaa3fc3805
fix: remove deprecated LDAP username format support (#13165) 2021-09-08 13:31:51 -07:00
Andreas Auernhammer
e438dccf19
sts: add support for certificate-based authentication (#12748) 
This commit adds a new STS API for X.509 certificate
authentication.

A client can make an HTTP POST request over a TLS connection
and MinIO will verify the provided client certificate, map it to an 
S3 policy and return temp. S3 credentials to the client.

So, this STS API allows clients to authenticate with X.509
certificates over TLS and obtain temp. S3 credentials.

For more details and examples refer to the docs/sts/tls.md
documentation.

Signed-off-by: Andreas Auernhammer <hi@aead.dev>
 2021-09-07 19:03:48 -07:00
Klaus Post
b2c92cdaaa
select: Add more compression formats (#13142) 
Support Zstandard, LZ4, S2, and snappy as additional 
compression formats for S3 Select.
 2021-09-06 09:09:53 -07:00
Harshavardhana
1250312287
fail ready/liveness if etcd is unhealthy in gateway mode (#13146) 2021-09-03 17:05:41 -07:00
Minio Trusted
e2b6fb0a6a Update yaml files to latest version RELEASE.2021-09-03T03-56-13Z 2021-09-03 06:45:54 +00:00
Harshavardhana
f486cfae86 point all docs for containers to quay.io 2021-09-01 18:48:26 -07:00
Minio Trusted
5c6dc63577 Update yaml files to latest version RELEASE.2021-08-31T05-46-54Z 2021-08-31 07:27:43 +00:00
Klaus Post
470553ff5d
Tweak readall allocation and renameData buffer reuse (#13108) 
Use a single allocation for reading the file, not the growing buffer of `io.ReadAll`.

Reuse the write buffer if we can when writing metadata in RenameData.
 2021-08-30 08:38:11 -07:00
Minio Trusted
abf079135e Update yaml files to latest version RELEASE.2021-08-25T00-41-18Z 2021-08-25 02:01:05 +00:00
Nitish Tiwari
60394ddf83
Add support for changing job name in Grafana dashboard (#13050) 2021-08-24 09:51:09 -07:00
Minio Trusted
7802088e71 Update yaml files to latest version RELEASE.2021-08-20T18-32-01Z 2021-08-20 20:42:00 +00:00
Harshavardhana
202d0b64eb
fix: enable go1.17 github ci/cd (#12997) 2021-08-18 18:35:22 -07:00
Minio Trusted
16aeb68c28 Update yaml files to latest version RELEASE.2021-08-17T20-53-08Z 2021-08-17 22:34:33 +00:00
Krishnan Parthasarathi
30b77f59b1
doc: Add ilm prometheus metrics information (#12994) 2021-08-17 12:19:36 -07:00
Aditya Manthramurthy
9b7d593e28
Add helper script to call assume role (#12978) 2021-08-17 01:46:59 -07:00
Ricardo Katz
a526ad2e80
Add headers into AMQP notifications (#12911) 
Signed-off-by: Ricardo Katz <rkatz@vmware.com>
 2021-08-11 22:24:19 -07:00
Harshavardhana
39f81d2c5b update max_delay to max_sleep under healing docs 2021-08-10 08:52:42 -07:00
Minio Trusted
7b0b0f9101 Update yaml files to latest version RELEASE.2021-08-05T22-01-19Z 2021-08-06 02:19:42 +00:00
Harshavardhana
c13cbc64d1 fix multiple typos in documentation 2021-08-04 14:15:45 -07:00
Matt Sarrel
109c8acf4f
fixed typo in metrics README.md (#12874) 2021-08-04 12:48:57 -07:00
Harshavardhana
035882d292
fix: remove parentIsObject() check (#12851) 
we will allow situations such as

```
a/b/1.txt
a/b
```

and

```
a/b
a/b/1.txt
```

we are going to document that this usecase is
not supported and we will never support it, if
any application does this users have to delete
the top level parent to make sure namespace is
accessible at lower level.

rest of the situations where the prefixes get
created across sets are supported as is.
 2021-08-03 13:26:57 -07:00
Harshavardhana
ea64a9263c
add deprecation notice for LDAP username format (#12849) 2021-08-02 18:20:06 -07:00
Harshavardhana
7281e86d9e update docker-compose to RELEASE.2021-07-30T00-02-00Z 2021-07-29 18:07:06 -07:00
Harshavardhana
a78bc7bfdb rename all access token to id_tokens 2021-07-27 22:38:12 -07:00
Harshavardhana
3735450e7e
fix: allow audience claim to be an array (#12810) 
Some incorrect setups might have multiple audiences
where they are trying to use a single authentication
endpoint for multiple services.

Nevertheless OpenID spec allows it to make it
even more confusin for no good reason.

> It MUST contain the OAuth 2.0 client_id of the
> Relying Party as an audience value. It MAY also
> contain identifiers for other audiences. In the
> general case, the aud value is an array of case
> sensitive strings. In the common special case
> when there is one audience, the aud value MAY
> be a single case sensitive string.

fixes #12809
 2021-07-27 18:37:51 -07:00
Minio Trusted
471b4fd0c9 Update yaml files to latest version RELEASE.2021-07-27T02-40-15Z 2021-07-27 05:02:00 +00:00
Aditya Manthramurthy
7106e9394c
[LDAP] Document new LDAP synchronization behavior (#12786) 2021-07-25 10:45:53 -07:00
Aditya Manthramurthy
9a31030e74
DEPRECATION Warning: Avoid configuring default LDAP STS Expiry (#12781) 
- Show notice when `MINIO_IDENTITY_LDAP_STS_EXPIRY` or the 
  corresponding to the configuration option is used at server startup.
- Once support is removed, the default will be fixed at 1 hour.
- Users may specify expiry directly in the STS API.
- Update docs and help message
- Adds example in ldap.go to configure expiry in STS API.
 2021-07-22 16:43:57 -07:00
Aditya Manthramurthy
a3079a7de2
fix: Add support for DurationSeconds in LDAP STS API (#12778) 2021-07-22 12:13:21 -07:00
Minio Trusted
026a005b64 Update yaml files to latest version RELEASE.2021-07-22T05-23-32Z 2021-07-22 06:12:29 +00:00
Minio Trusted
dbd7f74bb9 Update yaml files to latest version RELEASE.2021-07-21T22-15-23Z 2021-07-21 23:46:28 +00:00
Ali Malek
c25ad94b7e
Refactor docker-compose, Use YAML variable for common minio configs (#12735) 2021-07-18 18:06:40 -07:00
Minio Trusted
ead8778305 Update yaml files to latest version RELEASE.2021-07-15T22-27-34Z 2021-07-16 04:27:54 +00:00
Aditya Manthramurthy
eae9c2f65b
Add changes to ensure session policy is enforced in LDAP STS (#12716) 
- Bonus: Fix bug in webidentity sts that doesnt parse session policy correctly.

- update ldap.go to support session policy argument
 2021-07-15 15:27:34 -07:00
Ravind Kumar
ce800ed347
DOCS: Remove http links in favor of HTTPS (#12725) 2021-07-15 13:31:59 -07:00
Harshavardhana
8d19efe7e0
feat: increase allowed maximum STS expiration timeout to 365 days (#12704) 2021-07-14 00:08:22 -07:00
Harshavardhana
e316873f84
feat: Add support for kakfa audit logger target (#12678) 2021-07-13 09:39:13 -07:00
Harshavardhana
e25ae1c01f add console-address flag for spoonfed users 2021-07-12 17:05:38 -07:00
Minio Trusted
bc8962102f Update yaml files to latest version RELEASE.2021-07-12T02-44-53Z 2021-07-12 03:45:14 +00:00
Harshavardhana
55b08403cb update docker docs with --console-address 
also update the orchestration docs for nginx
proxy for console.

fixes #12660
 2021-07-11 19:44:53 -07:00
Klaus Post
d6a2fe02d3
Add admin file inspector (#12635) 
Download files from *any* bucket/path as an encrypted zip file.

The key is included in the response but can be separated so zip 
and the key doesn't have to be sent on the same channel.

Requires https://github.com/minio/pkg/pull/6
 2021-07-09 11:29:16 -07:00
Harshavardhana
28adb29db3
feat: Add support to poll users on external SSO (#12592) 
Additional support for vendor-specific admin API
integrations for OpenID, to ensure validity of
credentials on MinIO.

Every 5minutes check for validity of credentials
on MinIO with vendor specific IDP.
 2021-07-09 11:17:21 -07:00
Minio Trusted
b6dd9b55a7 Update yaml files to latest version RELEASE.2021-07-08T19-43-25Z 2021-07-08 22:53:49 -07:00
Christian Weiske
95d29a6a53
fix: Mention GetBucketyPolicy in S3 gateway docs policy (#12642) 
That action is needed for the gateway to obtain the policy
setting for public access.

Resolves: https://github.com/minio/minio/issues/12638
 2021-07-08 10:34:56 -07:00
Minio Trusted
c14f965293 Update yaml files to latest version RELEASE.2021-07-08T01-15-01Z 2021-07-08 03:12:44 +00:00
Harshavardhana
2ce60d772b
fix: update README.md for new release (#12637) 
rename all 'docker run' commands to 'podman run'

fixes #12633
 2021-07-07 18:07:29 -07:00
Harshavardhana
3137dc2eb3
support implicit flow in web-identity.go example (#12600) 
when a client secret is not provided,
automatically assume implicit flow
for authentication and invoke
relevant code accordingly.
 2021-06-30 07:43:04 -07:00
Harshavardhana
b043e61e8e remove deprecated warning for 'mc encrypt' 2021-06-25 08:12:40 -07:00
iternity-dotcom
d40268d899
docs: add limits for bucket and object name length (#12564) 2021-06-23 12:51:31 -07:00
Harshavardhana
cdeccb5510
feat: Deprecate embedded browser and import console (#12460) 
This feature also changes the default port where
the browser is running, now the port has moved
to 9001 and it can be configured with

```
--console-address ":9001"
```
 2021-06-17 20:27:04 -07:00
Harshavardhana
e4fbc6a9ff update to RELEASE.2021-06-17T00-10-46Z 2021-06-16 17:50:32 -07:00
Klaus Post
0d1d26a4ea
Simplify s3zip example (#12500) 
There is no need to create a custom transport wrapper.

Just set the header in the options.
 2021-06-14 11:04:14 -07:00
Harshavardhana
264ee97219 update to RELEASE.2021-06-14T01-29-23Z release 2021-06-13 20:58:56 -07:00
Poorna Krishnamoorthy
92e4c8be10
Update replication docs to fix mc reference (#12490) 
Signed-off-by: Poorna Krishnamoorthy <poorna@minio.io>
 2021-06-13 11:37:22 -07:00
Poorna Krishnamoorthy
f2a3872301
Update design.md for replication (#12486) 
Fixes #12483
 2021-06-10 16:05:39 -07:00
Anis Elleuch
ba5fb2365c
feat: support of ZIP list/get/head as S3 extension (#12267) 
When enabled, it is possible to list/get files
inside a zip file without uncompressing it.

Signed-off-by: Anis Elleuch <anis@min.io>
 2021-06-10 08:17:03 -07:00
Minio Trusted
22c58be421 Update yaml files to latest version RELEASE.2021-06-09T18-51-39Z 2021-06-09 19:21:21 +00:00
iternity-dotcom
dd5e9493f1
docs: add missing admin actions (#12442) 2021-06-09 10:37:20 -07:00
Nitish Tiwari
32017454ee
fix typo in Grafana dashboard json (#12471) 2021-06-09 08:04:12 -07:00
Minio Trusted
1bf80a6eea Update yaml files to latest version RELEASE.2021-06-07T21-40-51Z 2021-06-07 22:13:41 +00:00
Nitish Tiwari
00c5d7e1b3
Add healing related metrics in official dashboard (#12456) 2021-06-07 12:46:54 -07:00
Poorna Krishnamoorthy
f199afcd6c
tiering: add aws role support for s3 (#12424) 
Signed-off-by: Poorna Krishnamoorthy <poorna@minio.io>
 2021-06-04 12:47:00 -07:00
Poorna Krishnamoorthy
dbea8d2ee0
Add support for existing object replication. (#12109) 
Also adding an API to allow resyncing replication when
existing object replication is enabled and the remote target
is entirely lost. With the `mc replicate reset` command, the
objects that are eligible for replication as per the replication
config will be resynced to target if existing object replication
is enabled on the rule.
 2021-06-01 19:59:11 -07:00
Harshavardhana
1f262daf6f
rename all remaining packages to internal/ (#12418) 
This is to ensure that there are no projects
that try to import `minio/minio/pkg` into
their own repo. Any such common packages should
go to `https://github.com/minio/pkg`
 2021-06-01 14:59:40 -07:00
Poorna Krishnamoorthy
3690de0c6b
Drop Pending size and count from replication metrics (#12378) 
Real-time metrics calculated in-memory rely on the initial
replication metrics saved with data usage. However, this can
lag behind the actual state of the cluster at the time of server 
restart leading to inaccurate Pending size/counts reported to
Prometheus. Dropping the Pending metrics as this can be more 
reliably monitored by applications with replication notifications.

Signed-off-by: Poorna Krishnamoorthy <poorna@minio.io>
 2021-05-31 20:26:52 -07:00
Harshavardhana
ab7410af11 docs: add hostname entry for docker stack deploy 2021-05-31 19:11:52 -07:00
Harshavardhana
5cd6253508 docs: mention docker stack deploy doesn't work with compose.yml 2021-05-31 18:59:32 -07:00
Harshavardhana
f27513ff28 remove all docker swarm references 2021-05-31 18:55:21 -07:00
Harshavardhana
4444ba13a4
support ldap:username for policy substitution (#12390) 
LDAPusername is the simpler form of LDAPUser (userDN),
using a simpler version is convenient from policy
conditions point of view, since these are unique id's
used for LDAP login.
 2021-05-28 10:33:07 -07:00
Klaus Post
62b6615a2a
Add xl-meta data extraction (#12396) 
* Add `-export` flag to export data.
* Support wildcard inputs.
 2021-05-28 08:10:41 -07:00
Minio Trusted
ba75281d5a Update yaml files to latest version RELEASE.2021-05-27T22-06-31Z 2021-05-27 22:35:50 +00:00
Klaus Post
acc452b7ce
Add more erasure codes on degraded systems. (#11852) 
In cases where a cluster is degraded, we do not uphold our consistency 
guarantee and we will write fewer erasure codes and rely on healing 
to recreate the missing shards.

In some cases replacing known bad disks in practice take days.
We want to change the behavior of a known degraded system to keep
the erasure code promise of the storage class for each object.

This will create the objects with the same confidence as a fully 
functional cluster. The tradeoff will be that objects created 
during a partial outage will take up slightly more space.

This means that when the storage class is EC:4, there should 
always be written 4 parity shards, even if some disks are unavailable.

When an object is created on a set, the disks are immediately 
checked. If any disks are unavailable additional parity shards 
will be made for each offline disk, up to 50% of the number of disks.

We add an internal metadata field with the actual and intended 
erasure code level, this can optionally be picked up later by 
the scanner if we decide that data like this should be re-sharded.
 2021-05-27 11:38:09 -07:00
Harshavardhana
9773b16f6f update TLS docs to use new certgen tool 2021-05-26 10:41:29 -07:00
Minio Trusted
7b06b754a6 Update yaml files to latest version RELEASE.2021-05-26T00-22-46Z 2021-05-26 00:52:38 +00:00
Harshavardhana
2baabd455b docs: fix per tenant limits docs formatting 2021-05-24 09:37:17 -07:00
Harshavardhana
41e9c6572f fix: docs links use non-raw links for markdown 2021-05-22 10:52:47 -07:00
Harshavardhana
0da84a0c72 Update yaml files to latest version RELEASE.2021-05-22T02-34-39Z 2021-05-21 20:32:14 -07:00
Harshavardhana
df4914b6f3 fix: update docs, fix wording and links 2021-05-21 12:36:03 -07:00
Minio Trusted
783ea5eb5c Update yaml files to latest version RELEASE.2021-05-20T22-31-44Z 2021-05-20 23:02:31 +00:00
Poorna Krishnamoorthy
a27339826c
Fix replication README.md docs (#12330) 
Signed-off-by: Poorna Krishnamoorthy <poorna@minio.io>
 2021-05-20 08:17:14 -07:00
Harshavardhana
bb7fbcdc09
fix: generating service accounts for group only LDAP accounts (#12318) 
fixes #12315
 2021-05-18 15:19:20 -07:00
Harshavardhana
267f12a2a1 Update to release RELEASE.2021-05-18T00-53-28Z 2021-05-17 18:24:38 -07:00
Harshavardhana
3d9873106d
feat: distributed setup can start now with default credentials (#12303) 
In lieu of new changes coming for server command line, this
change is to deprecate strict requirement for distributed setups
to provide root credentials.

Bonus: remove MINIO_WORM warning from April 2020, it is time to
remove this warning.
 2021-05-17 08:45:22 -07:00
Minio Trusted
ce3d9dc9fa Update yaml files to latest version RELEASE.2021-05-16T05-32-34Z 2021-05-16 06:03:09 +00:00
Harshavardhana
7334247c98 update docs about NFS consistency model 2021-05-14 11:34:56 -07:00
Poorna Krishnamoorthy
951acf561c
Add support for syncing replica modifications (#11104) 
when bidirectional replication is set up.

If ReplicaModifications is enabled in the replication
configuration, sync metadata updates to source if
replication rules are met. By default, if this
configuration is unset, MinIO automatically sync's
metadata updates on replica back to the source.
 2021-05-13 19:20:45 -07:00
Nitish Tiwari
a592d3be19
fix the dashboard to use $rate_interval (#12277) 
refer https://grafana.com/blog/2020/09/28/new-in-grafana-7.2-__rate_interval-for-prometheus-rate-queries-that-just-work/
for further information
 2021-05-12 08:06:47 -07:00
Minio Trusted
f63eedb2b4 Update yaml files to latest version RELEASE.2021-05-11T23-27-41Z 2021-05-12 00:16:44 +00:00
Harshavardhana
477cd85bef purge deprecate docker swarm documentation 
documentation license moved to CC-BY-4.0
 2021-05-10 09:50:06 -07:00
Harshavardhana
d09806c887 add docs LICENSE file 2021-05-07 09:32:51 -07:00
Harshavardhana
2fd9c13b50 rename minio-cluster to minio-job as per prometheus config 2021-05-06 12:39:58 -07:00
Nitish Tiwari
ddc1e4b5b3
Update Grafana dashboard to use the new v2 cluster metrics (#12220) 
Fixes #11543
 2021-05-06 14:44:03 +05:30
Harshavardhana
804a23a06d update docs to remove _OLD credential references 
also update the docs about config, IAM on encryption.
 2021-05-04 10:27:51 -07:00
Harshavardhana
f7a87b30bf Revert "deprecate embedded browser (#12163)" 
This reverts commit 736d8cbac4.

Bring contrib files for older contributions
 2021-04-30 08:50:39 -07:00
Andreas Auernhammer
e5ec1325fc
docs: add QuickStart section to KMS encryption of IAM data (#12190) 
This commit enhances the docs about IAM encryption.
It adds a quick-start section that explains how to
get started quickly with `MINIO_KMS_SECRET_KEY`
instead of setting up KES.

It also removes the startup message that gets printed
when the server migrates IAM data to plaintext.
We will point this out in the release notes.

Signed-off-by: Andreas Auernhammer <aead@mail.de>
 2021-04-29 14:20:28 -07:00
Harshavardhana
736d8cbac4
deprecate embedded browser (#12163) 
https://github.com/minio/console takes over the functionality for the
future object browser development

Signed-off-by: Harshavardhana <harsha@minio.io>
 2021-04-27 10:52:12 -07:00
Harshavardhana
f420996dfa
fix: allow parsing keys in both new and old format (#12144) 
Bonus fix fallback to decrypt previously
encrypted content as well using older master
key ciphertext format.

Signed-off-by: Harshavardhana <harsha@minio.io>
 2021-04-24 19:05:25 -07:00
Krishnan Parthasarathi
c829e3a13b Support for remote tier management (#12090) 
With this change, MinIO's ILM supports transitioning objects to a remote tier.
This change includes support for Azure Blob Storage, AWS S3 compatible object
storage incl. MinIO and Google Cloud Storage as remote tier storage backends.

Some new additions include:

 - Admin APIs remote tier configuration management

 - Simple journal to track remote objects to be 'collected'
   This is used by object API handlers which 'mutate' object versions by
   overwriting/replacing content (Put/CopyObject) or removing the version
   itself (e.g DeleteObjectVersion).

 - Rework of previous ILM transition to fit the new model
   In the new model, a storage class (a.k.a remote tier) is defined by the
   'remote' object storage type (one of s3, azure, GCS), bucket name and a
   prefix.

* Fixed bugs, review comments, and more unit-tests

- Leverage inline small object feature
- Migrate legacy objects to the latest object format before transitioning
- Fix restore to particular version if specified
- Extend SharedDataDirCount to handle transitioned and restored objects
- Restore-object should accept version-id for version-suspended bucket (#12091)
- Check if remote tier creds have sufficient permissions
- Bonus minor fixes to existing error messages

Co-authored-by: Poorna Krishnamoorthy <poorna@minio.io>
Co-authored-by: Krishna Srinivas <krishna@minio.io>
Signed-off-by: Harshavardhana <harsha@minio.io>
 2021-04-23 11:58:53 -07:00
Harshavardhana
069432566f update license change for MinIO 
Signed-off-by: Harshavardhana <harsha@minio.io>
 2021-04-23 11:58:53 -07:00
Minio Trusted
e05e14309c Update yaml files to latest version RELEASE.2021-04-22T15-44-28Z 2021-04-22 16:32:11 +00:00
Poorna Krishnamoorthy
28f0ded1a4
Update replication design.md for sync mode (#12100) 2021-04-20 17:31:36 -07:00
Minio Trusted
49b7923309 Update yaml files to latest version RELEASE.2021-04-18T19-26-29Z 2021-04-18 20:15:32 +00:00
Ravind Kumar
ca9b48b3b4
Update Replication README to point at new docs (#12069) 
This is a minor change to call out the new documentation and warn 
users to change  their bookmarks. Once we are ready to set up 
some redirects, we can remove this page from Gluegun TOC.
 2021-04-15 16:32:44 -07:00
Klaus Post
d50193d30f
xl-meta.go dump skip CRC check (#12031) 
Skip CRC of metadata.
 2021-04-09 17:38:23 -07:00
Aditya Manthramurthy
05a9108c24
Update etcd doc to clarify that any STS setup should work (#12022) 2021-04-08 17:30:17 -07:00
Harshavardhana
0e4794ea50
fix: allow S3 gateway passthrough for SSE-S3 header (#12020) 
only in case of S3 gateway we have a case where we
need to allow for SSE-S3 headers as passthrough,

If SSE-C headers are passed then they are rejected
if KMS is not configured.
 2021-04-08 16:40:38 -07:00
Poorna Krishnamoorthy
2899cc92b4
Update replication docs for required permission (#12010) 2021-04-07 15:56:02 -07:00
Minio Trusted
a772379dc5 Update yaml files to latest version RELEASE.2021-04-06T23-11-00Z 2021-04-06 17:59:45 -07:00
Harshavardhana
8a9d15ace2 update prometheus metrics with failed_count 2021-04-04 09:52:37 -07:00
Poorna Krishnamoorthy
47c09a1e6f
Various improvements in replication (#11949) 
- collect real time replication metrics for prometheus.
- add pending_count, failed_count metric for total pending/failed replication operations.

- add API to get replication metrics

- add MRF worker to handle spill-over replication operations

- multiple issues found with replication
- fixes an issue when client sends a bucket
 name with `/` at the end from SetRemoteTarget
 API call make sure to trim the bucket name to 
 avoid any extra `/`.

- hold write locks in GetObjectNInfo during replication
  to ensure that object version stack is not overwritten
  while reading the content.

- add additional protection during WriteMetadata() to
  ensure that we always write a valid FileInfo{} and avoid
  ever writing empty FileInfo{} to the lowest layers.

Co-authored-by: Poorna Krishnamoorthy <poorna@minio.io>
Co-authored-by: Harshavardhana <harsha@minio.io>
 2021-04-03 09:03:42 -07:00
Klaus Post
2623338dc5
Inline small file data in xl.meta file (#11758) 2021-03-29 17:00:55 -07:00
Minio Trusted
91eb1fe2ef Update yaml files to latest version RELEASE.2021-03-26T00-00-41Z 2021-03-26 00:23:59 +00:00
Ritesh H Shukla
23b03dadb8
Add process uptime metric (#11844) 2021-03-20 21:23:27 -07:00
Minio Trusted
b379ca3bb0 Update yaml files to latest version RELEASE.2021-03-17T02-33-02Z 2021-03-17 02:56:28 +00:00
Ravind Kumar
980311fdfd
Fix STANDARD defaults, point to new docs site. (#11800) 2021-03-16 12:04:28 -07:00
Minio Trusted
afd346417d Update yaml files to latest version RELEASE.2021-03-12T00-00-47Z 2021-03-12 00:23:57 +00:00
Minio Trusted
9d118b372e Update yaml files to latest version RELEASE.2021-03-10T05-11-33Z 2021-03-10 05:34:48 +00:00
Minio Trusted
464fa08f2e Update yaml files to latest version RELEASE.2021-03-04T00-53-13Z 2021-03-04 01:15:49 +00:00
Harshavardhana
2c198ae7b6
fix: prometheus metrics disks_online count when disks are down (#11689) 
prometheus metrics was using total disks instead
of online disk count, when disks were down, this
PR fixes this and also adds a new metric for
total_disk_count
 2021-03-03 11:18:41 -08:00
Nitish Tiwari
bbd1244a88
Add support for mTLS for Audit log target (#11645) 2021-03-01 09:19:13 -08:00
Minio Trusted
e3f8830ab7 Update yaml files to latest version RELEASE.2021-03-01T04-20-55Z 2021-03-01 04:43:28 +00:00
cbows
c67d1bf120
add unauthenticated lookup-bind mode to LDAP identity (#11655) 
Closes #11646
 2021-02-28 12:57:31 -08:00
Harshavardhana
9171d6ef65
rename all references from crawl -> scanner (#11621) 2021-02-26 15:11:42 -08:00
Minio Trusted
b912e9ab41 Update yaml files to latest version RELEASE.2021-02-24T18-44-45Z 2021-02-24 19:08:36 +00:00
Minio Trusted
1b63291ee2 Update yaml files to latest version RELEASE.2021-02-23T20-05-01Z 2021-02-23 20:28:30 +00:00
Aditya Manthramurthy
02e7de6367
LDAP config: fix substitution variables (#11586) 
- In username search filter and username format variables we support %s for
replacing with the username.

- In group search filter we support %s for username and %d for the full DN of
the username.
 2021-02-22 13:20:36 -08:00
Harshavardhana
8cad407e0b
fix: Bring support for symlink on regular files on NAS (#11383) 
fixes #11203
 2021-02-20 00:30:12 -08:00
WangYuMu
c70240b893
fix incorrect values in sizing guide (#11583) 2021-02-19 10:05:04 -08:00
Harshavardhana
8ba2136e06 Update yaml files to latest version RELEASE.2021-02-19T04-38-02Z 2021-02-18 21:02:25 -08:00
Poorna Krishnamoorthy
8e8a792d9d
Allow delete marker replication from replica (#11566) 
in the case of active-active replication.

This PR also has the following changes:

- add docs on replication design
- fix corner case of completing versioned delete on a delete marker
  when the target is down and `mc rm --vid` is performed repeatedly. Instead
  the version should still be retained in the `PENDING|FAILED` state until
  replication sync completes.
- remove `s3:Replication:OperationCompletedReplication` and
   `s3:Replication:OperationFailedReplication` from ObjectCreated 
  events type
 2021-02-18 00:33:51 -08:00
Harshavardhana
ffea6fcf09
fix: rename crawler as scanner in config (#11549) 2021-02-17 12:04:11 -08:00
Minio Trusted
cfc8b92dff Update yaml files to latest version RELEASE.2021-02-14T04-01-33Z 2021-02-14 04:25:52 +00:00
Minio Trusted
0ef3e359d8 Update yaml files to latest version RELEASE.2021-02-11T08-23-43Z 2021-02-11 08:47:10 +00:00
Aditya Manthramurthy
466e95bb59
Return group DN instead of group name in LDAP STS (#11501) 
- Additionally, check if the user or their groups has a policy attached during
the STS call.

- Remove the group name attribute configuration value.
 2021-02-10 16:52:49 -08:00
Krishna Srinivas
876b79b8d8
read-health check endpoint returns success if cluster can serve read requests (#11310) 2021-02-09 01:00:44 -08:00
Sarasa Kisaragi
152d7cd95b
HDFS support keytab (#11473) 2021-02-07 17:29:47 -08:00
Minio Trusted
647a209c73 Update yaml files to latest version RELEASE.2021-02-07T01-31-02Z 2021-02-07 01:53:27 +00:00
Harshavardhana
9497dfd804 docs: add deprecation notice for federation 2021-02-04 17:18:37 -08:00
Bahram Aghaei
f2d49ec21a
Update ldap.md: add a link to ldap.go (#11409) 2021-02-02 15:47:04 -08:00
Ritesh H Shukla
c4848f9b4f
Add process start time to cluster metrics. (#11405) 2021-02-01 23:02:18 -08:00
swartz-k
8c663f93f7
fix: typo in chinese docs (#11401) 2021-02-01 18:42:58 -08:00
Minio Trusted
b4cb7edf85 Update yaml files to latest version RELEASE.2021-02-01T22-56-52Z 2021-02-01 23:28:23 +00:00
Andreas Auernhammer
33554651e9
crypto: deprecate native Hashicorp Vault support (#11352) 
This commit deprecates the native Hashicorp Vault
support and removes the legacy Vault documentation.

The native Hashicorp Vault documentation is marked as
outdated and deprecated for over a year now. We give
another 6 months before we start removing Hashicorp Vault
support and show a deprecation warning when a MinIO server
starts with a native Vault configuration.
 2021-01-29 17:55:37 -08:00
Minio Trusted
451d9057f3 Update yaml files to latest version RELEASE.2021-01-30T00-20-58Z 2021-01-30 00:45:11 +00:00
Harshavardhana
6717295e18 fix: rename audit log docs and datastructure 2021-01-26 13:39:55 -08:00
Anis Elleuch
00cff1aac5
audit: per object send pool number, set number and servers per operation (#11233) 2021-01-26 13:21:51 -08:00
Aditya Manthramurthy
5f51ef0b40
Add LDAP Lookup-Bind mode (#11318) 
This change allows the MinIO server to be configured with a special (read-only)
LDAP account to perform user DN lookups.

The following configuration parameters are added (along with corresponding
environment variables) to LDAP identity configuration (under `identity_ldap`):

- lookup_bind_dn / MINIO_IDENTITY_LDAP_LOOKUP_BIND_DN
- lookup_bind_password / MINIO_IDENTITY_LDAP_LOOKUP_BIND_PASSWORD
- user_dn_search_base_dn / MINIO_IDENTITY_LDAP_USER_DN_SEARCH_BASE_DN
- user_dn_search_filter / MINIO_IDENTITY_LDAP_USER_DN_SEARCH_FILTER

This lookup-bind account is a service account that is used to lookup the user's
DN from their username provided in the STS API. When configured, searching for
the user DN is enabled and configuration of the base DN and filter for search is
required. In this "lookup-bind" mode, the username format is not checked and must
not be specified. This feature is to support Active Directory setups where the
DN cannot be simply derived from the username.

When the lookup-bind is not configured, the old behavior is enabled: the minio
server performs LDAP lookups as the LDAP user making the STS API request and the
username format is checked and configuring it is required.
 2021-01-25 14:26:10 -08:00
Ritesh H Shukla
0bf2d84f96
update new metrics url docs (#11342) 2021-01-25 01:03:07 -08:00
Ritesh H Shukla
7575c24037
Add open FD and FD limit to cluster metrics (#11328) 2021-01-22 18:30:16 -08:00
Harshavardhana
c080f04e66 fix: prometheus metrics link typo update to latest 2021-01-22 01:53:23 -08:00
Ritesh H Shukla
b4add82bb6
Updated Prometheus metrics (#11141) 
* Add metrics for nodes online and offline
* Add cluster capacity metrics
* Introduce v2 metrics
 2021-01-18 20:35:38 -08:00
Aditya Manthramurthy
3163a660aa
Fix support for multiple LDAP user formats (#11276) 
Fixes support for using multiple base DNs for user search in the LDAP directory
allowing users from different subtrees in the LDAP hierarchy to request
credentials.

- The username in the produced credentials is now the full DN of the LDAP user
to disambiguate users in different base DNs.
 2021-01-17 21:54:32 -08:00
Harshavardhana
98f76008c7 fix: bucket lifecycle again to remove Days parameter 2021-01-17 01:50:56 -08:00
Harshavardhana
8da0b7cf03 fix: lifecycle documentation for DeleteMarker 2021-01-17 01:37:25 -08:00
Harshavardhana
f903cae6ff
Support variable server pools (#11256) 
Current implementation requires server pools to have
same erasure stripe sizes, to facilitate same SLA
and expectations.

This PR allows server pools to be variadic, i.e they
do not have to be same erasure stripe sizes - instead
they should have SLA for parity ratio.

If the parity ratio cannot be guaranteed by the new
server pool, the deployment is rejected i.e server
pool expansion is not allowed.
 2021-01-16 12:08:02 -08:00
Minio Trusted
40d59c1961 Update yaml files to latest version RELEASE.2021-01-16T02-19-44Z 2021-01-16 02:43:53 +00:00
Poorna Krishnamoorthy
7090bcc8e0
fix: doc links and delete replication permissions enforcement (#11285) 2021-01-15 15:22:55 -08:00
Harshavardhana
cc2d887e0e fix: whitespace and formatting in replication docs 2021-01-14 22:58:53 -08:00
Poorna Krishnamoorthy
c1b4b24236
Update replication docs (#11279) 2021-01-15 10:22:57 +05:30
Poorna Krishnamoorthy
feaf8dfb9a
Fix replication status reported on completion (#11273) 
Fixes: #11272
 2021-01-13 11:52:28 -08:00
Poorna Krishnamoorthy
b97d53b29c
fix remote target healthcheck (#11267) 2021-01-12 20:48:04 -08:00
Aditya Manthramurthy
00af9881b0
LDAP doc fix: remove repeated paragraph and add emphasis (#11266) 2021-01-12 15:44:31 -08:00
Poorna Krishnamoorthy
7824e19d20
Allow synchronous replication if enabled. (#11165) 
Synchronous replication can be enabled by setting the --sync
flag while adding a remote replication target.

This PR also adds proxying on GET/HEAD to another node in a
active-active replication setup in the event of a 404 on the current node.
 2021-01-11 22:36:51 -08:00
George Tsatsis
e8176fe978
Use -new during OpenSSL certificate generation (#11199) 
As per https://stackoverflow.com/a/3758443/8156177, OpenSSL expects a certificate via STDIN. 
`-new` will allow a new certificate to be generated instead.
 2021-01-11 02:24:50 -08:00
Minio Trusted
d9224fbc65 Update yaml files to latest version RELEASE.2021-01-08T21-18-21Z 2021-01-08 21:37:35 +00:00
Harshavardhana
76e2713ffe
fix: use buffers only when necessary for io.Copy() (#11229) 
Use separate sync.Pool for writes/reads

Avoid passing buffers for io.CopyBuffer()
if the writer or reader implement io.WriteTo or io.ReadFrom
respectively then its useless for sync.Pool to allocate
buffers on its own since that will be completely ignored
by the io.CopyBuffer Go implementation.

Improve this wherever we see this to be optimal.

This allows us to be more efficient on memory usage.
```
   385  // copyBuffer is the actual implementation of Copy and CopyBuffer.
   386  // if buf is nil, one is allocated.
   387  func copyBuffer(dst Writer, src Reader, buf []byte) (written int64, err error) {
   388  	// If the reader has a WriteTo method, use it to do the copy.
   389  	// Avoids an allocation and a copy.
   390  	if wt, ok := src.(WriterTo); ok {
   391  		return wt.WriteTo(dst)
   392  	}
   393  	// Similarly, if the writer has a ReadFrom method, use it to do the copy.
   394  	if rt, ok := dst.(ReaderFrom); ok {
   395  		return rt.ReadFrom(src)
   396  	}
```

From readahead package
```
// WriteTo writes data to w until there's no more data to write or when an error occurs.
// The return value n is the number of bytes written.
// Any error encountered during the write is also returned.
func (a *reader) WriteTo(w io.Writer) (n int64, err error) {
	if a.err != nil {
		return 0, a.err
	}
	n = 0
	for {
		err = a.fill()
		if err != nil {
			return n, err
		}
		n2, err := w.Write(a.cur.buffer())
		a.cur.inc(n2)
		n += int64(n2)
		if err != nil {
			return n, err
		}
```
 2021-01-06 09:36:55 -08:00
Harshavardhana
b5d291ea88
fix: rename remaining zone -> pool (#11231) 2021-01-06 09:35:47 -08:00
Klaus Post
eb9172eecb
Allow Compression + encryption (#11103) 2021-01-05 20:08:35 -08:00
Harshavardhana
cb0eaeaad8
feat: migrate to ROOT_USER/PASSWORD from ACCESS/SECRET_KEY (#11185) 2021-01-05 10:22:57 -08:00
Minio Trusted
f3f0041ad0 Update yaml files to latest version RELEASE.2021-01-05T05-22-38Z 2021-01-05 05:42:54 +00:00
Baptiste Mille-Mathias
c1f6ca6697
Fix caddy project url (#11198) 2020-12-31 09:44:07 -08:00
0xflotus
cde801282d
chore: enabled syntax highlighting in docs (#11182) 2020-12-29 17:38:28 -08:00
Ravind Kumar
6cf0008469
fix: docs typo in object lock docs (#11181) 2020-12-29 16:14:10 -08:00
Minio Trusted
7b0330a98c Update yaml files to latest version RELEASE.2020-12-29T23-29-29Z 2020-12-29 23:46:24 +00:00
Minio Trusted
d3c853a3be Update yaml files to latest version RELEASE.2020-12-26T01-35-54Z 2020-12-26 01:53:30 +00:00
Poorna Krishnamoorthy
7b8a456f68
Update lifecycle README docs (#11160) 
Removing reference to transition feature in docs as this
feature is being revamped to provide better extensibility
across different cloud targets.
 2020-12-23 19:56:55 -08:00
Harshavardhana
b43906f6ee fix: docs typos and keywords 2020-12-23 11:59:20 -08:00
Minio Trusted
bfb92a27b7 Update yaml files to latest version RELEASE.2020-12-23T02-24-12Z 2020-12-23 02:43:25 +00:00
Minio Trusted
386dd56856 Update yaml files to latest version RELEASE.2020-12-18T03-27-42Z 2020-12-18 03:45:13 +00:00
Minio Trusted
26624552be Update yaml files to latest version RELEASE.2020-12-16T05-05-17Z 2020-12-16 05:23:19 +00:00
Minio Trusted
31bf6f0c25 Update yaml files to latest version RELEASE.2020-12-12T08-39-07Z 2020-12-12 08:56:30 +00:00
Harshavardhana
14792cdbc6
docs: fix the metrics formatting (#11081) 2020-12-10 18:15:47 -08:00
Harshavardhana
97856bfebf
fix: grafana double counting for bucket usage, histrogram and objects (#11070) 2020-12-09 20:30:37 -08:00
Minio Trusted
83a67a1d21 Update yaml files to latest version RELEASE.2020-12-10T01-54-29Z 2020-12-10 02:12:01 +00:00
Nitish Tiwari
54d243cd98
fix: grafana dashboard calculating online nodes (#11041) 
Also use a generic name instead of diff names per revision
 2020-12-09 00:26:42 -08:00
Ritesh H Shukla
04848dfa1c
Add documentation for bucket replication related metrics (#11055) 2020-12-08 12:48:10 -08:00
Harshavardhana
4a564336fe Revert "Add metrics for nodes online and offline (#11050)" 
This reverts commit f60bbdf86b.
 2020-12-08 09:23:35 -08:00
Ritesh H Shukla
f60bbdf86b
Add metrics for nodes online and offline (#11050) 2020-12-08 01:06:27 -08:00
Harshavardhana
8c79f87f02
add dynamic config docs (#11048) 
Co-authored-by: Eco <41090896+eco-minio@users.noreply.github.com>
 2020-12-07 19:02:20 -08:00
Poorna Krishnamoorthy
f3beb1236a
Add cache usage, total capacity to prometheus metrics (#11026) 2020-12-07 16:35:11 -08:00
Poorna Krishnamoorthy
934bed47fa
Add transition event notification (#11047) 
This is a MinIO specific extension to allow monitoring of transition events.
 2020-12-07 13:53:28 -08:00
Nitish Tiwari
6ff12f5f01
Add the dashboard json file (#11028) 
This will allow users to contribute to the dashboard as needed.
 2020-12-04 16:27:41 -08:00
Nitish Tiwari
de9b64834e
fix: update grafana dashboard docs (#11023) 
Refer to the official Grafana dashboard
 2020-12-03 15:56:15 -08:00
Minio Trusted
d54cf77356 Update yaml files to latest version RELEASE.2020-12-03T05-49-24Z 2020-12-03 06:05:56 +00:00
Minio Trusted
1ab4d6a6aa Update yaml files to latest version RELEASE.2020-12-03T00-03-10Z 2020-12-03 00:24:16 +00:00
Anis Elleuch
fe11e9047d
deprecate CommonName from TLS docs (#11017) 
CommonName is not supported anymore in Go 1.15

fix the TLS documentation to use subjAltNames
 2020-12-02 10:18:39 -08:00
Sudarshan (Sid)
3ca046b408
Added set keyword in the command set to enable encryption on buckets (#11010) 2020-12-01 16:00:49 -08:00
Harshavardhana
4ec45753e6 rename server sets to server pools 2020-12-01 13:50:33 -08:00
Harshavardhana
790833f3b2 Revert "Support variable server sets (#10314)" 
This reverts commit aabf053d2f.
 2020-12-01 12:02:29 -08:00
Harshavardhana
aabf053d2f
Support variable server sets (#10314) 2020-11-25 16:28:47 -08:00
Minio Trusted
f839bb5a0a Update yaml files to latest version RELEASE.2020-11-25T22-36-25Z 2020-11-25 22:53:02 +00:00
Anis Elleuch
75a8e81f8f
azure: Specify different Azure storage in the shell env (#10943) 
AZURE_STORAGE_ACCOUNT and AZURE_STORAGE_KEY are used in 
azure CLI to specify the azure blob storage access & secret keys. With this commit, 
it is possible to set them if you want the gateway's own credentials to be
different from the Azure blob credentials.

Co-authored-by: Harshavardhana <harsha@minio.io>
 2020-11-23 16:45:56 -08:00
Poorna Krishnamoorthy
39f3d5493b
Show Delete replication status header (#10946) 
X-Minio-Replication-Delete-Status header shows the
status of the replication of a permanent delete of a version.

All GETs are disallowed and return 405 on this object version.
In the case of replicating delete markers.

X-Minio-Replication-DeleteMarker-Status shows the status 
of replication, and would similarly return 405.

Additionally, this PR adds reporting of delete marker event completion
and updates documentation
 2020-11-21 23:48:50 -08:00
Shireesh Anjal
7bc47a14cc
Rename OBD to Health (#10842) 
Also, Remove thread stats and openfds from the health report 
as we already have process stats and numfds
 2020-11-20 12:52:53 -08:00
Harshavardhana
9263be8cca
docs: fix missing event types in notifications (#10944) 2020-11-20 11:27:27 -08:00
Poorna Krishnamoorthy
1ebf6f146a Add support for ILM transition (#10565) 
This PR adds transition support for ILM
to transition data to another MinIO target
represented by a storage class ARN. Subsequent
GET or HEAD for that object will be streamed from
the transition tier. If PostRestoreObject API is
invoked, the transitioned object can be restored for
duration specified to the source cluster.
 2020-11-19 18:47:17 -08:00
Harshavardhana
9a34fd5c4a Revert "Revert "Add delete marker replication support (#10396)"" 
This reverts commit 267d7bf0a9.
 2020-11-19 18:43:58 -08:00
Minio Trusted
b9e3a8b5ac Update yaml files to latest version RELEASE.2020-11-19T23-48-16Z 2020-11-20 00:06:03 +00:00
Minio Trusted
0784a0c33a Update yaml files to latest version RELEASE.2020-11-13T20-10-18Z 2020-11-13 20:27:01 +00:00
Minio Trusted
3595cb1267 Update yaml files to latest version RELEASE.2020-11-12T22-33-34Z 2020-11-12 22:51:11 +00:00
Omar Alvarez
07859ef48b
docs: clarify notifications support for gateways (#10729) 
Co-authored-by: Harshavardhana <harsha@minio.io>
 2020-11-12 12:19:04 -08:00
Harshavardhana
267d7bf0a9 Revert "Add delete marker replication support (#10396)" 
This reverts commit 50c10a5087.

PR is moved to origin/dev branch
 2020-11-12 11:43:14 -08:00
Anton Melser
2c1e37197b
fix: bad example json for policy in replication docs (#10869) 2020-11-10 17:49:49 -08:00
Poorna Krishnamoorthy
50c10a5087
Add delete marker replication support (#10396) 
Delete marker replication is implemented for V2
configuration specified in AWS spec (though AWS
allows it only in the V1 configuration).

This PR also brings in a MinIO only extension of
replicating permanent deletes, i.e. deletes specifying
version id are replicated to target cluster.
 2020-11-10 15:24:14 -08:00
Minio Trusted
9f4ad873bc Update yaml files to latest version RELEASE.2020-11-10T21-02-24Z 2020-11-10 21:18:35 +00:00
Anis Elleuch
8e8ddf7233
doc: Add definition of 1KB and 1MB in prometheus (#10857) 2020-11-09 10:05:01 -08:00
Minio Trusted
21016265e5 Update yaml files to latest version RELEASE.2020-11-06T23-17-07Z 2020-11-06 23:34:19 +00:00
Minio Trusted
51222cc664 Update yaml files to latest version RELEASE.2020-10-28T08-16-50Z 2020-10-28 08:33:22 +00:00
Minio Trusted
ca77ee1c0e Update yaml files to latest version RELEASE.2020-10-27T04-03-55Z 2020-10-27 04:20:10 +00:00
Harshavardhana
4442382c16 update STS examples to use latest v7 APIs 2020-10-24 11:16:59 -07:00
Anis Elleuch
6fd088f448
docs: Update ILM doc with versioning features (#10714) 2020-10-20 09:23:27 -07:00
Minio Trusted
ba5215561f Update yaml files to latest version RELEASE.2020-10-18T21-54-12Z 2020-10-18 22:15:30 +00:00
Márk Sági-Kazár
4eb45c9a0f
fix: dex getting started guide URL (#10701) 2020-10-17 16:47:28 -07:00
Harshavardhana
ad726b49b4
rename zones to serverSets to avoid terminology conflict (#10679) 
we are bringing in availability zones, we should avoid
zones as per server expansion concept.
 2020-10-15 14:28:50 -07:00
Harshavardhana
9c042a503b
remove deprecate readiness from healthcheck docs (#10659) 2020-10-12 18:56:03 -07:00
Minio Trusted
4a678ad70f Update yaml files to latest version RELEASE.2020-10-12T21-53-21Z 2020-10-12 22:15:06 +00:00
Minio Trusted
071c004f8b Update yaml files to latest version RELEASE.2020-10-09T22-55-05Z 2020-10-09 23:16:30 +00:00
miraculli
1738eb24b1
fix: caching doc README.md missing high watermark (#10646) 2020-10-09 08:32:51 -07:00
Poorna Krishnamoorthy
01498a3e34
fix: add docs for new event types in notification (#10636) 2020-10-06 13:33:23 -07:00
Ravind Kumar
57f0176759
Update KES table to include additional supported KMS providers (#10631) 2020-10-06 11:09:43 -07:00
Minio Trusted
017954e7ea Update yaml files to latest version RELEASE.2020-10-03T02-19-42Z 2020-10-03 02:36:54 +00:00
Minio Trusted
511424a287 Update yaml files to latest version RELEASE.2020-09-26T03-44-56Z 2020-09-26 04:02:42 +00:00
Praveen raj Mani
b880796aef
Set the maximum open connections limit in PG and MySQL target configs (#10558) 
As the bulk/recursive delete will require multiple connections to open at an instance,
The default open connections limit will be reached which results in the following error

```FATAL:  sorry, too many clients already```

By setting the open connections to a reasonable value - `2`, We ensure that the max open connections
will not be exhausted and lie under bounds.

The queries are simple inserts/updates/deletes which is operational and sufficient with the
the maximum open connection limit is 2.

Fixes #10553

Allow user configuration for MaxOpenConnections
 2020-09-24 22:20:30 -07:00
poornas
e6ab4db6b8
Fix minimum replication workers started (#10560) 
This PR also fixes GetReplicationConfiguration permission
in web-handlers.go to use bucket as resource
 2020-09-24 12:25:41 -07:00
Espen Finnesand
df08fd1f03
Update Kubernetes Helm Readme.md (#10559) 
The link to the Minio Helm Chart repository was wrong.
 2020-09-24 08:38:46 -07:00
Minio Trusted
ac82f416a4 Update yaml files to latest version RELEASE.2020-09-23T19-18-30Z 2020-09-23 19:38:14 +00:00
飞雪无情
5fdf47b118
Add Chinese documentation for distributed design (#10549) 2020-09-23 09:26:47 -07:00
Harshavardhana
8b74a72b21
fix: rename READY deadline to CLUSTER deadline ENV (#10535) 2020-09-23 09:14:33 -07:00
poornas
a4006e23a0
Update replication docs to clarify permissions (#10536) 
Co-authored-by: Klaus Post <klauspost@gmail.com>
 2020-09-22 11:58:04 -07:00
Minio Trusted
d73c4f09f3 Update yaml files to latest version RELEASE.2020-09-21T22-31-59Z 2020-09-21 22:58:40 +00:00
飞雪无情
dd4a2d7419
update distributed zh_CN document. (#10531) 2020-09-21 10:54:56 -07:00
Derek Bender
3168e93730
fix typo in healthcheck README.md (#10518) 2020-09-18 09:52:37 -07:00
Aleksey Pogibelev
6f45e303f5
fix tls flag (#10487) 2020-09-18 15:35:54 +05:30