Commit Graph

9074 Commits

Author SHA1 Message Date
Harshavardhana 737a3f0bad
fix: decommission bugfixes found during migration of .minio.sys/config (#14078) 2022-01-10 17:26:00 -08:00
Harshavardhana 3bd9636a5b
do not remove Sid from svcaccount policies (#14064)
fixes #13905
2022-01-10 14:26:26 -08:00
Harshavardhana 76b21de0c6
feat: decommission feature for pools (#14012)
```
λ mc admin decommission start alias/ http://minio{1...2}/data{1...4}
```

```
λ mc admin decommission status alias/
┌─────┬─────────────────────────────────┬──────────────────────────────────┬────────┐
│ ID  │ Pools                           │ Capacity                         │ Status │
│ 1st │ http://minio{1...2}/data{1...4} │ 439 GiB (used) / 561 GiB (total) │ Active │
│ 2nd │ http://minio{3...4}/data{1...4} │ 329 GiB (used) / 421 GiB (total) │ Active │
└─────┴─────────────────────────────────┴──────────────────────────────────┴────────┘
```

```
λ mc admin decommission status alias/ http://minio{1...2}/data{1...4}
Progress: ===================> [1GiB/sec] [15%] [4TiB/50TiB]
Time Remaining: 4 hours (started 3 hours ago)
```

```
λ mc admin decommission status alias/ http://minio{1...2}/data{1...4}
ERROR: This pool is not scheduled for decommissioning currently.
```

```
λ mc admin decommission cancel alias/
┌─────┬─────────────────────────────────┬──────────────────────────────────┬──────────┐
│ ID  │ Pools                           │ Capacity                         │ Status   │
│ 1st │ http://minio{1...2}/data{1...4} │ 439 GiB (used) / 561 GiB (total) │ Draining │
└─────┴─────────────────────────────────┴──────────────────────────────────┴──────────┘
```

> NOTE: Canceled decommission will not make the pool active again, since we might have
> Potentially partial duplicate content on the other pools, to avoid this scenario be
> very sure to start decommissioning as a planned activity.

```
λ mc admin decommission cancel alias/ http://minio{1...2}/data{1...4}
┌─────┬─────────────────────────────────┬──────────────────────────────────┬────────────────────┐
│ ID  │ Pools                           │ Capacity                         │ Status             │
│ 1st │ http://minio{1...2}/data{1...4} │ 439 GiB (used) / 561 GiB (total) │ Draining(Canceled) │
└─────┴─────────────────────────────────┴──────────────────────────────────┴────────────────────┘
```
2022-01-10 09:07:49 -08:00
Harshavardhana dabb058167 release helm v3.4.6 2022-01-07 22:24:06 -08:00
Minio Trusted f394313fee Update yaml files to latest version RELEASE.2022-01-08T03-11-54Z 2022-01-08 05:53:59 +00:00
Harshavardhana b7c5e45fff
heal: isObjectDangling should return false when it cannot decide (#14053)
In a multi-pool setup when disks are coming up, or in a single pool
setup let's say with 100's of erasure sets with a slow network.

It's possible when healing is attempted on `.minio.sys/config`
folder, it can lead to healing unexpectedly deleting some policy
files as dangling due to a mistake in understanding when `isObjectDangling`
is considered to be 'true'.

This issue happened in commit 30135eed86
when we assumed the validMeta with empty ErasureInfo is considered
to be fully dangling. This implementation issue gets exposed when
the server is starting up.

This is most easily seen with multiple-pool setups because of the
disconnected fashion pools that come up. The decision to purge the
object as dangling is taken incorrectly prior to the correct state
being achieved on each pool, when the corresponding drive let's say
returns 'errDiskNotFound', a 'delete' is triggered. At this point,
the 'drive' comes online because this is part of the startup sequence
as drives can come online lazily.

This kind of situation exists because we allow (totalDisks/2) number
of drives to be online when the server is being restarted.

Implementation made an incorrect assumption here leading to policies
getting deleted.

Added tests to capture the implementation requirements.
2022-01-07 19:11:54 -08:00
Aditya Manthramurthy 0a224654c2
fix: progagation of service accounts for site replication (#14054)
- Only non-root-owned service accounts are replicated for now.
- Add integration tests for OIDC with site replication
2022-01-07 17:41:43 -08:00
dharmendra kariya 47e4a36d7e
update NOTES.txt in helm help (#14049) 2022-01-07 10:42:16 -08:00
Minio Trusted e420a1de4d Update yaml files to latest version RELEASE.2022-01-07T01-53-23Z 2022-01-07 05:57:36 +00:00
Aditya Manthramurthy 62dc0f7698
Update site replication docs (#14044)
- Now OpenID external IDP is also supported for site replication.

- IAM users/groups are also replicated
2022-01-06 17:53:23 -08:00
Aditya Manthramurthy 2d31d92271
Fix redigo dep to latest unretracted version (#14043)
To avoid error message like:

```
go: warning: github.com/gomodule/redigo@v2.0.0+incompatible: retracted by module author: Old development version not maintained or published.
go: to switch to the latest unretracted version, run:
	go get github.com/gomodule/redigo@latest
```
2022-01-06 16:00:31 -08:00
Aditya Manthramurthy 1981fe2072
Add internal IDP and OIDC users support for site-replication (#14041)
- This allows site-replication to be configured when using OpenID or the
  internal IDentity Provider.

- Internal IDP IAM users and groups will now be replicated to all members of the
  set of replicated sites.

- When using OpenID as the external identity provider, STS and service accounts
  are replicated.

- Currently this change dis-allows root service accounts from being
  replicated (TODO: discuss security implications).
2022-01-06 15:52:43 -08:00
Aditya Manthramurthy f68bd37acf
Do not clean golangci-lint's cache (#14042)
- This speeds up running the linters during local development. With a fully
cached run, linter completes in 8 seconds.

- Any caching issues if present would be local and would not impact CI anyway
which always starts with a clean state.
2022-01-06 14:19:51 -08:00
Minio Trusted 76877eb6fa move gofumpt to golang-ci 2022-01-06 13:08:21 -08:00
Klaus Post 3d66d053c7
Add small client TLS PSK cache (#14039) 2022-01-06 11:34:02 -08:00
Harshavardhana 0d3ae3810f
make sure to comply with MQTT spec (#14037)
- keep-alive cannot be 0 by default anymore
- client_id cannot be empty

fixes #13993
2022-01-06 11:25:39 -08:00
Klaus Post 0e31cff762
fix: DeleteMultipleObjects to finish even if cancelled + concurrent sets (#14038)
* Process sets concurrently.
* Disconnect context from request.
* Insert context cancellation checks.
* errFileNotFound and errFileVersionNotFound are ok, unless creating delete markers.
2022-01-06 10:47:49 -08:00
Shireesh Anjal c27110e37d
Add timeinfo to health data (#14013)
Capture RoundtripDuration to figure out 
NTP issues in subnet health analyzer.
2022-01-06 01:51:10 -08:00
Harshavardhana 89441a22aa
enforceRetentionForDeletion should return false early for delete-marker (#14033) 2022-01-05 17:05:28 -08:00
Minio Trusted 557135185c update helm to v3.4.5 2022-01-05 11:32:55 -08:00
Poorna 4d39fd4165
Add API for cluster replication status visibility (#13885) 2022-01-05 02:44:08 -08:00
Minio Trusted f4c03e56b8 Update yaml files to latest version RELEASE.2022-01-04T07-41-07Z 2022-01-04 23:01:29 +00:00
Minio Trusted d2b6aa9033 update console to v0.13.2 2022-01-03 23:41:07 -08:00
Minio Trusted 5dd40b9377 Update yaml files to latest version RELEASE.2022-01-03T18-22-58Z 2022-01-04 00:12:07 +00:00
Harshavardhana 001b77e7e1
use readConfig/saveConfig to simplify I/O on usage/tracker info (#14019) 2022-01-03 10:22:58 -08:00
Anis Elleuch 9d91d32d82
typo: Low capital in some JSON field names in log/audit output (#14020)
Use a low capital in some fields in JSON log/audit output to follow
other fields names.
2022-01-03 09:26:26 -08:00
Harshavardhana a60ac7ca17
fix: audit log to support object names in multipleObjectNames() handler (#14017) 2022-01-03 01:28:52 -08:00
Harshavardhana 42ba0da6b0
fix: initialize new drwMutex for each attempt in 'for {' loop. (#14009)
It is possible that GetLock() call remembers a previously
failed releaseAll() when there are networking issues, now
this state can have potential side effects.

This PR tries to avoid this side affect by making sure
to initialize NewNSLock() for each GetLock() attempts
made to avoid any prior state in the memory that can
interfere with the new lock grants.
2022-01-02 09:15:34 -08:00
Harshavardhana f527c708f2
run gofumpt cleanup across code-base (#14015) 2022-01-02 09:15:06 -08:00
Classmate Zhou 6f474982ed
docs: rename 'baremetal' to 'bare metal' (#14014) 2022-01-01 08:39:02 -08:00
Minio Trusted fd6cd52728 update helm to v3.4.4 2021-12-30 20:21:24 -08:00
Minio Trusted c9e49f4366 Update yaml files to latest version RELEASE.2021-12-29T06-49-06Z 2021-12-29 07:38:12 +00:00
Harshavardhana 46fd9f4a53 fix: update storage-class properly
fixes #14005
2021-12-28 22:49:06 -08:00
Harshavardhana 4da641533d add compliance markdown 2021-12-28 15:59:11 -08:00
Harshavardhana 79df2c7ce7
correctly calculate read quorum based on the available fileInfo (#14000)
The current usage of assuming `default` parity of `4` is not correct
for all objects stored on MinIO, objects in .minio.sys have maximum
parity, healing won't trigger on these objects due to incorrect
verification of quorum.
2021-12-28 15:33:03 -08:00
Yoann Guillerme 3e28af1723
docs: update TLS doc use -keyout instead of -key (#14001) 2021-12-28 12:51:38 -08:00
Harshavardhana 866a95de38
fix: choose appropriate quorum for a given erasure set (#13998)
multiObject delete should honor expected quorum
2021-12-28 12:41:52 -08:00
Minio Trusted 6aa0574a53 Update yaml files to latest version RELEASE.2021-12-27T07-23-18Z 2021-12-27 19:52:13 +00:00
Minio Trusted bb97eafa82 madmin-go v1.1.23 and pkg v1.1.11 2021-12-26 23:23:18 -08:00
Minio Trusted 51d5efee1b update go v0.13.1 2021-12-26 22:36:49 -08:00
Harshavardhana c980804514
trim values from envrionment files (#13991)
trim values to remove any spaces, newlines
from the files while importing credentials
and other values.
2021-12-25 22:02:54 -08:00
Harshavardhana b883803b21
fix: healing across pools removing dangling objects (#13990)
adds other simplifications to the code when running
namespace heals across pools.
2021-12-25 09:01:44 -08:00
Harshavardhana 7e3a7d7044
add healing for invalid shards by skipping the blocks (#13978)
Built on top of #13945, now we need to simply skip the
shards and its automated.
2021-12-23 23:01:46 -08:00
Harshavardhana 9ad6012782
simplify logger time and avoid possible crashes (#13986)
time.Format() is not necessary prematurely for JSON
marshalling, since JSON marshalling indeed defaults
to RFC3339Nano.

This also ensures the 'time' is remembered until its
logged and it is the same time when the 'caller'
invoked 'log' functions.
2021-12-23 15:33:54 -08:00
Aditya Manthramurthy 5a96cbbeaa
Fix user privilege escalation bug (#13976)
The AddUser() API endpoint was accepting a policy field. 
This API is used to update a user's secret key and account 
status, and allows a regular user to update their own secret key. 

The policy update is also applied though does not appear to 
be used by any existing client-side functionality.

This fix changes the accepted request body type and removes 
the ability to apply policy changes as that is possible via the 
policy set API.

NOTE: Changing passwords can be disabled as a workaround
for this issue by adding an explicit "Deny" rule to disable the API
for users.
2021-12-23 09:21:21 -08:00
Harshavardhana 416977436e rename MINIO_CACHE_.._MASTER_KEY to MINIO_CACHE_.._SECRET_KEY
fixes #13975
2021-12-22 12:11:07 -08:00
Harshavardhana 54ec0a1308
add configurable delta for skipping shards (#13967)
This PR is an attempt to make this configurable
as not all situations have same level of tolerable
delta, i.e disks are replaced days apart or even
hours.

There is also a possibility that nodes have drifted
in time, when NTP is not configured on the system.
2021-12-22 11:43:01 -08:00
Klaus Post ebd78e983f
Limit key size to 3K (#13974)
User is reporting `Error 1071 :Specified key was too long,max key 
length is 3072 bytes`.

Regression caused by #13414
2021-12-22 11:41:51 -08:00
Harshavardhana 1cf726348f
return meaningful error for disabled users (#13968)
fixes #13958
2021-12-22 11:40:21 -08:00
Harshavardhana 41f75e6d1b update helm to v3.4.3 2021-12-21 12:55:50 -08:00