939fbb3c38
ilm: Make per-tier stats available via admin-tier-info ( #13381 )
2021-10-23 18:38:33 -07:00
3b9dfa9d29
Add IAM service account tests ( #13502 )
2021-10-23 09:36:57 -07:00
0c76fb57f2
Update yaml files to latest version RELEASE.2021-10-23T03-28-24Z
2021-10-23 07:46:29 +00:00
9694fa8d3a
update console to v0.11.0 release
2021-10-22 20:28:24 -07:00
20761e053e
replication: Fix replica stats during crawling ( #13499 )
...
Also show replica stats with an ARN in Prometheus output.
2021-10-22 19:13:50 -07:00
29d885b40f
Add IAM system tests ( #13487 )
...
For internal IDP user, policy and groups
2021-10-22 01:33:28 -07:00
087dc13965
fix: server in shutdown should return 503 instead of 403 ( #13496 )
...
various situations where the client is retrying the request
server going through shutdown might incorrectly send 403
which is a non-retriable error, this PR allows for clients
when they retry an attempt to go to another healthy pod
or server in a distributed cluster - assuming it is a properly
load-balanced setup.
2021-10-22 01:30:27 -07:00
e7f559c582
Fixes to replication metrics ( #13493 )
...
For reporting ReplicaSize and loading initial
replication metrics correctly.
2021-10-21 18:52:55 -07:00
52c5f6e152
remove extraneous whitespaces
2021-10-21 14:43:13 -07:00
26ca59859f
update LDAP doc ( #13492 )
...
- clarify the login flow
- add some sections on configuration
- minor fixes to improve readability
2021-10-21 14:41:32 -07:00
23d6770ff9
Inspect: Preserve permission flags ( #13490 )
...
Preserve permission from disk files. Can help identify issues.
Refactor GetRawData function to be cleaner.
2021-10-21 11:20:13 -07:00
ac36a377b0
fix: remove deprecated jwks_url from config KV ( #13477 )
2021-10-20 11:31:09 -07:00
1642867136
Add documentation for debugging tools ( #13484 )
...
Move `xl-meta` so it can be installed out-of-repo with a single command.
2021-10-20 10:12:46 -07:00
ce40392803
Capture TLS info in health report ( #13470 )
...
So that TLS related checks can be added in subnet health-analyzer
2021-10-20 10:12:01 -07:00
5f1af8a69d
For IAM with etcd backend, avoid sending notifications ( #13472 )
...
As we use etcd's watch interface, we do not need the
network notifications as they are no-ops anyway.
Bonus: Remove globalEtcdClient global usage in IAM
2021-10-20 03:22:35 -07:00
c57ff2640e
recognize slow networks to step down faster during netperf ( #13473 )
2021-10-20 03:22:07 -07:00
d7fd396b7c
update minio-go v7.0.15
2021-10-18 19:13:29 -07:00
45d145a823
fix: immediate tiering for NoncurrentVersionTransition ( #13464 )
2021-10-18 17:24:30 -07:00
221ef78faa
Remove IAMSys dependency from IAMStorageAPI ( #13436 )
...
IAMSys is a higher-level object, that should not be called by the lower-level
storage API interface for IAM. This is to prepare for further improvements in
IAM code.
2021-10-18 11:21:57 -07:00
d86513cbba
tls: Better error message when certificate curve is not supported ( #13462 )
2021-10-18 09:32:16 -07:00
25b5904b84
Enable sanity tests for internal IDP ( #13457 )
...
Co-authored-by: Harshavardhana <harsha@minio.io>
2021-10-18 09:31:55 -07:00
c2eb60df4a
bz2: limit max concurrent CPU ( #13458 )
...
Ensure that bz2 decompression will never take more than 50% CPU.
2021-10-18 08:44:36 -07:00
feabd0430c
etcd: Add logs for unusual failures ( #13460 )
...
etcd operations, get/put/delete, should be logged when failed
with errors other than not found error. It will make it easier to
see connections issues from MinIO to etcd.
2021-10-18 08:43:04 -07:00
838de23357
re-use rand.New() do not repeat allocate. ( #13448 )
...
also simplify readerLocks to be just like
writeLocks, DRWMutex() is never shared
and there are order guarantees that need
for such a thing to work for RLock's
2021-10-18 08:39:59 -07:00
d7b7040408
tls: Avoid 3DES cipher ( #13459 )
...
3DES is enabled by default in Golang, this commit will use
tls.CipherSuites() which returns all ciphers excluding those with
security issues, such as 3DES.
2021-10-18 08:39:15 -07:00
44e4bdc6f4
restrict multi object delete > 1000 objects ( #13454 )
...
AWS S3 returns error if > 1000 objects are sent
per MultiObject delete request, we should comply
no reason to not comply.
2021-10-18 08:38:33 -07:00
779060bc16
Locker: Improve Refresh speed ( #13430 )
...
Refresh was doing a linear scan of all locked resources. This was adding
up to significant delays in locking on high load systems with long
running requests.
Add a secondary index for O(log(n)) UID -> resource lookups.
Multiple resources are stored in consecutive strings.
Bonus fixes:
* On multiple Unlock entries unlock the write locks we can.
* Fix `expireOldLocks` skipping checks on entry after expiring one.
* Return fast on canTakeUnlock/canTakeLock.
* Prealloc some places.
2021-10-15 03:12:13 -07:00
76239fa1ae
Fix s3zip not returning data ( #13442 )
...
We do not reliably know the length of compressed data, including headers.
Request until the end-of-stream. Results will still be properly truncated.
Fixes #13441
2021-10-14 12:37:30 -07:00
5e53f767c4
Use concurrent bz2 decompression ( #13360 )
...
Testing with `mc sql --compression BZIP2 --csv-input "rd=\n,fh=USE,fd=;" --query="select COUNT(*) from S3Object" local2/testbucket/nyc-taxi-data-10M.csv.bz2`
Before 96.98s, after 10.79s. Uses about 70% CPU while running.
2021-10-14 11:11:07 -07:00
974073a2e5
directio: Check if buffers are set. ( #13440 )
...
Check if directio buffers have actually been fetched and prevent errors on double Close. Return error on Read after Close.
Fixes
```
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xf8582f]
goroutine 210 [running]:
github.com/minio/minio/internal/ioutil.(*ODirectReader).Read(0xc0054f8320, {0xc0014560b0, 0xa8, 0x44d012})
github.com/minio/minio/internal/ioutil/odirect_reader.go:88 +0x10f
io.ReadAtLeast({0x428c5c0, 0xc0054f8320}, {0xc0014560b0, 0xa8, 0xa8}, 0xa8)
io/io.go:328 +0x9a
io.ReadFull(...)
io/io.go:347
github.com/minio/minio/internal/ioutil.ReadFile({0xc001bf60e0, 0x6})
github.com/minio/minio/internal/ioutil/read_file.go:48 +0x19b
github.com/minio/minio/cmd.(*FSObjects).scanBucket.func1({{0xc00444e1e0, 0x4d}, 0x0, {0xc0040cf240, 0xe}, {0xc0040cf24f, 0x18}, {0xc0040cf268, 0x18}, 0x0, ...})
github.com/minio/minio/cmd/fs-v1.go:366 +0x1ea
github.com/minio/minio/cmd.(*folderScanner).scanFolder.func1({0xc00474a6a8, 0xc0065d6793}, 0x0)
github.com/minio/minio/cmd/data-scanner.go:494 +0xb15
github.com/minio/minio/cmd.readDirFn({0xc002803e80, 0x34}, 0xc000670270)
github.com/minio/minio/cmd/os-readdir_unix.go:172 +0x638
github.com/minio/minio/cmd.(*folderScanner).scanFolder(0xc002deeb40, {0x42dc9d0, 0xc00068cbc0}, {{0xc001c6e2d0, 0x27}, 0xc0023db8e0, 0x1}, 0xc0001c7ab0)
github.com/minio/minio/cmd/data-scanner.go:427 +0xa8f
github.com/minio/minio/cmd.(*folderScanner).scanFolder.func2({{0xc001c6e2d0, 0x27}, 0xc0023db8e0, 0x27})
github.com/minio/minio/cmd/data-scanner.go:549 +0xd0
github.com/minio/minio/cmd.(*folderScanner).scanFolder(0xc002deeb40, {0x42dc9d0, 0xc00068cbc0}, {{0xc0013fa9e0, 0xe}, 0x0, 0x1}, 0xc000670dd8)
github.com/minio/minio/cmd/data-scanner.go:623 +0x205d
github.com/minio/minio/cmd.scanDataFolder({_, _}, {_, _}, {{{0xc0013fa9e0, 0xe}, 0x802, {0x210f15d2, 0xed8f903b8, 0x5bc0e80}, ...}, ...}, ...)
github.com/minio/minio/cmd/data-scanner.go:333 +0xc51
github.com/minio/minio/cmd.(*FSObjects).scanBucket(_, {_, _}, {_, _}, {{{0xc0013fa9e0, 0xe}, 0x802, {0x210f15d2, 0xed8f903b8, ...}, ...}, ...})
github.com/minio/minio/cmd/fs-v1.go:364 +0x305
github.com/minio/minio/cmd.(*FSObjects).NSScanner(0x42dc9d0, {0x42dc9d0, 0xc00068cbc0}, 0x0, 0xc003bcfda0, 0x802)
github.com/minio/minio/cmd/fs-v1.go:307 +0xa16
github.com/minio/minio/cmd.runDataScanner({0x42dc9d0, 0xc00068cbc0}, {0x436a6c0, 0xc000bfcf50})
github.com/minio/minio/cmd/data-scanner.go:150 +0x749
created by github.com/minio/minio/cmd.initDataScanner
github.com/minio/minio/cmd/data-scanner.go:73 +0xb0
```
2021-10-14 10:19:17 -07:00
d693431183
fix: ReadFileStream should return an error when size mismatches ( #13435 )
...
offset+length should match the Size() of the individual parts
return 'errFileCorrupt' otherwise, to trigger healing of the individual
parts do not error out prematurely when healing such bitrot's upon
successful parts being written to the client.
another issue this PR fixes is to not return and error to
the client if we have just triggered a heal on a specific
part of the object, instead continue to read all the content
and let the heal happen asynchronously later.
2021-10-13 19:49:14 -07:00
bedf739d16
update required IAM policies
2021-10-13 12:28:53 -07:00
082755de1a
update helm to v3.2.0
2021-10-12 19:16:24 -07:00
6299e42aa9
Update yaml files to latest version RELEASE.2021-10-13T00-23-17Z
2021-10-13 01:14:23 +00:00
129f41cee9
update dependencies for minio/console and minio/pkg
...
IAM policy parser was incorrectly dropping duplicate
statements, this update brings the right fix for
these situations.
2021-10-12 17:23:17 -07:00
415bbc74aa
checkKeyValid() should return owner true for rootCreds ( #13422 )
...
Looks like policy restriction was not working properly
for normal users when they are not svc or STS accounts.
- svc accounts are now properly fixed to get
right permissions when its inherited, so
we do not have to set 'owner = true'
- sts accounts have always been using right
permissions, do not need an explicit lookup
- regular users always have proper policy mapping
2021-10-12 13:18:02 -07:00
13e41f2c68
fix: simplify loading IAM users to avoid using regular ListObjects() ( #13392 )
...
- avoids relying in listQuorum from the underlying listObjects()
and potentially missing entries if any.
- avoid the entire merging logic etc, listing raw set by set
and loading whatever is found is cleaner when dealing with
a large cluster for IAM metadata.
2021-10-12 09:53:17 -07:00
1e117b780a
fix: validate exclusivity with partNumber regardless of valid Range ( #13418 )
...
To mimic an exact AWS S3 behavior this fix is needed.
2021-10-12 09:24:19 -07:00
f8c5c24159
force delete should just use rename() ( #13417 )
...
use rename() instead of forced blocking
delete call, faster for large namespaces.
2021-10-12 09:24:00 -07:00
f5a55c44d4
fix: do not overwrite error on fallback. ( #13415 )
...
older content was returning '404' upon headObject()
due to swallowing of the error, make sure the
error is handling independently.
fixes #13397
2021-10-11 19:48:42 -07:00
91a0e7bdaa
update mysql notification key length, character set and collation ( #13414 )
...
fixes #13227
2021-10-11 17:40:11 -07:00
b07e309627
fix: ignore empty values while parsing tlsEnabled value
2021-10-11 17:04:02 -07:00
9ea45399ce
fix: enable AssumeRoleWithCertificate API only when asked ( #13410 )
...
This is a breaking change but we need to do this to avoid
issues discussed in #13409 based on discussions from #13371
fixes #13371
fixes #13409
2021-10-11 14:23:51 -07:00
c19b1a143e
fix: allow service accounts for root credentials ( #13412 )
...
fixes #13407
2021-10-11 13:40:13 -07:00
02c24a860d
fix: crash in hard quota enforcement ( #13403 )
...
due to data structure change after multi-site
replication, hard quota was broken due to
data structure change.
This PR fixes this.
2021-10-11 11:03:54 -07:00
9f652708ee
Fix Elastic crash with no index ( #13406 )
...
Removed naked assert.
Fixes #13389
2021-10-11 10:07:38 -07:00
05fa790584
update helm to v3.1.9
2021-10-10 14:28:04 -07:00
e0db822a9b
Update yaml files to latest version RELEASE.2021-10-10T16-53-30Z
2021-10-10 21:24:01 +00:00
ec0fee6208
fix: the returned object key when object is directory ( #13391 )
2021-10-10 09:53:30 -07:00
a188554fe1
Add missing keys to API config help ( #13255 )
...
Added missing `apiClusterDeadline` and `apiListQuorum` to API config.HelpKVS structure
2021-10-10 09:52:21 -07:00
Krishnan Parthasarathi
Aditya Manthramurthy
Minio Trusted
Harshavardhana
Anis Elleuch
Poorna K
Klaus Post
Shireesh Anjal
Sidhartha Mani
jiangfucheng
David Regla