MyFavMirrors/minio
1
0
Fork 0
mirror of https://github.com/minio/minio.git synced 2024-12-25 22:55:54 -05:00
Code Issues Packages Projects Releases Wiki Activity
9,093 Commits 4 Branches 506 Tags 189 MiB
70e1cbda21
Commit Graph

9093 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
moon
d158607f8e
fix(AuditLog): panic while st is nil (#13510) 2021-10-27 09:29:42 -07:00
Krishnan Parthasarathi
939fbb3c38
ilm: Make per-tier stats available via admin-tier-info (#13381) 2021-10-23 18:38:33 -07:00
Aditya Manthramurthy
3b9dfa9d29
Add IAM service account tests (#13502) 2021-10-23 09:36:57 -07:00
Minio Trusted
0c76fb57f2 Update yaml files to latest version RELEASE.2021-10-23T03-28-24Z 2021-10-23 07:46:29 +00:00
Harshavardhana
9694fa8d3a update console to v0.11.0 release 2021-10-22 20:28:24 -07:00
Anis Elleuch
20761e053e
replication: Fix replica stats during crawling (#13499) 
Also show replica stats with an ARN in Prometheus output.
 2021-10-22 19:13:50 -07:00
Aditya Manthramurthy
29d885b40f
Add IAM system tests (#13487) 
For internal IDP user, policy and groups
 2021-10-22 01:33:28 -07:00
Harshavardhana
087dc13965
fix: server in shutdown should return 503 instead of 403 (#13496) 
various situations where the client is retrying the request
server going through shutdown might incorrectly send 403
which is a non-retriable error, this PR allows for clients
when they retry an attempt to go to another healthy pod
or server in a distributed cluster - assuming it is a properly
load-balanced setup.
 2021-10-22 01:30:27 -07:00
Poorna K
e7f559c582
Fixes to replication metrics (#13493) 
For reporting ReplicaSize and loading initial
replication metrics correctly.
 2021-10-21 18:52:55 -07:00
Harshavardhana
52c5f6e152 remove extraneous whitespaces 2021-10-21 14:43:13 -07:00
Aditya Manthramurthy
26ca59859f
update LDAP doc (#13492) 
- clarify the login flow
- add some sections on configuration
- minor fixes to improve readability
 2021-10-21 14:41:32 -07:00
Klaus Post
23d6770ff9
Inspect: Preserve permission flags (#13490) 
Preserve permission from disk files. Can help identify issues.

Refactor GetRawData function to be cleaner.
 2021-10-21 11:20:13 -07:00
Harshavardhana
ac36a377b0
fix: remove deprecated jwks_url from config KV (#13477) 2021-10-20 11:31:09 -07:00
Klaus Post
1642867136
Add documentation for debugging tools (#13484) 
Move `xl-meta` so it can be installed out-of-repo with a single command.
 2021-10-20 10:12:46 -07:00
Shireesh Anjal
ce40392803
Capture TLS info in health report (#13470) 
So that TLS related checks can be added in subnet health-analyzer
 2021-10-20 10:12:01 -07:00
Aditya Manthramurthy
5f1af8a69d
For IAM with etcd backend, avoid sending notifications (#13472) 
As we use etcd's watch interface, we do not need the 
network notifications as they are no-ops anyway.

Bonus: Remove globalEtcdClient global usage in IAM
 2021-10-20 03:22:35 -07:00
Sidhartha Mani
c57ff2640e
recognize slow networks to step down faster during netperf (#13473) 2021-10-20 03:22:07 -07:00
Harshavardhana
d7fd396b7c update minio-go v7.0.15 2021-10-18 19:13:29 -07:00
Krishnan Parthasarathi
45d145a823
fix: immediate tiering for NoncurrentVersionTransition (#13464) 2021-10-18 17:24:30 -07:00
Aditya Manthramurthy
221ef78faa
Remove IAMSys dependency from IAMStorageAPI (#13436) 
IAMSys is a higher-level object, that should not be called by the lower-level
storage API interface for IAM. This is to prepare for further improvements in
IAM code.
 2021-10-18 11:21:57 -07:00
Anis Elleuch
d86513cbba
tls: Better error message when certificate curve is not supported (#13462) 2021-10-18 09:32:16 -07:00
Aditya Manthramurthy
25b5904b84
Enable sanity tests for internal IDP (#13457) 
Co-authored-by: Harshavardhana <harsha@minio.io>
 2021-10-18 09:31:55 -07:00
Klaus Post
c2eb60df4a
bz2: limit max concurrent CPU (#13458) 
Ensure that bz2 decompression will never take more than 50% CPU.
 2021-10-18 08:44:36 -07:00
Anis Elleuch
feabd0430c
etcd: Add logs for unusual failures (#13460) 
etcd operations, get/put/delete, should be logged when failed
with errors other than not found error. It will make it easier to
see connections issues from MinIO to etcd.
 2021-10-18 08:43:04 -07:00
Harshavardhana
838de23357
re-use rand.New() do not repeat allocate. (#13448) 
also simplify readerLocks to be just like
writeLocks, DRWMutex() is never shared
and there are order guarantees that need
for such a thing to work for RLock's
 2021-10-18 08:39:59 -07:00
Anis Elleuch
d7b7040408
tls: Avoid 3DES cipher (#13459) 
3DES is enabled by default in Golang, this commit will use
tls.CipherSuites() which returns all ciphers excluding those with
security issues, such as 3DES.
 2021-10-18 08:39:15 -07:00
Harshavardhana
44e4bdc6f4
restrict multi object delete > 1000 objects (#13454) 
AWS S3 returns error if > 1000 objects are sent
per MultiObject delete request, we should comply
no reason to not comply.
 2021-10-18 08:38:33 -07:00
Klaus Post
779060bc16
Locker: Improve Refresh speed (#13430) 
Refresh was doing a linear scan of all locked resources. This was adding 
up to significant delays in locking on high load systems with long 
running requests.

Add a secondary index for O(log(n)) UID -> resource lookups. 
Multiple resources are stored in consecutive strings.

Bonus fixes:

 * On multiple Unlock entries unlock the write locks we can.
 * Fix `expireOldLocks` skipping checks on entry after expiring one.
 * Return fast on canTakeUnlock/canTakeLock.
 * Prealloc some places.
 2021-10-15 03:12:13 -07:00
Klaus Post
76239fa1ae
Fix s3zip not returning data (#13442) 
We do not reliably know the length of compressed data, including headers.

Request until the end-of-stream. Results will still be properly truncated.

Fixes #13441
 2021-10-14 12:37:30 -07:00
Klaus Post
5e53f767c4
Use concurrent bz2 decompression (#13360) 
Testing with `mc sql --compression BZIP2 --csv-input "rd=\n,fh=USE,fd=;" --query="select COUNT(*) from S3Object" local2/testbucket/nyc-taxi-data-10M.csv.bz2`

Before 96.98s, after 10.79s. Uses about 70% CPU while running.
 2021-10-14 11:11:07 -07:00
Klaus Post
974073a2e5
directio: Check if buffers are set. (#13440) 
Check if directio buffers have actually been fetched and prevent errors on double Close. Return error on Read after Close.

Fixes

```
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xf8582f]

goroutine 210 [running]:
github.com/minio/minio/internal/ioutil.(*ODirectReader).Read(0xc0054f8320, {0xc0014560b0, 0xa8, 0x44d012})
	github.com/minio/minio/internal/ioutil/odirect_reader.go:88 +0x10f
io.ReadAtLeast({0x428c5c0, 0xc0054f8320}, {0xc0014560b0, 0xa8, 0xa8}, 0xa8)
	io/io.go:328 +0x9a
io.ReadFull(...)
	io/io.go:347
github.com/minio/minio/internal/ioutil.ReadFile({0xc001bf60e0, 0x6})
	github.com/minio/minio/internal/ioutil/read_file.go:48 +0x19b
github.com/minio/minio/cmd.(*FSObjects).scanBucket.func1({{0xc00444e1e0, 0x4d}, 0x0, {0xc0040cf240, 0xe}, {0xc0040cf24f, 0x18}, {0xc0040cf268, 0x18}, 0x0, ...})
	github.com/minio/minio/cmd/fs-v1.go:366 +0x1ea
github.com/minio/minio/cmd.(*folderScanner).scanFolder.func1({0xc00474a6a8, 0xc0065d6793}, 0x0)
	github.com/minio/minio/cmd/data-scanner.go:494 +0xb15
github.com/minio/minio/cmd.readDirFn({0xc002803e80, 0x34}, 0xc000670270)
	github.com/minio/minio/cmd/os-readdir_unix.go:172 +0x638
github.com/minio/minio/cmd.(*folderScanner).scanFolder(0xc002deeb40, {0x42dc9d0, 0xc00068cbc0}, {{0xc001c6e2d0, 0x27}, 0xc0023db8e0, 0x1}, 0xc0001c7ab0)
	github.com/minio/minio/cmd/data-scanner.go:427 +0xa8f
github.com/minio/minio/cmd.(*folderScanner).scanFolder.func2({{0xc001c6e2d0, 0x27}, 0xc0023db8e0, 0x27})
	github.com/minio/minio/cmd/data-scanner.go:549 +0xd0
github.com/minio/minio/cmd.(*folderScanner).scanFolder(0xc002deeb40, {0x42dc9d0, 0xc00068cbc0}, {{0xc0013fa9e0, 0xe}, 0x0, 0x1}, 0xc000670dd8)
	github.com/minio/minio/cmd/data-scanner.go:623 +0x205d
github.com/minio/minio/cmd.scanDataFolder({_, _}, {_, _}, {{{0xc0013fa9e0, 0xe}, 0x802, {0x210f15d2, 0xed8f903b8, 0x5bc0e80}, ...}, ...}, ...)
	github.com/minio/minio/cmd/data-scanner.go:333 +0xc51
github.com/minio/minio/cmd.(*FSObjects).scanBucket(_, {_, _}, {_, _}, {{{0xc0013fa9e0, 0xe}, 0x802, {0x210f15d2, 0xed8f903b8, ...}, ...}, ...})
	github.com/minio/minio/cmd/fs-v1.go:364 +0x305
github.com/minio/minio/cmd.(*FSObjects).NSScanner(0x42dc9d0, {0x42dc9d0, 0xc00068cbc0}, 0x0, 0xc003bcfda0, 0x802)
	github.com/minio/minio/cmd/fs-v1.go:307 +0xa16
github.com/minio/minio/cmd.runDataScanner({0x42dc9d0, 0xc00068cbc0}, {0x436a6c0, 0xc000bfcf50})
	github.com/minio/minio/cmd/data-scanner.go:150 +0x749
created by github.com/minio/minio/cmd.initDataScanner
	github.com/minio/minio/cmd/data-scanner.go:73 +0xb0
```
 2021-10-14 10:19:17 -07:00
Harshavardhana
d693431183
fix: ReadFileStream should return an error when size mismatches (#13435) 
offset+length should match the Size() of the individual parts
return 'errFileCorrupt' otherwise, to trigger healing of the individual 
parts do not error out prematurely when healing such bitrot's upon
successful parts being written to the client.

another issue this PR fixes is to not return and error to
the client if we have just triggered a heal on a specific
part of the object, instead continue to read all the content
and let the heal happen asynchronously later.
 2021-10-13 19:49:14 -07:00
Harshavardhana
bedf739d16 update required IAM policies 2021-10-13 12:28:53 -07:00
Harshavardhana
082755de1a update helm to v3.2.0 2021-10-12 19:16:24 -07:00
Minio Trusted
6299e42aa9 Update yaml files to latest version RELEASE.2021-10-13T00-23-17Z 2021-10-13 01:14:23 +00:00
Harshavardhana
129f41cee9 update dependencies for minio/console and minio/pkg 
IAM policy parser was incorrectly dropping duplicate
statements, this update brings the right fix for
these situations.
 2021-10-12 17:23:17 -07:00
Harshavardhana
415bbc74aa
checkKeyValid() should return owner true for rootCreds (#13422) 
Looks like policy restriction was not working properly
for normal users when they are not svc or STS accounts.

- svc accounts are now properly fixed to get
  right permissions when its inherited, so
  we do not have to set 'owner = true'

- sts accounts have always been using right
  permissions, do not need an explicit lookup

- regular users always have proper policy mapping
 2021-10-12 13:18:02 -07:00
Harshavardhana
13e41f2c68
fix: simplify loading IAM users to avoid using regular ListObjects() (#13392) 
- avoids relying in listQuorum from the underlying listObjects()
  and potentially missing entries if any.

- avoid the entire merging logic etc, listing raw set by set
  and loading whatever is found is cleaner when dealing with
  a large cluster for IAM metadata.
 2021-10-12 09:53:17 -07:00
Harshavardhana
1e117b780a
fix: validate exclusivity with partNumber regardless of valid Range (#13418) 
To mimic an exact AWS S3 behavior this fix is needed.
 2021-10-12 09:24:19 -07:00
Harshavardhana
f8c5c24159
force delete should just use rename() (#13417) 
use rename() instead of forced blocking
delete call, faster for large namespaces.
 2021-10-12 09:24:00 -07:00
Harshavardhana
f5a55c44d4
fix: do not overwrite error on fallback. (#13415) 
older content was returning '404' upon headObject()
due to swallowing of the error, make sure the
error is handling independently.

fixes #13397
 2021-10-11 19:48:42 -07:00
Aditya Manthramurthy
91a0e7bdaa
update mysql notification key length, character set and collation (#13414) 
fixes #13227
 2021-10-11 17:40:11 -07:00
Harshavardhana
b07e309627 fix: ignore empty values while parsing tlsEnabled value 2021-10-11 17:04:02 -07:00
Harshavardhana
9ea45399ce
fix: enable AssumeRoleWithCertificate API only when asked (#13410) 
This is a breaking change but we need to do this to avoid
issues discussed in #13409 based on discussions from #13371

fixes #13371
fixes #13409
 2021-10-11 14:23:51 -07:00
Harshavardhana
c19b1a143e
fix: allow service accounts for root credentials (#13412) 
fixes #13407
 2021-10-11 13:40:13 -07:00
Harshavardhana
02c24a860d
fix: crash in hard quota enforcement (#13403) 
due to data structure change after multi-site
replication, hard quota was broken due to
data structure change.

This PR fixes this.
 2021-10-11 11:03:54 -07:00
Klaus Post
9f652708ee
Fix Elastic crash with no index (#13406) 
Removed naked assert.

Fixes #13389
 2021-10-11 10:07:38 -07:00
Harshavardhana
05fa790584 update helm to v3.1.9 2021-10-10 14:28:04 -07:00
Minio Trusted
e0db822a9b Update yaml files to latest version RELEASE.2021-10-10T16-53-30Z 2021-10-10 21:24:01 +00:00
jiangfucheng
ec0fee6208
fix: the returned object key when object is directory (#13391) 2021-10-10 09:53:30 -07:00