1
0
mirror of https://github.com/minio/minio.git synced 2025-02-13 14:48:09 -05:00

9515 Commits

Author SHA1 Message Date
Harshavardhana
14d8a931fe
re-use io.Copy buffers with 32k pools ()
Borrowed idea from Go's usage of this
optimization for ReadFrom() on client
side, we should re-use the 32k buffers
io.Copy() allocates for generic copy
from a reader to writer.

the performance increase for reads for
really tiny objects is at this range
after this change.

> * Fastest: +7.89% (+1.3 MiB/s) throughput, +7.89% (+1308.1) obj/s
2021-11-02 08:11:50 -07:00
Harshavardhana
30ba85bc67
no need to write storageClass globally ()
fixes 
2021-11-02 08:11:20 -07:00
Aditya Manthramurthy
caadcc3ed8
Move all IAM storage functionality into iam store type ()
- Ensure all actions accessing storage lock properly.

- Behavior change: policies can be deleted only when they 
  are not associated with any active credentials.
2021-11-01 21:58:07 -07:00
Poorna K
26f55472c6
fix: clean up dangling buckets during bucket delete () 2021-11-01 21:52:45 -07:00
Aditya Manthramurthy
79a58e275c
fix: race in delete user functionality ()
- The race happens with a goroutine that refreshes IAM cache data from storage.
- It could lead to deleted users re-appearing as valid live credentials.
- This change also causes CI to run tests without a race flag (in addition to
running it with).
2021-11-01 15:03:07 -07:00
Aditya Manthramurthy
900e584514
CI: Cancel in-progress jobs when a PR is updated ()
- This should lead to faster results as jobs will be queued for shorter periods
when PRs are updated.

- Current behavior is that previously running CI jobs for an updated PR run to
completion needlessly, and cause new CI jobs to be queued.

Ref: https://docs.github.com/en/actions/learn-github-actions/workflow-syntax-for-github-actions#concurrency
2021-11-01 13:42:48 -07:00
Harshavardhana
bb639d9f29
remove double reads delete versions ()
deleting collection of versions belonging
to same object, we can avoid re-reading
the xl.meta from the disk instead purge
all the requested versions in-memory,

the tradeoff is to allocate a map to de-dup
the versions, allow disks to be read only
once per object.

additionally reduce the data transfer between
nodes by shortening msgp data values.
2021-11-01 10:50:07 -07:00
Poorna K
15dcacc1fc
Add support for caching multipart in writethrough mode () 2021-11-01 08:11:58 -07:00
Harshavardhana
6d53e3c2d7
reduce number of middleware handlers ()
- combine similar looking functionalities into single
  handlers, and remove unnecessary proxying of the
  requests at handler layer.

- remove bucket forwarding handler as part of default setup
  add it only if bucket federation is enabled.

Improvements observed for 1kiB object reads.
```
-------------------
Operation: GET
Operations: 4538555 -> 4595804
* Average: +1.26% (+0.2 MiB/s) throughput, +1.26% (+190.2) obj/s
* Fastest: +4.67% (+0.7 MiB/s) throughput, +4.67% (+739.8) obj/s
* 50% Median: +1.15% (+0.2 MiB/s) throughput, +1.15% (+173.9) obj/s
```
2021-11-01 08:04:03 -07:00
Klaus Post
8ed7346273
Disable AVX512 on Darwin ()
Preemptively disable AVX512 until https://github.com/golang/go/issues/49233 has been resolved.

This potentially affects reedsolomon, simdjson, sha256-simd, md5-simd packages.

Init order requires a separate package since main itself is initialized last, but imports are initialized in the order they are imported from main (confirmed).
2021-11-01 08:03:16 -07:00
Harshavardhana
3c1220adca add tests for default governance replication 2021-10-30 08:57:59 -07:00
Harshavardhana
4ed0eb7012
remove double reads updating object metadata ()
Removes RLock/RUnlock for updating metadata,
since we already take a write lock to update
metadata, this change removes reading of xl.meta
as well as an additional lock, the performance gain
should increase 3x theoretically for

- PutObjectRetention
- PutObjectLegalHold

This optimization is mainly for Veeam like
workloads that require a certain level of iops
from these API calls, we were losing iops.
2021-10-30 08:22:04 -07:00
Harshavardhana
2af5445309 update 3-site replication tests 2021-10-29 22:09:55 -07:00
Harshavardhana
abb1916bda
update list objects limit to match S3 spec 2021-10-28 18:21:51 -07:00
Klaus Post
9424dca9e4
jwt: Improve allocations ()
Avoid string -> byte allocations.

```
BenchmarkParseJWTStandardClaims-32       3527152           343.2 ns/op      1489 B/op         21 allocs/op
BenchmarkParseJWTStandardClaims-32       4713199           259.2 ns/op       706 B/op         16 allocs/op

BenchmarkParseJWTMapClaims-32        2666668           448.7 ns/op      1883 B/op         32 allocs/op
BenchmarkParseJWTMapClaims-32        3120709           377.1 ns/op      1227 B/op         28 allocs/op
```
2021-10-28 17:04:48 -07:00
Harshavardhana
db84bb9bd3
avoid atomics for self contained reader/writers ()
read/writers are not concurrent in handlers
and self contained - no need to use atomics on
them.

avoids unnecessary contentions where it's not
required.
2021-10-28 17:03:00 -07:00
Klaus Post
c603f85488
readAllData: Reuse small file buffers ()
(Re)use small buffers for small readAllData operations.
2021-10-28 17:02:22 -07:00
Aditya Manthramurthy
2f1ee25f50
Add test for AssumeRole with internal IDP () 2021-10-28 09:05:51 -07:00
Klaus Post
7bdf9005e5
Remove HTTP flushes for returning handlers ()
When handlers return they are automatically flushed. Manual flushing can force responsewriters to use suboptimal paths and generally just wastes CPU.
2021-10-28 07:36:34 -07:00
Klaus Post
d9c1d79e30
Protect logger targets ()
Logger targets were not race protected against concurrent updates from for example `HTTPConsoleLoggerSys`.

Restrict direct access to targets and make slices immutable so a returned slice can be processed safely without locks.
2021-10-28 07:35:28 -07:00
Harshavardhana
bd88b86919 update console to latest to fix CVE-2021-42836 2021-10-27 21:14:02 -07:00
Minio Trusted
8e29ae8c44 Update yaml files to latest version RELEASE.2021-10-27T16-29-42Z 2021-10-28 02:45:22 +00:00
moon
d158607f8e
fix(AuditLog): panic while st is nil () RELEASE.2021-10-27T16-29-42Z 2021-10-27 09:29:42 -07:00
Krishnan Parthasarathi
939fbb3c38
ilm: Make per-tier stats available via admin-tier-info () 2021-10-23 18:38:33 -07:00
Aditya Manthramurthy
3b9dfa9d29
Add IAM service account tests () 2021-10-23 09:36:57 -07:00
Minio Trusted
0c76fb57f2 Update yaml files to latest version RELEASE.2021-10-23T03-28-24Z 2021-10-23 07:46:29 +00:00
Harshavardhana
9694fa8d3a update console to v0.11.0 release RELEASE.2021-10-23T03-28-24Z 2021-10-22 20:28:24 -07:00
Anis Elleuch
20761e053e
replication: Fix replica stats during crawling ()
Also show replica stats with an ARN in Prometheus output.
2021-10-22 19:13:50 -07:00
Aditya Manthramurthy
29d885b40f
Add IAM system tests ()
For internal IDP user, policy and groups
2021-10-22 01:33:28 -07:00
Harshavardhana
087dc13965
fix: server in shutdown should return 503 instead of 403 ()
various situations where the client is retrying the request
server going through shutdown might incorrectly send 403
which is a non-retriable error, this PR allows for clients
when they retry an attempt to go to another healthy pod
or server in a distributed cluster - assuming it is a properly
load-balanced setup.
2021-10-22 01:30:27 -07:00
Poorna K
e7f559c582
Fixes to replication metrics ()
For reporting ReplicaSize and loading initial
replication metrics correctly.
2021-10-21 18:52:55 -07:00
Harshavardhana
52c5f6e152 remove extraneous whitespaces 2021-10-21 14:43:13 -07:00
Aditya Manthramurthy
26ca59859f
update LDAP doc ()
- clarify the login flow
- add some sections on configuration
- minor fixes to improve readability
2021-10-21 14:41:32 -07:00
Klaus Post
23d6770ff9
Inspect: Preserve permission flags ()
Preserve permission from disk files. Can help identify issues.

Refactor GetRawData function to be cleaner.
2021-10-21 11:20:13 -07:00
Harshavardhana
ac36a377b0
fix: remove deprecated jwks_url from config KV () 2021-10-20 11:31:09 -07:00
Klaus Post
1642867136
Add documentation for debugging tools ()
Move `xl-meta` so it can be installed out-of-repo with a single command.
2021-10-20 10:12:46 -07:00
Shireesh Anjal
ce40392803
Capture TLS info in health report ()
So that TLS related checks can be added in subnet health-analyzer
2021-10-20 10:12:01 -07:00
Aditya Manthramurthy
5f1af8a69d
For IAM with etcd backend, avoid sending notifications ()
As we use etcd's watch interface, we do not need the 
network notifications as they are no-ops anyway.

Bonus: Remove globalEtcdClient global usage in IAM
2021-10-20 03:22:35 -07:00
Sidhartha Mani
c57ff2640e
recognize slow networks to step down faster during netperf () 2021-10-20 03:22:07 -07:00
Harshavardhana
d7fd396b7c update minio-go v7.0.15 2021-10-18 19:13:29 -07:00
Krishnan Parthasarathi
45d145a823
fix: immediate tiering for NoncurrentVersionTransition () 2021-10-18 17:24:30 -07:00
Aditya Manthramurthy
221ef78faa
Remove IAMSys dependency from IAMStorageAPI ()
IAMSys is a higher-level object, that should not be called by the lower-level
storage API interface for IAM. This is to prepare for further improvements in
IAM code.
2021-10-18 11:21:57 -07:00
Anis Elleuch
d86513cbba
tls: Better error message when certificate curve is not supported () 2021-10-18 09:32:16 -07:00
Aditya Manthramurthy
25b5904b84
Enable sanity tests for internal IDP ()
Co-authored-by: Harshavardhana <harsha@minio.io>
2021-10-18 09:31:55 -07:00
Klaus Post
c2eb60df4a
bz2: limit max concurrent CPU ()
Ensure that bz2 decompression will never take more than 50% CPU.
2021-10-18 08:44:36 -07:00
Anis Elleuch
feabd0430c
etcd: Add logs for unusual failures ()
etcd operations, get/put/delete, should be logged when failed
with errors other than not found error. It will make it easier to
see connections issues from MinIO to etcd.
2021-10-18 08:43:04 -07:00
Harshavardhana
838de23357
re-use rand.New() do not repeat allocate. ()
also simplify readerLocks to be just like
writeLocks, DRWMutex() is never shared
and there are order guarantees that need
for such a thing to work for RLock's
2021-10-18 08:39:59 -07:00
Anis Elleuch
d7b7040408
tls: Avoid 3DES cipher ()
3DES is enabled by default in Golang, this commit will use
tls.CipherSuites() which returns all ciphers excluding those with
security issues, such as 3DES.
2021-10-18 08:39:15 -07:00
Harshavardhana
44e4bdc6f4
restrict multi object delete > 1000 objects ()
AWS S3 returns error if > 1000 objects are sent
per MultiObject delete request, we should comply
no reason to not comply.
2021-10-18 08:38:33 -07:00
Klaus Post
779060bc16
Locker: Improve Refresh speed ()
Refresh was doing a linear scan of all locked resources. This was adding 
up to significant delays in locking on high load systems with long 
running requests.

Add a secondary index for O(log(n)) UID -> resource lookups. 
Multiple resources are stored in consecutive strings.

Bonus fixes:

 * On multiple Unlock entries unlock the write locks we can.
 * Fix `expireOldLocks` skipping checks on entry after expiring one.
 * Return fast on canTakeUnlock/canTakeLock.
 * Prealloc some places.
2021-10-15 03:12:13 -07:00