Commit Graph

9675 Commits

Author SHA1 Message Date
Harshavardhana
32b2f6117e
fix: do not pass around sync.Map (#15250)
it is not safe to pass around sync.Map
through pointers, as it may be concurrently
updated by different callers.

this PR simplifies by avoiding sync.Map
altogether, we do not need sync.Map
to keep object->erasureMap association.

This PR fixes a crash when concurrently
using this value when audit logs are
configured.

```
fatal error: concurrent map iteration and map write

goroutine 247651580 [running]:
runtime.throw({0x277a6c1?, 0xc002381400?})
        runtime/panic.go:992 +0x71 fp=0xc004d29b20 sp=0xc004d29af0 pc=0x438671
runtime.mapiternext(0xc0d6e87f18?)
        runtime/map.go:871 +0x4eb fp=0xc004d29b90 sp=0xc004d29b20 pc=0x41002b
```
2022-07-07 17:04:25 -07:00
Harshavardhana
ae92521310
remove unnecessary nAgreed value in partial() func (#15242) 2022-07-07 13:45:34 -07:00
Harshavardhana
5802df4365
retry and resume decom operation upon retriable failures (#15244)
it is possible in a k8s-like system reading pool.bin
might not have quorum during startup, however, add
a way to retry after this failure.
2022-07-07 12:31:44 -07:00
Minio Trusted
c1901f4e12 Update yaml files to latest version RELEASE.2022-07-06T20-29-49Z 2022-07-07 00:24:36 +00:00
Anis Elleuch
8d98282afd
Better reporting of total/free usable capacity of the cluster (#15230)
The current code uses approximation using a ratio. The approximation 
can skew if we have multiple pools with different disk capacities.

Replace the algorithm with a simpler one which counts data 
disks and ignore parity disks.
2022-07-06 13:29:49 -07:00
Harshavardhana
dd839bf295
add NATS JetStream support (#15201) 2022-07-06 13:29:08 -07:00
Harshavardhana
3af6073576
no 'replicate status' without replication config (#15233)
'replicate status' shouldn't be displaying historic
values unless replication config is present on the
relevant bucket.
2022-07-06 09:53:33 -07:00
Harshavardhana
2518af5f9e
fix: allow certain mutations on objects during decommissioning (#15231)
fix: allow certain mutation on objects during decommission

currently by mistake deletion of objects was skipped,
if the object resided on the pool being decommissioned.

delete's are okay to be allowed since decommission is
designed to run on a cluster with active I/O.
2022-07-06 09:53:16 -07:00
Harshavardhana
7b793d84c8
fix: calculate scanner metric paths for single drive (#15232)
Additionally use pathJoin() to avoid double `//`
in path names.
2022-07-06 07:48:38 -07:00
Aditya Manthramurthy
af9bc7ea7d
Add external IDP management Admin API for OpenID (#15152) 2022-07-05 18:18:04 -07:00
Klaus Post
ac055b09e9
Add detailed scanner metrics (#15161) 2022-07-05 14:45:49 -07:00
haslersn
df42914da6
Fix missing whitespace in error message for IncompleteBody (#15227) 2022-07-05 12:19:57 -07:00
Klaus Post
2471bdda00
fix: for DiskInfo call cache disk metrics (#15229)
Small uploads spend a significant amount of time (~5%) fetching disk info metrics. Also maps are allocated for each call.

Add a 100ms cache to disk metrics.
2022-07-05 11:02:30 -07:00
dorman
c7e01b139d
helm: service port set to minioAPIPort in helm (#15223) 2022-07-05 07:38:04 -07:00
Harshavardhana
9d80ff5a05
fix: decommission delete markers for non-current objects (#15225)
versioned buckets were not creating the delete markers
present in the versioned stack of an object, this essentially
would stop decommission to succeed.

This PR fixes creating such delete markers properly during
a decommissioning process, adds tests as well.
2022-07-05 07:37:24 -07:00
Minio Trusted
39b3941892 Update yaml files to latest version RELEASE.2022-07-04T21-02-54Z 2022-07-04 21:51:54 +00:00
Harshavardhana
b311abed31
decom IAM, Bucket metadata properly (#15220)
Current code incorrectly passed the
config asset object name while decommissioning,
make sure that we pass the right object name
to be hashed on the newer set of pools.

This PR fixes situations after a successful
decommission, the users and policies might go
missing due to wrong hashed set.
2022-07-04 14:02:54 -07:00
Harshavardhana
ce667ddae0
do not print errFileNotFound in entries.resolve() (#15216) 2022-07-04 06:40:46 -07:00
Harshavardhana
0fee993a4b
return appropriate error under 'decom status' (#15213)
fixes #15208
2022-07-01 16:21:23 -07:00
Poorna
0ea5c9d8e8
site healing: Skip stale iam asset updates from peer. (#15203)
Allow healing to apply IAM change only when peer
gave the most recent update.
2022-07-01 13:19:13 -07:00
Harshavardhana
63ac260bd5
Simplify Prometheus metrics gather (#15210) 2022-07-01 13:18:39 -07:00
Minio Trusted
a01a39b153 Update yaml files to latest version RELEASE.2022-06-30T20-58-09Z 2022-07-01 00:44:04 +00:00
Harshavardhana
f9a4ad7904
update banner with version+runtime (#15206) 2022-06-30 13:58:09 -07:00
Minio Trusted
e60b67d246 Revert "Tighten enforcement of object retention (#14993)"
This reverts commit 5e3010d455.

This commit causes regression on object locked buckets causine
delete-markers to be not created.
2022-06-30 13:06:32 -07:00
Klaus Post
9004d69c6f
Make ReqInfo concurrency safe (#15204)
Some read/writes of ReqInfo did not get appropriate locks, leading to races.

Make sure reading and writing holds appropriate locks.
2022-06-30 10:48:50 -07:00
Harshavardhana
8856a2d77b
finalize startup-banner and remove unnecessary logs (#15202) 2022-06-29 16:32:04 -07:00
Anis Elleuch
54a061bdda
Save minio version information centrally (#15181) 2022-06-29 14:45:49 -07:00
Harshavardhana
65b4b100a8
de-couple caller context to avoid internal races (#15195)
```
fatal error: concurrent map iteration and map write
fatal error: concurrent map iteration and map write

goroutine 745335841 [running]:
runtime.throw({0x273e67b?, 0x80?})
        runtime/panic.go:992 +0x71 fp=0xc0390bc240 sp=0xc0390bc210 pc=0x438671
runtime.mapiternext(0x40d987?)
        runtime/map.go:871 +0x4eb fp=0xc0390bc2b0 sp=0xc0390bc240 pc=0x41002b
runtime.mapiterinit(0x46bec7?, 0x4ef76c?, 0xc0017cc9c0?)
        runtime/map.go:861 +0x228 fp=0xc0390bc2d0 sp=0xc0390bc2b0 pc=0x40fae8
reflect.mapiterinit(0x1b5?, 0xc0?, 0x235bcc0?)
```

```
github.com/minio/minio/internal/rest/client.go:151 +0x5f4 fp=0xc0390bd988 sp=0xc0390bd730 pc=0x153e434
```
2022-06-29 14:44:26 -07:00
Poorna
7cc9286e0f
site healing: Skip stale bucket metadata updates from peer (#15186)
Allow healing to apply bucket metadata change only when peer
gave the most recent update.
2022-06-28 18:09:20 -07:00
Harshavardhana
2f25639ea0
update banner to reflect the final agreed UI (#15192) 2022-06-28 16:37:40 -07:00
Harshavardhana
2070c215a2
handle missing funcNames for handlers (#15188)
also use designated names for internal
calls

- storageREST calls are storageR
- lockREST calls are lockR
- peerREST calls are just peer

Named in this fashion to facilitate wildcard matches
by having prefixes of the same name.

Additionally, also enable funcNames for generic handlers
that return errors, currently we disable '<unknown>'
2022-06-28 05:04:10 -07:00
Minio Trusted
94b98222c2 update minio-go/v7 to v7.0.30 2022-06-27 21:12:22 -07:00
Harshavardhana
9c605ad153
allow support for parity '0', '1' enabling support for 2,3 drive setups (#15171)
allows for further granular setups

- 2 drives (1 parity, 1 data)
- 3 drives (1 parity, 2 data)

Bonus: allows '0' parity as well.
2022-06-27 20:22:18 -07:00
Anis Elleuch
b7c7e59dac
Revert proxying requests with precondition errors (#15180)
In a replicated setup, when an object is updated in one cluster but
still waiting to be replicated to the other cluster, GET requests with
if-match, and range headers will likely fail. It is better to proxy
requests instead.

Also, this commit avoids printing verbose logs about precondition &
range errors.
2022-06-27 14:03:44 -07:00
Klaus Post
767c1436d3
Upgrade reedsolomon/compression packages (#15182)
reedsolomon/cpuid would take a long time to start up on Xen VMs with 
AMD processors due to a bug in the VM CPUID implementation.

Compression upgraded for better speed/compression.
2022-06-27 13:07:42 -07:00
Harshavardhana
699cf6ff45
perform object sweep after equeue the latest CopyObject() (#15183)
keep it similar to PutObject/CompleteMultipart
2022-06-27 12:11:33 -07:00
Anis Elleuch
9201870f6c
Remove unnecessary code in WalkDir() (#15168)
Recalculating forward is useless. It is never used and it will be
computed again when calling scanDir() again.
2022-06-27 10:26:56 -07:00
Harshavardhana
6722f58668
save MinIO version with each version (8-bytes extra) (#15170)
store MinIO version along with each version in 'xl.meta'
for future purposes, can be used as ways to add specific
code for bug fixes if any.
2022-06-27 03:59:41 -07:00
Harshavardhana
7b9b7cef11
add license banner for GNU AGPLv3 (#15178)
Bonus: rewrite subnet re-use of Transport
2022-06-27 03:58:25 -07:00
Minio Trusted
7d4fce09dc update RedHat UBI image to 8.6 2022-06-26 09:14:23 -07:00
Minio Trusted
2075501d86 Update yaml files to latest version RELEASE.2022-06-25T15-50-16Z 2022-06-26 16:09:28 +00:00
Harshavardhana
bd099f5e71
fix: change timedValue to return the previously cached value (#15169)
fix: change timedvalue to return previous cached value

caller can interpret the underlying error and decide
accordingly, places where we do not interpret the
errors upon timedValue.Get() - we should simply use
the previously cached value instead of returning "empty".

Bonus: remove some unused code
2022-06-25 08:50:16 -07:00
Klaus Post
baf257adcb
fix: health client leak when calling UpdateAllTargets (#15167)
When `LoadBucketMetadataHandler` is called and `UpdateAllTargets` gets called.

Since targets are rebuilt we cancel all.
2022-06-24 11:12:52 -07:00
Anis Elleuch
4fd1986885
Trace all http requests (#15064)
Add a generic handler that adds a new tracing context to the request if
tracing is enabled. Other handlers are free to modify the tracing
context to update information on the fly, such as, func name, enable
body logging etc..

With this commit, requests like this 

```
curl -H "Host: ::1:3000" http://localhost:9000/
```

will be traced as well.
2022-06-23 23:19:24 -07:00
Harshavardhana
e1afac9439
reduce sha256 CPU usage by turning it off for speedtests (#15154)
continuation of the PR #15151, keeping signature v4 for
the headers however avoiding sha256 for the body.
2022-06-23 11:26:53 -07:00
Poorna
580d9db85e
Add APIs to import/export IAM data (#15014) 2022-06-23 09:25:15 -07:00
Anis Elleuch
42e2fd35d8
heal: Include dir markers when healing a fresh disk (#15158)
Directories markers are not healed when healing a new fresh disk. A
a proper fix would be moving object names encoding/decoding to erasure
object level but it is too late now since the object to set distribution is
calculated at a higher level.
2022-06-23 06:47:33 -07:00
Harshavardhana
1a40c7c27c
use signature-v2 for 'object perf' tests to avoid CPU using sha256 (#15151)
It is observed in a local 8 drive system the CPU seems to be
bottlenecked at

```
(pprof) top
Showing nodes accounting for 1385.31s, 88.47% of 1565.88s total
Dropped 1304 nodes (cum <= 7.83s)
Showing top 10 nodes out of 159
      flat  flat%   sum%        cum   cum%
      724s 46.24% 46.24%       724s 46.24%  crypto/sha256.block
   219.04s 13.99% 60.22%    226.63s 14.47%  syscall.Syscall
   158.04s 10.09% 70.32%    158.04s 10.09%  runtime.memmove
   127.58s  8.15% 78.46%    127.58s  8.15%  crypto/md5.block
    58.67s  3.75% 82.21%     58.67s  3.75%  github.com/minio/highwayhash.updateAVX2
    40.07s  2.56% 84.77%     40.07s  2.56%  runtime.epollwait
    33.76s  2.16% 86.93%     33.76s  2.16%  github.com/klauspost/reedsolomon._galMulAVX512Parallel84
     8.88s  0.57% 87.49%     11.56s  0.74%  runtime.step
     7.84s   0.5% 87.99%      7.84s   0.5%  runtime.memclrNoHeapPointers
     7.43s  0.47% 88.47%     22.18s  1.42%  runtime.pcvalue
```

Bonus changes:

- re-use transport for bucket replication clients, also site replication clients.
- use 32KiB buffer for all read and writes at transport layer seems to help
  TLS read connections.
- Do not have 'MaxConnsPerHost' this is problematic to be used with net/http
  connection pooling 'MaxIdleConnsPerHost' is enough.
2022-06-22 16:28:25 -07:00
Anis Elleuch
f3bec41eb9
s3-verify: Add a flag to exclude younger than a certain age (#15142)
--minimum-object-age 1h can help exclude objects that are newly
uploaded but not replicated yet
2022-06-22 08:12:47 -07:00
Andreas Auernhammer
825634d24e
fips: fix order of elliptic curves (#15141)
This commit fixes the order of elliptic curves.
As documented by https://pkg.go.dev/crypto/tls#Config
```
// CurvePreferences contains the elliptic curves that will be used in
// an ECDHE handshake, in preference order. If empty, the default will
// be used. The client will use the first preference as the type for
// its key share in TLS 1.3. This may change in the future.
```

In general, we should prefer `X25519` over the NIST curves.

Signed-off-by: Andreas Auernhammer <hi@aead.dev>
2022-06-22 08:09:28 -07:00