Commit Graph

11290 Commits

Author SHA1 Message Date
Minio Trusted 99c1a642a4 Update yaml files to latest version RELEASE.2023-10-25T06-33-25Z 2023-10-25 07:52:49 +00:00
Harshavardhana c60f54e5be
make ListMultipart/ListParts more reliable skip healing disks (#18312)
this PR also fixes old flaky tests, by properly marking disk offline-based tests.
2023-10-24 23:33:25 -07:00
Harshavardhana 483389f2e2 set diskMaxConcurrent to 32 if nrRequests is lower 2023-10-24 17:21:12 -07:00
Harshavardhana c0f2f84285
avoid racy replicationCount checks (#18311)
resync status may not be upto-date by
the time the resync is over due to how
the timer is triggered.

diff is sufficient to know if replication
happened or not.
2023-10-24 15:30:42 -07:00
Harshavardhana 069d118329
fix: listObjectParts to prefer local and single disks (#18309) 2023-10-24 13:51:57 -07:00
Harshavardhana a7b1834772
fix: flaky and stupid tests in root lockdown (#18308) 2023-10-24 13:22:44 -07:00
Klaus Post 6415dec37a
Improve multipart listing speed (#18307) 2023-10-24 12:06:06 -07:00
Klaus Post 74253e1ddc
Fix BackendInfo() race (#18305)
`GetParityForSC` has a value receiver, so Config is copied before the lock is obtained.

Make it pointer receiver.

Fixes:

```
WARNING: DATA RACE
Read at 0x0000079cdd10 by goroutine 190:
  github.com/minio/minio/cmd.(*erasureServerPools).BackendInfo()
      github.com/minio/minio/cmd/erasure-server-pool.go:579 +0x6f
  github.com/minio/minio/cmd.(*erasureServerPools).LocalStorageInfo()
      github.com/minio/minio/cmd/erasure-server-pool.go:614 +0x3c6
  github.com/minio/minio/cmd.(*peerRESTServer).LocalStorageInfoHandler()
      github.com/minio/minio/cmd/peer-rest-server.go:347 +0x4ea
  github.com/minio/minio/cmd.(*peerRESTServer).LocalStorageInfoHandler-fm()
...

WARNING: DATA RACE
Read at 0x0000079cdd10 by goroutine 190:
  github.com/minio/minio/cmd.(*erasureServerPools).BackendInfo()
      github.com/minio/minio/cmd/erasure-server-pool.go:579 +0x6f
  github.com/minio/minio/cmd.(*erasureServerPools).LocalStorageInfo()
      github.com/minio/minio/cmd/erasure-server-pool.go:614 +0x3c6
  github.com/minio/minio/cmd.(*peerRESTServer).LocalStorageInfoHandler()
      github.com/minio/minio/cmd/peer-rest-server.go:347 +0x4ea
  github.com/minio/minio/cmd.(*peerRESTServer).LocalStorageInfoHandler-fm()
```
2023-10-24 08:15:41 -07:00
Minio Trusted 01b3fb91e5 Update yaml files to latest version RELEASE.2023-10-24T04-42-36Z 2023-10-24 05:05:12 +00:00
Harshavardhana 2dc917e87f
maxConcurrent must be set only once per node (#18303) 2023-10-23 21:42:36 -07:00
Aditya Manthramurthy 0a284a1a10
fix: SR: Add more info when IAM config differs (#18302)
Provide details on what IAM info mismatched when the validation fails
2023-10-23 21:16:40 -07:00
Harshavardhana 5c8339e1e8
fix: veeam SOS API to higher layers (#18287)
- support populating usage info from scanner info
- support populating quota for the bucket via quota
  settings for the bucket
2023-10-23 13:55:45 -07:00
Harshavardhana fd37418da2
fix: allow server not initialized error to be retried (#18300)
Since relaxing quorum the error across pools
for ListBuckets(), GetBucketInfo() we hit a
situation where loading IAM could potentially
return an error for second pool that server
is not initialized.

We need to handle this, let the pool come online
and retry transparently - this PR fixes that.
2023-10-23 12:30:20 -07:00
Harshavardhana bbfea29c2b
use object modTime for the event sequencer ID (#18285)
always set modTime after lock is acquired in
completemultipart stage to make sure that the
modTime is not racy.
2023-10-20 19:28:05 -07:00
Harshavardhana aa703dc903
relax write quorum requirement for ListBuckets()/HeadBucket() (#18288)
Also fix error handling for HeadBucket() to be pool specific
2023-10-20 17:50:21 -07:00
Krishnan Parthasarathi 8cd80fec8c
Add unit test for lifecycle.FilterRules (#18284) 2023-10-19 21:33:28 -07:00
Harshavardhana 780882efcf
do not check for query params to be signed headers (#18283)
x-amz-signed-headers is meant for HTTP headers only
not for query params, using that to verify things
further can lead to failure.

The generated presigned URL with custom metadata
is already kosher (tamper proof).

fixes #18281
2023-10-19 21:32:49 -07:00
dependabot[bot] c5636143c6
build(deps): bump github.com/nats-io/nats-server/v2 from 2.9.20 to 2.9.23 (#18282)
build(deps): bump github.com/nats-io/nats-server/v2

Bumps [github.com/nats-io/nats-server/v2](https://github.com/nats-io/nats-server) from 2.9.20 to 2.9.23.
- [Release notes](https://github.com/nats-io/nats-server/releases)
- [Changelog](https://github.com/nats-io/nats-server/blob/main/.goreleaser.yml)
- [Commits](https://github.com/nats-io/nats-server/compare/v2.9.20...v2.9.23)

---
updated-dependencies:
- dependency-name: github.com/nats-io/nats-server/v2
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-10-19 16:40:59 -07:00
Klaus Post ba6218b354
fix: resource metrics "concurrent map iteration and map write" (#18273)
`resourceMetricsMap` has no protection against concurrent reads and writes.

Add a mutex and don't use maps from the last iteration.

Bug introduced in #18057

Fixes #18271
2023-10-18 13:28:50 -07:00
Harshavardhana 8e32de3ba9
cache DiskInfo() metrics call separately (#18270) 2023-10-18 11:17:32 -07:00
Klaus Post e37508fb8f
fix: linter errors in Windows specific code (#18276) 2023-10-18 11:08:15 -07:00
Klaus Post b46a717425
Remove unused config migration (#18277)
None of the migration is called. Remove dead code.
2023-10-18 11:05:24 -07:00
Klaus Post 7926df0b80
Fix globalDeploymentID race (#18275)
globalDeploymentID was being read while it was being set.

Fixes race:

```
WARNING: DATA RACE
Write at 0x0000079605a0 by main goroutine:
  github.com/minio/minio/cmd.connectLoadInitFormats()
      github.com/minio/minio/cmd/prepare-storage.go:269 +0x14f0
  github.com/minio/minio/cmd.waitForFormatErasure()
      github.com/minio/minio/cmd/prepare-storage.go:294 +0x21d
...

Previous read at 0x0000079605a0 by goroutine 105:
  github.com/minio/minio/cmd.newContext()
      github.com/minio/minio/cmd/utils.go:817 +0x31e
  github.com/minio/minio/cmd.adminMiddleware.func1()
      github.com/minio/minio/cmd/admin-router.go:110 +0x96
  net/http.HandlerFunc.ServeHTTP()
      net/http/server.go:2136 +0x47
  github.com/minio/minio/cmd.setBucketForwardingMiddleware.func1()
      github.com/minio/minio/cmd/generic-handlers.go:460 +0xb1a
  net/http.HandlerFunc.ServeHTTP()
      net/http/server.go:2136 +0x47
...
```
2023-10-18 08:06:57 -07:00
Krishnan Parthasarathi 557df666fd
Don't skip rules with ExpiredObjectDeleteMarker (#18256) 2023-10-16 22:46:46 -07:00
Harshavardhana f91b257f50
choose different max_concurrent requests per drive based on HDD/NVMe (#18254)
currently the default for all drives is 512, which is a lot
for HDDs the recent testing has revealed moving this to 32
for HDDs seems like a fair value.
2023-10-16 17:18:13 -07:00
Aditya Manthramurthy 28a2d1eb3d
Allow OpenID ARN resource ID to start with a `-` (#18255) 2023-10-16 13:50:51 -07:00
Minio Trusted a0ae1489e5 Update yaml files to latest version RELEASE.2023-10-16T04-13-43Z 2023-10-16 05:58:49 +00:00
Harshavardhana edfb310a59
fix: always load ENVs from files first as soon as server starts (#18247)
This is a regression from #18231, however reading from ENV files
must happen well before any parsing logic is invoked.
2023-10-15 21:13:43 -07:00
Minio Trusted a2312028b9 Update yaml files to latest version RELEASE.2023-10-14T05-17-22Z 2023-10-14 06:27:13 +00:00
Poorna 78f1f69d57
fix site replication resync status (#18245)
To persist status changes on disk upon completion.

Adds new tests to handle this functionality.
2023-10-13 22:17:22 -07:00
Harshavardhana e1e33077e8
fix: tests and resync replication status (#18244) 2023-10-13 17:03:34 -07:00
Aditya Manthramurthy b3e7de010d
Remove usage of errors.Join for go1.19 compat (#18243) 2023-10-13 15:14:16 -07:00
Satish Michael f5b04865f4
Helm Chart: Added "MINIO_IDENTITY_OPENID_REDIRECT_URI" Env Var (#18236)
added redirect uri env

Signed-off-by: Satish Kumar Kadarkarai Main <michael.satish@gmail.com>
2023-10-13 07:46:54 -07:00
Shireesh Anjal bf1c6edb76
Revert "Capture network device info in health report" (#18241)
Introducing a new version of healthinfo struct for adding this info is
not correct. It needs to be implemented differently without adding a new
version.

This reverts commit 8737025d940f80360ed4b3686b332db5156f6659.
2023-10-13 07:46:36 -07:00
jiuker 2ac7fee017
fix: missing fileName will upload failed when PostPolicyBucketHandler (#18240) 2023-10-13 07:31:23 -07:00
Klaus Post 128256e3ab
Add event counters (#18232)
Export metric for global events sent and skipped for the lifetime of the server.
2023-10-12 15:39:22 -07:00
Shireesh Anjal a66a7f3e97
Capture network device info in health report (#18213) 2023-10-12 15:33:31 -07:00
jiuker 20b79f8945
fix: env depend on the flag (#18231) 2023-10-12 15:32:38 -07:00
Klaus Post 9a877734b2
Fix various poolmeta races (#18230)
There is a fundamental race condition in `newErasureServerPools`, where setObjectLayer is 
called before the poolMeta has been loaded/populated.

We add a placeholder value to this field but disable all saving of the value, so we don't risk 
overwriting the value on disk. Once the value has been loaded or created, it is replaced with 
the proper value, which will also be saved.

Also fixes various accesses of `poolMeta` that were done without locks.

We make the `poolMeta.IsSuspended` return false, even if we shouldn't risk out-of-bounds 
reads anymore.
2023-10-12 15:30:42 -07:00
Harshavardhana 409c391850
implement helpers to get relevant info instead of FileInfo() (#18228) 2023-10-12 15:29:59 -07:00
Klaus Post 763ff085a6
Add CI tests for next branch (#18224) 2023-10-12 06:15:10 -07:00
Shubhendu 5b9656374c
Error if target went offline (#18221)
If target went offline while MinIO was down, error once
while trying to send message. If target goes offline during
MinIO server running, it already comes through ping() call
and errors out if target offline.

Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
2023-10-12 06:13:57 -07:00
dependabot[bot] b32014549c
build(deps): bump golang.org/x/net from 0.15.0 to 0.17.0 (#18219)
Bumps [golang.org/x/net](https://github.com/golang/net) from 0.15.0 to 0.17.0.
- [Commits](https://github.com/golang/net/compare/v0.15.0...v0.17.0)

---
updated-dependencies:
- dependency-name: golang.org/x/net
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-10-12 01:54:36 -07:00
dependabot[bot] 9476d212bc
build(deps): bump golang.org/x/net from 0.14.0 to 0.17.0 in /docs/debugging/s3-verify (#18218)
build(deps): bump golang.org/x/net in /docs/debugging/s3-verify

Bumps [golang.org/x/net](https://github.com/golang/net) from 0.14.0 to 0.17.0.
- [Commits](https://github.com/golang/net/compare/v0.14.0...v0.17.0)

---
updated-dependencies:
- dependency-name: golang.org/x/net
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-10-12 00:39:12 -07:00
jiuker 000928d34e
fix: should call func globalOSMetrics.time(s)() when updateOSMetrics (#18209) 2023-10-12 00:08:13 -07:00
Harshavardhana 6829ae5b13
completely remove drive caching layer from gateway days (#18217)
This has already been deprecated for close to a year now.
2023-10-11 21:18:17 -07:00
jiuker f09756443d
fix: a dynamic config will make a panic for addOrUpdateIDP (#18208) 2023-10-11 09:06:40 -07:00
jiuker 5512016885
fix: siteResyncMetrics init will make a deadlock when len(siteReplication) >= 3 (#18206) 2023-10-10 23:27:27 -07:00
Harshavardhana 21ecb941fe
fix: avoid counting out of band deletes during disk heal (#18205) 2023-10-10 14:39:48 -07:00
Harshavardhana 77e94087cf
fix: calling statfs() call moves the disk head (#18203)
if erasure upgrade is needed rely on the in-memory
values, instead of performing a "DiskInfo()" call.

https://brendangregg.com/blog/2016-09-03/sudden-disk-busy.html

for HDDs these are problematic, lets avoid this because
there is no value in "being" absolutely strict here
in terms of parity. We are okay to increase parity
as we see based on the in-memory online/offline ratio.
2023-10-10 13:47:35 -07:00