Commit Graph

12352 Commits

Author SHA1 Message Date
Harshavardhana
54ecce66f0 move readLock on uploadId at EOF in PutObjectPart (#258)
we do not need to hold the read locks at the higher
layer instead before reading the body, instead hold
the read locks properly at the time of renamePart()
for protection from racy part overwrites to compete
with concurrent completeMultipart().
2024-12-13 14:06:18 -08:00
Artur Melanchyk
2b008c598b
fix: replace mutex with atomic (#20762) 2024-12-13 19:32:46 +05:30
dependabot[bot]
86d02b17cf
Bump golang.org/x/crypto from 0.23.0 to 0.31.0 in /docs/debugging/s3-verify (#20757)
Bump golang.org/x/crypto in /docs/debugging/s3-verify

Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.23.0 to 0.31.0.
- [Commits](https://github.com/golang/crypto/compare/v0.23.0...v0.31.0)

---
updated-dependencies:
- dependency-name: golang.org/x/crypto
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-12-12 05:00:05 -08:00
Anis Eleuch
c1a95a70ac
heal: Move CheckParts from single handler to streaming RPC (#20755)
CheckParts call can take time to verify 10k parts of a single object in a single drive.
To avoid an internal dealine of one minute in the single handler RPC, this commit will
switch to streaming RPC instead.
2024-12-12 11:20:57 +05:30
Aditya Manthramurthy
f246c9053f
fix: Privilege escalation in IAM import API (#20756)
This API had missing permissions checking, allowing a user to change
their policy mapping by:

1. Craft iam-info.zip file: Update own user permission in
user_mappings.json
2. Upload it via `mc admin cluster iam import nobody iam-info.zip`

Here `nobody` can be a user with pretty much any kind of permission (but
not anonymous) and this ends up working.

Some more detailed steps - start from a fresh setup:

```
./minio server /tmp/d{1...4} &
mc alias set myminio http://localhost:9000 minioadmin minioadmin
mc admin user add myminio nobody nobody123
mc admin policy attach myminio readwrite nobody nobody123
mc alias set nobody http://localhost:9000 nobody nobody123

mc admin cluster iam export myminio
mkdir /tmp/x && mv myminio-iam-info.zip /tmp/x
cd /tmp/x
unzip myminio-iam-info.zip
echo '{"nobody":{"version":1,"policy":"consoleAdmin","updatedAt":"2024-08-13T19:47:10.1Z"}}' > \
      iam-assets/user_mappings.json
zip -r myminio-iam-info-updated.zip iam-assets/

mc admin cluster iam import nobody ./myminio-iam-info-updated.zip
mc admin service restart nobody
```
2024-12-12 07:39:40 +05:30
Cesar N.
9cdd204ae4
Upgrade Console version to v1.7.5 (#20748) 2024-12-11 16:23:53 +05:30
Harshavardhana
7b3eb9f7f8
fix: groups lookup performance issue with users with lots of groups (#20740)
fixes https://github.com/minio/minio/issues/20717
2024-12-11 16:23:28 +05:30
ebozduman
d56ef8dbe1
Adds AIstore documentation link (#20738) 2024-12-11 16:22:17 +05:30
Mark Theunissen
a248ed5ff5
Fixes for POST policy checks and the x-ignore implementation (#20674) 2024-12-11 16:21:34 +05:30
Klaus Post
5bb31e4883
Disable mint full object tests (#20743)
Remove expected failures from https://github.com/minio/minio-go/pull/2026
2024-12-09 18:59:22 -08:00
Taran Pelkey
aff2a76d80
Return error when attempting to create a policy with commas in name (#20724) 2024-12-04 03:51:26 -08:00
Anis Eleuch
eddbe6bca2
heal: Report bucket healing result correctly (#20721) 2024-12-04 04:42:25 +05:30
Anis Eleuch
734d1e320a
heal: Single object heal to look for older versions as well (#203) (#20723)
`mc admin heal ALIAS/bucket/object` does not have any flag to heal
object noncurrent versions, this commit will make healing of the object
noncurrent versions implicitly asked.

This also fixes the 'mc admin heal ALIAS/bucket/object' that does not work 
correctly when the bucket is versioned. This has been broken since Apr 2023.
2024-12-04 04:42:04 +05:30
Anis Eleuch
b8dab7b1a9
Set http server read/write timeout from --idle-timeout (#228) (#20715)
Golang http.Server will call SetReadDeadline overwriting the previous
deadline configuration set after a new connection Accept in the custom
listener code. Therefore, --idle-timeout was not correctly respected.

Make http.Server read/write timeout similar to --idle-timeout.
2024-12-02 18:51:17 +05:30
Klaus Post
abd6bf060d
Add 'X-Forwarded-For' to (s)FTP requests (#20709)
Fixes #20707
2024-11-29 18:25:37 +05:30
Alex
f0d4ef604c
Updated Console to v1.7.4 (#20693)
Signed-off-by: Benjamin Perez <benjamin@bexsoft.net>
2024-11-28 13:14:11 +05:30
Ramon de Klein
2712f75762
prevent IAM cleanup errors (#20691) 2024-11-28 13:13:51 +05:30
Krutika Dhananjay
4c46668da8
Add a test case for fix #20684 (#20688)
The test fails without the change.
Also, removed a duplicate test case involving lifecycle config with no rules.
2024-11-28 13:13:31 +05:30
Anis Eleuch
02e93fd6ba
heal: Better reporting to mc with dangling/timeout errors (#20690)
The code assigns corrupted state to a drive for any unexpected error,
which is confusing for users. This change will make sure to assign
corrupted state only for corrupted parts or xl.meta. Use unknown state
with a explanation for any unexpected error, like canceled, deadline
errors, drive timeout, ...

Also make sure to return the bucket/object name when the object is not
found or marked not found by the heal dangling code.
2024-11-26 10:45:35 -08:00
Krutika Dhananjay
366876e98b
Fix prefix validation in lifecycle rule (#20684) 2024-11-25 12:12:21 -08:00
Mark Theunissen
d202fdd022
Add the policy name to the audit logs tags when doing policy-based API calls. Add retention settings to tags (#20638)
* Add the policy name to the audit log tags when doing policy-based API calls

* Audit log the retention settings requested in the API call

* Audit log of retention on PutObjectRetention API path too
2024-11-25 09:17:12 -08:00
Eng Zer Jun
c07e5b49d4
refactor: replace experimental maps and slices with stdlib (#20679)
The experimental functions are now available in the standard library in
Go 1.23 [1].

[1]: https://go.dev/doc/go1.23#new-unique-package

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2024-11-25 09:10:22 -08:00
Aditya Manthramurthy
9a39f8ad4d
fix: Remove User should fail for a service account (#20677)
The RemoveUser API only removes internal users, and it reports success
when it didnt find the internal user account for deletion. When provided
with a service account, it should not report success as that is misleading.
2024-11-21 18:24:04 -08:00
Nao Yonashiro
7e0c1c9413
feat: bump github.com/cosnicolaou/pbzip2 from 1.0.3 to 1.0.5 (#20671)
Update github.com/cosnicolaou/pbzip2 to latest version for
significant performance improvements. This update brings a 45%
reduction in processing time.
2024-11-20 18:53:52 -08:00
John Morales
485d833cd7
fix: API label casing and count value for +Inf bucket v2 metrics (#20656) 2024-11-18 11:17:52 -08:00
Klaus Post
e8a476ef5a
Keep larger merge buffers for RPC (#20654)
Keep larger merge buffers

When sending large messages >1K, the merge buffer would continuously be reallocated.

This could happen on listings, where blocks are typically 4->8K.

Keep merge buffer of up to 256KB.

Benchmark with 4096b messages:

```
benchmark                                          old ns/op     new ns/op     delta
BenchmarkRequests/servers=2/bytes/par=32-32        8271          6360          -23.10%
BenchmarkRequests/servers=2/bytes/par=64-32        7840          4731          -39.66%
BenchmarkRequests/servers=2/bytes/par=128-32       7291          4740          -34.99%
BenchmarkRequests/servers=2/bytes/par=256-32       7095          4580          -35.45%
BenchmarkRequests/servers=2/bytes/par=512-32       6757          4584          -32.16%
BenchmarkRequests/servers=2/bytes/par=1024-32      6429          4453          -30.74%

benchmark                                          old bytes     new bytes     delta
BenchmarkRequests/servers=2/bytes/par=32-32        12090         821           -93.21%
BenchmarkRequests/servers=2/bytes/par=64-32        17423         820           -95.29%
BenchmarkRequests/servers=2/bytes/par=128-32       18493         822           -95.56%
BenchmarkRequests/servers=2/bytes/par=256-32       18892         821           -95.65%
BenchmarkRequests/servers=2/bytes/par=512-32       19064         826           -95.67%
BenchmarkRequests/servers=2/bytes/par=1024-32      19038         842           -95.58%
```
2024-11-16 09:18:48 -08:00
Krutika Dhananjay
267f0ecea2
Fix 0 httpTimeout for logger webhook (#20653)
Previously, not setting http.Config.HTTPTimeout for logger webhook
was resulting in a timeout of 0, and causing "deadline exceeded"
errors in log webhook.

This change introduces a new env variable for configuring log webhook
timeout and more importantly sets it when the config is initialised.
2024-11-16 02:01:36 -08:00
Harshavardhana
4ee3434854
updating all dependencies as per regular cadence (#20646) 2024-11-14 12:33:18 -08:00
Anis Eleuch
0e9854372e
heal/batch: Fix missing redirection to the first node (#20642)
Manual heal can return XMinioHealInvalidClientToken if the manual
healing is started in the first node, and the next mc call to get the
heal status is landed on another node. The reason is that redirection
based on the token ID is not able to redirect requests to the first node
due to a typo.

This also affects the batch cancel command if the batch is being done in
the first node, the user will never be able to cancel it due to the same
bug.
2024-11-13 04:07:28 -08:00
Klaus Post
b5177993b3
Make DeadlineConn http.Listener compatible (#20635)
HTTP likes to slap an infinite read deadline on a connection and 
do a blocking read while the response is being written.

This effectively means that a reading deadline becomes the 
request-response deadline.

Instead of enforcing our timeout, we pass it through and keep 
"infinite deadline" is sticky on connections.

However, we still "record" when reads are aborted, so we never overwrite that.

The HTTP server should have `ReadTimeout` and `IdleTimeout` set for the deadline to be effective.

Use --idle-timeout for incoming connections.
2024-11-12 12:41:41 -08:00
Klaus Post
55f5c18fd9
Harden internode DeadlineConn (#20631)
Since DeadlineConn would send deadline updates directly upstream,
it would race with Read/Write operations. The stdlib will perform a read, 
but do an async SetReadDeadLine(unix(1)) to cancel the Read in 
`abortPendingRead`. In this case, the Read may override the 
deadline intended to cancel the read.

Stop updating deadlines if a deadline in the past is seen and when Close is called. 
A mutex now protects all upstream deadline calls to avoid races. 

This should fix the short-term buildup of...

```
365 @ 0x44112e 0x4756b9 0x475699 0x483525 0x732286 0x737407 0x73816b 0x479601
#	0x475698	sync.runtime_notifyListWait+0x138		runtime/sema.go:569
#	0x483524	sync.(*Cond).Wait+0x84				sync/cond.go:70
#	0x732285	net/http.(*connReader).abortPendingRead+0xa5	net/http/server.go:729
#	0x737406	net/http.(*response).finishRequest+0x86		net/http/server.go:1676
#	0x73816a	net/http.(*conn).serve+0x62a			net/http/server.go:2050
```

AFAICT Only affects internode calls that create a connection (non-grid).
2024-11-11 09:15:17 -08:00
Harshavardhana
8ce101c174 fix: LDAP service port number in tests 2024-11-11 07:01:29 -08:00
Klaus Post
4972735507
Fix lint issues from v1.62.0 upgrade (#20633)
* Fix lint issues from v1.62.0 upgrade

* Fix xlMetaV2TrimData version checks.
2024-11-11 06:51:43 -08:00
Minio Trusted
e6ca6de194 Update yaml files to latest version RELEASE.2024-11-07T00-52-20Z 2024-11-07 23:58:42 +00:00
Harshavardhana
cefc43e4da simplify the Get()/GetMultiple() re-use GetRaw() for both (#179)
Remember GetMultiple() must be used if your target is calling
PutMultiple(), without that the multiple events will not be
replayed.
2024-11-06 16:52:20 -08:00
Ramon de Klein
25e34fda5f
decompress audit log properly before sending to remote target (#20619) 2024-11-06 13:25:24 -08:00
Erfan
4208d7af5a
docs: remove redundant prometheus metric (#20618) 2024-11-06 07:44:21 -08:00
Klaus Post
8d42f37e4b
Fix msgUnPath crash (#20614)
These are needed checks for the functions to be un-crashable with any input 
given to `msgUnPath` (tested with fuzzing).

Both conditions would result in a crash, which prevents that. Some 
additional upstream checks are needed.

Fixes #20610
2024-11-05 04:37:59 -08:00
dependabot[bot]
7cb4b5c636
Bump github.com/golang-jwt/jwt/v4 from 4.5.0 to 4.5.1 (#20611)
Bumps [github.com/golang-jwt/jwt/v4](https://github.com/golang-jwt/jwt) from 4.5.0 to 4.5.1.
- [Release notes](https://github.com/golang-jwt/jwt/releases)
- [Changelog](https://github.com/golang-jwt/jwt/blob/main/VERSION_HISTORY.md)
- [Commits](https://github.com/golang-jwt/jwt/compare/v4.5.0...v4.5.1)

---
updated-dependencies:
- dependency-name: github.com/golang-jwt/jwt/v4
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-11-05 02:22:36 -08:00
Harshavardhana
1615920f48 fix typos reported in CI/CD 2024-11-04 11:06:02 -08:00
Cesar N.
7ee42b3ff5
Update console package to v1.7.3 (#20606) 2024-11-04 08:47:08 -08:00
Harshavardhana
a6f1e727fb
add tests for ILM transition and healing (#166) (#20601)
This PR fixes a regression introduced in https://github.com/minio/minio/pull/19797
by restoring the healing ability of transitioned objects

Bonus: support for transitioned objects to carry original
The object name is for future reverse lookups if necessary.

Also fix parity calculation for tiered objects to n/2 for n/2 == (parity)
2024-10-31 15:10:24 -07:00
Aditya Manthramurthy
c1fc7779ca
Remove expires field from list objects metadata (#20600)
This field was always 0 regardless of whether the object had an expiry
so we are basically removing dead code.
2024-10-31 12:27:06 -07:00
Allan Roger Reid
b3ab7546ee
Fix typos in README.md (#20599) 2024-10-31 10:38:53 -07:00
Minio Trusted
ad88a81e3d Update yaml files to latest version RELEASE.2024-10-29T16-01-48Z 2024-10-30 21:24:58 +00:00
Aditya Manthramurthy
c4239ced22
run IAM purge routines deterministically every hr (#20587)
Existing implementation runs IAM purge routines for expired LDAP and
OIDC accounts with a probability of 0.25 after every IAM refresh. This
change ensures that they are run once in each hour.
2024-10-29 09:01:48 -07:00
Anis Eleuch
f85c28e960
heal: large objects fix and avoid .healing.bin corner case premature exit (#20577)
xlStorage.Healing() returns nil if there is an error reading
.healing.bin or if this latter is empty. healing.bin update()
call returns early if .healing.bin is empty; hence, no further update
of .healing.bin is possible.

A .healing.bin can be empty if os.Open() with O_TRUNC is successful
but the next Write returns an error.

To avoid this weird situation, avoid making healingTracker.update()
to return early if .healing.bin is empty, so write again.

This commit also fixes wrong error log printing when an object is 
healed in another drive in the same erasure set but not in the drive 
that is actively healing by fresh drive healing code. Currently, it prints 
<nil> instead of a factual error.

* heal: Scan .minio.sys metadata only during site-wide heal (#137)

mc admin heal always invoke .minio.sys heal, but sometimes, this latter
contains a lot of data, many service accounts, STS accounts etc, which
makes mc admin heal command very slow.

Only invoke .minio.sys healing when no bucket was specified in `mc admin
heal` command.
2024-10-26 02:58:27 -07:00
Anis Eleuch
f7e176d4ca
heal: Avoid deadline error with very large objects (#140) (#20586)
Healing a large object with a normal scan mode where no parts read 
is involved can still fail after 30 seconds if an object has

There are too many parts when hard disks are being used mainly. 
The reason is there is a general deadline that checks for all parts we 
do a deadline per part.
2024-10-26 02:56:26 -07:00
Aditya Manthramurthy
72a0d14195
fix: avoid useless expires value in listing meta (#20584)
When listing objects with metadata, avoid returning an "expires" time
metadata value when its value is the zero time as this means that no
expires value is set on the object.
2024-10-24 19:13:19 -07:00
Klaus Post
6abe4128d7
Fix ILM expire workers exiting (#20578)
Fix expire workers exiting

Under 2 conditions ILM expire workers would exit, eventually causing all workers to terminate.
2024-10-23 08:35:37 -07:00