12376 Commits

Author SHA1 Message Date
Klaus Post
827004cd6d
Add Full Object Checksums and CRC64-NVME (#20855)
Backport of AIStor PR 247.

Add support for full object checksums as described here:

https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html

New checksum types are fully supported. Mint tests from https://github.com/minio/minio-go/pull/2026 are now passing.

Includes fixes from https://github.com/minio/minio/pull/20743 for mint tests.

Add using checksums as validation for object content. Fixes #20845 #20849

Fixes checksum replication (downstream PR 250)
RELEASE.2025-01-20T14-49-07Z
2025-01-20 06:49:07 -08:00
Harshavardhana
779ec8f0d4
do not list buckets without local quorum (#20852)
ListBuckets() would result in listing buckets
without quorum, this PR fixes the behavior.
2025-01-19 15:13:17 -08:00
Minio Trusted
3d0f513ee2 Update yaml files to latest version RELEASE.2025-01-18T00-31-37Z 2025-01-18 01:38:52 +00:00
Harshavardhana
4b6eadbd80
update deps (#20851) RELEASE.2025-01-18T00-31-37Z 2025-01-17 16:31:37 -08:00
Shubhendu
6f47414b23
Correct bucket metrics name (#20823)
Earlier, cluster and bucket metrics were named 
`minio_usage_last_activity_nano_seconds`.

The bucket level is now named as 
`minio_bucket_usage_last_activity_nano_seconds`

Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
2025-01-17 13:12:21 -08:00
Harshavardhana
224a27992a update go-deps on debugging tools 2025-01-17 11:41:46 -08:00
Harshavardhana
232544e1d8 update inspect export, which was missed from #20846 2025-01-17 11:38:44 -08:00
Anis Eleuch
dbcb71828d
s3: Provide enough buffer when the object final size is unknown (#20847)
When compression is enabled, the final object size is not calculated in
that case, we need to make sure that the provided buffer is always
more significant than the shard size, the bitrot will always calculate 
the hash of blocks with shard size, except the last block.
2025-01-17 11:19:30 -08:00
Klaus Post
b9196757fd
Fix inconsistently written compressed files. (#20846)
Before https://github.com/minio/minio/pull/20575, files could pick up indices 
from unrelated files if no index was added.

This would result in these files not being consistent across a set.

When loading, search for the compression indicators and check if they 
are within the problematic date range, and clean up any parts that have 
an index but shouldn't.

The test validates that the signature matches the one in files stored without an index.

Bumps xlMetaVersion, so this check doesn't have to be made for future versions.
2025-01-17 11:17:18 -08:00
Andreas Auernhammer
b4ac53d157
update github.com/minio/kms-go/kes to v0.3.1 (#20843)
Signed-off-by: Andreas Auernhammer <github@aead.dev>
2025-01-16 01:13:28 -08:00
Poorna
4952bdb770
decom: avoid skipping single delete markers for replication (#20836)
It is possible delete marker was received on old pool as decom
move in progress, this PR allows decom retry to ensure these
delete markers are moved to new pool so that decommission can
be completed.

Fixes #20819
2025-01-14 11:53:02 -08:00
Anis Eleuch
00b2ef2932
Bump golang.org/x/net to silence wrong vuln checker (#20814) 2025-01-08 16:39:24 +05:30
Klaus Post
4536ecfaa4
Add cpuio profiling potential crash workaround (#20809)
Add profiling potential crash wourkaround

Using admin traces could potentially crash the server (or handler more likely) due to upstream divide by 0: https://github.com/felixge/fgprof/pull/34

Ensure the profile always runs 100ms before stopping, so sample count isn't 0 (default sample rate ~10ms/sample, but allow for cpu starvation)
2025-01-06 21:21:54 +05:30
Harshavardhana
43a7402968 update helm release v5.4.0 2025-01-02 21:34:47 -08:00
Allan Roger Reid
330dca9a35
Add resiliency tests (#20786) 2024-12-20 20:24:45 -08:00
Klaus Post
ddd137d317
ListObjectParts should return actual size (#20782)
Fixes #20781:

```
λ aws --endpoint-url http://127.0.0.1:9001 s3api list-parts --bucket testbucket --key test.testcompress --upload-id "ZDM0YzUwM2YtZWM1Zi00NWI2LTgxMzYtZTIwMGE3Yjc0Y2Y1LjYyMzgyMmFhLWU2N2QtNGUyYS04NDE1LWUzZDFlZmJmMWUyZHgxNzM0NjI1MjgyMDkyNzY4MDAw"
{
    "Parts": [
        {
            "PartNumber": 1,
            "LastModified": "2024-12-19T16:47:04.334000+00:00",
            "ETag": "\"7025f242f56479e06c435c0b500cdbb2\"",
            "Size": 2002
        }
    ],
    "ChecksumAlgorithm": "",
    "Initiator": {
        "ID": "02d6176db174dc93cb1b899f7c6078f08654445fe8cf1b6ce98d8855f66bdbf4",
        "DisplayName": "02d6176db174dc93cb1b899f7c6078f08654445fe8cf1b6ce98d8855f66bdbf4"
    },
    "Owner": {
        "DisplayName": "02d6176db174dc93cb1b899f7c6078f08654445fe8cf1b6ce98d8855f66bdbf4",
        "ID": "02d6176db174dc93cb1b899f7c6078f08654445fe8cf1b6ce98d8855f66bdbf4"
    },
    "StorageClass": "STANDARD"
}
```

(for whatever reason the python script generated a 2002 byte file ;)
2024-12-19 23:51:46 +05:30
Minio Trusted
06ddd8770e Update yaml files to latest version RELEASE.2024-12-18T13-15-44Z 2024-12-19 00:24:40 +00:00
Anis Eleuch
16f8cf1c52
heal: Include more use case of not healable but readable objects (#248) (#20776)
If one object has many parts where all parts are readable but some parts
are missing from some drives, this object can be sometimes un-healable,
which is wrong.

This commit will avoid reading from drives that have missing, corrupted or
outdated xl.meta. It will also check if any part is unreadable to avoid
healing in that case.
RELEASE.2024-12-18T13-15-44Z
2024-12-18 05:15:44 -08:00
Mark Theunissen
01e520eb23
s3: Sanitize the source object name in CopyObject handler (#20774) 2024-12-17 07:01:07 -08:00
Harshavardhana
02f770a0c0
update all dependencies and use latest msgp (#20768) 2024-12-16 04:20:12 +05:30
dependabot[bot]
2f4c79bc0f
Bump golang.org/x/crypto from 0.29.0 to 0.31.0 (#20767)
Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.29.0 to 0.31.0.
- [Commits](https://github.com/golang/crypto/compare/v0.29.0...v0.31.0)

---
updated-dependencies:
- dependency-name: golang.org/x/crypto
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-12-14 23:04:23 -08:00
dependabot[bot]
969ee7dfbe
Bump golang.org/x/crypto from 0.23.0 to 0.31.0 in /docs/debugging/inspect (#20760)
Bump golang.org/x/crypto in /docs/debugging/inspect

Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.23.0 to 0.31.0.
- [Commits](https://github.com/golang/crypto/compare/v0.23.0...v0.31.0)

---
updated-dependencies:
- dependency-name: golang.org/x/crypto
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-12-14 22:04:58 -08:00
Minio Trusted
5f0b086b05 Update yaml files to latest version RELEASE.2024-12-13T22-19-12Z 2024-12-15 03:52:57 +00:00
Artur Melanchyk
68b004a48f
fix: specify size in some map allocations (#20764) RELEASE.2024-12-13T22-19-12Z 2024-12-13 14:19:12 -08:00
Harshavardhana
54ecce66f0 move readLock on uploadId at EOF in PutObjectPart (#258)
we do not need to hold the read locks at the higher
layer instead before reading the body, instead hold
the read locks properly at the time of renamePart()
for protection from racy part overwrites to compete
with concurrent completeMultipart().
2024-12-13 14:06:18 -08:00
Artur Melanchyk
2b008c598b
fix: replace mutex with atomic (#20762) 2024-12-13 19:32:46 +05:30
dependabot[bot]
86d02b17cf
Bump golang.org/x/crypto from 0.23.0 to 0.31.0 in /docs/debugging/s3-verify (#20757)
Bump golang.org/x/crypto in /docs/debugging/s3-verify

Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.23.0 to 0.31.0.
- [Commits](https://github.com/golang/crypto/compare/v0.23.0...v0.31.0)

---
updated-dependencies:
- dependency-name: golang.org/x/crypto
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-12-12 05:00:05 -08:00
Anis Eleuch
c1a95a70ac
heal: Move CheckParts from single handler to streaming RPC (#20755)
CheckParts call can take time to verify 10k parts of a single object in a single drive.
To avoid an internal dealine of one minute in the single handler RPC, this commit will
switch to streaming RPC instead.
2024-12-12 11:20:57 +05:30
Aditya Manthramurthy
f246c9053f
fix: Privilege escalation in IAM import API (#20756)
This API had missing permissions checking, allowing a user to change
their policy mapping by:

1. Craft iam-info.zip file: Update own user permission in
user_mappings.json
2. Upload it via `mc admin cluster iam import nobody iam-info.zip`

Here `nobody` can be a user with pretty much any kind of permission (but
not anonymous) and this ends up working.

Some more detailed steps - start from a fresh setup:

```
./minio server /tmp/d{1...4} &
mc alias set myminio http://localhost:9000 minioadmin minioadmin
mc admin user add myminio nobody nobody123
mc admin policy attach myminio readwrite nobody nobody123
mc alias set nobody http://localhost:9000 nobody nobody123

mc admin cluster iam export myminio
mkdir /tmp/x && mv myminio-iam-info.zip /tmp/x
cd /tmp/x
unzip myminio-iam-info.zip
echo '{"nobody":{"version":1,"policy":"consoleAdmin","updatedAt":"2024-08-13T19:47:10.1Z"}}' > \
      iam-assets/user_mappings.json
zip -r myminio-iam-info-updated.zip iam-assets/

mc admin cluster iam import nobody ./myminio-iam-info-updated.zip
mc admin service restart nobody
```
2024-12-12 07:39:40 +05:30
Cesar N.
9cdd204ae4
Upgrade Console version to v1.7.5 (#20748) 2024-12-11 16:23:53 +05:30
Harshavardhana
7b3eb9f7f8
fix: groups lookup performance issue with users with lots of groups (#20740)
fixes https://github.com/minio/minio/issues/20717
2024-12-11 16:23:28 +05:30
ebozduman
d56ef8dbe1
Adds AIstore documentation link (#20738) 2024-12-11 16:22:17 +05:30
Mark Theunissen
a248ed5ff5
Fixes for POST policy checks and the x-ignore implementation (#20674) 2024-12-11 16:21:34 +05:30
Klaus Post
5bb31e4883
Disable mint full object tests (#20743)
Remove expected failures from https://github.com/minio/minio-go/pull/2026
2024-12-09 18:59:22 -08:00
Taran Pelkey
aff2a76d80
Return error when attempting to create a policy with commas in name (#20724) 2024-12-04 03:51:26 -08:00
Anis Eleuch
eddbe6bca2
heal: Report bucket healing result correctly (#20721) 2024-12-04 04:42:25 +05:30
Anis Eleuch
734d1e320a
heal: Single object heal to look for older versions as well (#203) (#20723)
`mc admin heal ALIAS/bucket/object` does not have any flag to heal
object noncurrent versions, this commit will make healing of the object
noncurrent versions implicitly asked.

This also fixes the 'mc admin heal ALIAS/bucket/object' that does not work 
correctly when the bucket is versioned. This has been broken since Apr 2023.
2024-12-04 04:42:04 +05:30
Anis Eleuch
b8dab7b1a9
Set http server read/write timeout from --idle-timeout (#228) (#20715)
Golang http.Server will call SetReadDeadline overwriting the previous
deadline configuration set after a new connection Accept in the custom
listener code. Therefore, --idle-timeout was not correctly respected.

Make http.Server read/write timeout similar to --idle-timeout.
2024-12-02 18:51:17 +05:30
Klaus Post
abd6bf060d
Add 'X-Forwarded-For' to (s)FTP requests (#20709)
Fixes #20707
2024-11-29 18:25:37 +05:30
Alex
f0d4ef604c
Updated Console to v1.7.4 (#20693)
Signed-off-by: Benjamin Perez <benjamin@bexsoft.net>
2024-11-28 13:14:11 +05:30
Ramon de Klein
2712f75762
prevent IAM cleanup errors (#20691) 2024-11-28 13:13:51 +05:30
Krutika Dhananjay
4c46668da8
Add a test case for fix #20684 (#20688)
The test fails without the change.
Also, removed a duplicate test case involving lifecycle config with no rules.
2024-11-28 13:13:31 +05:30
Anis Eleuch
02e93fd6ba
heal: Better reporting to mc with dangling/timeout errors (#20690)
The code assigns corrupted state to a drive for any unexpected error,
which is confusing for users. This change will make sure to assign
corrupted state only for corrupted parts or xl.meta. Use unknown state
with a explanation for any unexpected error, like canceled, deadline
errors, drive timeout, ...

Also make sure to return the bucket/object name when the object is not
found or marked not found by the heal dangling code.
2024-11-26 10:45:35 -08:00
Krutika Dhananjay
366876e98b
Fix prefix validation in lifecycle rule (#20684) 2024-11-25 12:12:21 -08:00
Mark Theunissen
d202fdd022
Add the policy name to the audit logs tags when doing policy-based API calls. Add retention settings to tags (#20638)
* Add the policy name to the audit log tags when doing policy-based API calls

* Audit log the retention settings requested in the API call

* Audit log of retention on PutObjectRetention API path too
2024-11-25 09:17:12 -08:00
Eng Zer Jun
c07e5b49d4
refactor: replace experimental maps and slices with stdlib (#20679)
The experimental functions are now available in the standard library in
Go 1.23 [1].

[1]: https://go.dev/doc/go1.23#new-unique-package

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2024-11-25 09:10:22 -08:00
Aditya Manthramurthy
9a39f8ad4d
fix: Remove User should fail for a service account (#20677)
The RemoveUser API only removes internal users, and it reports success
when it didnt find the internal user account for deletion. When provided
with a service account, it should not report success as that is misleading.
2024-11-21 18:24:04 -08:00
Nao Yonashiro
7e0c1c9413
feat: bump github.com/cosnicolaou/pbzip2 from 1.0.3 to 1.0.5 (#20671)
Update github.com/cosnicolaou/pbzip2 to latest version for
significant performance improvements. This update brings a 45%
reduction in processing time.
2024-11-20 18:53:52 -08:00
John Morales
485d833cd7
fix: API label casing and count value for +Inf bucket v2 metrics (#20656) 2024-11-18 11:17:52 -08:00
Klaus Post
e8a476ef5a
Keep larger merge buffers for RPC (#20654)
Keep larger merge buffers

When sending large messages >1K, the merge buffer would continuously be reallocated.

This could happen on listings, where blocks are typically 4->8K.

Keep merge buffer of up to 256KB.

Benchmark with 4096b messages:

```
benchmark                                          old ns/op     new ns/op     delta
BenchmarkRequests/servers=2/bytes/par=32-32        8271          6360          -23.10%
BenchmarkRequests/servers=2/bytes/par=64-32        7840          4731          -39.66%
BenchmarkRequests/servers=2/bytes/par=128-32       7291          4740          -34.99%
BenchmarkRequests/servers=2/bytes/par=256-32       7095          4580          -35.45%
BenchmarkRequests/servers=2/bytes/par=512-32       6757          4584          -32.16%
BenchmarkRequests/servers=2/bytes/par=1024-32      6429          4453          -30.74%

benchmark                                          old bytes     new bytes     delta
BenchmarkRequests/servers=2/bytes/par=32-32        12090         821           -93.21%
BenchmarkRequests/servers=2/bytes/par=64-32        17423         820           -95.29%
BenchmarkRequests/servers=2/bytes/par=128-32       18493         822           -95.56%
BenchmarkRequests/servers=2/bytes/par=256-32       18892         821           -95.65%
BenchmarkRequests/servers=2/bytes/par=512-32       19064         826           -95.67%
BenchmarkRequests/servers=2/bytes/par=1024-32      19038         842           -95.58%
```
2024-11-16 09:18:48 -08:00