Commit Graph

62 Commits

Author SHA1 Message Date
M Alvee
07c3a429bf fix: conditional checks write for multipart (#21567) 2025-09-07 09:13:09 -07:00
Klaus Post
f0b91e5504 Run modernize (#21546)
`go run golang.org/x/tools/gopls/internal/analysis/modernize/cmd/modernize@latest -fix -test ./...` executed.

`go generate ./...` ran afterwards to keep generated.
2025-08-28 19:39:48 -07:00
Mark Theunissen
4ea6f3b06b fix: invalid checksum on site replication with conforming checksum types (#21535) 2025-08-22 07:15:21 -07:00
Andreas Auernhammer
1d50cae43d remove support for FIPS 140-2 with boringcrypto (#21292)
This commit removes FIPS 140-2 related code for the following
reasons:
 - FIPS 140-2 is a compliance, not a security requirement. Being
   FIPS 140-2 compliant has no security implication on its own.
   From a tech. perspetive, a FIPS 140-2 compliant implementation
   is not necessarily secure and a non-FIPS 140-2 compliant implementation
   is not necessarily insecure. It depends on the concret design and
   crypto primitives/constructions used.
 - The boringcrypto branch used to achieve FIPS 140-2 compliance was never
   officially supported by the Go team and is now in maintainance mode.
   It is replaced by a built-in FIPS 140-3 module. It will be removed
   eventually. Ref: https://github.com/golang/go/issues/69536
 - FIPS 140-2 modules are no longer re-certified after Sep. 2026.
   Ref: https://csrc.nist.gov/projects/cryptographic-module-validation-program

Signed-off-by: Andreas Auernhammer <github@aead.dev>
2025-05-16 07:27:42 -07:00
Harshavardhana
8c70975283 make sure to validate signature unsigned trailer stream (#21103)
This is a security incident fix, it would seem like since
the implementation of unsigned payload trailer on PUTs,
we do not validate the signature of the incoming request.

The signature can be invalid and is totally being ignored,
this in-turn allows any arbitrary secret to upload objects
given the user has "WRITE" permissions on the bucket, since
acces-key is a public information in general exposes these
potential users with WRITE on the bucket to be used by any
arbitrary client to make a fake request to MinIO the signature
under Authorization: header is totally ignored.

A test has been added to cover this scenario and fail
appropriately.
2025-04-03 07:55:52 -07:00
Taran Pelkey
e88d494775 Migrate golanglint-ci config to V2 (#21081) 2025-03-29 17:56:02 -07:00
Klaus Post
aeabac9181 Test checksum types for invalid combinations (#20953) 2025-02-18 08:24:01 -08:00
Klaus Post
827004cd6d Add Full Object Checksums and CRC64-NVME (#20855)
Backport of AIStor PR 247.

Add support for full object checksums as described here:

https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html

New checksum types are fully supported. Mint tests from https://github.com/minio/minio-go/pull/2026 are now passing.

Includes fixes from https://github.com/minio/minio/pull/20743 for mint tests.

Add using checksums as validation for object content. Fixes #20845 #20849

Fixes checksum replication (downstream PR 250)
2025-01-20 06:49:07 -08:00
Klaus Post
ddd137d317 ListObjectParts should return actual size (#20782)
Fixes #20781:

```
λ aws --endpoint-url http://127.0.0.1:9001 s3api list-parts --bucket testbucket --key test.testcompress --upload-id "ZDM0YzUwM2YtZWM1Zi00NWI2LTgxMzYtZTIwMGE3Yjc0Y2Y1LjYyMzgyMmFhLWU2N2QtNGUyYS04NDE1LWUzZDFlZmJmMWUyZHgxNzM0NjI1MjgyMDkyNzY4MDAw"
{
    "Parts": [
        {
            "PartNumber": 1,
            "LastModified": "2024-12-19T16:47:04.334000+00:00",
            "ETag": "\"7025f242f56479e06c435c0b500cdbb2\"",
            "Size": 2002
        }
    ],
    "ChecksumAlgorithm": "",
    "Initiator": {
        "ID": "02d6176db174dc93cb1b899f7c6078f08654445fe8cf1b6ce98d8855f66bdbf4",
        "DisplayName": "02d6176db174dc93cb1b899f7c6078f08654445fe8cf1b6ce98d8855f66bdbf4"
    },
    "Owner": {
        "DisplayName": "02d6176db174dc93cb1b899f7c6078f08654445fe8cf1b6ce98d8855f66bdbf4",
        "ID": "02d6176db174dc93cb1b899f7c6078f08654445fe8cf1b6ce98d8855f66bdbf4"
    },
    "StorageClass": "STANDARD"
}
```

(for whatever reason the python script generated a 2002 byte file ;)
2024-12-19 23:51:46 +05:30
Harshavardhana
7a34c88d73 add consistent nonce to make multipart deterministic per part (#20359)
This change adds a consistent nonce to ensure
that multipart uploads are deterministic on a 
per-part basis.

Thanks to @klauspost for the work here minio/sio@3cd3734
2024-08-31 11:25:48 -07:00
Klaus Post
f1302c40fe Fix uninitialized replication stats (#20260)
Services are unfrozen before `initBackgroundReplication` is finished. This means that 
the globalReplicationStats write is racy. Switch to an atomic pointer.

Provide the `ReplicationPool` with the stats, so it doesn't have to be grabbed 
from the atomic pointer on every use.

All other loads and checks are nil, and calls return empty values when stats 
still haven't been initialized.
2024-08-15 05:04:40 -07:00
Harshavardhana
c91d1ec2e3 fix: avoid metadata cache without data for all callers (#19935) 2024-06-14 06:28:35 -07:00
Klaus Post
a2cab02554 Fix SSE-C checksums (#19896)
Compression will be disabled by default if SSE-C is specified. So we can still honor SSE-C.
2024-06-10 08:31:51 -07:00
Poorna
5aaef9790f replication: pass checksum headers to replica (#19834) 2024-06-06 02:36:42 -07:00
Shubhendu
21b6204692 Test proxying of DEL marker for bucket replication (#19870)
Make sure to avoid proxying for DEL markers

Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
2024-06-04 04:38:26 -07:00
Aditya Manthramurthy
5f78691fcf ldap: Add user DN attributes list config param (#19758)
This change uses the updated ldap library in minio/pkg (bumped
up to v3). A new config parameter is added for LDAP configuration to
specify extra user attributes to load from the LDAP server and to store
them as additional claims for the user.

A test is added in sts_handlers.go that shows how to access the LDAP
attributes as a claim.

This is in preparation for adding SSH pubkey authentication to MinIO's SFTP
integration.
2024-05-24 16:05:23 -07:00
Harshavardhana
08d74819b6 handle racy updates to globalSite config (#19750)
```
==================
WARNING: DATA RACE
Read at 0x0000082be990 by goroutine 205:
  github.com/minio/minio/cmd.setCommonHeaders()

Previous write at 0x0000082be990 by main goroutine:
  github.com/minio/minio/cmd.lookupConfigs()
```
2024-05-16 16:13:47 -07:00
Harshavardhana
e8d14c0d90 verify preconditions during CompleteMultipart (#19713)
Bonus: hold the write lock properly to apply
optimistic concurrency during NewMultipartUpload()
2024-05-10 17:31:22 -07:00
Shubhendu
468a9fae83 Enable replication of SSE-C objects (#19107)
If site replication enabled across sites, replicate the SSE-C
objects as well. These objects could be read from target sites
using the same client encryption keys.

Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
2024-03-28 10:44:56 -07:00
Harshavardhana
c32f699105 turn-off md5sum for SSE-KMS/SSE-C as optimization for multipart (#19106)
only enable md5sum if explicitly asked by the client, otherwise
its not necessary to compute md5sum when SSE-KMS/SSE-C is enabled.

this is continuation of #17958
2024-02-22 04:24:11 -08:00
Harshavardhana
afd19de5a9 fix: allow configuring excess versions alerting (#19028)
Bonus: enable audit alerts for object versions
beyond the configured value, default is '100'
versions per object beyond which scanner will
alert for each such objects.
2024-02-11 23:41:53 -08:00
Harshavardhana
7c948adf88 allow pre-allocating buffers to reduce frequent GCs during growth (#18686)
This PR also increases per node bpool memory from 1024 entries
to 2048 entries; along with that, it also moves the byte pool
centrally instead of being per pool.
2023-12-21 08:59:38 -08:00
Harshavardhana
fba883839d feat: bring new HDD related performance enhancements (#18239)
Optionally allows customers to enable 

- Enable an external cache to catch GET/HEAD responses 
- Enable skipping disks that are slow to respond in GET/HEAD 
  when we have already achieved a quorum
2023-11-22 13:46:17 -08:00
Anis Eleuch
70fbcfee4a Implement batch snowball (#18485) 2023-11-22 10:51:46 -08:00
Poorna
96ec8fcba1 Preserve replica timestamps in multipart (#18318)
Also a backward compatibility fix to use x-amz-replica-status
if present as replication status.
2023-10-25 21:24:10 -07:00
Harshavardhana
6829ae5b13 completely remove drive caching layer from gateway days (#18217)
This has already been deprecated for close to a year now.
2023-10-11 21:18:17 -07:00
Harshavardhana
74e0c9ab9b reduce unnecessary logging, simplify certain error handling (#18196)
remove a bunch of unnecessary logs
2023-10-10 00:33:42 -07:00
Harshavardhana
d6446cb096 do not return an error in AbortMultipartUpload() (#18135)
returning an error is a bit undefined in AWS S3
as it may return an error or not depending on the
time from AbortMultipartUpload().
2023-09-29 10:28:19 -07:00
Harshavardhana
c34bdc33fb make sure to set Versioned field to ensure rename2 is not called (#18141)
without this the rename2() can rename the previous dataDir
causing issues for different versions of the object, only
latest version is preserved due to this bug.

Added healing code to ensure recovery of such content.
2023-09-29 09:08:24 -07:00
jiuker
9947c01c8e feat: SSE-KMS use uuid instead of read all data to md5. (#17958) 2023-09-18 10:00:54 -07:00
Harshavardhana
fa6d082bfd reduce all major allocations in replication path (#18032)
- remove targetClient for passing around via replicationObjectInfo{}
- remove cloing to object info unnecessarily
- remove objectInfo from replicationObjectInfo{} (only require necessary fields)
2023-09-16 02:28:06 -07:00
Aditya Manthramurthy
1c99fb106c Update to minio/pkg/v2 (#17967) 2023-09-04 12:57:37 -07:00
Harshavardhana
62c9e500de remove mTime requirement from pre-condition checks (#17916)
given a versionId the mtime is always the same, it
can never be different than its original value.

versionIds also do not conflict, since they are uuid's
and unique practically forever.
2023-08-24 14:33:58 -07:00
Harshavardhana
dde1a12819 fix: validate incoming uploadID to be base64 encoded (#17865)
Bonus fixes include

- do not have to write final xl.meta (renameData) does this
  already, saves some IOPs.

- make sure to purge the multipart directory properly using
  a recursive delete, otherwise this can easily pile up and
  rely on the stale uploads cleanup.

fixes #17863
2023-08-17 09:37:55 -07:00
Harshavardhana
331bdc2245 fix: remove CompleteMultipartUpload() 200 OK response for blocking calls (#17699)
sending whitespace character with CompleteMultipartUpload()
with 200 OK was an AWS S3 compatible implementation detail,
and it was expected that the client SDK must look for both
successful XML as well as error XML for 200 OK.

But this is not useful anymore on MinIO, since we do not
have any large delayed coalescing of parts anymore.
2023-07-20 22:14:38 -07:00
Harshavardhana
73de721a63 fix: handle copyObjectPart encryption properly (#17530)
- look for requested encryption while compressing
  not just via HTTP Headers, but also via multipart
  metadata

- look for SSE-S3 etag decryption not just via HTTP
  Headers, but also via multipart metadata

fixes #17519
2023-06-28 09:43:50 -07:00
Harshavardhana
f32efd5429 more compliance related fixes (#17408)
- lifecycle must return InvalidArgument for rule errors
- do not return `null` versionId in HTTP header
- reject mixed SSE uploads with correct error message
2023-06-13 13:52:33 -07:00
Harshavardhana
43468f4d47 return InvalidRequest when no parts are provided (#17395) 2023-06-10 21:59:51 -07:00
Harshavardhana
75c6fc4f02 only allow decryption of etag for only sse-s3 (#17335) 2023-06-05 13:08:51 -07:00
Klaus Post
66156b8230 Stricter partNumber checks (#17270)
Fixes #17269
2023-05-24 08:00:47 -07:00
Krishnan Parthasarathi
3e128c116e Add lifecycle event source to audit log tags (#17248) 2023-05-22 15:28:56 -07:00
Klaus Post
76913a9fd5 Signed trailers for signature v4 (#16484) 2023-05-05 19:53:12 -07:00
Harshavardhana
8fd07bcd51 simplify sort.Sort by using sort.Slice (#17066) 2023-04-24 13:28:18 -07:00
Harshavardhana
6825bd7e75 fix: inlined objects don't need to honor long locks (#17039) 2023-04-17 12:16:37 -07:00
Harshavardhana
8e02660a0d update all our deps (#16899) 2023-03-28 03:45:24 -07:00
Klaus Post
fd6622458b Add detailed scanner trace output and notifications (#16668) 2023-02-21 09:33:33 -08:00
Harshavardhana
e64b9f6751 fix: disallow SSE-C encrypted objects on replicated buckets (#16467) 2023-01-24 15:46:33 -08:00
Harshavardhana
31b0decd46 migrate to minio/mux from gorilla/mux (#16456) 2023-01-23 16:42:47 +05:30
Harshavardhana
b4ef5ff294 remove unnecessary code checking for supported features (#16423) 2023-01-17 19:37:47 +05:30
Harshavardhana
2fc182d8e6 fix: iso8601TimeFormat padding issue for certain nanoseconds (#16207) 2022-12-12 10:28:30 -08:00