Commit Graph

344 Commits

Author SHA1 Message Date
Harshavardhana
0bc34952eb
fix: under FanOut API avoid repeated md5sum calculation (#17572)
md5sum calculation has a high CPU overhead, avoid calculating
it repeatedly for similar fanOut calls.

To fix following CPU profiler result
```
(pprof) top10
Showing nodes accounting for 678.68s, 84.67% of 801.54s total
Dropped 1072 nodes (cum <= 4.01s)
Showing top 10 nodes out of 156
      flat  flat%   sum%        cum   cum%
   332.54s 41.49% 41.49%    332.54s 41.49%  runtime/internal/syscall.Syscall6
   228.39s 28.49% 69.98%    228.39s 28.49%  crypto/md5.block
    48.07s  6.00% 75.98%     48.07s  6.00%  runtime.memmove
    28.91s  3.61% 79.59%     28.91s  3.61%  github.com/minio/highwayhash.updateAVX2
     8.25s  1.03% 80.61%      8.25s  1.03%  runtime.futex
     8.25s  1.03% 81.64%     10.81s  1.35%  runtime.step
     6.99s  0.87% 82.52%     22.35s  2.79%  runtime.pcvalue
     6.67s  0.83% 83.35%     38.90s  4.85%  runtime.mallocgc
     5.77s  0.72% 84.07%     32.61s  4.07%  runtime.gentraceback
     4.84s   0.6% 84.67%     10.49s  1.31%  runtime.lock2
```
2023-07-05 03:16:05 -07:00
Klaus Post
216069d0da
Remove 'null' version ID from directory object response (#17495)
Fixes #17494

Regression from #17132
2023-06-23 13:26:00 -07:00
Aditya Manthramurthy
5a1612fe32
Bump up madmin-go and pkg deps (#17469) 2023-06-19 17:53:08 -07:00
Harshavardhana
1443b5927a
allow quorum fileInfo to pick same parityBlocks (#17454)
Bonus: allow replication to proceed for 503 errors such as
with error code SlowDownRead
2023-06-18 18:20:15 -07:00
Harshavardhana
6806537eb3
event args list for fanOut notification must be sized same (#17450)
without this fan-out API can crash if client cancels
the on-going request.
2023-06-18 07:09:20 -07:00
Klaus Post
4a562d6732
fix: fanout error response - error must be string for marshaling (#17433)
Uses https://github.com/minio/minio-go/pull/1839
2023-06-15 09:21:53 -07:00
Harshavardhana
f32efd5429
more compliance related fixes (#17408)
- lifecycle must return InvalidArgument for rule errors
- do not return `null` versionId in HTTP header
- reject mixed SSE uploads with correct error message
2023-06-13 13:52:33 -07:00
Harshavardhana
26a0803388
various compliance related fixes (#17401)
- getObjectTagging to be allowed for anonymous policies
- return correct errors for invalid retention period
- return sorted list of tags for an object
- putObjectTagging must return 200 OK not 204 OK
- return 409 ErrObjectLockConfigurationNotAllowed for existing buckets
2023-06-12 13:22:07 -07:00
Harshavardhana
be45ffd8a4
return 204 status code for DeleteBucketTagging (#17400) 2023-06-11 20:49:02 -07:00
jiuker
14f6ac9222
fix: fail large content in DeleteMultipleObjects() early (#17321) 2023-05-31 10:58:14 -07:00
Anis Eleuch
6425fec366
s3: Add x-minio-error-code header for S3 HEAD requests (#17283) 2023-05-26 10:13:18 -07:00
Harshavardhana
d0a0eb9738
support fan-out objects via PostUpload() (#17233) 2023-05-24 22:51:07 -07:00
Klaus Post
5677f73794
Add PostObject Checksum (#17244) 2023-05-23 07:58:33 -07:00
Harshavardhana
ef2fc0f99e
fix: reduce using memory and temporary files. (#17206) 2023-05-15 14:08:54 -07:00
Harshavardhana
1d0211d395
allow deletes on directory objects to perform permanent deletes (#17132) 2023-05-04 14:43:52 -07:00
Praveen raj Mani
72802a5972
Use 'minio/pkg/sync/errgroup' and 'minio/pkg/workers' (#17069) 2023-04-25 22:57:40 -07:00
Harshavardhana
d1737199ed
fix: delete DNS upon success, update failure message (#17059) 2023-04-21 12:12:31 -07:00
Harshavardhana
e700be8cd6
fix: return appropriate Location header for MakeBucket() (#16820) 2023-03-15 13:40:40 -07:00
jiuker
cd3a2de5a3
add isValidLocation to common parseLocation (#16690) 2023-02-25 08:09:20 +05:30
Klaus Post
fd6622458b
Add detailed scanner trace output and notifications (#16668) 2023-02-21 09:33:33 -08:00
Harshavardhana
84fe4fd156
fix: multiObjectDelete by passing versionId for authorization (#16562) 2023-02-08 08:01:00 +05:30
Harshavardhana
14cf8f1b22
upgrade deps for minio/pkg v1.6.1 to include groups conditions (#16538) 2023-02-06 09:27:29 -08:00
Harshavardhana
d19cbc81b5
fix: do not return IAM/Bucket metadata replication errors to client (#16486) 2023-01-26 11:11:54 -08:00
Harshavardhana
e64b9f6751
fix: disallow SSE-C encrypted objects on replicated buckets (#16467) 2023-01-24 15:46:33 -08:00
Harshavardhana
31b0decd46
migrate to minio/mux from gorilla/mux (#16456) 2023-01-23 16:42:47 +05:30
Harshavardhana
b4ef5ff294
remove unnecessary code checking for supported features (#16423) 2023-01-17 19:37:47 +05:30
Harshavardhana
f1bbb7fef5
vectorize cluster-wide calls such as bucket operations (#16313) 2023-01-03 08:16:39 -08:00
Harshavardhana
b882310e2b
avoid locks for internal and invalid buckets in MakeBucket() (#16302) 2022-12-23 07:46:00 -08:00
Harshavardhana
2433698372
fix: remove unnecessary logs for client conn errors (#16261) 2022-12-15 08:25:05 -08:00
Harshavardhana
b03dd1af17
remove hard limit for number of buckets (#16194) 2022-12-08 12:24:03 -08:00
Aditya Manthramurthy
a30cfdd88f
Bump up madmin-go to v2 (#16162) 2022-12-06 13:46:50 -08:00
Poorna
8dd3c41b2a
allow MakeBucket errors to be handled lazily (#15945)
remote error is not required to be passed back to the 
client - this is mostly because we have healing that should 
eventually, catch up on this and heal the bucket.
2022-10-25 23:32:37 -07:00
Harshavardhana
23b329b9df
remove gateway completely (#15929) 2022-10-24 17:44:15 -07:00
Harshavardhana
a8332efa94
fix: Delete() of bucket metadata should not parse the config (#15904) 2022-10-19 17:55:09 -07:00
Harshavardhana
f696a221af
allow tagging policy condition for GetObject (#15777) 2022-10-02 12:29:29 -07:00
Harshavardhana
b579163802
limit number of buckets to 500k (#15668)
500k is a reasonable limit for any single MinIO
cluster deployment, in future we may increase this
value.

However for now we are going to keep this limit.
2022-09-09 03:06:34 -07:00
Harshavardhana
433b6fa8fe
upgrade golang-lint to the latest (#15600) 2022-08-26 12:52:29 -07:00
Anis Elleuch
1d2ff46a89
Ensure lock/versioning permissions when creating a bucket (#15432)
Currently, the code doesn't check if the user creating a bucket with
locking feature has bucket locking and versioning permissions enabled,
adding it in accordance with S3 spec.

https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateBucket.html

Object Lock - If ObjectLockEnabledForBucket is set to true in your CreateBucket request,
s3:PutBucketObjectLockConfiguration and s3:PutBucketVersioning permissions are required.
2022-08-05 16:27:09 -07:00
Poorna
426c902b87
site replication: fix healing of bucket deletes. (#15377)
This PR changes the handling of bucket deletes for site 
replicated setups to hold on to deleted bucket state until 
it syncs to all the clusters participating in site replication.
2022-07-25 17:51:32 -07:00
Andreas Auernhammer
242d06274a
kms: add context.Context to KMS API calls (#15327)
This commit adds a `context.Context` to the
the KMS `{Stat, CreateKey, GenerateKey}` API
calls.

The context will be used to terminate external calls
as soon as the client requests gets canceled.

A follow-up PR will add a `context.Context` to
the remaining `DecryptKey` API call.

Signed-off-by: Andreas Auernhammer <hi@aead.dev>
2022-07-18 18:54:27 -07:00
Harshavardhana
1cd6713e24
copy query values before update to preserve the expected keys (#15310)
in success_action_redirect we were missing required
query params as per S3 spec - updated tests.
2022-07-15 15:04:48 -07:00
Klaus Post
0149382cdc
Add padding to compressed+encrypted files (#15282)
Add up to 256 bytes of padding for compressed+encrypted files.

This will obscure the obvious cases of extremely compressible content 
and leave a similar output size for a very wide variety of inputs.

This does *not* mean the compression ratio doesn't leak information 
about the content, but the outcome space is much smaller, 
so often *less* information is leaked.
2022-07-13 07:52:15 -07:00
Harshavardhana
788fd3df81
preserve incoming query params in success_action_redirect (#15280)
fixes #15274
2022-07-13 07:46:44 -07:00
Harshavardhana
0a8b78cb84
fix: simplify passing auditLog eventType (#15278)
Rename Trigger -> Event to be a more appropriate
name for the audit event.

Bonus: fixes a bug in AddMRFWorker() it did not
cancel the waitgroup, leading to waitgroup leaks.
2022-07-12 10:43:32 -07:00
Minio Trusted
e60b67d246 Revert "Tighten enforcement of object retention (#14993)"
This reverts commit 5e3010d455.

This commit causes regression on object locked buckets causine
delete-markers to be not created.
2022-06-30 13:06:32 -07:00
Poorna
7cc9286e0f
site healing: Skip stale bucket metadata updates from peer (#15186)
Allow healing to apply bucket metadata change only when peer
gave the most recent update.
2022-06-28 18:09:20 -07:00
Harshavardhana
2bb6a3f4d0
cleanup site replication error handling (#15113)
site replication errors were printed at
various random locations, repeatedly - this
PR attempts to remove double logging and
capture all of them at a common place.

This PR also enhances the code to show
partial success and errors as well.
2022-06-20 10:48:11 -07:00
Harshavardhana
48e367ff7d
reject resync start on misconfigured replication rules (#15041)
we expect resync to start on buckets with replication
rule ExistingObjects enabled, if not we reject such
calls.
2022-06-06 02:54:39 -07:00
Harshavardhana
f1abb92f0c
feat: Single drive XL implementation (#14970)
Main motivation is move towards a common backend format
for all different types of modes in MinIO, allowing for
a simpler code and predictable behavior across all features.

This PR also brings features such as versioning, replication,
transitioning to single drive setups.
2022-05-30 10:58:37 -07:00
Poorna
5e3010d455
Tighten enforcement of object retention (#14993)
Ref issue#14991 - in the rare case that object in bucket under
retention has null version, make sure to enforce retention
rules.
2022-05-28 02:21:19 -07:00