Commit Graph

9854 Commits

Author SHA1 Message Date
Harshavardhana
d350b666ff
feat: add idempotent delete marker support (#15521)
The bottom line is delete markers are a nuisance,
most applications are not version aware and this
has simply complicated the version management.

AWS S3 gave an unnecessary complication overhead
for customers, they need to now manage these
markers by applying ILM settings and clean
them up on a regular basis.

To make matters worse all these delete markers
get replicated as well in a replicated setup,
requiring two ILM settings on each site.

This PR is an attempt to address this inferior
implementation by deviating MinIO towards an
idempotent delete marker implementation i.e
MinIO will never create any more than single
consecutive delete markers.

This significantly reduces operational overhead
by making versioning more useful for real data.

This is an S3 spec deviation for pragmatic reasons.
2022-08-18 16:41:59 -07:00
Aditya Manthramurthy
21831b3fe2
fix: cleanup config KV parsing using madmin helpers (#15552) 2022-08-18 15:55:17 -07:00
Harshavardhana
895357607a
avoid using errors.As for 'errors.New' use errors.Is (#15549)
Bonus: ignore coredns CVE, for now, there is no fix yet

https://github.com/coredns/coredns/issues/5574
2022-08-18 11:10:49 -07:00
Minio Trusted
ac240a8477 fix: CoreDNS vulnerability CVE-2022-2837 dependency 2022-08-18 10:10:06 -07:00
Harshavardhana
bf38c0c0d1
fix: increase concurrency of DeleteObjects() to N/10th (#15546)
instead of keeping the value 10 and static, make
the concurrency a function of incoming number of
objects being deleted.
2022-08-18 09:33:56 -07:00
Pierre KELBERT
67cf15d036
mount export volume even if persistence is disabled (#15543) 2022-08-17 08:50:32 -07:00
Aditya Manthramurthy
701a82642b
Ensure subnet+callhome configs are single sub-sys (#15542) 2022-08-16 17:46:46 -07:00
Poorna
21fe14201f
replication: centralize healthcheck for remote targets (#15516)
This PR moves health check from minio-go client to being
managed on the server.

Additionally integrating health check into site replication
2022-08-16 17:46:22 -07:00
Harshavardhana
48640b1de2
fix: trim arn:aws:kms from incoming SSE aws-kms-key-id (#15540) 2022-08-16 11:28:30 -07:00
Anis Elleuch
5682685c80
Introduce disk io stats metrics (#15512) 2022-08-16 07:13:49 -07:00
Sébastien Rodriguez
9c025b8cce
helm: fix correct deployment environment variable (#15534) 2022-08-14 18:13:56 -07:00
Minio Trusted
eef9f13360 update helm v4.0.12 2022-08-13 22:50:43 -07:00
Minio Trusted
fa9b361a3d Update yaml files to latest version RELEASE.2022-08-13T21-54-44Z 2022-08-14 05:33:08 +00:00
reyerdam
49862ba347
helm: (Bugfix) enable objectlocking (#15524) 2022-08-13 14:54:44 -07:00
Aditya Manthramurthy
ee2afcf70b
Move some config constants to madmin (#15529) 2022-08-13 14:50:16 -07:00
Harshavardhana
c7d535c648
init console after IAM init() (#15531)
fixes #15527
2022-08-13 12:54:41 -07:00
Aditya Manthramurthy
9986e103cf
Fix env var output in config get/export APIs (#15528)
Fix a bug where env vars are not output when the config for the
subsystem is specified solely via env vars.
2022-08-13 10:39:01 -07:00
Minio Trusted
c5b3666089 Update yaml files to latest version RELEASE.2022-08-11T04-37-28Z 2022-08-11 22:36:34 +00:00
Minio Trusted
d265fe7f9e update console v0.19.5 2022-08-10 21:37:28 -07:00
Krishnan Parthasarathi
91e6af4470
Add trace support for decommissioning (#15502)
* Add trace support for decommissioning
* Add support for tracing errors during decommission
2022-08-10 12:46:45 -07:00
Noah Gao
b940fe8fca
chore: bad title syntax of helm chart README (#15513) 2022-08-10 11:16:22 -07:00
reyerdam
73fe2e95fe
helm: Support adding objectlocking for buckets (#15505)
implemented object locking during bucket creation in helm chart
2022-08-10 08:13:51 -07:00
Shireesh Anjal
316c492842
Upgrade madmin-go to latest version (v1.4.15) (#15510) 2022-08-10 07:36:13 -07:00
Harshavardhana
74418b542a
fix: incorrect context timeout during listPath() (#15509)
This PR cleans up the listing code for single drive
to ensure that we do not add an incorrect context
timeout, while resuming the listing.

fixes #15508
2022-08-10 07:35:29 -07:00
Poorna
172e63dbb6
fix: site replication group updates to set status correctly (#15507)
Fixes: #15486
2022-08-09 15:17:43 -07:00
Poorna
21bf5b4db7
replication: heal proactively upon access (#15501)
Queue failed/pending replication for healing during listing and GET/HEAD
API calls. This includes healing of existing objects that were never
replicated or those in the middle of a resync operation.

This PR also fixes a bug in ListObjectVersions where lifecycle filtering
should be done.
2022-08-09 15:00:24 -07:00
Harshavardhana
a406bb0288
restrict number of disks used for scanning buckets upto GOMAXPROCS (#15492)
control scanner parallelism to avoid higher CPU
usage on nodes that have more drives but an old CPU.
2022-08-08 16:16:44 -07:00
Harshavardhana
1823ab6808
LDAP/OpenID must be initialized IAM Init() (#15491)
This allows for LDAP/OpenID to be non-blocking,
allowing for unreachable Identity targets to be
initialized in IAM.
2022-08-08 16:16:27 -07:00
Harshavardhana
8eec49304d use logger.Info instead of logger.LogIf 2022-08-08 16:13:58 -07:00
Harshavardhana
ecdc2f2f5f
fix: maxConcurrent '0' is an invalid value (#15500)
log and continue with defaults instead of
crashing the service.
2022-08-08 15:18:45 -07:00
Minio Trusted
6a6c772ff2 Update yaml files to latest version RELEASE.2022-08-08T18-34-09Z 2022-08-08 21:30:40 +00:00
Harshavardhana
e178c55bc3
remove non-working GetRawData() from FS mode (#15498) 2022-08-08 11:34:09 -07:00
Poorna
2c137c0d04
fix: handle invalid endpoint errors in site replication(#15499)
fixes #15497
2022-08-08 11:12:05 -07:00
Minio Trusted
1d35f2b58f update madmin-go to v1.4.13 2022-08-08 10:41:38 -07:00
Harshavardhana
638c57e466 revert changes in FS implementation for umask
fixes #15494
2022-08-08 09:48:24 -07:00
Harshavardhana
5e4213b3be
fix: keep writing previous speedtest result (#15484)
when object speedtest is running keep writing
previous speedtest result back to client until
we have a new result - this avoids sending back
blank entries in between the speedtest when it
is running in 'autotune' mode.
2022-08-07 23:04:03 -07:00
Minio Trusted
102295f58a update helm v4.0.11 2022-08-06 22:41:47 -07:00
Jan Šafařík
a0d14f8ff7
helm: add a new line to the end of the credentials file (#15485) 2022-08-06 15:01:01 -07:00
Harshavardhana
e0b0a351c6
remove IAM old migration code (#15476)
```
commit 7bdaf9bc50
Author: Aditya Manthramurthy <donatello@users.noreply.github.com>
Date:   Wed Jul 24 17:34:23 2019 -0700

    Update on-disk storage format for users system (#7949)
```

Bonus: fixes a bug when etcd keys were being re-encrypted.
2022-08-05 17:53:23 -07:00
Minio Trusted
fcd4b3ba9b Update yaml files to latest version RELEASE.2022-08-05T23-27-09Z 2022-08-06 00:08:21 +00:00
Anis Elleuch
1d2ff46a89
Ensure lock/versioning permissions when creating a bucket (#15432)
Currently, the code doesn't check if the user creating a bucket with
locking feature has bucket locking and versioning permissions enabled,
adding it in accordance with S3 spec.

https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateBucket.html

Object Lock - If ObjectLockEnabledForBucket is set to true in your CreateBucket request,
s3:PutBucketObjectLockConfiguration and s3:PutBucketVersioning permissions are required.
2022-08-05 16:27:09 -07:00
Harshavardhana
8f7c739328
feat: add SpeedTest ResponseTimes and TTFB (#15479)
Capture average, p50, p99, p999 response times
and ttfb values. These are needed for latency
measurements and overall understanding of our
speedtest results.
2022-08-05 09:40:03 -07:00
Poorna
1beea3daba
fix: import bucket metadata import to return a summary (#15462) 2022-08-05 01:52:50 -07:00
Minio Trusted
1ffd063939 update CREDITS for latest deps 2022-08-04 23:30:31 -07:00
Aditya Manthramurthy
3d94c38ec4
Add env variables to configuration APIs output (#15465)
Config export and config get APIs now include environment 
variables set on the server
2022-08-04 22:21:52 -07:00
Harshavardhana
f4af2d3cdc
fix: decodeDirObject() in single drive DeleteObjects() call (#15477)
Thanks to @bh4t for reproducing this issue.
2022-08-04 18:57:43 -07:00
ebozduman
b57e7321e7
Replaces 'disk'=>'drive' visible to end user (#15464) 2022-08-04 16:10:08 -07:00
Anis Elleuch
e93867488b
actively cancel listIAMConfigItems to avoid goroutine leak (#15471)
listConfigItems creates a goroutine but sometimes callers will
exit without properly asking listAllIAMConfigItems() to stop sending
results, hence a goroutine leak.

Create a new context and cancel it for each listAllIAMConfigItems
call.
2022-08-04 13:20:43 -07:00
Minio Trusted
c08790edd2 upgrade helm v4.0.10 2022-08-04 09:09:22 -07:00
Kourosh Tafreshi
a46baddbc4
Add OIDC to the HelmChart (#15469) 2022-08-04 09:07:51 -07:00