MyFavMirrors/minio
1
0
Fork 0
mirror of https://github.com/minio/minio.git synced 2024-12-26 23:25:54 -05:00
Code Issues Packages Projects Releases Wiki Activity
11,372 Commits 4 Branches 506 Tags 189 MiB
60aff22931
Commit Graph

1493 Commits

Author SHA1 Message Date
Minio Trusted
7f2c90a0ed Update yaml files to latest version RELEASE.2023-02-17T17-52-43Z 2023-02-17 19:07:39 +00:00
Minio Trusted
bf749eec61 Update yaml files to latest version RELEASE.2023-02-10T18-48-39Z 2023-02-10 20:55:45 +00:00
k0i
6b41f941b6
fix: README.md in docs/config (#16564) 2023-02-10 03:02:11 -08:00
Minio Trusted
1141187bf2 Update yaml files to latest version RELEASE.2023-02-09T05-16-53Z 2023-02-09 06:21:34 +00:00
Harshavardhana
c97f50e274
replication syncs happen in 60sec intervals (#16571) 2023-02-08 08:23:26 -08:00
Harshavardhana
14cf8f1b22
upgrade deps for minio/pkg v1.6.1 to include groups conditions (#16538) 2023-02-06 09:27:29 -08:00
Harshavardhana
e47a31f9fc
fix: object size distribution in metrics for all objects (#16539) 2023-02-04 21:10:10 -08:00
Minio Trusted
a24037bfec Update yaml files to latest version RELEASE.2023-01-31T02-24-19Z 2023-01-31 07:49:15 +00:00
Jan Zhanal
8fa80874a6
doc: LDAP/AD - nested groups (#16483) 2023-01-26 22:17:59 +05:30
Minio Trusted
1bf1fafc86 Update yaml files to latest version RELEASE.2023-01-25T00-19-54Z 2023-01-25 02:09:27 +00:00
Harshavardhana
e64b9f6751
fix: disallow SSE-C encrypted objects on replicated buckets (#16467) 2023-01-24 15:46:33 -08:00
Harshavardhana
3683673fb0
add missing gorilla/mux migration, update credits (#16461) 2023-01-23 08:46:37 -08:00
Anis Elleuch
e73894fa50
grafana: Show one metric for the total data growth (#16449) 2023-01-20 09:39:28 -08:00
Minio Trusted
3fa7218c44 Update yaml files to latest version RELEASE.2023-01-20T02-05-44Z 2023-01-20 08:11:58 +00:00
Minio Trusted
5c1a4ba5f9 Update yaml files to latest version RELEASE.2023-01-18T04-36-38Z 2023-01-18 07:46:44 +00:00
Minio Trusted
d33c527e39 Update yaml files to latest version RELEASE.2023-01-12T02-06-16Z 2023-01-12 07:27:07 +00:00
Minio Trusted
52b88b52f0 Update yaml files to latest version RELEASE.2023-01-06T18-11-18Z 2023-01-08 07:51:31 +00:00
Poorna
b29e159604
docs: Update replication setup commands (#16361) 2023-01-04 13:39:37 -08:00
Harshavardhana
b197623ed2
remove unnecessary kernel-tuning docs (#16354) 2023-01-04 01:33:40 -08:00
Harshavardhana
14d29b77ae
update replication tests with latest 'mc' (#16348) 2023-01-03 22:54:39 -08:00
Minio Trusted
72394a8319 Update yaml files to latest version RELEASE.2023-01-02T09-40-09Z 2023-01-03 10:16:34 +00:00
Anis Elleuch
b8943fdf19
doc: Update prometheus metrics list (#16329) 2022-12-29 15:08:22 -08:00
Harshavardhana
2937711390
fix: DeleteObject() API with versionId under replication (#16325) 2022-12-28 22:48:33 -08:00
Aditya Manthramurthy
9e6cc847f8
Add HTTP2 config option for policy plugin (#16225) 2022-12-13 14:28:48 -08:00
Minio Trusted
76905b7a67 Update yaml files to latest version RELEASE.2022-12-12T19-27-27Z 2022-12-13 01:10:31 +00:00
Aditya Manthramurthy
a469e6768d
Add LDAP DNS SRV record lookup support (#16201) 2022-12-12 11:27:27 -08:00
Harshavardhana
b03dd1af17
remove hard limit for number of buckets (#16194) 2022-12-08 12:24:03 -08:00
Klaus Post
3eb2d086b2
Replace filepathx with fork (#16192) 2022-12-08 10:42:44 -08:00
Minio Trusted
9f71369b67 Update yaml files to latest version RELEASE.2022-12-07T00-56-37Z 2022-12-07 01:30:51 +00:00
Aditya Manthramurthy
a30cfdd88f
Bump up madmin-go to v2 (#16162) 2022-12-06 13:46:50 -08:00
Harshavardhana
a9f5b58a01
fix: update the JSON keys for latest 'mc' release (#16171) 2022-12-05 10:28:22 -08:00
Minio Trusted
d6351879f3 Update yaml files to latest version RELEASE.2022-12-02T19-19-22Z 2022-12-02 20:15:36 +00:00
Harshavardhana
71133105d7
re-order the top-level config keys for priority (#16150) 2022-12-01 07:50:08 -08:00
Minio Trusted
76943ac05e Update yaml files to latest version RELEASE.2022-11-29T23-40-49Z 2022-11-30 21:13:07 +00:00
Aditya Manthramurthy
87cbd41265
feat: Allow at most one claim based OpenID IDP (#16145) 2022-11-29 15:40:49 -08:00
Minio Trusted
53cbc020b9 Update yaml files to latest version RELEASE.2022-11-26T22-43-32Z 2022-11-27 04:08:35 +00:00
Minio Trusted
3661aaf8a1 Update yaml files to latest version RELEASE.2022-11-17T23-20-09Z 2022-11-18 08:51:38 +00:00
Minio Trusted
6dc7109a9f Update yaml files to latest version RELEASE.2022-11-11T03-44-20Z 2022-11-11 04:57:16 +00:00
Minio Trusted
4fe9cbb973 Update yaml files to latest version RELEASE.2022-11-10T18-20-21Z 2022-11-10 19:36:54 +00:00
yanggang
52769e1e71
remove io/util for advanced golang (#16011) 2022-11-08 07:58:02 -08:00
Minio Trusted
808739867c Update yaml files to latest version RELEASE.2022-11-08T05-27-07Z 2022-11-08 05:59:18 +00:00
Anis Elleuch
7e73fc2870
Implement inspect data API v2 (#15474) 
Co-authored-by: Klaus Post <klauspost@gmail.com>
 2022-11-02 13:36:38 -07:00
Minio Trusted
0303cd8625 Update yaml files to latest version RELEASE.2022-10-29T06-21-33Z 2022-10-29 09:29:37 +00:00
Poorna
dc92bb4646
xl-meta: update metadata version (#15958) 2022-10-28 02:48:43 -07:00
Harshavardhana
23b329b9df
remove gateway completely (#15929) 2022-10-24 17:44:15 -07:00
Minio Trusted
630dabf4b9 Update yaml files to latest version RELEASE.2022-10-24T18-35-07Z 2022-10-24 19:29:30 +00:00
Minio Trusted
c921dc75c7 Update yaml files to latest version RELEASE.2022-10-21T22-37-48Z 2022-10-22 00:04:37 +00:00
Klaus Post
86d543d0f6
Check for s3zip content offset (#15924) 2022-10-21 15:37:48 -07:00
Minio Trusted
bd77f1df4c Update yaml files to latest version RELEASE.2022-10-20T00-55-09Z 2022-10-20 06:38:33 +00:00
Minio Trusted
196fab6834 Update yaml files to latest version RELEASE.2022-10-15T19-57-03Z 2022-10-15 23:20:56 +00:00
Minio Trusted
0c8dd8046a Update yaml files to latest version RELEASE.2022-10-08T20-11-00Z 2022-10-09 19:44:19 +00:00
Harshavardhana
2a13cc28f2 feat: implement support batch replication (#15554) 2022-10-05 23:00:43 -07:00
Minio Trusted
4d761fda81 Update yaml files to latest version RELEASE.2022-10-05T14-58-27Z 2022-10-05 22:50:50 +00:00
Minio Trusted
dc6733dacc Update yaml files to latest version RELEASE.2022-10-02T19-29-29Z 2022-10-03 02:29:56 +00:00
Daryl White
d44f3526dc
Update links to documentation site (#15750) 2022-09-28 21:28:45 -07:00
Minio Trusted
1480340830 Update yaml files to latest version RELEASE.2022-09-25T15-44-53Z 2022-09-26 04:54:15 +00:00
Minio Trusted
334f1ed45a Update yaml files to latest version RELEASE.2022-09-22T18-57-27Z 2022-09-23 06:01:11 +00:00
Klaus Post
ff12080ff5
Remove deprecated io/ioutil (#15707) 2022-09-19 11:05:16 -07:00
Minio Trusted
98874c3baf Update yaml files to latest version RELEASE.2022-09-17T00-09-45Z 2022-09-17 08:07:32 +00:00
Ashish Kumar Sinha
fb8d512f58
update docs to add cycle in scanner (#15686) 2022-09-13 22:46:02 -07:00
Harshavardhana
b579163802
limit number of buckets to 500k (#15668) 
500k is a reasonable limit for any single MinIO
cluster deployment, in future we may increase this
value.

However for now we are going to keep this limit.
 2022-09-09 03:06:34 -07:00
Minio Trusted
87f0c8e7e8 Update yaml files to latest version RELEASE.2022-09-07T22-25-02Z 2022-09-08 00:44:12 +00:00
Minio Trusted
240164560f Update yaml files to latest version RELEASE.2022-09-01T23-53-36Z 2022-09-02 01:52:19 +00:00
Minio Trusted
cd18599e7b Update yaml files to latest version RELEASE.2022-08-26T19-53-15Z 2022-08-29 00:10:22 +00:00
Minio Trusted
1fd72d5aea Update yaml files to latest version RELEASE.2022-08-25T07-17-05Z 2022-08-25 07:47:18 +00:00
Anis Elleuch
b737c83a66
Ensure that only one node performs site replication healing (#15584) 
When a node finds a change in the other replication cluster and applies
to itself will already notify other peers. No need for all nodes in a
given cluster to do site replication healing, only one node is
sufficient.
 2022-08-24 13:46:09 -07:00
Minio Trusted
2c48f6a02b Update yaml files to latest version RELEASE.2022-08-22T23-53-06Z 2022-08-23 05:45:31 +00:00
Klaus Frank
a2e037f0ec
doc: certgen typo in parameter name (#15555) 
"-ca" is not a valid parameter remove it.
 2022-08-19 10:48:34 -07:00
Harshavardhana
d350b666ff
feat: add idempotent delete marker support (#15521) 
The bottom line is delete markers are a nuisance,
most applications are not version aware and this
has simply complicated the version management.

AWS S3 gave an unnecessary complication overhead
for customers, they need to now manage these
markers by applying ILM settings and clean
them up on a regular basis.

To make matters worse all these delete markers
get replicated as well in a replicated setup,
requiring two ILM settings on each site.

This PR is an attempt to address this inferior
implementation by deviating MinIO towards an
idempotent delete marker implementation i.e
MinIO will never create any more than single
consecutive delete markers.

This significantly reduces operational overhead
by making versioning more useful for real data.

This is an S3 spec deviation for pragmatic reasons.
 2022-08-18 16:41:59 -07:00
Minio Trusted
fa9b361a3d Update yaml files to latest version RELEASE.2022-08-13T21-54-44Z 2022-08-14 05:33:08 +00:00
Minio Trusted
c5b3666089 Update yaml files to latest version RELEASE.2022-08-11T04-37-28Z 2022-08-11 22:36:34 +00:00
Minio Trusted
6a6c772ff2 Update yaml files to latest version RELEASE.2022-08-08T18-34-09Z 2022-08-08 21:30:40 +00:00
Minio Trusted
fcd4b3ba9b Update yaml files to latest version RELEASE.2022-08-05T23-27-09Z 2022-08-06 00:08:21 +00:00
ebozduman
b57e7321e7
Replaces 'disk'=>'drive' visible to end user (#15464) 2022-08-04 16:10:08 -07:00
Minio Trusted
e956369c4e Update yaml files to latest version RELEASE.2022-08-02T23-59-16Z 2022-08-03 01:45:42 +00:00
Harshavardhana
fd349103e8
fix: allow P-384/P-512 constant time implementation (#15445) 
since go1.18.x P-384/P-512 are now constant time
implementations, enable them.
 2022-08-01 09:27:16 -07:00
Minio Trusted
6b4cb35f4f Update yaml files to latest version RELEASE.2022-07-30T05-21-40Z 2022-07-30 05:50:06 +00:00
Minio Trusted
824d52a82b Update yaml files to latest version RELEASE.2022-07-29T19-40-48Z 2022-07-29 22:06:57 +00:00
Minio Trusted
067ebab9d8 update object-locking docs and word them appropriately 2022-07-29 12:40:48 -07:00
Harshavardhana
5e763b71dc
use logger.LogOnce to reduce printing disconnection logs (#15408) 
fixes #15334

- re-use net/url parsed value for http.Request{}
- remove gosimple, structcheck and unusued due to https://github.com/golangci/golangci-lint/issues/2649
- unwrapErrs upto leafErr to ensure that we store exactly the correct errors
 2022-07-27 09:44:59 -07:00
Minio Trusted
bfc70bc74e Update yaml files to latest version RELEASE.2022-07-26T00-53-03Z 2022-07-26 06:56:37 +00:00
Poorna
426c902b87
site replication: fix healing of bucket deletes. (#15377) 
This PR changes the handling of bucket deletes for site 
replicated setups to hold on to deleted bucket state until 
it syncs to all the clusters participating in site replication.
 2022-07-25 17:51:32 -07:00
Minio Trusted
e465c3587b Update yaml files to latest version RELEASE.2022-07-24T17-09-31Z 2022-07-24 17:37:58 +00:00
Minio Trusted
f802d2ba83 Update yaml files to latest version RELEASE.2022-07-24T01-54-52Z 2022-07-24 02:31:24 +00:00
MohammadReza
f4d5c861f3
update grafana dashboard (#15357) 2022-07-21 15:17:44 -07:00
Minio Trusted
e7eb94de6b Update yaml files to latest version RELEASE.2022-07-17T15-43-14Z 2022-07-17 22:06:11 +00:00
Harshavardhana
b6eb8dff64
Add decommission compression+encryption enabled tests (#15322) 
update compression environment variables to follow
the expected sub-system style, however support fallback
mode.
 2022-07-17 08:43:14 -07:00
Harshavardhana
7da9e3a6f8
support encrypted/compressed objects properly during decommission (#15320) 
fixes #15314
 2022-07-16 19:35:24 -07:00
Minio Trusted
4aecd8d039 Update yaml files to latest version RELEASE.2022-07-15T03-44-22Z 2022-07-15 06:05:11 +00:00
Poorna
7e32a17742
fix: site replication healing of missing buckets (#15298) 
fixes a regression from #15186

- Adding tests to cover healing of buckets.
- Also dereference quota in SiteReplicationStatus only when non-nil
 2022-07-14 14:27:47 -07:00
Minio Trusted
688215e787 Update yaml files to latest version RELEASE.2022-07-13T23-29-44Z 2022-07-14 00:11:19 +00:00
daniel-bogusz95
00e235a1ee
fix grammatic errors and minor rewrites (#15264) 
Thank you @djwfyi for the help
 2022-07-11 07:59:49 -07:00
Minio Trusted
cf9e3069f2 Update yaml files to latest version RELEASE.2022-07-08T00-05-23Z 2022-07-08 00:44:43 +00:00
Minio Trusted
c1901f4e12 Update yaml files to latest version RELEASE.2022-07-06T20-29-49Z 2022-07-07 00:24:36 +00:00
Harshavardhana
9d80ff5a05
fix: decommission delete markers for non-current objects (#15225) 
versioned buckets were not creating the delete markers
present in the versioned stack of an object, this essentially
would stop decommission to succeed.

This PR fixes creating such delete markers properly during
a decommissioning process, adds tests as well.
 2022-07-05 07:37:24 -07:00
Minio Trusted
39b3941892 Update yaml files to latest version RELEASE.2022-07-04T21-02-54Z 2022-07-04 21:51:54 +00:00
Harshavardhana
b311abed31
decom IAM, Bucket metadata properly (#15220) 
Current code incorrectly passed the
config asset object name while decommissioning,
make sure that we pass the right object name
to be hashed on the newer set of pools.

This PR fixes situations after a successful
decommission, the users and policies might go
missing due to wrong hashed set.
 2022-07-04 14:02:54 -07:00
Poorna
0ea5c9d8e8
site healing: Skip stale iam asset updates from peer. (#15203) 
Allow healing to apply IAM change only when peer
gave the most recent update.
 2022-07-01 13:19:13 -07:00
Minio Trusted
a01a39b153 Update yaml files to latest version RELEASE.2022-06-30T20-58-09Z 2022-07-01 00:44:04 +00:00
Poorna
7cc9286e0f
site healing: Skip stale bucket metadata updates from peer (#15186) 
Allow healing to apply bucket metadata change only when peer
gave the most recent update.
 2022-06-28 18:09:20 -07:00
Harshavardhana
9c605ad153
allow support for parity '0', '1' enabling support for 2,3 drive setups (#15171) 
allows for further granular setups

- 2 drives (1 parity, 1 data)
- 3 drives (1 parity, 2 data)

Bonus: allows '0' parity as well.
 2022-06-27 20:22:18 -07:00
Minio Trusted
2075501d86 Update yaml files to latest version RELEASE.2022-06-25T15-50-16Z 2022-06-26 16:09:28 +00:00
Anis Elleuch
f3bec41eb9
s3-verify: Add a flag to exclude younger than a certain age (#15142) 
--minimum-object-age 1h can help exclude objects that are newly
uploaded but not replicated yet
 2022-06-22 08:12:47 -07:00
Anis Elleuch
95b51c48be
s3-verify: Fix endpoint and missing comparaison (#15129) 
- Fix a typo where target s3 client uses the source endpoint
- Fix a missing necessary comparison: if source name is lexically lower than target name
 2022-06-21 05:35:41 -07:00
Minio Trusted
17ab8145b5 Update yaml files to latest version RELEASE.2022-06-20T23-13-45Z 2022-06-21 00:16:07 +00:00
Harshavardhana
e83e947ca3
debug/s3-verify: simplify the tool to use lower memory footprint (#15110) 2022-06-20 10:45:35 -07:00
daniel-bogusz95
ce6c23a360 docs: some grammatical, typo fixes 
includes #15104, #15105, #15106, #15107
 2022-06-19 15:35:51 -07:00
Minio Trusted
1e1d9acb1b Update yaml files to latest version RELEASE.2022-06-17T02-00-35Z 2022-06-17 02:56:57 +00:00
Harshavardhana
8082d1fed6
add bucket level S3 received/sent bytes (#15084) 
adds bucket level metrics for bytes received and sent bytes on all S3 API calls.
 2022-06-14 15:14:24 -07:00
Minio Trusted
f34b2ef90b update dashboard Data Usage Growth as time series 2022-06-13 22:05:36 -07:00
George Costea
ce894665a8
examples: support configuration of a session policy file (#15078) 2022-06-13 15:36:58 -07:00
Anis Elleuch
e9efee0e64
debug: Close object after check (#15077) 2022-06-13 07:21:04 -07:00
Minio Trusted
4b3e7aee0b Update yaml files to latest version RELEASE.2022-06-11T19-55-32Z 2022-06-11 21:04:23 +00:00
Harshavardhana
7413045f0e
fix: add missing minio_s3_requests_total (#15070) 
PR #15052 caused a regression, add the missing metrics back.

Bonus:

- internode information should be only for distributed setups 
- update the dashboard to include 4xx and 5xx error panels.
 2022-06-11 00:50:31 -07:00
Harshavardhana
d76c508566
debug: verify diff on latest objects on source and target buckets (#15069) 2022-06-10 16:56:51 -07:00
Minio Trusted
8fb46de5e4 Update yaml files to latest version RELEASE.2022-06-10T16-59-15Z 2022-06-10 20:12:04 +00:00
Anis Elleuch
5fb420c703
prometheus: Add S3 4xx and 5xx S3 monitoring (#15052) 
Currently minio_s3_requests_errors_total covers 4xx and 
5xx S3 responses which can be confusing when s3 applications 
sent a lot of HEAD requests with obvious 404 responses or 
when the replication is enabled.

Add 
- minio_s3_requests_4xx_errors_total
- minio_s3_requests_5xx_errors_total

to help users monitor 4xx and 5xx HTTP status codes separately.
 2022-06-08 11:22:34 -07:00
Minio Trusted
f63645546d update minimum goroutine threshold on dashboard 2022-06-06 22:13:54 -07:00
Minio Trusted
27ab780317 Update yaml files to latest version RELEASE.2022-06-07T00-33-41Z 2022-06-07 01:06:59 +00:00
Minio Trusted
ac8cb6ba0d Update yaml files to latest version RELEASE.2022-06-06T23-14-52Z 2022-06-06 23:47:31 +00:00
Harshavardhana
48e367ff7d
reject resync start on misconfigured replication rules (#15041) 
we expect resync to start on buckets with replication
rule ExistingObjects enabled, if not we reject such
calls.
 2022-06-06 02:54:39 -07:00
Minio Trusted
3a398775fb Update yaml files to latest version RELEASE.2022-06-03T01-40-53Z 2022-06-03 02:36:56 +00:00
Minio Trusted
197d6fb644 Update yaml files to latest version RELEASE.2022-06-02T16-16-26Z 2022-06-02 17:46:32 +00:00
Minio Trusted
1fce2b180f Update yaml files to latest version RELEASE.2022-06-02T02-11-04Z 2022-06-02 02:42:14 +00:00
Harshavardhana
befbf48563 fix: s3-check-md5 to not panic for incomplete md5 2022-05-30 20:58:42 -07:00
Harshavardhana
f1abb92f0c
feat: Single drive XL implementation (#14970) 
Main motivation is move towards a common backend format
for all different types of modes in MinIO, allowing for
a simpler code and predictable behavior across all features.

This PR also brings features such as versioning, replication,
transitioning to single drive setups.
 2022-05-30 10:58:37 -07:00
Harshavardhana
c2630bb3a3 add total usage pie chart based on total/free bytes 2022-05-28 09:53:53 -07:00
Aditya Manthramurthy
464b9d7c80
Add support for Identity Management Plugin (#14913) 
- Adds an STS API `AssumeRoleWithCustomToken` that can be used to 
  authenticate via the Id. Mgmt. Plugin.
- Adds a sample identity manager plugin implementation
- Add doc for plugin and STS API
- Add an example program using go SDK for AssumeRoleWithCustomToken
 2022-05-26 17:58:09 -07:00
Praveen raj Mani
62cd643868
Add --insecure flag to skip TLS verification in s3-md5-check tool (#14980) 2022-05-26 06:02:05 -07:00
Minio Trusted
1b7dd70f72 Update yaml files to latest version RELEASE.2022-05-26T05-48-41Z 2022-05-26 06:27:03 +00:00
Klaus Post
fc08fcab52
hash-set: Add file input for debug tool (#14965) 
Add input option for a file list to display total distribution.
 2022-05-24 09:05:39 -07:00
Minio Trusted
5be76856bd Update yaml files to latest version RELEASE.2022-05-23T18-45-11Z 2022-05-24 00:29:45 +00:00
Minio Trusted
18a4276e25 Update yaml files to latest version RELEASE.2022-05-19T18-20-59Z 2022-05-19 20:18:49 +00:00
Aditya Manthramurthy
4629abd5a2
Add tests for Access Management Plugin (#14909) 2022-05-12 15:24:19 -07:00
Krishna Srinivas
389ec21d0c
Update documentation for /minio/health/cluster (#14889) 2022-05-12 09:54:07 -07:00
Eric Qiu
c293c2e9a3
docs: update new name for MINIO_POLICY_OPA_URL (#14898) 2022-05-11 13:04:15 -07:00
Aditya Manthramurthy
83071a3459
Add support for Access Management Plugin (#14875) 
- This change renames the OPA integration as Access Management Plugin - there is
nothing specific to OPA in the integration, it is just a webhook.

- OPA configuration is automatically migrated to Access Management Plugin and
OPA specific configuration is marked as deprecated.

- OPA doc is updated and moved.
 2022-05-10 17:14:55 -07:00
Minio Trusted
95d1a12422 Update yaml files to latest version RELEASE.2022-05-08T23-50-31Z 2022-05-09 03:46:40 +00:00
Harshavardhana
62aa42cccf
avoid replication proxy on version excluded paths (#14878) 
no need to attempt proxying objects that were
never replicated, but do have local `null`
versions on them.
 2022-05-08 16:50:31 -07:00
Harshavardhana
5cffd3780a
fix: multiple fixes in prefix exclude implementation (#14877) 
- do not need to restrict prefix exclusions that do not
  have `/` as suffix, relax this requirement as spark may
  have staging folders with other autogenerated characters
  , so we are better off doing full prefix March and skip. 

- multiple delete objects was incorrectly creating a
  null delete marker on a versioned bucket instead of
  creating a proper versioned delete marker.

- do not suspend paths on the excluded prefixes during
  delete operations to avoid creating `null` delete markers,
  honor suspension of versioning only at bucket level for
  delete markers.
 2022-05-07 22:06:44 -07:00
Krishnan Parthasarathi
ad8e611098
feat: implement prefix-level versioning exclusion (#14828) 
Spark/Hadoop workloads which use Hadoop MR 
Committer v1/v2 algorithm upload objects to a 
temporary prefix in a bucket. These objects are 
'renamed' to a different prefix on Job commit. 
Object storage admins are forced to configure 
separate ILM policies to expire these objects 
and their versions to reclaim space.

Our solution:

This can be avoided by simply marking objects 
under these prefixes to be excluded from versioning, 
as shown below. Consequently, these objects are 
excluded from replication, and don't require ILM 
policies to prune unnecessary versions.

-  MinIO Extension to Bucket Version Configuration
```xml
<VersioningConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"> 
        <Status>Enabled</Status>
        <ExcludeFolders>true</ExcludeFolders>
        <ExcludedPrefixes>
          <Prefix>app1-jobs/*/_temporary/</Prefix>
        </ExcludedPrefixes>
        <ExcludedPrefixes>
          <Prefix>app2-jobs/*/__magic/</Prefix>
        </ExcludedPrefixes>

        <!-- .. up to 10 prefixes in all -->     
</VersioningConfiguration>
```
Note: `ExcludeFolders` excludes all folders in a bucket 
from versioning. This is required to prevent the parent 
folders from accumulating delete markers, especially
those which are shared across spark workloads 
spanning projects/teams.

- To enable version exclusion on a list of prefixes

```
mc version enable --excluded-prefixes "app1-jobs/*/_temporary/,app2-jobs/*/_magic," --exclude-prefix-marker myminio/test
```
 2022-05-06 19:05:28 -07:00
Aditya Manthramurthy
2b7e75e079
Add OPA doc and remove deprecation marking (#14863) 2022-05-04 23:53:42 -07:00
Minio Trusted
2fc65dcb99 Update yaml files to latest version RELEASE.2022-05-04T07-45-27Z 2022-05-04 08:54:16 +00:00
Minio Trusted
0a256053ee Update yaml files to latest version RELEASE.2022-05-03T20-36-08Z 2022-05-03 21:27:19 +00:00
Minio Trusted
f427dbbd60 Update yaml files to latest version RELEASE.2022-04-30T22-23-53Z 2022-05-01 01:33:28 +00:00
Harshavardhana
85f3a9f3b0 Remove Azure gateway implementation (#14418) 
refer #14331
 2022-04-29 12:51:23 -07:00
Minio Trusted
96f27a4965 Update yaml files to latest version RELEASE.2022-04-29T01-27-09Z 2022-04-29 06:32:50 +00:00
Aditya Manthramurthy
0e502899a8
Add support for multiple OpenID providers with role policies (#14223) 
- When using multiple providers, claim-based providers are not allowed. All
providers must use role policies.

- Update markdown config to allow `details` HTML element
 2022-04-28 18:27:09 -07:00
Harshavardhana
990fbeb3a4 rename true/false to on/off in bucket notification docs 2022-04-27 23:51:31 -07:00
Minio Trusted
8a6b2b4447 Update yaml files to latest version RELEASE.2022-04-26T01-20-24Z 2022-04-26 02:08:20 +00:00
Harshavardhana
0cc993f403 Remove GCS, HDFS gateway implementations #14418 
refer #14331
 2022-04-24 10:19:17 -07:00
Minio Trusted
1494ba2e6e Update yaml files to latest version RELEASE.2022-04-16T04-26-02Z 2022-04-16 05:03:00 +00:00
Minio Trusted
646350fa7f Update yaml files to latest version RELEASE.2022-04-12T06-55-35Z 2022-04-12 07:23:20 +00:00
Minio Trusted
33d267fa1b Update yaml files to latest version RELEASE.2022-04-09T15-09-52Z 2022-04-09 20:23:18 +00:00
Minio Trusted
f630d7c3fa Update yaml files to latest version RELEASE.2022-04-08T19-44-35Z 2022-04-08 23:35:38 +00:00
Minio Trusted
c7962118f8 Update yaml files to latest version RELEASE.2022-04-01T03-41-39Z 2022-04-01 08:23:40 +00:00
Andreas Auernhammer
e955aa7f2a
kes: add support for encrypted private keys (#14650) 
This commit adds support for encrypted KES
client private keys.

Now, it is possible to encrypt the KES client
private key (`MINIO_KMS_KES_KEY_FILE`) with
a password.

For example, KES CLI already supports the
creation of encrypted private keys:
```
kes identity new --encrypt --key client.key --cert client.crt MinIO
```

To decrypt an encrypted private key, the password
needs to be provided:
```
MINIO_KMS_KES_KEY_PASSWORD=<password>
```

Signed-off-by: Andreas Auernhammer <hi@aead.dev>
 2022-03-29 09:53:33 -07:00
Eco
81d2b54dfd
doc: typo fix for ttfb entry in table (#14647) 2022-03-29 09:42:02 -07:00
Minio Trusted
be5d394e56 Update yaml files to latest version RELEASE.2022-03-26T06-49-28Z 2022-03-26 07:32:25 +00:00
Minio Trusted
849a27ee61 update hotfixes instructions and fix some typo 2022-03-25 23:49:28 -07:00
Andreas Auernhammer
4d2fc530d0
add support for SSE-S3 bulk ETag decryption (#14627) 
This commit adds support for bulk ETag
decryption for SSE-S3 encrypted objects.

If KES supports a bulk decryption API, then
MinIO will check whether its policy grants
access to this API. If so, MinIO will use
a bulk API call instead of sending encrypted
ETags serially to KES.

Note that MinIO will not use the KES bulk API
if its client certificate is an admin identity.

MinIO will process object listings in batches.
A batch has a configurable size that can be set
via `MINIO_KMS_KES_BULK_API_BATCH_SIZE=N`.
It defaults to `500`.

This env. variable is experimental and may be
renamed / removed in the future.

Signed-off-by: Andreas Auernhammer <hi@aead.dev>
 2022-03-25 15:01:41 -07:00
Minio Trusted
cf220be9b5 Update yaml files to latest version RELEASE.2022-03-24T00-43-44Z 2022-03-24 01:28:05 +00:00
Minio Trusted
751e9fb7be Update yaml files to latest version RELEASE.2022-03-22T02-05-10Z 2022-03-22 02:45:24 +00:00
Minio Trusted
e9bc0e7e98 Update yaml files to latest version RELEASE.2022-03-17T06-34-49Z 2022-03-17 00:11:59 -07:00
Minio Trusted
7a733a8d54 Update yaml files to latest version RELEASE.2022-03-17T02-57-36Z 2022-03-16 22:27:48 -07:00
Aditya Manthramurthy
ce97313fda
Add extra LDAP configuration validation (#14535) 
- The result now contains suggestions on fixing common configuration issues.
- These suggestions will subsequently be exposed in console/mc
 2022-03-16 19:57:36 -07:00
Minio Trusted
721c053712 Update yaml files to latest version RELEASE.2022-03-14T18-25-24Z 2022-03-14 19:32:22 +00:00
Minio Trusted
57b7c3494f Update yaml files to latest version RELEASE.2022-03-11T23-57-45Z 2022-03-13 08:47:27 +00:00
Poorna
f8d6eaaa96
fix: regression from range GET proxy on replicated buckets #14345 (#14532) 
Fixes: #14531
 2022-03-11 15:56:49 -08:00
Minio Trusted
80039f60d5 Update yaml files to latest version RELEASE.2022-03-11T11-08-23Z 2022-03-11 11:47:17 +00:00
Aditya Manthramurthy
b7ed3b77bd
Indicate required fields in LDAP configuration correctly (#14526) 2022-03-10 19:03:38 -08:00
Minio Trusted
fce380a044 Update yaml files to latest version RELEASE.2022-03-08T22-28-51Z 2022-03-09 01:36:59 +00:00
Minio Trusted
1f11af42f1 Update yaml files to latest version RELEASE.2022-03-05T06-32-39Z 2022-03-05 09:27:28 +00:00
Jan Madera
a026c8748f
Update nginx.conf for large file uploads (#14481) 2022-03-04 22:32:39 -08:00
Harshavardhana
b0c84e3de7
fix: deleteVersions causing xl.meta to have empty Versions[] slice (#14483) 
This is a side-affect of the optimization done in PR #13544 which
causes a certain type of delete operations on given object versions
can cause lastVersion indication to be skipped, which leads to
an `xl.meta` where Versions[] slice is empty while the entire
file is intact by itself.

This PR tries to ensure that such files are visible and deletable
by regular means of listing as null 'delete-marker' and also
avoid the situation where this potential issue might arise.
 2022-03-04 20:01:26 -08:00
Minio Trusted
04bbede17d Update yaml files to latest version RELEASE.2022-03-03T21-21-16Z 2022-03-03 22:16:10 +00:00
Emmet McPoland
27fad98179
Replace HeadBucket permission with GetBucketAcl (#14436) 
Resolves https://github.com/minio/minio/issues/14379
 2022-03-01 21:18:23 -08:00
Minio Trusted
7d4579e737 Update yaml files to latest version RELEASE.2022-02-26T02-54-46Z 2022-02-26 03:36:08 +00:00
Harshavardhana
890e526bde rename 'mc admin inspect' to 'mc support inspect' 2022-02-24 17:17:53 -08:00
Harshavardhana
16ce455fca update docker release to RELEASE.2022-02-24T22-12-01Z 2022-02-24 15:35:14 -08:00
Minio Trusted
b23b19e5c3 Update yaml files to latest version RELEASE.2022-02-18T01-50-10Z 2022-02-17 19:12:27 -08:00
Minio Trusted
1dbb3f6f43 Update yaml files to latest version RELEASE.2022-02-17T23-22-26Z 2022-02-18 00:33:01 +00:00
Minio Trusted
27d94c64ed Update yaml files to latest version RELEASE.2022-02-16T00-35-27Z 2022-02-16 05:34:56 +00:00
Pierre Kancir
d0862ddf86
doc: add console-address on all example (#14307) 
--console-address ":9001" is  missing on docker example for regular user.
 2022-02-15 09:26:04 -08:00
Harshavardhana
f088e8960b
docs: turn-on more markdown rules and fix them (#14301) 2022-02-14 08:50:42 -08:00
Anis Elleuch
1f92fc3fc0
Always check for root disks unless MINIO_CI_CD is set (#14232) 
The current code considers a pool with all root disks to be as part
of a testing environment even if there are other pools with mounted
disks. This will result to illegitimate writing in root disks.

Fix this by simplifing the logic: require MINIO_CI_CD in order to skip
root disk check.
 2022-02-13 15:42:07 -08:00
Minio Trusted
f71b114a84 Update yaml files to latest version RELEASE.2022-02-12T00-51-25Z 2022-02-13 19:31:37 +00:00
Harshavardhana
e3e0532613
cleanup markdown docs across multiple files (#14296) 
enable markdown-linter
 2022-02-11 16:51:25 -08:00
Harshavardhana
97b5bf1fb7 update gateway docs to indicate code-freeze 2022-02-10 11:39:41 -08:00
Poorna
ed3418c046
Refactor replication resync to be an active process (#14266) 
When resync is triggered, walk the bucket namespace and
resync objects that are unreplicated. This PR also adds
an API to report resync progress.
 2022-02-10 10:16:52 -08:00
Harshavardhana
a2230868e0 remove all stale old docs about 2018 releases 2022-02-10 09:54:27 -08:00
Krishnan Parthasarathi
0ee2933234
Export tier metrics via Prometheus (#13413) 
e.g
```
minio_cluster_ilm_transitioned_bytes{server="minio3:9000",tier="S3TIER-1"} 1.36317772e+08
minio_cluster_ilm_transitioned_bytes{server="minio3:9000",tier="S3TIER-2"} 2892
minio_cluster_ilm_transitioned_bytes{server="minio3:9000",tier="STANDARD"}
1.3631488e+08

minio_cluster_ilm_transitioned_objects{server="minio3:9000",tier="S3TIER-1"} 1
minio_cluster_ilm_transitioned_objects{server="minio3:9000",tier="S3TIER-2"} 0
minio_cluster_ilm_transitioned_objects{server="minio3:9000",tier="STANDARD"} 1

minio_cluster_ilm_transitioned_versions{server="minio3:9000",tier="S3TIER-1"} 3
minio_cluster_ilm_transitioned_versions{server="minio3:9000",tier="S3TIER-2"} 2
minio_cluster_ilm_transitioned_versions{server="minio3:9000",tier="STANDARD"} 1
```
 2022-02-08 12:45:28 -08:00
Minio Trusted
2480c66857 Update yaml files to latest version RELEASE.2022-02-07T08-17-33Z 2022-02-07 09:19:24 +00:00
Minio Trusted
570670be8c Update yaml files to latest version RELEASE.2022-02-05T04-40-59Z 2022-02-05 18:33:46 +00:00
Minio Trusted
9417fd933e Update yaml files to latest version RELEASE.2022-02-01T18-00-14Z 2022-02-01 23:37:07 +00:00
Harshavardhana
74faed166a
Add quota usage as part of prometheus metrics (#14222) 
Bonus: pass caller context when needed to all bucket metadata handling calls.
 2022-01-31 17:27:43 -08:00
Harshavardhana
dbd05d6e82
remove FIFO bucket quota, use ILM expiration instead (#14206) 2022-01-31 11:07:04 -08:00
Minio Trusted
c7df5fb119 Update yaml files to latest version RELEASE.2022-01-28T02-28-16Z 2022-01-28 05:12:35 +00:00
Minio Trusted
a66071099c Update yaml files to latest version RELEASE.2022-01-27T03-53-02Z 2022-01-27 04:47:45 +00:00
Aditya Manthramurthy
7dfa565d00
Identity LDAP: Allow multiple search base DNs (#14191) 
This change allows the MinIO server to lookup users in different directory
sub-trees by allowing specification of multiple search bases separated by
semicolons.
 2022-01-26 15:05:59 -08:00