jiuker
62cd918061
fix: close helmInfo file descriptor ( #16319 )
2023-01-01 23:26:59 -08:00
Klaus Post
6a04067514
fix: tweak read buffer size to reduce over-reading ( #16338 )
2023-01-01 08:14:20 -08:00
Taran Pelkey
49b3908635
fix: misplaced write response command in DetachPolicy() ( #16333 )
2022-12-30 20:04:03 -08:00
Harshavardhana
f93183f66e
fix: a deadlock by refactoring listBuckets() under site replication ( #16323 )
2022-12-29 00:08:31 -08:00
Harshavardhana
2937711390
fix: DeleteObject() API with versionId under replication ( #16325 )
2022-12-28 22:48:33 -08:00
Anis Elleuch
27417459fb
metrics: Show healing info for all nodes ( #16315 )
2022-12-26 08:35:32 -08:00
Harshavardhana
5b8fe2e89a
allow locks with object affinity to spread across pools ( #16312 )
2022-12-23 20:55:45 -08:00
Anis Elleuch
acc9c033ed
debug: Add X-Amz-Request-ID to lock/unlock calls ( #16309 )
2022-12-23 19:49:07 -08:00
Poorna
8528b265a9
Validate replication target update to avoid duplicate endpoints ( #16311 )
2022-12-23 15:44:48 -08:00
Harshavardhana
b882310e2b
avoid locks for internal and invalid buckets in MakeBucket() ( #16302 )
2022-12-23 07:46:00 -08:00
Poorna
de0b43de32
persist replication stats with leader lock ( #16282 )
2022-12-22 14:25:13 -08:00
jiuker
29dd7f1d68
tier verification leaks fd, that must be closed ( #16296 )
...
Co-authored-by: Harshavardhana <harsha@minio.io>
2022-12-22 10:35:54 -08:00
Poorna
6423e4c767
Remove site replication config if it succeeded locally ( #16279 )
2022-12-22 01:31:20 -08:00
Krishnan Parthasarathi
2fa35def2c
Fix DeleteObject when only free versions remain ( #16289 )
2022-12-21 16:24:07 -08:00
Anis Elleuch
34167c51d5
trace: Add bootstrap tracing events ( #16286 )
2022-12-21 15:52:29 -08:00
Harshavardhana
a5f8af4efb
serialize replication stats() only when needed ( #16280 )
2022-12-20 00:07:53 -08:00
Harshavardhana
5a218f38a1
allow retries for transaction lock on startup ( #16273 )
2022-12-19 22:00:00 -08:00
Anis Elleuch
e57e946206
Do not save credentials in config.json ( #16275 )
2022-12-19 12:27:06 -08:00
Klaus Post
b4f71362e9
Avoid config migration on every startup ( #16278 )
2022-12-19 11:10:14 -08:00
Taran Pelkey
ed37b7a9d5
Add API to fetch policy user/group associations ( #16239 )
2022-12-19 10:37:03 -08:00
Anis Elleuch
89db3fdb5d
Do not return an error when version disparity is detected ( #16269 )
2022-12-16 08:52:12 -08:00
Harshavardhana
80fc3a8a52
use newDynamicTimeoutWithOpts() when appropriate ( #16266 )
2022-12-15 13:11:37 -08:00
Klaus Post
988a2e8fed
Faster startup of large distributed systems with latency ( #16259 )
2022-12-15 08:31:21 -08:00
Harshavardhana
2433698372
fix: remove unnecessary logs for client conn errors ( #16261 )
2022-12-15 08:25:05 -08:00
Harshavardhana
5d7e8f79ed
fix: remove scanner healing with unnecessary logs ( #16260 )
2022-12-14 16:39:18 -08:00
Harshavardhana
bad229e16e
fix: support event name s3:Restore:* ( #16257 )
2022-12-14 05:12:07 -08:00
Poorna
d37e514733
Cleanup remote targets automatically on replication config removal. ( #16221 )
2022-12-14 03:24:06 -08:00
Harshavardhana
c73ea27ed7
do not log checksum mismatch error, client received the error ( #16246 )
2022-12-14 01:57:40 -08:00
Krishnan Parthasarathi
0159b56717
fix: rebalance to account for object's on-disk size ( #16240 )
2022-12-14 00:15:14 -08:00
Aditya Manthramurthy
9e6cc847f8
Add HTTP2 config option for policy plugin ( #16225 )
2022-12-13 14:28:48 -08:00
Taran Pelkey
709eb283d9
Add endpoints for managing IAM policies ( #15897 )
...
Co-authored-by: Taran <taran@minio.io>
Co-authored-by: ¨taran-p¨ <¨taran@minio.io¨>
Co-authored-by: Aditya Manthramurthy <donatello@users.noreply.github.com>
2022-12-13 12:13:23 -08:00
Anis Elleuch
76dde82b41
Implement STS account info API ( #16115 )
2022-12-13 08:38:50 -08:00
Anis Elleuch
939c0100a6
log: Do not interpret verbs in object names in console output ( #16233 )
2022-12-13 08:27:40 -08:00
Aditya Manthramurthy
2d60bf8c50
Refactor HTTP transports ( #16222 )
2022-12-12 20:31:21 -08:00
Harshavardhana
37e20f6ef2
feat: allow listening specific addrs for API port ( #16223 )
2022-12-12 18:48:46 -08:00
Harshavardhana
2fc182d8e6
fix: iso8601TimeFormat padding issue for certain nanoseconds ( #16207 )
2022-12-12 10:28:30 -08:00
Shireesh Anjal
a2cbeaa9e6
Use different subnet public key during dev/test ( #16216 )
2022-12-12 10:28:15 -08:00
Harshavardhana
444ff20bc5
do not rename multipart failed transactions back to tmp ( #16204 )
2022-12-12 01:40:29 -08:00
Harshavardhana
20ef5e7a6a
avoid double deletes() when no more versions ( #16206 )
2022-12-12 01:40:04 -08:00
Aditya Manthramurthy
e06127566d
Add IAM API to attach/detach policies for LDAP ( #16182 )
2022-12-09 13:08:33 -08:00
Harshavardhana
dfe73629a3
fix: delete marker discrepancies via DeleteObject() API ( #16195 )
2022-12-08 18:15:16 -08:00
Harshavardhana
b03dd1af17
remove hard limit for number of buckets ( #16194 )
2022-12-08 12:24:03 -08:00
Harshavardhana
4bc367c490
fix: translate tier add errors properly ( #16191 )
2022-12-08 11:18:07 -08:00
Klaus Post
3eb2d086b2
Replace filepathx with fork ( #16192 )
2022-12-08 10:42:44 -08:00
Klaus Post
70986b6e6e
Add version id to healresult ( #16193 )
2022-12-08 07:49:10 -08:00
Klaus Post
ebe395788b
feat: Encrypt s3zip file index ( #16179 )
2022-12-07 14:56:07 -08:00
Klaus Post
12fd6678ee
Encrypt checksums with KMS on CompleteMultipartUpload ( #16177 )
2022-12-07 10:18:18 -08:00
Harshavardhana
90d35b70b4
remove unnecessary logs for truncated XML inputs ( #16184 )
2022-12-07 08:30:52 -08:00
Javier Adriel
04ae9058ed
Populate end_session_endpoint ( #16183 )
2022-12-06 16:56:37 -08:00
Aditya Manthramurthy
a30cfdd88f
Bump up madmin-go to v2 ( #16162 )
2022-12-06 13:46:50 -08:00
Anis Elleuch
1bae32dc96
xl: Delete older data-dir when replacing an existing version-id ( #16176 )
2022-12-06 13:43:18 -08:00
Anis Elleuch
932d2c3c62
Add X-Amz-Request-Id to internode calls ( #16146 )
2022-12-06 09:27:26 -08:00
jiuker
8d8d07ac5c
use readlock instead of writelock to get heal information ( #16175 )
2022-12-06 08:08:22 -08:00
Anis Elleuch
44735be38e
s3: Return correct error when Version is invalid in policy document ( #16178 )
2022-12-06 08:07:24 -08:00
Klaus Post
3fd9059b4e
opt: Only stream big data usage caches ( #16168 )
2022-12-05 13:01:11 -08:00
Klaus Post
a713aee3d5
Run staticcheck on CI ( #16170 )
2022-12-05 11:18:50 -08:00
Andreas Auernhammer
d882ba2cb4
kms: add support for KES enclaves ( #16139 )
...
Signed-off-by: Andreas Auernhammer <hi@aead.dev>
2022-12-04 02:34:24 -08:00
jiuker
6086f45d25
fix: in disk cache readCacheFileStream should closed upon return ( #16138 )
2022-12-04 02:28:10 -08:00
Klaus Post
98cffbce03
s3zip: Limit over-read for single file ( #16161 )
2022-12-02 08:53:24 -08:00
Klaus Post
1cd875de1e
Persist updated metadata ( #16160 )
2022-12-02 08:35:04 -08:00
Harshavardhana
5a8df7efb3
re-implement StorageInfo to be a peer call ( #16155 )
2022-12-01 14:31:35 -08:00
Anis Elleuch
c84e2939e4
trace: Publish storage layer errors ( #16153 )
2022-12-01 12:10:54 -08:00
Anis Elleuch
641ab24aec
repl: resync orchestrator to use global shared lock ( #16154 )
2022-12-01 12:10:09 -08:00
Harshavardhana
71133105d7
re-order the top-level config keys for priority ( #16150 )
2022-12-01 07:50:08 -08:00
Aditya Manthramurthy
87cbd41265
feat: Allow at most one claim based OpenID IDP ( #16145 )
2022-11-29 15:40:49 -08:00
Klaus Post
cc1d8f0057
Check for abandoned data when healing ( #16122 )
2022-11-28 10:20:55 -08:00
Anis Elleuch
1f1dcdce65
move HTTP recorder to an internal library ( #16128 )
2022-11-28 10:20:27 -08:00
Shireesh Anjal
98a67a3776
Improvements in logger and audit webhooks ( #16102 )
2022-11-28 08:03:26 -08:00
Poorna
63fc6ba2cd
preserve replicated ETag properly on target ( #16129 )
2022-11-26 14:43:32 -08:00
jiuker
ce53d7f6c2
add disk.Close() in healFreshDisk to indicate idiomatic flow of code ( #16124 )
2022-11-26 00:26:15 -08:00
jiuker
fe8eed963e
fix: wrapped error will not equal in decommissioning ( #16113 )
2022-11-24 08:00:42 -08:00
Shireesh Anjal
59f877fc64
fix: Timestamp not added in diagnostics report ( #16114 )
2022-11-23 07:11:22 -08:00
Klaus Post
f96fe9773c
fix: duplicated shared prefix with custom delimiter when listing ( #16111 )
2022-11-22 08:51:04 -08:00
Anis Elleuch
04948b4d55
fix: checking for stale STS account under site replication ( #16109 )
2022-11-22 07:26:33 -08:00
Klaus Post
98ba622679
Reduce temporary file clean-up waits ( #16110 )
2022-11-22 07:23:36 -08:00
Harshavardhana
08103870a5
update single drive setup error message ( #16098 )
2022-11-18 14:47:38 -08:00
Anis Elleuch
993e586855
config: return XMinioConfigNotFound code for non existing config ( #16065 )
2022-11-18 10:28:14 -08:00
Harshavardhana
58ec835af0
fix: skip free version ID and marker in metadata equality ( #16093 )
2022-11-18 05:48:22 -08:00
Harshavardhana
6aea950d74
avoid partID lock validating uploadID exists prematurely ( #16086 )
2022-11-18 03:09:35 -08:00
Poorna
7198be5be9
bucket resync: persist reset id to bucket metadata ( #16088 )
2022-11-18 01:39:05 -08:00
Klaus Post
a22b4adf4c
distribute replication ops based on names ( #16083 )
2022-11-17 15:20:09 -08:00
Klaus Post
b7bb122be8
fix: replication auto-scaling deadlock ( #16084 )
2022-11-17 07:35:02 -08:00
Krishnan Parthasarathi
8441a3bf5f
fix: update metacache entry only once ( #16072 )
2022-11-16 11:25:00 -08:00
Harshavardhana
853c4de75a
allow changing endpoints in distributed setups ( #16071 )
2022-11-16 07:59:10 -08:00
jiuker
3597af789e
allow resultCh to be closed() after clusterMetaHealthInfo() ( #16073 )
2022-11-16 03:04:36 -08:00
Shireesh Anjal
5246e3be84
Send health diagnostics data as part of callhome ( #16006 )
2022-11-15 13:53:05 -08:00
Klaus Post
8a07000e58
fix: refactor getReplicationDiff for safe use ( #16051 )
2022-11-15 07:59:21 -08:00
Krishnan Parthasarathi
3bb82ef60d
top-locks: Include lock-held duration ( #16061 )
2022-11-15 07:57:52 -08:00
Harshavardhana
91f45c4aa6
avoid inconsistent versions healing when versions are large ( #16066 )
2022-11-14 18:35:26 -08:00
Poorna
d6bc141bd1
feat: Add support for site level resync ( #15753 )
2022-11-14 07:16:40 -08:00
jiuker
7ac64ad24a
fix: use errors.Is for wrapped returns ( #16062 )
2022-11-14 07:15:46 -08:00
Harshavardhana
6d76db9d6c
improve server startup error when pools are incorrect ( #16056 )
2022-11-11 19:40:45 -08:00
jiuker
bdcb485740
netPerfRX Reset() should use write Lock() ( #16043 )
2022-11-10 19:44:20 -08:00
Poorna
e32b948a49
fix: parsing multipart uploadID under site replicated setup ( #16048 )
...
continue the fix from #16034
2022-11-10 16:17:45 -08:00
Klaus Post
5b242f1d11
Add Audit target metrics ( #16044 )
2022-11-10 10:20:21 -08:00
Poorna
34d28dd79f
replication: Avoid blocking on mrf save ( #16045 )
2022-11-10 10:20:02 -08:00
Krishnan Parthasarathi
6eef9b4a23
lifecycle: simplify Eval and HasActiveRules ( #16036 )
2022-11-10 07:17:45 -08:00
Aditya Manthramurthy
5f1999cc71
fix: avoid URL unsafe chars in multipart upload ID ( #16034 )
2022-11-09 16:41:16 -08:00
Krishnan Parthasarathi
40a2c6b882
Return remote tier as StorageClass for transitioned objects ( #16035 )
2022-11-09 15:57:34 -08:00
jiuker
7b7356f04c
close the reader under disk cache bitrot verification ( #16024 )
2022-11-09 04:20:11 -08:00
Klaus Post
bbc312fce6
Add notification queue metrics ( #16026 )
2022-11-08 16:36:47 -08:00
Anis Elleuch
7260241511
Remove some logs caused by external apps ( #16027 )
2022-11-08 13:29:05 -08:00
Anis Elleuch
3b1a9b9fdf
Use the same lock for the scanner and site replication healing ( #15985 )
2022-11-08 08:55:55 -08:00
Harshavardhana
72afc2727a
rebalance status must return appropriate error initially ( #16022 )
2022-11-08 07:56:45 -08:00
Aditya Manthramurthy
76d822bf1e
Add LDAP policy entities API ( #15908 )
2022-11-07 14:35:09 -08:00
Klaus Post
ddeca9f12a
fix: filter rest errors and logs returned ( #16019 )
2022-11-07 10:38:08 -08:00
Harshavardhana
1f3db03bf0
allow changing argument for path for SNSD setup ( #16013 )
2022-11-07 00:11:58 -08:00
Harshavardhana
944c62daf4
skip flaky tests on windows OS ( #16015 )
2022-11-07 00:11:21 -08:00
Harshavardhana
9547b7d0e9
add deadlineConnections on remoteTransport ( #16010 )
2022-11-05 11:09:21 -07:00
Klaus Post
808ecfe0f2
merge versions across sets when listing ( #16003 )
2022-11-04 11:33:22 -07:00
Klaus Post
2894dd4d1a
fix: hold lock while serializing replication stats ( #16007 )
2022-11-04 09:59:14 -07:00
jiuker
fd8750e959
fix: http body must be drained in downloadBinary() ( #16001 )
2022-11-04 08:22:38 -07:00
Poorna
4f5d38a4b1
site replication edit: validate endpoint belongs to deployment ( #16000 )
2022-11-03 16:23:45 -07:00
Anis Elleuch
7e73fc2870
Implement inspect data API v2 ( #15474 )
...
Co-authored-by: Klaus Post <klauspost@gmail.com>
2022-11-02 13:36:38 -07:00
Harshavardhana
0d49b365ff
converge SNSD deployments into single code ( #15988 )
2022-11-01 16:41:01 -07:00
Anis Elleuch
7721595aa9
config: Deprecated delay/max_wait/scanner and introduce speed ( #15941 )
2022-11-01 08:04:07 -07:00
Harshavardhana
fd6f6fc8df
cleanup stale parent multipart directories ( #15980 )
2022-11-01 08:00:02 -07:00
Aditya Manthramurthy
4fb47cd568
fix: update admin IDP APIs to be more RESTful ( #15896 )
2022-10-31 14:52:26 -07:00
Klaus Post
ecc932d5dd
Clean entire tmp-old on restart ( #15979 )
2022-10-31 07:27:50 -07:00
Harshavardhana
b57fbff7c1
ignore background healInfo in single drive setup ( #15968 )
2022-10-31 07:26:10 -07:00
Poorna
d765b89a63
improve validation for replication resync API ( #15964 )
2022-10-28 23:21:33 -07:00
Harshavardhana
6e4acf0504
add a message of removal for gateway and hide the command ( #15965 )
2022-10-28 14:11:20 -07:00
Klaus Post
71954faa3a
mark pubsub type safe via generics ( #15961 )
2022-10-28 10:55:42 -07:00
Klaus Post
0f0e154315
fix: inconsistent replication delete marker timestamps ( #15956 )
2022-10-27 09:46:52 -07:00
Harshavardhana
136d41775f
remove numAvailableDisks check as it doesn't serve any purpose ( #15954 )
2022-10-27 09:05:24 -07:00
Harshavardhana
ec77d28e62
make subnet subsys dynamic and simplify callhome ( #15927 )
2022-10-27 00:20:01 -07:00
Klaus Post
86420a1f46
Store multipart checksums ( #15953 )
2022-10-26 18:14:58 -07:00
Poorna
7dd8b6c8ed
ensure ILM expiry creates non null deleteMarker for versioned bucket ( #15947 )
2022-10-26 16:09:27 -07:00
Anis Elleuch
533c9d4fe3
fix: lockName to disallow parallel same erasure set healing ( #15951 )
2022-10-26 12:43:54 -07:00
Anis Elleuch
a35ef155fc
return appropriate error status code in the lock handler ( #15950 )
2022-10-26 09:51:26 -07:00
Poorna
8dd3c41b2a
allow MakeBucket errors to be handled lazily ( #15945 )
...
remote error is not required to be passed back to the
client - this is mostly because we have healing that should
eventually, catch up on this and heal the bucket.
2022-10-25 23:32:37 -07:00
Krishnan Parthasarathi
4523da6543
feat: introduce pool-level rebalance ( #15483 )
2022-10-25 12:36:57 -07:00
Poorna
ce8456a1a9
proxy multipart to peers via multipart uploadID ( #15926 )
2022-10-25 10:52:29 -07:00
Poorna
9ce1884732
reject editing bucket replication config when site replication is enabled ( #15937 )
2022-10-24 20:24:32 -07:00
Harshavardhana
23b329b9df
remove gateway completely ( #15929 )
2022-10-24 17:44:15 -07:00
Krishnan Parthasarathi
0c34e51a75
Filter out tiering metadata during CopyObject ( #15936 )
2022-10-24 16:32:31 -07:00
Anis Elleuch
fc6c794972
Audit dangling object removal ( #15933 )
2022-10-24 11:35:07 -07:00
Klaus Post
86d543d0f6
Check for s3zip content offset ( #15924 )
2022-10-21 15:37:48 -07:00
Poorna
e4e90b53c1
fix: delete-marker replication check properly ( #15923 )
2022-10-21 14:45:06 -07:00
Anis Elleuch
58d776daa0
Set CONSOLE_MINIO_SERVER to 127.0.0.1 by default ( #15887 )
2022-10-21 14:42:28 -07:00
Krishnan Parthasarathi
f6b2e89109
Pass encrypted etag as is for immediate tiering ( #15925 )
2022-10-21 14:40:50 -07:00
Anis Elleuch
ac85c2af76
lifecycle: refactor rules filtering and tagging support ( #15914 )
2022-10-21 10:46:53 -07:00
Shireesh Anjal
5aba2aedb3
Do not freeze s3 traffic in healthinfo api ( #15912 )
2022-10-21 00:34:32 -07:00
Harshavardhana
a8332efa94
fix: Delete() of bucket metadata should not parse the config ( #15904 )
2022-10-19 17:55:09 -07:00
Aditya Manthramurthy
3dbef72dc7
fix: AccountInfo API for roleARN based accounts ( #15907 )
2022-10-19 17:54:41 -07:00
Aditya Manthramurthy
2d16e74f38
Add LDAP IDP Configuration APIs ( #15840 )
2022-10-19 11:00:10 -07:00
Anis Elleuch
de5070446d
Deprecate --listeners flag ( #15900 )
2022-10-19 08:45:50 -07:00
Harshavardhana
374abd1e7d
add filter support for tags and metadata in batch replication ( #15885 )
2022-10-18 21:22:21 -07:00
Anis Elleuch
0506d9e83d
storage: Return errDiskNotFound when a peer is during shutdown ( #15868 )
2022-10-18 13:50:46 -07:00
Klaus Post
bd3dfad8b9
Add concurrent Snowball extraction + options ( #15836 )
2022-10-18 13:50:21 -07:00
Harshavardhana
9fff315555
do not need to trace ignored objects ( #15894 )
2022-10-18 13:47:55 -07:00
Anis Elleuch
18fb86b7be
convert context.DeadlineExceed to offline disk in DiskInfo() ( #15886 )
2022-10-18 03:01:16 -07:00
Harshavardhana
58a8275e84
do not assume invalid buf to be non-xl.meta ( #15843 )
2022-10-17 09:39:21 -07:00
Aditya Manthramurthy
85fc7cea97
Pass role ARN for OIDC providers to console ( #15862 )
2022-10-15 12:57:03 -07:00
Harshavardhana
328d660106
support CRC32 Checksums on single drive setup ( #15873 )
2022-10-15 11:58:47 -07:00
Harshavardhana
c68910005b
validate bucket before attempting batch replication ( #15861 )
2022-10-15 11:58:31 -07:00
Harshavardhana
c79bcc8838
Revert "convert context.DeadlineExceed to offline disk in DiskInfo() ( #15869 )"
...
This reverts commit 0fe58dbb34
2022-10-14 20:37:50 -07:00
Anis Elleuch
0fe58dbb34
convert context.DeadlineExceed to offline disk in DiskInfo() ( #15869 )
2022-10-14 19:32:13 -07:00
Harshavardhana
6cb2f56395
Revert "Revert "tests: Add context cancelation ( #15374 )""
...
This reverts commit 564a0afae1
2022-10-14 03:08:40 -07:00
Harshavardhana
59e33b3b21
validate setBucketTarget properly as per BucketExists() call ( #15860 )
2022-10-13 17:46:49 -07:00
Poorna
0e3c92c027
attempt delete marker replication after object is replicated ( #15857 )
...
Ensure delete marker replication success, especially since the
recent optimizations to heal on HEAD, LIST and GET can force
replication attempts on delete marker before underlying object
version could have synced.
2022-10-13 17:45:23 -07:00
Anis Elleuch
db7a9b2c37
heal-info: Return the endpoint of a disk with unknown state ( #15854 )
2022-10-13 16:41:44 -07:00
Harshavardhana
44097faec1
support deleteMarkers and all versions in batch replication ( #15858 )
2022-10-13 14:42:10 -07:00
Klaus Post
bf3da5081f
Omit empty checksums in responses ( #15850 )
2022-10-13 00:49:46 -07:00
Harshavardhana
5532982857
do not disable IsKubernetes(), IsDocker() checks with MINIO_CI_CD ( #15852 )
2022-10-12 23:40:48 -07:00
Anis Elleuch
783dd875f7
refactor objectQuorumFromMeta() to search for parity quorum ( #15844 )
2022-10-12 16:42:45 -07:00
Harshavardhana
97112c69be
fix: replication stats() to not crash under any situation ( #15851 )
...
Co-authored-by: Daniel Valdivia <18384552+dvaldivia@users.noreply.github.com>
2022-10-12 15:47:41 -07:00
Javier Adriel
2939000342
Add metrics, version and apis handlers ( #15839 )
2022-10-12 12:08:03 -07:00
Harshavardhana
41e1654f9a
remove spurious logging for object not found ( #15842 )
2022-10-12 04:28:21 -07:00
Harshavardhana
e3cb0278ce
honor specified target prefix under batch replication ( #15834 )
2022-10-11 14:36:06 -07:00
Harshavardhana
0c81f1bdb3
indicate how long it took to bring the drive online ( #15835 )
2022-10-11 11:33:56 -07:00
Klaus Post
6220875803
Add missing server info fields ( #15826 )
2022-10-11 11:31:26 -07:00
Aditya Manthramurthy
64cf887b28
use LDAP config from minio/pkg to share with console ( #15810 )
2022-10-07 22:12:36 -07:00
Harshavardhana
927a879052
authenticate the request first for headObject() ( #15820 )
2022-10-07 21:45:53 -07:00
Anis Elleuch
dfe0c96b87
preserve Version and DeleteMarker sort order in the list XML response ( #15819 )
2022-10-07 16:12:36 -07:00
Anis Elleuch
e856e10ac2
ignore VersionNotFound in addition to ObjectNotFound while replicating ( #15814 )
2022-10-07 16:11:41 -07:00
Harshavardhana
928feb0889
remove unused debug param from evalActionFromLifecycle ( #15813 )
2022-10-07 10:24:12 -07:00
Anis Elleuch
158d0e26a2
decom: Ignore object/version error during deletion ( #15806 )
2022-10-06 09:41:58 -07:00
Harshavardhana
78385bfbeb
set bucket creation timestamp properly for legacy FS backend ( #15800 )
2022-10-06 02:46:31 -07:00
Harshavardhana
2a13cc28f2
feat: implement support batch replication ( #15554 )
2022-10-05 23:00:43 -07:00
Lenin Alevski
4bdf41a6c7
Removing unused getUpdateReaderFromFile function ( #15794 )
...
Signed-off-by: Lenin Alevski <alevsk.8772@gmail.com>
2022-10-05 07:58:27 -07:00
Klaus Post
3c605c93fe
warn when 0 parity has been set as default parity ( #15790 )
2022-10-04 22:41:42 -07:00
Anis Elleuch
121f18a443
Use admin request check for ReplicationDiff handler ( #15793 )
2022-10-04 17:47:31 -07:00
Harshavardhana
538aeef27a
fix: heal service accounts for LDAP users in site replication ( #15785 )
2022-10-04 10:41:47 -07:00
Poorna
be0d2537b7
site replication: fix typo in meta collection ( #15792 )
2022-10-04 10:19:17 -07:00
Javier Adriel
3307aa1260
Implement KMS handlers ( #15737 )
2022-10-04 10:05:09 -07:00
Harshavardhana
57cfdfd8fb
remove 'perf' tests from health diagnostics ( #15780 )
2022-10-03 00:18:41 -07:00
Harshavardhana
f696a221af
allow tagging policy condition for GetObject ( #15777 )
2022-10-02 12:29:29 -07:00
Harshavardhana
2aac50571d
fix: de-duplicate conflicting object names on namespace ( #15772 )
2022-09-30 15:44:21 -07:00
Shireesh Anjal
45edd27ad7
Re-load config after 'mc admin config reset' ( #15771 )
2022-09-30 10:55:53 -07:00
Daryl White
d44f3526dc
Update links to documentation site ( #15750 )
2022-09-28 21:28:45 -07:00
Harshavardhana
41b633f5ea
support tagging based policy conditions ( #15763 )
2022-09-28 11:25:46 -07:00
Anis Elleuch
86bb48792c
non-blocking initialization of bucket target notifications ( #15571 )
2022-09-27 17:23:28 -07:00
Harshavardhana
94dbb4a427
fix: generalize SC config and also skip healing sub-sys under SD ( #15757 )
2022-09-26 09:04:54 -07:00
Anis Elleuch
048a46ec2a
Add RPC tcp timeout/errs and AVG duration to prometheus ( #15747 )
2022-09-26 09:04:26 -07:00
Poorna
8ea6fb368d
Add auto configuration of replication workers ( #15636 )
2022-09-24 16:20:28 -07:00
Harshavardhana
b04c0697e1
validate correct ETag for the parts sent during CompleteMultipart ( #15751 )
2022-09-23 21:17:08 -07:00
Harshavardhana
50a8ba6a6f
fix: parse and save retainUntilDate in correct time format ( #15741 )
2022-09-23 08:49:27 -07:00
Anis Elleuch
20c89ebbb3
freeze before exit when _MINIO_DEBUG_NO_EXIT is defined ( #15709 )
...
this is to ensure keep k8s pods running, when they reach a "crashloop" stage
2022-09-22 11:57:27 -07:00
Krishnan Parthasarathi
6f56ba80b3
lifecycle: Assign unique id to rules with empty id ( #15731 )
2022-09-22 10:51:54 -07:00
Anis Elleuch
6e84283c66
fix: ignoring O_DIRECT in case of erasure single disk ( #15734 )
...
fixes #15733
fixes #15735
2022-09-22 10:41:06 -07:00
Harshavardhana
9d6fddcfdf
persist the non-default creds in config ( #15711 )
2022-09-21 16:14:47 -07:00
jiuker
749ce107ee
fix: context leak with replication endpoint hearbeat ( #15721 )
2022-09-21 03:08:45 -07:00
Poorna
aec2aa3497
site replication: clear config if remove --all specified ( #15716 )
2022-09-20 14:32:23 -07:00
Klaus Post
ff12080ff5
Remove deprecated io/ioutil ( #15707 )
2022-09-19 11:05:16 -07:00
Minio Trusted
d89f6af6c4
avoid replication stats crash in Prometheus
2022-09-16 17:09:45 -07:00
Harshavardhana
2c68a19dfd
upgrade all deps and update CREDITS ( #15650 )
2022-09-16 01:59:45 -07:00
Harshavardhana
9e5853ecc0
optimize double reads by reusing results from checkUploadIDExists() ( #15692 )
...
Move to using `xl.meta` data structure to keep temporary partInfo,
this allows for a future change where we move to different parts to
different drives.
2022-09-15 12:43:49 -07:00
Harshavardhana
124544d834
add pre-conditions support for PUT calls during replication ( #15674 )
...
PUT shall only proceed if pre-conditions are met, the new
code uses
- x-minio-source-mtime
- x-minio-source-etag
to verify if the object indeed needs to be replicated
or not, allowing us to avoid StatObject() call.
2022-09-14 18:44:04 -07:00
Poorna
b910904fa6
change replication stats save path for windows ( #15690 )
2022-09-14 13:49:13 -07:00
Klaus Post
eee1ce305c
When listing, do not count delete markers ( #15689 )
...
When limiting listing do not count delete, since they may be discarded.
Extend limit, since we may be discarding the forward-to marker.
Fix directories always being sent to resolve, since they didn't return as match.
2022-09-14 12:11:27 -07:00
Klaus Post
5c61c3ccdc
Fix flaky TestGetObjectWithOutdatedDisks ( #15687 )
...
On occasion this test fails:
```
2022-09-12T17:22:44.6562737Z === RUN TestGetObjectWithOutdatedDisks
2022-09-12T17:22:44.6563751Z erasure-object_test.go:1214: Test 2: Expected data to have md5sum = `c946b71bb69c07daf25470742c967e7c`, found `7d16d23f07072af1a809707ba101ae07`
2
```
Theory: Both objects are written with the same timestamp due to lower timer resolution on Windows. This results in secondary resolution, which is deterministic, but random.
Solution: Instead of hacking in a wait we request the specific version we want. Should still keep the test relevant.
Bonus: Remote action dependency for vulncheck
2022-09-14 08:17:39 -07:00
Poorna
a0fb0c1835
panic if replication config could not be read from disk ( #15685 )
...
If replication config could not be read from bucket metadata for some
reason, issue a panic so that unexpected replication outcomes can
be avoided for replicated buckets.
For similar reasons, adding a panic while fetching object-lock config
if it failed for reason other than non-existence of config.
2022-09-13 21:23:33 -07:00
Aditya Manthramurthy
e152b2a975
Pass groups claim into condition values ( #15679 )
...
This allows using `jwt:groups` as a multi-valued condition key in policies.
2022-09-13 09:45:36 -07:00
Poorna
6b9fd256e1
Persist in-memory replication stats to disk ( #15594 )
...
to avoid relying on scanner-calculated replication metrics.
This will improve the accuracy of the replication stats reported.
This PR also adds on to #15556 by handing replication
traffic that could not be queued by available workers to the
MRF queue so that entries in `PENDING` status are healed faster.
2022-09-12 12:40:02 -07:00
Klaus Post
ff9a74b91f
Add fast max-keys=1 support for Listing ( #15670 )
...
Add a listing option to stop when the limit is reached.
This can be used by stateless listings for fast results.
2022-09-09 08:13:06 -07:00
Harshavardhana
b579163802
limit number of buckets to 500k ( #15668 )
...
500k is a reasonable limit for any single MinIO
cluster deployment, in future we may increase this
value.
However for now we are going to keep this limit.
2022-09-09 03:06:34 -07:00
Krishnan Parthasarathi
96bfa77856
serialize updates to healing tracker ( #15647 )
...
When healing is parallelized by setting the ` _MINIO_HEAL_WORKERS`
environment variable, multiple goroutines may race while updating the disk's
healing tracker. This change serializes only these concurrent updates using a
channel. Note, the healing tracker is still not concurrency safe in other contexts.
2022-09-07 08:47:21 -07:00
Harshavardhana
8e997eba4a
fix: trigger Heal when xl.meta needs healing during PUT ( #15661 )
...
This PR is a continuation of the previous change instead
of returning an error, instead trigger a spot heal on the
'xl.meta' and return only after the healing is complete.
This allows for future GETs on the same resource to be
consistent for any version of the object.
2022-09-07 07:25:39 -07:00
Harshavardhana
228c6686f8
allow non-standards fallback for all http.TimeFormats ( #15662 )
...
fixes #15645
2022-09-07 07:24:54 -07:00
Harshavardhana
7776d064cf
allow non-standards fallback for Expires header ( #15655 )
...
fixes #15645
2022-09-05 19:18:18 -07:00
Harshavardhana
2d9b5a65f1
verify RenameData() versions to be consistent ( #15649 )
...
xl.meta gets written and never rolled back, however
we definitely need to validate the state that is
persisted on the disk, if there are inconsistencies
- more than write quorum we should return an error
to the client
- if write quorum was achieved however there are
inconsistent xl.meta's we should simply trigger
an MRF on them
2022-09-05 16:51:37 -07:00
Shireesh Anjal
c240da6568
Reuse madmin.ClusterRegistrationInfo ( #15654 )
...
The `clusterInfo` struct in admin-handlers is same as
madmin.ClusterRegistrationInfo, except for small differences in field
names.
Removing this and using madmin.ClusterRegistrationInfo in its place will
help in following ways:
- The JSON payload generated by mc in case of cluster registration will
be consistent (same keys) with cluster.info generated by minio as part
of the profile and inspect zip
- health-analyzer can parse the cluster.info using the same struct and
won't have to define it's own
2022-09-05 10:02:25 -07:00
Harshavardhana
157272dc5b
fix: use optimized json.NewEncoder instead for metrics ( #15648 )
2022-09-05 08:06:35 -07:00
yudoutingle
f4c56026a2
fix: potential deadLock caused by unlocking a non-existing lock ( #15635 )
2022-09-02 14:24:32 -07:00
Harshavardhana
37e3f5de10
do not print object not found errors in MRF healing ( #15646 )
2022-09-02 14:22:40 -07:00
Harshavardhana
5ea629beb2
avoid printing io.ErrUnexpectedEOF for .metacache objects ( #15642 )
2022-09-02 12:47:17 -07:00
Anis Elleuch
cf52691959
Save resync status in the backend using a last update timestamp ( #15638 )
...
Currently, there is a short time window where the code is allowed
to save the status of a replication resync. Currently, the window is
`now.Sub(st.EndTime) <= resyncTimeInterval`. Also, any failure to
write in the backend disks is not retried.
Refactor the code a little bit to rely on the last timestamp of a
successful write of the resync status of any given bucket in the
backend disks.
2022-09-01 16:53:36 -07:00
Anis Elleuch
10e75116ef
Avoid replicating dirs in listing with replication enabled ( #15641 )
...
When replication is enabled in a particular bucket, the listing will send
objects to bucket replication, but it is also sending prefixes for non
recursive listing which is useless and shows a lot of error logs.
This commit will ignore prefixes.
2022-09-01 15:22:11 -07:00
Harshavardhana
f649968c69
tier: avoid stats infinite loop in forwardTo method ( #15640 )
...
under some sequence of events following code would
reach an infinite loop.
```
idx1, idx2 := 0, 1
for ; idx2 != idx1; idx2++ {
fmt.Println(idx2)
}
```
fixes #15639
2022-09-01 13:51:06 -07:00
Harshavardhana
bcedc2b0d9
fix: add healing metric type for heal tracing ( #15631 )
...
changes the `heal.checkBucket` to `heal.Bucket` instead
since the latter is more meaningful.
2022-08-31 12:28:03 -07:00
Klaus Post
8e4a45ec41
fix: encrypt checksums in metadata ( #15620 )
2022-08-31 08:13:23 -07:00
Klaus Post
dec942beb6
feat: Add healing trace ( #15616 )
2022-08-31 01:56:12 -07:00
Abirdcfly
d4e0f13bb3
chore: remove duplicate word in comments ( #15607 )
...
Signed-off-by: Abirdcfly <fp544037857@gmail.com>
Signed-off-by: Abirdcfly <fp544037857@gmail.com>
2022-08-30 08:26:43 -07:00
Anis Elleuch
1f28a3bb80
Avoid messages from go test output ( #15601 )
...
A lot of warning messages are printed in CI/CD failures generated by go
test. Avoid that by requiring at least Error level for logging when
doing go test.
2022-08-30 08:23:40 -07:00
Krishnan Parthasarathi
3a1d3a7952
audit-log: Add time to get/restore object from remote-tier ( #15602 )
2022-08-29 21:33:59 -07:00
Klaus Post
a9f1ad7924
Add extended checksum support ( #15433 )
2022-08-29 16:57:16 -07:00
Poorna
929b9e164e
site replication: Avoid returning root svcacct info in sr metadata ( #15608 )
...
Service accounts of root users should not be replicated.
2022-08-29 11:19:51 -07:00
Harshavardhana
97376f6e8f
improve performance for inlined data ( #15603 )
...
inlined data often is bigger than the allowed
O_DIRECT alignment, so potentially we can write
'xl.meta' without O_DSYNC instead we can rely on
O_DIRECT + fdatasync() instead.
This PR allows O_DIRECT on inlined data that
would gain the benefits of performing O_DIRECT,
eventually performing an fdatasync() at the end.
Performance boost can be observed here for small
objects < 128KiB. The performance boost is mainly
seen on HDD, and marginal on NVMe setups.
2022-08-29 11:19:29 -07:00
Febriananda Wida Pramudita
1f22a16b15
fix: endpoints for single local disks must retain port info ( #15585 )
2022-08-26 12:53:15 -07:00
Harshavardhana
433b6fa8fe
upgrade golang-lint to the latest ( #15600 )
2022-08-26 12:52:29 -07:00
Krishnan Parthasarathi
99fbfe2421
Add concurrency to healing objects on a fresh disk ( #15575 )
2022-08-25 13:07:15 -07:00
Poorna
b1b6264bea
fix: validate deployment id when adding peer clusters ( #15591 )
...
Fixes : #15573
2022-08-25 11:30:52 -07:00
Aditya Manthramurthy
18dffb26e7
Allow querying a single target in config get API ( #15587 )
2022-08-25 00:17:05 -07:00
Harshavardhana
edba7c987b
fix: objects matching prefixes should not leave delete markers ( #15586 )
...
This is needed to ensure that we do not leave prefixes where
version is suspended, instead we never leave versions on
these paths.
2022-08-24 13:46:29 -07:00
Anis Elleuch
b737c83a66
Ensure that only one node performs site replication healing ( #15584 )
...
When a node finds a change in the other replication cluster and applies
to itself will already notify other peers. No need for all nodes in a
given cluster to do site replication healing, only one node is
sufficient.
2022-08-24 13:46:09 -07:00
Anis Elleuch
97a6322de1
Fix regression in notifying peers about new policy mapping ( #15583 )
...
Switch from mux.Vars() to r.Form to avoid the issue of missing arguments
passed to LoadPolicyMappingHandler.
2022-08-24 12:34:52 -07:00
Klaus Post
037fe4afdc
Add listing block reuse ( #15579 )
...
When streaming results, pool metadata slices when sent.
2022-08-24 09:11:16 -07:00
Aditya Manthramurthy
afbb63a197
Factor out external event notification funcs ( #15574 )
...
This change moves external event notification functionality into
`event-notification.go`. This simplifies notification related code.
2022-08-24 06:42:36 -07:00
Harshavardhana
8902561f3c
use new xxml for XML responses to support rare control characters ( #15511 )
...
use new xxml/XML responses to support rare control characters
fixes #15023
2022-08-23 17:04:11 -07:00
