MyFavMirrors
/
minio
mirror of https://github.com/minio/minio.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
11,450 Commits 5 Branches 503 Tags 186 MiB
Commit Graph

5752 Commits

Author SHA1 Message Date
Harshavardhana f7d29b4a53
cleanup of multipart per disk must cleanup itself only (#17223) 2023-05-17 01:45:58 -07:00
Harshavardhana 06557fe8be
allow decommissioned pools to be removed while others are finishing (#17221) 2023-05-16 16:00:57 -07:00
Poorna 2131046427
replication: fix audit log reporting (#17222) 2023-05-16 15:35:08 -07:00
Klaus Post aaf1abc993
simplify HardLimitReader by using LimitReader for internal usage (#17218) 2023-05-16 13:14:37 -07:00
jiuker 413549bcf5
fix: loadStatsFromDisk() should return nil for configNotFound (#17217) 2023-05-16 12:23:38 -07:00
jiuker 9a799065b3
fix: make slice cap of right size (#17192) 2023-05-16 08:10:07 -07:00
jiuker fd2959fa3a
fix: workers.New err must be returned (#17208) 2023-05-16 08:08:00 -07:00
jiuker 15bec32bb4
fix: tier handlers must write error only once (#17205) 2023-05-15 23:56:52 -07:00
Anis Eleuch e2b7a08c10
heal: Update object parity with the latest configured SC (#17187) 2023-05-15 21:32:13 -07:00
Harshavardhana ef2fc0f99e
fix: reduce using memory and temporary files. (#17206) 2023-05-15 14:08:54 -07:00
Harshavardhana d063596430
fix: veeam SOS API 'system.xml' strings (#17202) 2023-05-15 12:06:42 -07:00
jiuker bd2dc6c670
fix: in healing tracker printTo when err (#17207) 2023-05-15 10:14:48 -07:00
Harshavardhana b62791617c
fix: notify systemd as soon as we wait on the OS signal (#17199) 2023-05-12 16:42:17 -07:00
Poorna e07c2ab868
Use hash.NewLimitReader for internal multipart calls (#17191) 2023-05-12 11:19:08 -07:00
jiuker 203755793c
fix: in printEndpointError count error once per init() (#17193) 2023-05-12 10:41:54 -07:00
Anis Eleuch 883c98e26f
fix: remove objects when there are skipped versions due to ILM in decom (#17198) 2023-05-12 10:37:38 -07:00
Harshavardhana f5a20a5d06
allow nodes offline in k8s setups when expanding pools (#17183) 2023-05-11 17:41:33 -07:00
Poorna ef7177ebbd
disallow bucket replication setup with site replication (#17189) 2023-05-11 15:48:40 -07:00
Harshavardhana 3637aad36e
do not count ILM expired objects and other skipped objects (#17184) 2023-05-11 13:35:16 -07:00
Aditya Manthramurthy 77db9686fb
Update console to v0.27.0 (#17188) 2023-05-11 12:18:17 -07:00
Shireesh Anjal c326e5a34e
Add metrics for webhook endpoint stats (#17179) 2023-05-11 11:24:37 -07:00
jiuker c23c982593
xmlDecoder err use ErrMalformedXML when PutBucketACLHandler (#17185) 2023-05-11 11:11:15 -07:00
Shireesh Anjal a3d666356c
fix: error in capturing XFS error config in health report (#17176) 2023-05-10 15:20:48 -07:00
jiuker 3cdbc2f414
add validationErr to validateConfig When DeleteIdentityProviderCfg (#17173) 2023-05-10 09:37:30 -07:00
Harshavardhana b92cdea578
fix: start using pkg/workers to spawn parallel workers (#17170) 2023-05-09 16:37:31 -07:00
jiuker 5e629a99af
fix: for profiling duration parsing error reply use ErrInvalidRequest (#17169) 2023-05-09 14:27:49 -07:00
Harshavardhana a7f266c907
allow JWT parsing on large session policy based tokens (#17167) 2023-05-09 00:53:08 -07:00
Praveen raj Mani 57acacd5a7
Support persistent queue store for loggers (#17121) 2023-05-08 21:20:31 -07:00
jiuker 629503ff73
add Err to BucketExists when NoSuchBucket (#17155) 2023-05-08 07:51:59 -07:00
jiuker e3a070e3de
put *msgp.Reader back to pool (#17156) 2023-05-08 07:51:39 -07:00
Poorna c5c1426262
Validate if replication config being added is self referential (#17142) 2023-05-06 13:35:43 -07:00
Klaus Post 76913a9fd5
Signed trailers for signature v4 (#16484) 2023-05-05 19:53:12 -07:00
Harshavardhana 5569acd95c
disallow EC:0 if not set during server startup (#17141) 2023-05-04 14:44:30 -07:00
Harshavardhana 1d0211d395
allow deletes on directory objects to perform permanent deletes (#17132) 2023-05-04 14:43:52 -07:00
Anis Eleuch 06cd0a636e
Avoid calling KES Status when peers ping each other (#17140) 2023-05-04 11:28:33 -07:00
Klaus Post 7f7b489a3d
snowball: use latest time when mtime is missing (#17133) 2023-05-04 07:29:33 -07:00
Klaus Post bb6f4d7633
Remove redundant checkFormatJSON logging (#17134) 2023-05-04 07:28:37 -07:00
Alex 6e24dff26a
Added MINIO_BROWSER_LOGIN_ANIMATION env support for WebUI console (#17123) 
Signed-off-by: Benjamin Perez <benjamin@bexsoft.net>
 2023-05-03 15:32:50 -07:00
Harshavardhana e372e4e592
add nodeName to the log while taking drive offline (#17124) 2023-05-03 15:05:45 -07:00
Harshavardhana 9571b0825e
add configurable VRF interface and user-timeout (#17108) 2023-05-03 14:12:25 -07:00
Poorna 90e2cc3d4c
Add audit logging of site replication multipart proxying (#17122) 2023-05-03 11:19:45 -07:00
Harshavardhana 9112ca4e29
change ttfb_distribution metrics to histogramMetric (#17115) 2023-05-03 07:31:00 -07:00
Poorna ec84bad882
batch replication now supports arbitrary S3 targets (#17113) 2023-05-02 22:52:35 -07:00
Harshavardhana b53376a3a4
change directory objects to never create new versions (#17109) 2023-05-02 16:09:33 -07:00
Krishnan Parthasarathi 0ec722bc54
Add tags to NewerNoncurrentVersions audit event (#17110) 2023-05-02 12:56:33 -07:00
Praveen raj Mani 1704abaf6b
fix: store notification events immediately for persistent queues (#17112) 2023-05-02 07:53:13 -07:00
WGH ab34f0065c
Support systemd notify protocol (#17062) 2023-05-01 23:15:08 -07:00
Klaus Post e8c0a50862
optimization use small blocks up to 64KB (#17107) 2023-05-01 09:47:49 -07:00
Harshavardhana 02d8f3cdc8
fix: remove active healing on .minio.sys/ during startup (#17072) 2023-04-29 02:05:28 -07:00
Harshavardhana 7ae69accc0
allow root user to be disabled via config settings (#17089) 2023-04-28 12:24:14 -07:00
Klaus Post 7fad0c8b41
Remove checksums from HTTP range request, add part checksums (#17105) 2023-04-28 08:26:32 -07:00
Anis Eleuch d5aff735be
info: Add drives per set and sets count per pool information (#17100) 2023-04-27 15:24:03 -07:00
Poorna 98c26df53e
fix: allow past retention headers to be copied in batch replication (#17095) 2023-04-27 13:43:18 -07:00
Anis Eleuch 2448a9e047
grafana: Remove minio_s3_requests_errors_total metric (#17094) 2023-04-27 10:55:30 -07:00
Krishnan Parthasarathi e7cac8acef
Add tags to auditLogLifecycle (#17081) 2023-04-26 17:49:00 -07:00
Harshavardhana 6105997299
remove unnecessary log when listing resume fails (#17086) 2023-04-26 14:53:25 -07:00
Harshavardhana 8c874884fc
fix: do not copy context in DiskInfo cache (#17085) 2023-04-26 12:13:54 -07:00
Aditya Manthramurthy ebfe81e5fd
Fix put bucket policy error code (#17084) 2023-04-26 11:21:27 -07:00
Praveen raj Mani 72802a5972
Use 'minio/pkg/sync/errgroup' and 'minio/pkg/workers' (#17069) 2023-04-25 22:57:40 -07:00
Harshavardhana b1f3935c5b
allow ListObjects() when a prefix is an object (#17074) 2023-04-25 22:41:54 -07:00
Harshavardhana dbd53af369
fix: initialize reverse proxy forwarder with right public certs (#17080) 2023-04-25 15:50:32 -07:00
Harshavardhana b09fe0e50e
fix: DeleteBucket for peers() must recreate bucket upon errors (#17079) 2023-04-25 14:16:35 -07:00
Krishnan Parthasarathi fae9000304
heal: Pick maximally occuring modTime in quorum (#17071) 2023-04-25 10:13:57 -07:00
Harshavardhana 8fd07bcd51
simplify sort.Sort by using sort.Slice (#17066) 2023-04-24 13:28:18 -07:00
Anis Eleuch 6addc7a35d
server-info: Return initializing state properly (#17070) 2023-04-24 09:10:02 -07:00
Harshavardhana 477230c82e
avoid attempting to migrate old configs (#17004) 2023-04-21 13:56:08 -07:00
Harshavardhana d1737199ed
fix: delete DNS upon success, update failure message (#17059) 2023-04-21 12:12:31 -07:00
Harshavardhana 84f31ed45d
simplify MRF, converge it to regular healing (#17026) 2023-04-19 07:47:42 -07:00
jiuker 8a81e317d6
verify maxPartID in object options helpers (#17015) 2023-04-18 22:34:30 -07:00
Anis Eleuch 224d9a752f
fix: the race in healing tracker code (#17048) 2023-04-18 14:49:56 -07:00
Anis Eleuch 0db34e4b85
Listen bucket events to send empty events with new line (#17037) 2023-04-18 08:11:30 -07:00
Klaus Post f66625be67
Snowball: Extract headers for metadata (#17042) 2023-04-17 12:16:54 -07:00
Harshavardhana 6825bd7e75
fix: inlined objects don't need to honor long locks (#17039) 2023-04-17 12:16:37 -07:00
Klaus Post 839b9c9271
Reduce allocations in Walkdir (#17036) 2023-04-15 10:25:25 -07:00
Harshavardhana dd9ed85e22
implement support for FTP/SFTP server (#16952) 2023-04-15 07:34:02 -07:00
jiuker 6c1410f7f5
fix: Type of rejection for FIFO quota input (#17016) 2023-04-15 01:22:18 -07:00
Krishnan Parthasarathi f92450d8b3
commonParity should pick readable FileInfo (#17032) 2023-04-14 16:23:28 -07:00
Klaus Post c133979b8e
Add part count to checksum (#17035) 2023-04-14 09:44:45 -07:00
Poorna a9269cee29
heal: avoid logging version not found (#17031) 2023-04-13 19:45:52 -07:00
Klaus Post 958a480e53
fix: lambda function expiration when cred.Expiration is set (#17029) 2023-04-13 08:10:57 -07:00
Anis Eleuch a42650c065
Add minio_bucket_usage_version_total metric to Prometheus (#17023) 2023-04-12 20:08:07 -07:00
Harshavardhana bdad3730f7
fix: do not error out if the local bucket is missing (#17025) 2023-04-12 15:44:16 -07:00
Harshavardhana a5835cecbf
fix: regression in counting total requests (#17024) 2023-04-12 14:37:19 -07:00
Poorna cd6dec49c0
Add trace support for ilm activity (#16993) 2023-04-11 19:22:32 -07:00
Krishnan Parthasarathi 6877578bbc
Update minio_node_bucket_scans_finished metrics (#17006) 2023-04-11 19:21:34 -07:00
Harshavardhana f3682b6149
allow writes to pools with inconsistent xl.meta (#17008) 2023-04-11 11:17:46 -07:00
ferhat elmas 056ca0c68e
refactor: reuse open file in storage interface (#16970) 2023-04-09 23:09:28 -07:00
Krishnan Parthasarathi 25f7a8e406
Indicate RenameData is called by healObject (#16997) 2023-04-09 10:25:37 -07:00
Anis Eleuch 1f1c267b6c
Add used inodes to disk info (#16994) 2023-04-07 07:52:14 -07:00
Anis Eleuch fc94ea1ced
trace: Fix <unknown> func name of requests rejected by max clients (#16977) 2023-04-07 07:51:12 -07:00
jiuker 67d2cf8f30
add Err to Get getRemoteTargetClient. (#16982) 2023-04-07 07:50:46 -07:00
Harshavardhana 09a25ea7b7 lint: fix some lint issues on files 2023-04-06 22:42:10 -07:00
Klaus Post 260a63ca73
os/windows: do not return errFileNotFound in readDir's (#16988) 2023-04-06 22:28:58 -07:00
Shubhendu d3f70ea340
Enable audit log for global handlers (#16964) 
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
 2023-04-06 21:03:39 -07:00
Anis Eleuch 91b6fe1af3
trace: Bootstrap to show the correct source line number (#16989) 2023-04-06 17:51:53 -07:00
Klaus Post 9803f68522
snowball: Restrict zstd window size (#16987) 2023-04-06 17:47:38 -07:00
Shubhendu 4c204707fd
Correct to remove `null` version while ILM rule application (#16971) 
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
Co-authored-by: Harshavardhana <harsha@minio.io>
 2023-04-06 14:10:01 -07:00
Harshavardhana c06e0bfef9
set correct `Host:` value for replication event notification (#16984) 2023-04-06 10:20:53 -07:00
Klaus Post 8625a9dbb3
Make listing metadata permissions stricter (#16974) 2023-04-06 07:52:35 -07:00
jiuker 0320ac43cb
simplify bucketInfo return in GetBucketInfo peer call (#16983) 2023-04-06 01:30:50 -07:00
Anis Eleuch 111c7d4026
assumeRole return the correct http code for auth errors (#16967) 2023-04-05 22:19:31 -07:00
Klaus Post 62c3df0ca3
fix: directory listing on Go 1.20 windows (#16976) 2023-04-05 14:36:49 -07:00
Poorna 3158f2d12e
Add support for batch key rotation (#16844) 2023-04-04 10:56:54 -07:00
jiuker 6e359c586e
fix: close chan before return in scanner usage updates (#16960) 2023-04-04 10:51:05 -07:00
Poorna 699a24f7e5
batch: validate versioning on src/tgt buckets (#16955) 2023-04-04 10:50:11 -07:00
jiuker 5fa3665074
add isSysErrNotDir to openFileNoSync (#16958) 2023-04-04 08:00:08 -07:00
Harshavardhana 3b7781835e
add lock metrics per node (#16943) 2023-04-03 21:23:24 -07:00
Shubhendu 5fe1b46bfd
Enabled to send audit log while version deletion (#16954) 
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
 2023-04-03 11:58:04 -07:00
Poorna 407c9ddcbf
feat: add batch replicate from remote MinIO to local cluster (#16922) 2023-03-31 10:48:36 -07:00
Anis Eleuch d90d0c8931
Use one http response recorder per external http call (#16938) 2023-03-31 09:37:29 -07:00
Harshavardhana 216a471bbb
on quorum DeleteObject() errors attempt an MRF (#16932) 2023-03-31 08:15:41 -07:00
jiuker a7b7860e0e
fix: potential data conflicts save site-resync metadata (#16926) 2023-03-30 20:59:45 -07:00
Klaus Post d703daa480
Add metadata extension to ListObjectVersions (#16930) 2023-03-30 12:20:42 -07:00
Poorna dc8fdcb9c9
fix: error checking in DeleteBucket (#16929) 2023-03-30 11:54:08 -07:00
Harshavardhana 7a6c4e438e
allow more workers for ILM expiration (#16924) 2023-03-30 10:47:15 -07:00
jiuker c468b4e2a8
fix: avoid out of slice index (#16925) 2023-03-30 08:03:48 -07:00
jiuker b04956a676
fix: put *msgp.Reader back in pool (#16927) 2023-03-30 08:03:12 -07:00
Aditya Manthramurthy 518f6e4d39
fix: missing return after error response (#16920) 2023-03-29 16:21:13 -07:00
Allan Roger Reid 483b226cc1
fix: avoid logging when object/version not found in replication (#16919) 2023-03-29 15:02:45 -07:00
Harshavardhana 8e02660a0d
update all our deps (#16899) 2023-03-28 03:45:24 -07:00
jiuker 16feef2a2c
fix: time.Parse RFC3339Nano (#16892) 2023-03-27 11:51:54 -07:00
jiuker 66ff17e452
fix: Avoid multiple write responses (#16894) 2023-03-27 09:15:23 -07:00
Harshavardhana 4c5edacae2
ignore operation timedout errors (#16891) 2023-03-26 03:16:51 -07:00
Anis Eleuch 8b4d0255b7
Set Console global Root CAs early to trust provided certs (#16890) 2023-03-25 09:58:38 -07:00
Poorna 74040b457b
Allow setting sync mode for site replication (#16876) 2023-03-24 14:41:23 -07:00
Klaus Post 5e3bfd2148
Add user tags when listing with metadata (#16883) 2023-03-23 10:27:19 -07:00
Klaus Post 8b0ab6ead6
Revert "Make localLocker lock attempts cancellable (#16510)" (#16884) 2023-03-23 10:26:21 -07:00
Shubhendu b1b0aadabf
Revert query parameter `src` from diag upload if callhome enabled (#16881) 2023-03-23 00:24:58 -07:00
Harshavardhana ac7d9c449a
add missing expiration information from 'sts info' (#16878) 2023-03-22 16:47:02 -07:00
Anis Eleuch 1346561b9d
return quorum error instead of insufficient storage error (#16874) 2023-03-22 16:22:37 -07:00
Minio Trusted 4bc52897b2 Update yaml files to latest version RELEASE.2023-03-22T06-36-24Z 2023-03-22 21:16:15 +00:00
Krishnan Parthasarathi 6017b63a06
top-locks: Group by lock request ID (#16860) 2023-03-21 18:35:29 -07:00
Harshavardhana 0448728228
fix: add deadline conns and dnsCache for remote transports (#16865) 2023-03-21 08:49:20 -07:00
Harshavardhana 12047702f5
fix: tweak the maintenance=true to satisfy baremetal first (#16864) 2023-03-21 08:48:38 -07:00
Harshavardhana fb1492f531
check for quorum errors for DeleteBucket() (#16859) 2023-03-20 23:38:06 -07:00
Aditya Manthramurthy 05444a0f6a
Use the official pub key to always verify binary (#16857) 2023-03-20 13:16:18 -07:00
Harshavardhana b3c54ec81e
reject object names with '\' on windows (#16856) 2023-03-20 13:16:00 -07:00
Harshavardhana 6c11dbffd5
add crash protection from backend modifications (#16846) 2023-03-20 09:08:42 -07:00
Harshavardhana 3b5dbf9046
allow bootstrapping to validate internode tokens (#16853) 2023-03-20 01:40:24 -07:00
Aditya Manthramurthy 09c733677a
Add test for fixed post policy exploit (#16855) 2023-03-20 01:06:45 -07:00
Harshavardhana 8d6558b236
fix: convert '\' to '/' on windows (#16852) 2023-03-20 00:35:25 -07:00
Aditya Manthramurthy 67f4ba154a
fix: post policy request security bypass (#16849) 2023-03-19 21:15:20 -07:00
Poorna 440ad20c1d
Add support for batch job cancellation (#16843) 2023-03-17 23:42:43 -07:00
Krishnan Parthasarathi 31fba6f434
Save bootstrap trace events in a circular buffer (#16823) 2023-03-17 16:01:03 -07:00
Harshavardhana 280442e533
reduce 250ms to 50ms retry looking for metacache block (#16795) 2023-03-17 14:44:01 -07:00
Shubhendu 850a945a18
Added query parameter `src` to diag upload if callhome enabled (#16837) 
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
 2023-03-17 08:30:16 -07:00
Harshavardhana 46f9049fb4
simplify error responses for KMS (#16793) 2023-03-16 11:59:42 -07:00
Aditya Manthramurthy 58266c9e2c
Add enable flag for LDAP IDP config (#16805) 2023-03-16 11:58:59 -07:00
Poorna d1e775313d
support decommissioning of tiered objects (#16751) 2023-03-16 07:48:05 -07:00
Harshavardhana e700be8cd6
fix: return appropriate Location header for MakeBucket() (#16820) 2023-03-15 13:40:40 -07:00
Harshavardhana e0f4dd6027
remove unncessary logs from WalkDir(), PutObject() (#16818) 2023-03-15 11:52:23 -07:00
Harshavardhana de02eca467
restore rotating root credentials properly (#16812) 2023-03-15 08:07:42 -07:00
Harshavardhana c7f7e67a10
Do not allow adding root user to IAM subsystem (#16803) 2023-03-13 12:46:17 -07:00
Klaus Post 628042e65e
tests: Protect globalLocalDrives against races (#16800) 2023-03-13 06:04:20 -07:00
Aditya Manthramurthy 6305b206e1
fix: site-repl should heal STS with virtual parent (#16792) 2023-03-10 16:21:51 -08:00
Klaus Post d85da9236e
Add Object Version count histogram (#16739) 2023-03-10 08:53:59 -08:00
Anis Elleuch 5c087bdcad
fix: a cosmetic error reporting with a lock timeout (#16788) 2023-03-09 15:16:13 -08:00
Klaus Post a547bf517d
Remove locks on usage cache (#16786) 2023-03-09 15:15:46 -08:00
Harshavardhana b984bf8d1a
allow expiration of all versions during Listing() (#16757) 2023-03-09 15:15:30 -08:00
Poorna fb6ab1cca2
fix: allow replication of 'null' delete markers (#16773) 2023-03-08 07:03:29 -08:00
Krishnan Parthasarathi 56c57e2c53
tier-stats: Avoid repeated logs (#16774) 2023-03-07 16:43:05 -08:00
Poorna a6057c35cc
Avoid peer notification when peer is offline, tune retries (#16737) 2023-03-07 08:13:28 -08:00
Harshavardhana 901887e6bf
feat: add lambda transformation functions target (#16507) 2023-03-07 08:12:41 -08:00
Poorna ee54643004
Avoid unnecessary replication heal attempts (#16769) 2023-03-07 07:43:38 -08:00
Harshavardhana 0a17acdb34
return error if policy changes on disabled groups (#16766) 2023-03-06 10:46:24 -08:00
Harshavardhana 72e5212842
fix: handle syscall.EROFS also for osIsPermission() (#16765) 2023-03-06 08:56:29 -08:00
ferhat elmas 714283fae2
cleanup ignored static analysis (#16767) 2023-03-06 08:56:10 -08:00
ferhat elmas 3423028713
cleanup Go linter settings (#16736) 2023-03-04 20:57:35 -08:00
jiuker 9d062b37d7
return underlying error with BackendDown{} error (#16738) 2023-03-03 23:56:53 -08:00
jiuker f350953a19
calculate disk cache usage percent accurately (#16740) 2023-03-02 20:32:22 -08:00
Daniel Valdivia 958bba5b42
Attach creds, owner and region to madmin calls (#16658) 
Signed-off-by: Daniel Valdivia <18384552+dvaldivia@users.noreply.github.com>
 2023-03-02 14:35:24 -08:00
Poorna 0f2b95b497
fix a data race in IAM loading (#16742) 2023-03-02 14:32:54 -08:00
Klaus Post d07089ceac
Fix scanner deadlock on lost global lock (#16726) 2023-02-28 21:34:45 -08:00
Harshavardhana 0ff931dc76
fix: allow CORS to work by default (#16713) 2023-02-27 10:10:45 -08:00
Praveen raj Mani 4d708cebe9
Support adding service accounts with expiration (#16430) 
Co-authored-by: Harshavardhana <harsha@minio.io>
 2023-02-27 10:10:22 -08:00
dorman 4d7c8e3bb8
Remove redundant log (#16710) 
Co-authored-by: z30001483 <zekaifeng2@huawei.com>
 2023-02-27 09:59:47 -08:00
Aditya Manthramurthy 8cde38404d
Add metrics for custom auth plugin (#16701) 2023-02-27 09:55:18 -08:00
Krishnan Parthasarathi fe7bf6cbbc
Support tier-add if tier backend not empty (#16715) 2023-02-27 09:26:26 -08:00
Shubhendu 8b4eb2304b
Set logger webhook proxy on subnet proxy change (#16665) 
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
 2023-02-27 08:35:36 -08:00
Harshavardhana ae029191a3
liveness returns "busy" if queued requests > available capacity (#16719) 2023-02-27 08:34:52 -08:00
Harshavardhana bfedea9bad
fix: disk healing should honor the right pool/set index (#16712) 2023-02-27 04:55:32 -08:00
Aditya Manthramurthy 7777d3b43a
Remove globalSTSTLSConfig (#16709) 2023-02-26 23:37:00 -08:00
Aditya Manthramurthy 9ed4fc9687
Remove globalOpenIDConfig (#16708) 2023-02-25 21:01:37 -08:00
jiuker b49b39e99d
fix: errNoSuchPolicy should use errors.Is (#16656) 2023-02-25 00:01:37 -08:00
jiuker cd3a2de5a3
add isValidLocation to common parseLocation (#16690) 2023-02-25 08:09:20 +05:30
jiuker 6e8960ccdd
fix: delete globalProfiler should lock (#16697) 2023-02-25 08:07:44 +05:30
Aditya Manthramurthy e05f3d5d84
Remove globalLDAPConfig (#16706) 2023-02-25 08:07:22 +05:30
Anis Elleuch 94c6cb1323
tests: Add test for S3 API error codes (#16705) 2023-02-25 08:06:29 +05:30
Klaus Post 9acf1024e4
Remove bloom filter (#16682) 
Removes the bloom filter since it has so limited usability, often gets saturated anyway and adds a bunch of complexity to the scanner.

Also removes a tiny bit of CPU by each write operation.
 2023-02-24 09:03:31 +05:30
Harshavardhana a0f06eac2a
add Veeam SOS API first implementation (#16688) 2023-02-22 19:54:57 +05:30
jiuker 83fe1a2732
log: add more info about BROWSER_URL (#16678) 2023-02-22 19:54:05 +05:30
Harshavardhana 5c98223c89
add correct HostId instead of deploymentId for error responses (#16686) 2023-02-22 15:41:09 +05:30
jiuker 663a0b7783
save correct bucketInfo on it's indexes (#16685) 2023-02-22 14:08:34 +05:30
Anis Elleuch 6efe4d1df6
perf: Only remove generated data when no bucket name specified (#16610) 2023-02-21 21:21:40 -08:00
Daniel Valdivia fb17f97cf3
move audit and logger message structure to minio/pkg (#16655) 
Signed-off-by: Daniel Valdivia <18384552+dvaldivia@users.noreply.github.com>
 2023-02-21 21:21:17 -08:00
Poorna 9202c6e26a
Reload bucket targets when replication config is refreshed (#16684) 2023-02-21 21:18:49 -08:00
Poorna 59a5456091
fix: STS error translation to API error (#16683) 2023-02-22 09:57:48 +05:30
Allan Roger Reid 8bfe972bab
Set meaningful message from minio with env variable KMS_SECRET_KEY (#16584) 2023-02-22 07:13:01 +05:30
Klaus Post fd6622458b
Add detailed scanner trace output and notifications (#16668) 2023-02-21 09:33:33 -08:00
Poorna 8a08861dd9
fix: healing of replication config for endpoint changes (#16648) 2023-02-20 16:06:13 +05:30
Harshavardhana b66d7dc708
add missing x-amz-id-2 to event notification date (#16646) 2023-02-20 15:41:47 +05:30
Harshavardhana b94733ab31
avoid locks when unnecessary in SiteReplicationMetaInfo() (#16650) 2023-02-18 05:35:22 -08:00
Klaus Post 84bb7d05a9
fix: healing deadlocks and ordering (#16643) 2023-02-17 23:22:43 +05:30
Harshavardhana 98a84d88e2
fix: trim and ignore './' directories (#16642) 2023-02-17 07:15:03 -08:00
jiuker e470268c7c
fix: a possible closer leak in SelectObjectHandler (#16598) 2023-02-17 01:44:40 -08:00
jiuker 3a6cd4f73d
fix: sync.pool won't Put reader back (#16638) 2023-02-17 01:42:43 -08:00
Harshavardhana 6ea150fd68
fix: avoid printing certain errors under few locations (#16631) 2023-02-17 01:40:31 -08:00
Anis Elleuch a7188bc9d0
fix: evaluate BypassGov policy action in deletion correctly (#16635) 2023-02-17 07:53:34 +05:30
Krishnan Parthasarathi a1dd08f2e6
Include tier name in MinIO/S3 target user-agent (#16630) 2023-02-15 22:09:46 -08:00
Krishnan Parthasarathi d136ac0596
Don't close transition task channel on server exit (#16627) 2023-02-15 22:09:25 -08:00
Poorna c33a237067
fix: under site replication disallow remote target modification (#16628) 2023-02-15 20:22:13 -08:00
Poorna eb7d3da994
Fix site replication status reporting of quota (#16626) 2023-02-16 08:08:35 +05:30
Harshavardhana 37134e42d4
ignore io.EOF, io.ErrUnexpectedEOF on xl.meta reads in WalkDir() (#16625) 2023-02-15 07:12:48 -08:00
Harshavardhana 0c1f8b4e0f
add user-agent for all minio.Client usage (#16619) 2023-02-14 13:19:30 -08:00
Anis Elleuch 857674c3a0
heal: Do not mark buckets as done when there is no online disks (#16621) 2023-02-14 12:50:13 -08:00
Harshavardhana 15a75bd79b
ignore preconditionFailed error in batch replication (#16615) 2023-02-14 07:22:08 -08:00
Andreas Auernhammer 74887c7372
kms: add support for KES API keys and switch to KES Go SDK (#16617) 
Signed-off-by: Andreas Auernhammer <hi@aead.dev>
 2023-02-14 07:19:20 -08:00
Harshavardhana 31188e9327
add parallel workers in batch replication (#16609) 2023-02-13 12:07:58 -08:00
Harshavardhana ee6d96eb46
Periodically remove stale buckets from in-memory (#16597) 2023-02-13 08:09:52 -08:00
Anis Elleuch d6d01067a0
fix: allow global leader lock context merge to be canceled (#16603) 
Global leader lock was first designated to only acquired once
until the node is killed. However, currently, the code acquires
it repeatedly during the lifetime of the server, now there is a
goroutine leak.
 2023-02-13 01:26:38 -08:00
jiuker a15b6f21b8
remove incorrect use of WaitGroup (#16596) 2023-02-12 20:59:45 -08:00
Anis Elleuch 689179bf18
ServerInfo: return per erasure set information (#16583) 2023-02-11 18:31:56 +05:30
Klaus Post d0f4cc89a5
Re-add Veeam Listing workaround (#16593) 2023-02-10 10:48:39 -08:00
Harshavardhana d65debb6bc
fix: comply with RFC6750 UserInfo endpoint requirements (#16592) 2023-02-10 22:20:25 +05:30
Harshavardhana 72daccd468
fix: scanner in healing cycle must use actual size (#16589) 2023-02-10 06:53:03 -08:00
Harshavardhana b363400587
fix: username replacements for aws:username must use parentUser (#16591) 2023-02-10 06:52:31 -08:00
jiuker b9f5f9ba3f
fix: aclHandlers convert XML parse error to relevant client error (#16587) 2023-02-10 02:59:10 -08:00
Anis Elleuch c8ffa59d28
Periodically refresh buckets metadata from the backend disks (#16561) 
fixes #16553
 2023-02-09 10:29:20 -08:00
Poorna 52aeebebea
Fix site replication meta info call to be non-blocking (#16526) 
Co-authored-by: Harshavardhana <harsha@minio.io>
 2023-02-08 21:16:53 -08:00
Harshavardhana 71f02adfca Revert "Print golang http errors in MinIO log format (#16465)" 
This reverts commit 1fd7946dce.
 2023-02-09 09:27:27 +05:30
Krishnan Parthasarathi 9de26531e4
tiering: UpdateWorkers may be called before Init (#16573) 2023-02-08 19:13:34 -08:00
Anis Elleuch b1d98febfd
New disk healing goes through the healing workers (#16568) 2023-02-08 09:25:29 -08:00
jiuker 1828fb212a
fix: avoid goroutine leak after timeouts in PeerMetrics (#16569) 2023-02-08 09:11:16 -08:00
jiuker be92046dfd
Anonymous pools in health info output are double counted (#16566) 2023-02-08 06:58:50 -08:00
Krishnan Parthasarathi 990fc415f7
Ensure safety of transitionState at startup (#16563) 2023-02-07 23:11:42 -08:00
Harshavardhana d8daabae9b
make site replication healing safer (#16560) 2023-02-07 21:44:42 -08:00
Harshavardhana 84fe4fd156
fix: multiObjectDelete by passing versionId for authorization (#16562) 2023-02-08 08:01:00 +05:30
Harshavardhana 747d475e76
initialize subsystems that are not dependent on buckets first (#16559) 2023-02-07 12:46:47 -08:00
Anis Elleuch 095b518802
Show a better error msg when internal data encryption key is incorrect (#16549) 2023-02-07 05:22:54 -08:00
Harshavardhana 0319ae756a
fix: pass proper username (simple) string as expected (#16555) 2023-02-07 03:43:08 -08:00
Harshavardhana 11c7ecb5cf
support if-match/if-none-match with s3 uploads (#16551) 2023-02-06 18:58:29 -08:00
Anis Elleuch ffd57fde90
Convert 'server closed idle connection' to errDiskNotFound (#16548) 2023-02-06 10:42:08 -08:00
jiuker a451d1cb8d
fix: close resp.body checking for kubernetes version (#16547) 2023-02-06 10:41:41 -08:00
Harshavardhana 14cf8f1b22
upgrade deps for minio/pkg v1.6.1 to include groups conditions (#16538) 2023-02-06 09:27:29 -08:00
Harshavardhana 5996c8c4d5
feat: allow offline disks on a fresh start (#16541) 2023-02-06 09:26:09 -08:00
Harshavardhana 21885f9457
fix: liveness/readiness must return errors if KMS is unreachable (#16540) 2023-02-06 08:55:56 -08:00
Harshavardhana aa8b9572b9
remove double ENABLED help output (#16528) 2023-02-03 05:52:52 -08:00
Poorna 876e1a91b2
replication: Fix typo checking PreconditionFailed status code (#16517) 2023-02-02 19:22:02 +05:30
Anis Elleuch 2278fc8f47
Print original error when IAM load is failed in some places (#16511) 2023-02-01 17:32:22 +05:30
Harshavardhana a91f353621
support 'mc admin service restart' for windows (#16512) 2023-02-01 17:31:46 +05:30
Klaus Post cdb1b48ad9
Make localLocker lock attempts cancellable (#16510) 2023-01-31 09:41:17 -08:00
Kaan Kabalak 2d0f30f062
Fix typo in code comment (#16509) 2023-01-31 07:54:19 +05:30
Krishnan Parthasarathi cea2ca8c8e
Add restore-status header for multipart objects (#16508) 2023-01-31 07:53:45 +05:30
Klaus Post f713436dd0
Fix truncated list response on deleted replicated objects (#16504) 2023-01-30 09:13:53 -08:00
Klaus Post b923a62425
Check pool-index for invalid setups (#16501) 2023-01-30 18:33:07 +05:30
Harshavardhana 67fce4a5b3
fix: dangling delete() upon success should return 404 (#16494) 2023-01-27 12:43:45 -08:00
Poorna eaa65b7ade
fix replication healing on list to consider all versions (#16496) 2023-01-27 12:43:28 -08:00
Poorna 820d94447c
replication: fix target bucket passed on GET proxy (#16495) 2023-01-27 10:24:51 -08:00
Poorna ed20134a7b
replication: detect proxy header presence correctly (#16489) 2023-01-27 01:29:32 -08:00
Harshavardhana d19cbc81b5
fix: do not return IAM/Bucket metadata replication errors to client (#16486) 2023-01-26 11:11:54 -08:00
Anis Elleuch 1fd7946dce
Print golang http errors in MinIO log format (#16465) 2023-01-26 22:46:16 +05:30
Klaus Post 027ff0f3a8
fix: set modTime to current in snowball if archive shows empty (#16482) 2023-01-26 22:20:35 +05:30
Harshavardhana 54b561898f
fix: anonymize the x-amz-id-2 value from hostname (#16478) 2023-01-25 10:25:36 -08:00
Harshavardhana 65c104a589
add x-amz-id-2 to indicate the node that received the request (#16474) 2023-01-25 09:14:10 -08:00
Anis Elleuch 0a0416b6ea
Better error when setting up replication with a service account alias (#16472) 2023-01-25 21:50:12 +05:30
Anis Elleuch 441babdc41
Rename peer S3 prefix to avoid collision in the future (#16473) 2023-01-25 06:46:30 -08:00
Harshavardhana e64b9f6751
fix: disallow SSE-C encrypted objects on replicated buckets (#16467) 2023-01-24 15:46:33 -08:00
Florian Schwab d67a846ec4
allow restarting of decommissioning if completed, failed or canceld (#16464) 2023-01-24 07:07:59 -08:00
Poorna ca2a1c3f60
replication: clone metrics while loading metrics cache (#16462) 2023-01-24 02:10:32 -08:00
Poorna 93fbb228bf
Validate if parent user exists for service acct (#16443) 2023-01-24 08:17:18 +05:30
Anis Elleuch f37a5b6dae
Add CPU info in the check update user-agent (#16447) 2023-01-23 08:07:55 -08:00
Harshavardhana 31b0decd46
migrate to minio/mux from gorilla/mux (#16456) 2023-01-23 16:42:47 +05:30
Harshavardhana eb561e1c05
allow bootstrap platform checks to be pool specific (#16455) 2023-01-23 16:24:50 +05:30
Poorna ddad231921
replication: Avoid logging PreConditionFailed error (#16450) 2023-01-21 07:33:04 +05:30
Klaus Post 03b94f907f
fix: deleted object names for directory objects (#16448) 2023-01-20 21:16:06 +05:30
Shireesh Anjal 0f591d245d
fix: incorrect anonymization of drive endpoint (#16442) 2023-01-20 07:35:44 +05:30
Poorna 1b02e046c2
Fix bandwidth monitoring to be per remote target (#16360) 2023-01-19 18:52:16 +05:30
Harshavardhana d08e3cc895
add a way to avoid blocking queueHealTask() depending on caller (#16433) 2023-01-19 18:50:54 +05:30
Anis Elleuch d98116559b
Use async healing in PutObject call (#16431) 2023-01-19 00:54:22 -08:00
Krishnan Parthasarathi 71c95ad0d0
Signal stop-rebalance to all rebalancing pools (#16438) 2023-01-19 06:54:23 +05:30
Aditya Manthramurthy 698862ec5d
Fix transports/timeouts related regressions (#16427) 2023-01-18 10:06:38 +05:30
Harshavardhana b4ef5ff294
remove unnecessary code checking for supported features (#16423) 2023-01-17 19:37:47 +05:30
Harshavardhana 3db658e51e
use correct xml package for custom MarshalXML() (#16421) 2023-01-17 05:08:33 +05:30
Shireesh Anjal 5a9f7516d6
Add monthly license update job (#16391) 2023-01-17 05:08:15 +05:30
Anis Elleuch 3039fd4519
Optimize background heal status to use LocalStorageInfo (#16414) 2023-01-17 05:02:00 +05:30
Harshavardhana 095fc0561d
feat: allow decom of multiple pools (#16416) 2023-01-16 21:36:34 +05:30
Anis Elleuch beb1924437
Properly restart fresh disk healing when failed in some places (#16413) 2023-01-14 05:06:46 +05:30
jiuker c8e1154f1e
fix: reading from erasureDisks must be protected via read lock() (#16407) 2023-01-13 04:16:23 -08:00
Poorna b204c2dbec
fix: enforce deny on DeleteVersionAction (#16409) 2023-01-13 04:16:00 -08:00
Poorna b22b39de96
Avoid dangling deletes if disk not found (#16401) 2023-01-12 22:20:19 -08:00
Harshavardhana c242e6c391
fix: calculate common parity properly (#16406) 2023-01-13 03:28:16 +05:30
Anis Elleuch e05205756f
metrics: Add more logs when unable to read bucket usage (#16405) 2023-01-13 02:32:00 +05:30
Anis Elleuch 475a88b555
fix: error out if an object is found after a full decom (#16277) 2023-01-12 05:52:51 +05:30
Anis Elleuch 1ece3d1dfe
Add comment field to service accounts (#16380) 2023-01-10 21:57:52 +04:00
Anis Elleuch 2146ed4033
xl: Quit early when EC config is incorrect (#16390) 
Co-authored-by: Anis Elleuch <anis@min.io>
 2023-01-09 23:07:45 -08:00
Anis Elleuch ebd4388cca
s3: Return XMinioInvalidObjectName if the object contains null char (#16372) 2023-01-06 10:11:18 -08:00
Anis Elleuch 0333412148
fix: heal only once per disk per set among multiple disks (#16358) 2023-01-05 20:41:19 -08:00
Harshavardhana e0086c1be7
reduce startup delays on kubernetes (#16356) 2023-01-05 02:32:43 -08:00
Anis Elleuch 7883e55da2
Merge buckets list from different nodes in ListBuckets() call (#16357) 2023-01-04 08:53:58 -08:00
Harshavardhana a15a2556c3
converge listBuckets() as a peer call (#16346) 2023-01-03 23:39:40 -08:00
Harshavardhana f1bbb7fef5
vectorize cluster-wide calls such as bucket operations (#16313) 2023-01-03 08:16:39 -08:00
Harshavardhana 1cd8e1d8b6
remove the startup jitter before locks() (#16340) 2023-01-02 01:40:09 -08:00
jiuker 62cd918061
fix: close helmInfo file descriptor (#16319) 2023-01-01 23:26:59 -08:00
Klaus Post 6a04067514
fix: tweak read buffer size to reduce over-reading (#16338) 2023-01-01 08:14:20 -08:00
Taran Pelkey 49b3908635
fix: misplaced write response command in DetachPolicy() (#16333) 2022-12-30 20:04:03 -08:00
Harshavardhana f93183f66e
fix: a deadlock by refactoring listBuckets() under site replication (#16323) 2022-12-29 00:08:31 -08:00
Harshavardhana 2937711390
fix: DeleteObject() API with versionId under replication (#16325) 2022-12-28 22:48:33 -08:00
Anis Elleuch 27417459fb
metrics: Show healing info for all nodes (#16315) 2022-12-26 08:35:32 -08:00
Harshavardhana 5b8fe2e89a
allow locks with object affinity to spread across pools (#16312) 2022-12-23 20:55:45 -08:00
Anis Elleuch acc9c033ed
debug: Add X-Amz-Request-ID to lock/unlock calls (#16309) 2022-12-23 19:49:07 -08:00
Poorna 8528b265a9
Validate replication target update to avoid duplicate endpoints (#16311) 2022-12-23 15:44:48 -08:00
Harshavardhana b882310e2b
avoid locks for internal and invalid buckets in MakeBucket() (#16302) 2022-12-23 07:46:00 -08:00
Poorna de0b43de32
persist replication stats with leader lock (#16282) 2022-12-22 14:25:13 -08:00
jiuker 29dd7f1d68
tier verification leaks fd, that must be closed (#16296) 
Co-authored-by: Harshavardhana <harsha@minio.io>
 2022-12-22 10:35:54 -08:00
Poorna 6423e4c767
Remove site replication config if it succeeded locally (#16279) 2022-12-22 01:31:20 -08:00
Krishnan Parthasarathi 2fa35def2c
Fix DeleteObject when only free versions remain (#16289) 2022-12-21 16:24:07 -08:00
Anis Elleuch 34167c51d5
trace: Add bootstrap tracing events (#16286) 2022-12-21 15:52:29 -08:00
Harshavardhana a5f8af4efb
serialize replication stats() only when needed (#16280) 2022-12-20 00:07:53 -08:00
Harshavardhana 5a218f38a1
allow retries for transaction lock on startup (#16273) 2022-12-19 22:00:00 -08:00
Anis Elleuch e57e946206
Do not save credentials in config.json (#16275) 2022-12-19 12:27:06 -08:00
Klaus Post b4f71362e9
Avoid config migration on every startup (#16278) 2022-12-19 11:10:14 -08:00
Taran Pelkey ed37b7a9d5
Add API to fetch policy user/group associations (#16239) 2022-12-19 10:37:03 -08:00
Anis Elleuch 89db3fdb5d
Do not return an error when version disparity is detected (#16269) 2022-12-16 08:52:12 -08:00
Harshavardhana 80fc3a8a52
use newDynamicTimeoutWithOpts() when appropriate (#16266) 2022-12-15 13:11:37 -08:00
Klaus Post 988a2e8fed
Faster startup of large distributed systems with latency (#16259) 2022-12-15 08:31:21 -08:00
Harshavardhana 2433698372
fix: remove unnecessary logs for client conn errors (#16261) 2022-12-15 08:25:05 -08:00
Harshavardhana 5d7e8f79ed
fix: remove scanner healing with unnecessary logs (#16260) 2022-12-14 16:39:18 -08:00
Harshavardhana bad229e16e
fix: support event name s3:Restore:* (#16257) 2022-12-14 05:12:07 -08:00
Poorna d37e514733
Cleanup remote targets automatically on replication config removal. (#16221) 2022-12-14 03:24:06 -08:00
Harshavardhana c73ea27ed7
do not log checksum mismatch error, client received the error (#16246) 2022-12-14 01:57:40 -08:00
Krishnan Parthasarathi 0159b56717
fix: rebalance to account for object's on-disk size (#16240) 2022-12-14 00:15:14 -08:00
Aditya Manthramurthy 9e6cc847f8
Add HTTP2 config option for policy plugin (#16225) 2022-12-13 14:28:48 -08:00
Taran Pelkey 709eb283d9
Add endpoints for managing IAM policies (#15897) 
Co-authored-by: Taran <taran@minio.io>
Co-authored-by: ¨taran-p¨ <¨taran@minio.io¨>
Co-authored-by: Aditya Manthramurthy <donatello@users.noreply.github.com>
 2022-12-13 12:13:23 -08:00
Anis Elleuch 76dde82b41
Implement STS account info API (#16115) 2022-12-13 08:38:50 -08:00
Anis Elleuch 939c0100a6
log: Do not interpret verbs in object names in console output (#16233) 2022-12-13 08:27:40 -08:00
Aditya Manthramurthy 2d60bf8c50
Refactor HTTP transports (#16222) 2022-12-12 20:31:21 -08:00
Harshavardhana 37e20f6ef2
feat: allow listening specific addrs for API port (#16223) 2022-12-12 18:48:46 -08:00
Harshavardhana 2fc182d8e6
fix: iso8601TimeFormat padding issue for certain nanoseconds (#16207) 2022-12-12 10:28:30 -08:00
Shireesh Anjal a2cbeaa9e6
Use different subnet public key during dev/test (#16216) 2022-12-12 10:28:15 -08:00
Harshavardhana 444ff20bc5
do not rename multipart failed transactions back to tmp (#16204) 2022-12-12 01:40:29 -08:00
Harshavardhana 20ef5e7a6a
avoid double deletes() when no more versions (#16206) 2022-12-12 01:40:04 -08:00
Aditya Manthramurthy e06127566d
Add IAM API to attach/detach policies for LDAP (#16182) 2022-12-09 13:08:33 -08:00
Harshavardhana dfe73629a3
fix: delete marker discrepancies via DeleteObject() API (#16195) 2022-12-08 18:15:16 -08:00
Harshavardhana b03dd1af17
remove hard limit for number of buckets (#16194) 2022-12-08 12:24:03 -08:00
Harshavardhana 4bc367c490
fix: translate tier add errors properly (#16191) 2022-12-08 11:18:07 -08:00
Klaus Post 3eb2d086b2
Replace filepathx with fork (#16192) 2022-12-08 10:42:44 -08:00
Klaus Post 70986b6e6e
Add version id to healresult (#16193) 2022-12-08 07:49:10 -08:00
Klaus Post ebe395788b
feat: Encrypt s3zip file index (#16179) 2022-12-07 14:56:07 -08:00
Klaus Post 12fd6678ee
Encrypt checksums with KMS on CompleteMultipartUpload (#16177) 2022-12-07 10:18:18 -08:00
Harshavardhana 90d35b70b4
remove unnecessary logs for truncated XML inputs (#16184) 2022-12-07 08:30:52 -08:00
Javier Adriel 04ae9058ed
Populate end_session_endpoint (#16183) 2022-12-06 16:56:37 -08:00
Aditya Manthramurthy a30cfdd88f
Bump up madmin-go to v2 (#16162) 2022-12-06 13:46:50 -08:00
Anis Elleuch 1bae32dc96
xl: Delete older data-dir when replacing an existing version-id (#16176) 2022-12-06 13:43:18 -08:00
Anis Elleuch 932d2c3c62
Add X-Amz-Request-Id to internode calls (#16146) 2022-12-06 09:27:26 -08:00
jiuker 8d8d07ac5c
use readlock instead of writelock to get heal information (#16175) 2022-12-06 08:08:22 -08:00
Anis Elleuch 44735be38e
s3: Return correct error when Version is invalid in policy document (#16178) 2022-12-06 08:07:24 -08:00
Klaus Post 3fd9059b4e
opt: Only stream big data usage caches (#16168) 2022-12-05 13:01:11 -08:00
Klaus Post a713aee3d5
Run staticcheck on CI (#16170) 2022-12-05 11:18:50 -08:00
Andreas Auernhammer d882ba2cb4
kms: add support for KES enclaves (#16139) 
Signed-off-by: Andreas Auernhammer <hi@aead.dev>
 2022-12-04 02:34:24 -08:00
jiuker 6086f45d25
fix: in disk cache readCacheFileStream should closed upon return (#16138) 2022-12-04 02:28:10 -08:00
Klaus Post 98cffbce03
s3zip: Limit over-read for single file (#16161) 2022-12-02 08:53:24 -08:00
Klaus Post 1cd875de1e
Persist updated metadata (#16160) 2022-12-02 08:35:04 -08:00
Harshavardhana 5a8df7efb3
re-implement StorageInfo to be a peer call (#16155) 2022-12-01 14:31:35 -08:00
Anis Elleuch c84e2939e4
trace: Publish storage layer errors (#16153) 2022-12-01 12:10:54 -08:00
Anis Elleuch 641ab24aec
repl: resync orchestrator to use global shared lock (#16154) 2022-12-01 12:10:09 -08:00
Harshavardhana 71133105d7
re-order the top-level config keys for priority (#16150) 2022-12-01 07:50:08 -08:00
Aditya Manthramurthy 87cbd41265
feat: Allow at most one claim based OpenID IDP (#16145) 2022-11-29 15:40:49 -08:00
Klaus Post cc1d8f0057
Check for abandoned data when healing (#16122) 2022-11-28 10:20:55 -08:00
Anis Elleuch 1f1dcdce65
move HTTP recorder to an internal library (#16128) 2022-11-28 10:20:27 -08:00
Shireesh Anjal 98a67a3776
Improvements in logger and audit webhooks (#16102) 2022-11-28 08:03:26 -08:00
Poorna 63fc6ba2cd
preserve replicated ETag properly on target (#16129) 2022-11-26 14:43:32 -08:00
jiuker ce53d7f6c2
add disk.Close() in healFreshDisk to indicate idiomatic flow of code (#16124) 2022-11-26 00:26:15 -08:00
jiuker fe8eed963e
fix: wrapped error will not equal in decommissioning (#16113) 2022-11-24 08:00:42 -08:00
Shireesh Anjal 59f877fc64
fix: Timestamp not added in diagnostics report (#16114) 2022-11-23 07:11:22 -08:00
Klaus Post f96fe9773c
fix: duplicated shared prefix with custom delimiter when listing (#16111) 2022-11-22 08:51:04 -08:00
Anis Elleuch 04948b4d55
fix: checking for stale STS account under site replication (#16109) 2022-11-22 07:26:33 -08:00
Klaus Post 98ba622679
Reduce temporary file clean-up waits (#16110) 2022-11-22 07:23:36 -08:00
Harshavardhana 08103870a5
update single drive setup error message (#16098) 2022-11-18 14:47:38 -08:00
Anis Elleuch 993e586855
config: return XMinioConfigNotFound code for non existing config (#16065) 2022-11-18 10:28:14 -08:00
Harshavardhana 58ec835af0
fix: skip free version ID and marker in metadata equality (#16093) 2022-11-18 05:48:22 -08:00
Harshavardhana 6aea950d74
avoid partID lock validating uploadID exists prematurely (#16086) 2022-11-18 03:09:35 -08:00
Poorna 7198be5be9
bucket resync: persist reset id to bucket metadata (#16088) 2022-11-18 01:39:05 -08:00
Klaus Post a22b4adf4c
distribute replication ops based on names (#16083) 2022-11-17 15:20:09 -08:00
Klaus Post b7bb122be8
fix: replication auto-scaling deadlock (#16084) 2022-11-17 07:35:02 -08:00
Krishnan Parthasarathi 8441a3bf5f
fix: update metacache entry only once (#16072) 2022-11-16 11:25:00 -08:00
Harshavardhana 853c4de75a
allow changing endpoints in distributed setups (#16071) 2022-11-16 07:59:10 -08:00
jiuker 3597af789e
allow resultCh to be closed() after clusterMetaHealthInfo() (#16073) 2022-11-16 03:04:36 -08:00
Shireesh Anjal 5246e3be84
Send health diagnostics data as part of callhome (#16006) 2022-11-15 13:53:05 -08:00
Klaus Post 8a07000e58
fix: refactor getReplicationDiff for safe use (#16051) 2022-11-15 07:59:21 -08:00
Krishnan Parthasarathi 3bb82ef60d
top-locks: Include lock-held duration (#16061) 2022-11-15 07:57:52 -08:00
Harshavardhana 91f45c4aa6
avoid inconsistent versions healing when versions are large (#16066) 2022-11-14 18:35:26 -08:00
Poorna d6bc141bd1
feat: Add support for site level resync (#15753) 2022-11-14 07:16:40 -08:00
jiuker 7ac64ad24a
fix: use errors.Is for wrapped returns (#16062) 2022-11-14 07:15:46 -08:00
Harshavardhana 6d76db9d6c
improve server startup error when pools are incorrect (#16056) 2022-11-11 19:40:45 -08:00
jiuker bdcb485740
netPerfRX Reset() should use write Lock() (#16043) 2022-11-10 19:44:20 -08:00
Poorna e32b948a49
fix: parsing multipart uploadID under site replicated setup (#16048) 
continue the fix from #16034
 2022-11-10 16:17:45 -08:00
Klaus Post 5b242f1d11
Add Audit target metrics (#16044) 2022-11-10 10:20:21 -08:00
Poorna 34d28dd79f
replication: Avoid blocking on mrf save (#16045) 2022-11-10 10:20:02 -08:00
Krishnan Parthasarathi 6eef9b4a23
lifecycle: simplify Eval and HasActiveRules (#16036) 2022-11-10 07:17:45 -08:00
Aditya Manthramurthy 5f1999cc71
fix: avoid URL unsafe chars in multipart upload ID (#16034) 2022-11-09 16:41:16 -08:00
Krishnan Parthasarathi 40a2c6b882
Return remote tier as StorageClass for transitioned objects (#16035) 2022-11-09 15:57:34 -08:00
jiuker 7b7356f04c
close the reader under disk cache bitrot verification (#16024) 2022-11-09 04:20:11 -08:00
Klaus Post bbc312fce6
Add notification queue metrics (#16026) 2022-11-08 16:36:47 -08:00
Anis Elleuch 7260241511
Remove some logs caused by external apps (#16027) 2022-11-08 13:29:05 -08:00
Anis Elleuch 3b1a9b9fdf
Use the same lock for the scanner and site replication healing (#15985) 2022-11-08 08:55:55 -08:00
Harshavardhana 72afc2727a
rebalance status must return appropriate error initially (#16022) 2022-11-08 07:56:45 -08:00
Aditya Manthramurthy 76d822bf1e
Add LDAP policy entities API (#15908) 2022-11-07 14:35:09 -08:00
Klaus Post ddeca9f12a
fix: filter rest errors and logs returned (#16019) 2022-11-07 10:38:08 -08:00
Harshavardhana 1f3db03bf0
allow changing argument for path for SNSD setup (#16013) 2022-11-07 00:11:58 -08:00
Harshavardhana 944c62daf4
skip flaky tests on windows OS (#16015) 2022-11-07 00:11:21 -08:00
Harshavardhana 9547b7d0e9
add deadlineConnections on remoteTransport (#16010) 2022-11-05 11:09:21 -07:00
Klaus Post 808ecfe0f2
merge versions across sets when listing (#16003) 2022-11-04 11:33:22 -07:00
Klaus Post 2894dd4d1a
fix: hold lock while serializing replication stats (#16007) 2022-11-04 09:59:14 -07:00
jiuker fd8750e959
fix: http body must be drained in downloadBinary() (#16001) 2022-11-04 08:22:38 -07:00
Poorna 4f5d38a4b1
site replication edit: validate endpoint belongs to deployment (#16000) 2022-11-03 16:23:45 -07:00
Anis Elleuch 7e73fc2870
Implement inspect data API v2 (#15474) 
Co-authored-by: Klaus Post <klauspost@gmail.com>
 2022-11-02 13:36:38 -07:00
Harshavardhana 0d49b365ff
converge SNSD deployments into single code (#15988) 2022-11-01 16:41:01 -07:00
Anis Elleuch 7721595aa9
config: Deprecated delay/max_wait/scanner and introduce speed (#15941) 2022-11-01 08:04:07 -07:00
Harshavardhana fd6f6fc8df
cleanup stale parent multipart directories (#15980) 2022-11-01 08:00:02 -07:00
Aditya Manthramurthy 4fb47cd568
fix: update admin IDP APIs to be more RESTful (#15896) 2022-10-31 14:52:26 -07:00
Klaus Post ecc932d5dd
Clean entire tmp-old on restart (#15979) 2022-10-31 07:27:50 -07:00
Harshavardhana b57fbff7c1
ignore background healInfo in single drive setup (#15968) 2022-10-31 07:26:10 -07:00
Poorna d765b89a63
improve validation for replication resync API (#15964) 2022-10-28 23:21:33 -07:00
Harshavardhana 6e4acf0504
add a message of removal for gateway and hide the command (#15965) 2022-10-28 14:11:20 -07:00
Klaus Post 71954faa3a
mark pubsub type safe via generics (#15961) 2022-10-28 10:55:42 -07:00
Klaus Post 0f0e154315
fix: inconsistent replication delete marker timestamps (#15956) 2022-10-27 09:46:52 -07:00
Harshavardhana 136d41775f
remove numAvailableDisks check as it doesn't serve any purpose (#15954) 2022-10-27 09:05:24 -07:00
Harshavardhana ec77d28e62
make subnet subsys dynamic and simplify callhome (#15927) 2022-10-27 00:20:01 -07:00
Klaus Post 86420a1f46
Store multipart checksums (#15953) 2022-10-26 18:14:58 -07:00
Poorna 7dd8b6c8ed
ensure ILM expiry creates non null deleteMarker for versioned bucket (#15947) 2022-10-26 16:09:27 -07:00
Anis Elleuch 533c9d4fe3
fix: lockName to disallow parallel same erasure set healing (#15951) 2022-10-26 12:43:54 -07:00
Anis Elleuch a35ef155fc
return appropriate error status code in the lock handler (#15950) 2022-10-26 09:51:26 -07:00
Poorna 8dd3c41b2a
allow MakeBucket errors to be handled lazily (#15945) 
remote error is not required to be passed back to the 
client - this is mostly because we have healing that should 
eventually, catch up on this and heal the bucket.
 2022-10-25 23:32:37 -07:00
Krishnan Parthasarathi 4523da6543
feat: introduce pool-level rebalance (#15483) 2022-10-25 12:36:57 -07:00
Poorna ce8456a1a9
proxy multipart to peers via multipart uploadID (#15926) 2022-10-25 10:52:29 -07:00
Poorna 9ce1884732
reject editing bucket replication config when site replication is enabled (#15937) 2022-10-24 20:24:32 -07:00
Harshavardhana 23b329b9df
remove gateway completely (#15929) 2022-10-24 17:44:15 -07:00
Krishnan Parthasarathi 0c34e51a75
Filter out tiering metadata during CopyObject (#15936) 2022-10-24 16:32:31 -07:00
Anis Elleuch fc6c794972
Audit dangling object removal (#15933) 2022-10-24 11:35:07 -07:00
Klaus Post 86d543d0f6
Check for s3zip content offset (#15924) 2022-10-21 15:37:48 -07:00
Poorna e4e90b53c1
fix: delete-marker replication check properly (#15923) 2022-10-21 14:45:06 -07:00
Anis Elleuch 58d776daa0
Set CONSOLE_MINIO_SERVER to 127.0.0.1 by default (#15887) 2022-10-21 14:42:28 -07:00
Krishnan Parthasarathi f6b2e89109
Pass encrypted etag as is for immediate tiering (#15925) 2022-10-21 14:40:50 -07:00
Anis Elleuch ac85c2af76
lifecycle: refactor rules filtering and tagging support (#15914) 2022-10-21 10:46:53 -07:00
Shireesh Anjal 5aba2aedb3
Do not freeze s3 traffic in healthinfo api (#15912) 2022-10-21 00:34:32 -07:00
Harshavardhana a8332efa94
fix: Delete() of bucket metadata should not parse the config (#15904) 2022-10-19 17:55:09 -07:00
Aditya Manthramurthy 3dbef72dc7
fix: AccountInfo API for roleARN based accounts (#15907) 2022-10-19 17:54:41 -07:00
Aditya Manthramurthy 2d16e74f38
Add LDAP IDP Configuration APIs (#15840) 2022-10-19 11:00:10 -07:00
Anis Elleuch de5070446d
Deprecate --listeners flag (#15900) 2022-10-19 08:45:50 -07:00
Harshavardhana 374abd1e7d
add filter support for tags and metadata in batch replication (#15885) 2022-10-18 21:22:21 -07:00
Anis Elleuch 0506d9e83d
storage: Return errDiskNotFound when a peer is during shutdown (#15868) 2022-10-18 13:50:46 -07:00
Klaus Post bd3dfad8b9
Add concurrent Snowball extraction + options (#15836) 2022-10-18 13:50:21 -07:00
Harshavardhana 9fff315555
do not need to trace ignored objects (#15894) 2022-10-18 13:47:55 -07:00
Anis Elleuch 18fb86b7be
convert context.DeadlineExceed to offline disk in DiskInfo() (#15886) 2022-10-18 03:01:16 -07:00
Harshavardhana 58a8275e84
do not assume invalid buf to be non-xl.meta (#15843) 2022-10-17 09:39:21 -07:00
Aditya Manthramurthy 85fc7cea97
Pass role ARN for OIDC providers to console (#15862) 2022-10-15 12:57:03 -07:00
Harshavardhana 328d660106
support CRC32 Checksums on single drive setup (#15873) 2022-10-15 11:58:47 -07:00
Harshavardhana c68910005b
validate bucket before attempting batch replication (#15861) 2022-10-15 11:58:31 -07:00
Harshavardhana c79bcc8838 Revert "convert context.DeadlineExceed to offline disk in DiskInfo() (#15869)" 
This reverts commit 0fe58dbb34.
 2022-10-14 20:37:50 -07:00
Anis Elleuch 0fe58dbb34
convert context.DeadlineExceed to offline disk in DiskInfo() (#15869) 2022-10-14 19:32:13 -07:00
Harshavardhana 6cb2f56395 Revert "Revert "tests: Add context cancelation (#15374)"" 
This reverts commit 564a0afae1.
 2022-10-14 03:08:40 -07:00
Harshavardhana 59e33b3b21
validate setBucketTarget properly as per BucketExists() call (#15860) 2022-10-13 17:46:49 -07:00
Poorna 0e3c92c027 attempt delete marker replication after object is replicated (#15857) 
Ensure delete marker replication success, especially since the
recent optimizations to heal on HEAD, LIST and GET can force
replication attempts on delete marker before underlying object
version could have synced.
 2022-10-13 17:45:23 -07:00
Anis Elleuch db7a9b2c37
heal-info: Return the endpoint of a disk with unknown state (#15854) 2022-10-13 16:41:44 -07:00
Harshavardhana 44097faec1
support deleteMarkers and all versions in batch replication (#15858) 2022-10-13 14:42:10 -07:00
Klaus Post bf3da5081f
Omit empty checksums in responses (#15850) 2022-10-13 00:49:46 -07:00
Harshavardhana 5532982857
do not disable IsKubernetes(), IsDocker() checks with MINIO_CI_CD (#15852) 2022-10-12 23:40:48 -07:00
Anis Elleuch 783dd875f7
refactor objectQuorumFromMeta() to search for parity quorum (#15844) 2022-10-12 16:42:45 -07:00
Harshavardhana 97112c69be
fix: replication stats() to not crash under any situation (#15851) 
Co-authored-by: Daniel Valdivia <18384552+dvaldivia@users.noreply.github.com>
 2022-10-12 15:47:41 -07:00
Javier Adriel 2939000342
Add metrics, version and apis handlers (#15839) 2022-10-12 12:08:03 -07:00
Harshavardhana 41e1654f9a
remove spurious logging for object not found (#15842) 2022-10-12 04:28:21 -07:00
Harshavardhana e3cb0278ce
honor specified target prefix under batch replication (#15834) 2022-10-11 14:36:06 -07:00
Harshavardhana 0c81f1bdb3
indicate how long it took to bring the drive online (#15835) 2022-10-11 11:33:56 -07:00
Klaus Post 6220875803
Add missing server info fields (#15826) 2022-10-11 11:31:26 -07:00
Aditya Manthramurthy 64cf887b28
use LDAP config from minio/pkg to share with console (#15810) 2022-10-07 22:12:36 -07:00
Harshavardhana 927a879052
authenticate the request first for headObject() (#15820) 2022-10-07 21:45:53 -07:00
Anis Elleuch dfe0c96b87
preserve Version and DeleteMarker sort order in the list XML response (#15819) 2022-10-07 16:12:36 -07:00
Anis Elleuch e856e10ac2
ignore VersionNotFound in addition to ObjectNotFound while replicating (#15814) 2022-10-07 16:11:41 -07:00
Harshavardhana 928feb0889
remove unused debug param from evalActionFromLifecycle (#15813) 2022-10-07 10:24:12 -07:00
Anis Elleuch 158d0e26a2
decom: Ignore object/version error during deletion (#15806) 2022-10-06 09:41:58 -07:00
Harshavardhana 78385bfbeb
set bucket creation timestamp properly for legacy FS backend (#15800) 2022-10-06 02:46:31 -07:00
Harshavardhana 2a13cc28f2 feat: implement support batch replication (#15554) 2022-10-05 23:00:43 -07:00
Lenin Alevski 4bdf41a6c7
Removing unused getUpdateReaderFromFile function (#15794) 
Signed-off-by: Lenin Alevski <alevsk.8772@gmail.com>
 2022-10-05 07:58:27 -07:00
Klaus Post 3c605c93fe
warn when 0 parity has been set as default parity (#15790) 2022-10-04 22:41:42 -07:00
Anis Elleuch 121f18a443
Use admin request check for ReplicationDiff handler (#15793) 2022-10-04 17:47:31 -07:00
Harshavardhana 538aeef27a
fix: heal service accounts for LDAP users in site replication (#15785) 2022-10-04 10:41:47 -07:00
Poorna be0d2537b7
site replication: fix typo in meta collection (#15792) 2022-10-04 10:19:17 -07:00
Javier Adriel 3307aa1260
Implement KMS handlers (#15737) 2022-10-04 10:05:09 -07:00
Harshavardhana 57cfdfd8fb
remove 'perf' tests from health diagnostics (#15780) 2022-10-03 00:18:41 -07:00
Harshavardhana f696a221af
allow tagging policy condition for GetObject (#15777) 2022-10-02 12:29:29 -07:00
Harshavardhana 2aac50571d
fix: de-duplicate conflicting object names on namespace (#15772) 2022-09-30 15:44:21 -07:00
Shireesh Anjal 45edd27ad7
Re-load config after 'mc admin config reset' (#15771) 2022-09-30 10:55:53 -07:00
Daryl White d44f3526dc
Update links to documentation site (#15750) 2022-09-28 21:28:45 -07:00
Harshavardhana 41b633f5ea
support tagging based policy conditions (#15763) 2022-09-28 11:25:46 -07:00
Anis Elleuch 86bb48792c
non-blocking initialization of bucket target notifications (#15571) 2022-09-27 17:23:28 -07:00
Harshavardhana 94dbb4a427
fix: generalize SC config and also skip healing sub-sys under SD (#15757) 2022-09-26 09:04:54 -07:00
Anis Elleuch 048a46ec2a
Add RPC tcp timeout/errs and AVG duration to prometheus (#15747) 2022-09-26 09:04:26 -07:00
Poorna 8ea6fb368d
Add auto configuration of replication workers (#15636) 2022-09-24 16:20:28 -07:00
Harshavardhana b04c0697e1
validate correct ETag for the parts sent during CompleteMultipart (#15751) 2022-09-23 21:17:08 -07:00
Harshavardhana 50a8ba6a6f
fix: parse and save retainUntilDate in correct time format (#15741) 2022-09-23 08:49:27 -07:00
Anis Elleuch 20c89ebbb3
freeze before exit when _MINIO_DEBUG_NO_EXIT is defined (#15709) 
this is to ensure keep k8s pods running, when they reach a "crashloop" stage
 2022-09-22 11:57:27 -07:00
Krishnan Parthasarathi 6f56ba80b3
lifecycle: Assign unique id to rules with empty id (#15731) 2022-09-22 10:51:54 -07:00
Anis Elleuch 6e84283c66
fix: ignoring O_DIRECT in case of erasure single disk (#15734) 
fixes #15733 
fixes #15735
 2022-09-22 10:41:06 -07:00
Harshavardhana 9d6fddcfdf
persist the non-default creds in config (#15711) 2022-09-21 16:14:47 -07:00
jiuker 749ce107ee
fix: context leak with replication endpoint hearbeat (#15721) 2022-09-21 03:08:45 -07:00
Poorna aec2aa3497
site replication: clear config if remove --all specified (#15716) 2022-09-20 14:32:23 -07:00
Klaus Post ff12080ff5
Remove deprecated io/ioutil (#15707) 2022-09-19 11:05:16 -07:00
Minio Trusted d89f6af6c4 avoid replication stats crash in Prometheus 2022-09-16 17:09:45 -07:00
Harshavardhana 2c68a19dfd
upgrade all deps and update CREDITS (#15650) 2022-09-16 01:59:45 -07:00
Harshavardhana 9e5853ecc0
optimize double reads by reusing results from checkUploadIDExists() (#15692) 
Move to using `xl.meta` data structure to keep temporary partInfo,
this allows for a future change where we move to different parts to
different drives.
 2022-09-15 12:43:49 -07:00
Harshavardhana 124544d834
add pre-conditions support for PUT calls during replication (#15674) 
PUT shall only proceed if pre-conditions are met, the new
code uses

- x-minio-source-mtime
- x-minio-source-etag

to verify if the object indeed needs to be replicated
or not, allowing us to avoid StatObject() call.
 2022-09-14 18:44:04 -07:00
Poorna b910904fa6
change replication stats save path for windows (#15690) 2022-09-14 13:49:13 -07:00
Klaus Post eee1ce305c
When listing, do not count delete markers (#15689) 
When limiting listing do not count delete, since they may be discarded.

Extend limit, since we may be discarding the forward-to marker.

Fix directories always being sent to resolve, since they didn't return as match.
 2022-09-14 12:11:27 -07:00
Klaus Post 5c61c3ccdc
Fix flaky TestGetObjectWithOutdatedDisks (#15687) 
On occasion this test fails:

```
2022-09-12T17:22:44.6562737Z === RUN   TestGetObjectWithOutdatedDisks
2022-09-12T17:22:44.6563751Z     erasure-object_test.go:1214: Test 2: Expected data to have md5sum = `c946b71bb69c07daf25470742c967e7c`, found `7d16d23f07072af1a809707ba101ae07`
2
```

Theory: Both objects are written with the same timestamp due to lower timer resolution on Windows. This results in secondary resolution, which is deterministic, but random.

Solution: Instead of hacking in a wait we request the specific version we want. Should still keep the test relevant.

Bonus: Remote action dependency for vulncheck
 2022-09-14 08:17:39 -07:00
Poorna a0fb0c1835
panic if replication config could not be read from disk (#15685) 
If replication config could not be read from bucket metadata for some
reason, issue a panic so that unexpected replication outcomes can
be avoided for replicated buckets.

For similar reasons, adding a panic while fetching object-lock config
if it failed for reason other than non-existence of config.
 2022-09-13 21:23:33 -07:00
Aditya Manthramurthy e152b2a975
Pass groups claim into condition values (#15679) 
This allows using `jwt:groups` as a multi-valued condition key in policies.
 2022-09-13 09:45:36 -07:00
Poorna 6b9fd256e1
Persist in-memory replication stats to disk (#15594) 
to avoid relying on scanner-calculated replication metrics.
This will improve the accuracy of the replication stats reported.

This PR also adds on to #15556 by handing replication
traffic that could not be queued by available workers to the 
MRF queue so that entries in `PENDING` status are healed faster.
 2022-09-12 12:40:02 -07:00
Klaus Post ff9a74b91f
Add fast max-keys=1 support for Listing (#15670) 
Add a listing option to stop when the limit is reached.  
This can be used by stateless listings for fast results.
 2022-09-09 08:13:06 -07:00
Harshavardhana b579163802
limit number of buckets to 500k (#15668) 
500k is a reasonable limit for any single MinIO
cluster deployment, in future we may increase this
value.

However for now we are going to keep this limit.
 2022-09-09 03:06:34 -07:00
Krishnan Parthasarathi 96bfa77856
serialize updates to healing tracker (#15647) 
When healing is parallelized by setting the ` _MINIO_HEAL_WORKERS` 
environment variable, multiple goroutines may race while updating the disk's 
healing tracker. This change serializes only these concurrent updates using a
channel. Note, the healing tracker is still not concurrency safe in other contexts.
 2022-09-07 08:47:21 -07:00
Harshavardhana 8e997eba4a
fix: trigger Heal when xl.meta needs healing during PUT (#15661) 
This PR is a continuation of the previous change instead
of returning an error, instead trigger a spot heal on the
'xl.meta' and return only after the healing is complete.

This allows for future GETs on the same resource to be
consistent for any version of the object.
 2022-09-07 07:25:39 -07:00
Harshavardhana 228c6686f8
allow non-standards fallback for all http.TimeFormats (#15662) 
fixes #15645
 2022-09-07 07:24:54 -07:00
Harshavardhana 7776d064cf
allow non-standards fallback for Expires header (#15655) 
fixes #15645
 2022-09-05 19:18:18 -07:00
Harshavardhana 2d9b5a65f1
verify RenameData() versions to be consistent (#15649) 
xl.meta gets written and never rolled back, however
we definitely need to validate the state that is
persisted on the disk, if there are inconsistencies

- more than write quorum we should return an error
  to the client

- if write quorum was achieved however there are
  inconsistent xl.meta's we should simply trigger
  an MRF on them
 2022-09-05 16:51:37 -07:00
Shireesh Anjal c240da6568
Reuse madmin.ClusterRegistrationInfo (#15654) 
The `clusterInfo` struct in admin-handlers is same as
madmin.ClusterRegistrationInfo, except for small differences in field
names.

Removing this and using madmin.ClusterRegistrationInfo in its place will
help in following ways:

- The JSON payload generated by mc in case of cluster registration will
  be consistent (same keys) with cluster.info generated by minio as part
  of the profile and inspect zip
- health-analyzer can parse the cluster.info using the same struct and
  won't have to define it's own
 2022-09-05 10:02:25 -07:00
Harshavardhana 157272dc5b
fix: use optimized json.NewEncoder instead for metrics (#15648) 2022-09-05 08:06:35 -07:00
yudoutingle f4c56026a2
fix: potential deadLock caused by unlocking a non-existing lock (#15635) 2022-09-02 14:24:32 -07:00
Harshavardhana 37e3f5de10
do not print object not found errors in MRF healing (#15646) 2022-09-02 14:22:40 -07:00
Harshavardhana 5ea629beb2
avoid printing io.ErrUnexpectedEOF for .metacache objects (#15642) 2022-09-02 12:47:17 -07:00
Anis Elleuch cf52691959
Save resync status in the backend using a last update timestamp (#15638) 
Currently, there is a short time window where the code is allowed 
to save the status of a replication resync. Currently, the window is
`now.Sub(st.EndTime) <= resyncTimeInterval`. Also, any failure to 
write in the backend disks is not retried.

Refactor the code a little bit to rely on the last timestamp of a
successful write of the resync status of any given bucket in the 
backend disks.
 2022-09-01 16:53:36 -07:00
Anis Elleuch 10e75116ef
Avoid replicating dirs in listing with replication enabled (#15641) 
When replication is enabled in a particular bucket, the listing will send
objects to bucket replication, but it is also sending prefixes for non
recursive listing which is useless and shows a lot of error logs.

This commit will ignore prefixes.
 2022-09-01 15:22:11 -07:00
Harshavardhana f649968c69
tier: avoid stats infinite loop in forwardTo method (#15640) 
under some sequence of events following code would
reach an infinite loop.

```
idx1, idx2 := 0, 1
for ; idx2 != idx1; idx2++ {
        fmt.Println(idx2)
}
```

fixes #15639
 2022-09-01 13:51:06 -07:00
Harshavardhana bcedc2b0d9
fix: add healing metric type for heal tracing (#15631) 
changes the `heal.checkBucket` to `heal.Bucket` instead
since the latter is more meaningful.
 2022-08-31 12:28:03 -07:00
Klaus Post 8e4a45ec41
fix: encrypt checksums in metadata (#15620) 2022-08-31 08:13:23 -07:00
Klaus Post dec942beb6
feat: Add healing trace (#15616) 2022-08-31 01:56:12 -07:00
Abirdcfly d4e0f13bb3
chore: remove duplicate word in comments (#15607) 
Signed-off-by: Abirdcfly <fp544037857@gmail.com>

Signed-off-by: Abirdcfly <fp544037857@gmail.com>
 2022-08-30 08:26:43 -07:00
Anis Elleuch 1f28a3bb80
Avoid messages from go test output (#15601) 
A lot of warning messages are printed in CI/CD failures generated by go
test. Avoid that by requiring at least Error level for logging when
doing go test.
 2022-08-30 08:23:40 -07:00
Krishnan Parthasarathi 3a1d3a7952
audit-log: Add time to get/restore object from remote-tier (#15602) 2022-08-29 21:33:59 -07:00
Klaus Post a9f1ad7924
Add extended checksum support (#15433) 2022-08-29 16:57:16 -07:00
Poorna 929b9e164e
site replication: Avoid returning root svcacct info in sr metadata (#15608) 
Service accounts of root users should not be replicated.
 2022-08-29 11:19:51 -07:00
Harshavardhana 97376f6e8f
improve performance for inlined data (#15603) 
inlined data often is bigger than the allowed
O_DIRECT alignment, so potentially we can write
'xl.meta' without O_DSYNC instead we can rely on
O_DIRECT + fdatasync() instead.

This PR allows O_DIRECT on inlined data that
would gain the benefits of performing O_DIRECT,
eventually performing an fdatasync() at the end.

Performance boost can be observed here for small
objects < 128KiB. The performance boost is mainly
seen on HDD, and marginal on NVMe setups.
 2022-08-29 11:19:29 -07:00
Febriananda Wida Pramudita 1f22a16b15
fix: endpoints for single local disks must retain port info (#15585) 2022-08-26 12:53:15 -07:00
Harshavardhana 433b6fa8fe
upgrade golang-lint to the latest (#15600) 2022-08-26 12:52:29 -07:00
Krishnan Parthasarathi 99fbfe2421
Add concurrency to healing objects on a fresh disk (#15575) 2022-08-25 13:07:15 -07:00
Poorna b1b6264bea
fix: validate deployment id when adding peer clusters (#15591) 
Fixes: #15573
 2022-08-25 11:30:52 -07:00
Aditya Manthramurthy 18dffb26e7
Allow querying a single target in config get API (#15587) 2022-08-25 00:17:05 -07:00
Harshavardhana edba7c987b
fix: objects matching prefixes should not leave delete markers (#15586) 
This is needed to ensure that we do not leave prefixes where
version is suspended, instead we never leave versions on
these paths.
 2022-08-24 13:46:29 -07:00
Anis Elleuch b737c83a66
Ensure that only one node performs site replication healing (#15584) 
When a node finds a change in the other replication cluster and applies
to itself will already notify other peers. No need for all nodes in a
given cluster to do site replication healing, only one node is
sufficient.
 2022-08-24 13:46:09 -07:00
Anis Elleuch 97a6322de1
Fix regression in notifying peers about new policy mapping (#15583) 
Switch from mux.Vars() to r.Form to avoid the issue of missing arguments
passed to LoadPolicyMappingHandler.
 2022-08-24 12:34:52 -07:00
Klaus Post 037fe4afdc
Add listing block reuse (#15579) 
When streaming results, pool metadata slices when sent.
 2022-08-24 09:11:16 -07:00
Aditya Manthramurthy afbb63a197
Factor out external event notification funcs (#15574) 
This change moves external event notification functionality into
`event-notification.go`. This simplifies notification related code.
 2022-08-24 06:42:36 -07:00
Harshavardhana 8902561f3c
use new xxml for XML responses to support rare control characters (#15511) 
use new xxml/XML responses to support rare control characters

fixes #15023
 2022-08-23 17:04:11 -07:00
Anis Elleuch b8cdf060c8
Properly replicate policy mapping for virtual users (#15558) 
Currently, replicating policy mapping for STS users does not work. Fix
it is by passing user type to PolicyDBSet.
 2022-08-23 11:11:45 -07:00
Poorna 4155c5b695
replication: improve MRF healing. (#15556) 
This PR improves the replication failure healing by persisting
most recent failures to disk and re-queuing them until the replication
is successful.

While this does not eliminate the need for healing during a full scan, 
queuing MRF vastly improves the ETA to keeping replicated buckets 
in sync as it does not wait for the scanner visit to detect unreplicated 
object versions.
 2022-08-22 16:53:06 -07:00
Poorna 471467d310
fix: ensure metadata update happens after deletemarker replication (#15564) 
Fixes regression caused by #15521
 2022-08-22 15:59:06 -07:00
Harshavardhana ae4ee95d25
change default lock retry interval to 50ms (#15560) 
competing calls on the same object on versioned bucket
mutating calls on the same object may unexpected have
higher delays.

This can be reproduced with a replicated bucket
overwriting the same object writes, deletes repeatedly.

For longer locks like scanner keep the 1sec interval
 2022-08-19 16:21:05 -07:00
Harshavardhana e9055e9ef7
fix: walk() should cancel itself upon context cancellation (#15553) 
This PR fixes possible leaks that may emanate from not
listening on context cancelation or timeouts.

```
goroutine 60957610 [chan send, 16 minutes]:
github.com/minio/minio/cmd.(*erasureServerPools).Walk.func1.1.1(...)
        github.com/minio/minio/cmd/erasure-server-pool.go:1724 +0x368
github.com/minio/minio/cmd.listPathRaw({0x4a9a740, 0xc0666dffc0},...
        github.com/minio/minio/cmd/metacache-set.go:1022 +0xfc4
github.com/minio/minio/cmd.(*erasureServerPools).Walk.func1.1()
        github.com/minio/minio/cmd/erasure-server-pool.go:1764 +0x528
created by github.com/minio/minio/cmd.(*erasureServerPools).Walk.func1
        github.com/minio/minio/cmd/erasure-server-pool.go:1697 +0x1b7
```
 2022-08-18 17:49:08 -07:00
Harshavardhana d350b666ff
feat: add idempotent delete marker support (#15521) 
The bottom line is delete markers are a nuisance,
most applications are not version aware and this
has simply complicated the version management.

AWS S3 gave an unnecessary complication overhead
for customers, they need to now manage these
markers by applying ILM settings and clean
them up on a regular basis.

To make matters worse all these delete markers
get replicated as well in a replicated setup,
requiring two ILM settings on each site.

This PR is an attempt to address this inferior
implementation by deviating MinIO towards an
idempotent delete marker implementation i.e
MinIO will never create any more than single
consecutive delete markers.

This significantly reduces operational overhead
by making versioning more useful for real data.

This is an S3 spec deviation for pragmatic reasons.
 2022-08-18 16:41:59 -07:00
Harshavardhana 895357607a
avoid using errors.As for 'errors.New' use errors.Is (#15549) 
Bonus: ignore coredns CVE, for now, there is no fix yet

https://github.com/coredns/coredns/issues/5574
 2022-08-18 11:10:49 -07:00
Harshavardhana bf38c0c0d1
fix: increase concurrency of DeleteObjects() to N/10th (#15546) 
instead of keeping the value 10 and static, make
the concurrency a function of incoming number of
objects being deleted.
 2022-08-18 09:33:56 -07:00
Poorna 21fe14201f
replication: centralize healthcheck for remote targets (#15516) 
This PR moves health check from minio-go client to being
managed on the server.

Additionally integrating health check into site replication
 2022-08-16 17:46:22 -07:00
Harshavardhana 48640b1de2
fix: trim arn:aws:kms from incoming SSE aws-kms-key-id (#15540) 2022-08-16 11:28:30 -07:00
Anis Elleuch 5682685c80
Introduce disk io stats metrics (#15512) 2022-08-16 07:13:49 -07:00
Harshavardhana c7d535c648
init console after IAM init() (#15531) 
fixes #15527
 2022-08-13 12:54:41 -07:00
Aditya Manthramurthy 9986e103cf
Fix env var output in config get/export APIs (#15528) 
Fix a bug where env vars are not output when the config for the
subsystem is specified solely via env vars.
 2022-08-13 10:39:01 -07:00
Krishnan Parthasarathi 91e6af4470
Add trace support for decommissioning (#15502) 
* Add trace support for decommissioning
* Add support for tracing errors during decommission
 2022-08-10 12:46:45 -07:00
Shireesh Anjal 316c492842
Upgrade madmin-go to latest version (v1.4.15) (#15510) 2022-08-10 07:36:13 -07:00
Harshavardhana 74418b542a
fix: incorrect context timeout during listPath() (#15509) 
This PR cleans up the listing code for single drive
to ensure that we do not add an incorrect context
timeout, while resuming the listing.

fixes #15508
 2022-08-10 07:35:29 -07:00
Poorna 172e63dbb6
fix: site replication group updates to set status correctly (#15507) 
Fixes: #15486
 2022-08-09 15:17:43 -07:00
Poorna 21bf5b4db7
replication: heal proactively upon access (#15501) 
Queue failed/pending replication for healing during listing and GET/HEAD
API calls. This includes healing of existing objects that were never
replicated or those in the middle of a resync operation.

This PR also fixes a bug in ListObjectVersions where lifecycle filtering
should be done.
 2022-08-09 15:00:24 -07:00
Harshavardhana a406bb0288
restrict number of disks used for scanning buckets upto GOMAXPROCS (#15492) 
control scanner parallelism to avoid higher CPU
usage on nodes that have more drives but an old CPU.
 2022-08-08 16:16:44 -07:00
Harshavardhana 1823ab6808
LDAP/OpenID must be initialized IAM Init() (#15491) 
This allows for LDAP/OpenID to be non-blocking,
allowing for unreachable Identity targets to be
initialized in IAM.
 2022-08-08 16:16:27 -07:00
Harshavardhana 8eec49304d use logger.Info instead of logger.LogIf 2022-08-08 16:13:58 -07:00
Harshavardhana ecdc2f2f5f
fix: maxConcurrent '0' is an invalid value (#15500) 
log and continue with defaults instead of
crashing the service.
 2022-08-08 15:18:45 -07:00
Harshavardhana e178c55bc3
remove non-working GetRawData() from FS mode (#15498) 2022-08-08 11:34:09 -07:00
Poorna 2c137c0d04
fix: handle invalid endpoint errors in site replication(#15499) 
fixes #15497
 2022-08-08 11:12:05 -07:00
Harshavardhana 638c57e466 revert changes in FS implementation for umask 
fixes #15494
 2022-08-08 09:48:24 -07:00
Harshavardhana 5e4213b3be
fix: keep writing previous speedtest result (#15484) 
when object speedtest is running keep writing
previous speedtest result back to client until
we have a new result - this avoids sending back
blank entries in between the speedtest when it
is running in 'autotune' mode.
 2022-08-07 23:04:03 -07:00
Harshavardhana e0b0a351c6
remove IAM old migration code (#15476) 
```
commit 7bdaf9bc50
Author: Aditya Manthramurthy <donatello@users.noreply.github.com>
Date:   Wed Jul 24 17:34:23 2019 -0700

    Update on-disk storage format for users system (#7949)
```

Bonus: fixes a bug when etcd keys were being re-encrypted.
 2022-08-05 17:53:23 -07:00
Anis Elleuch 1d2ff46a89
Ensure lock/versioning permissions when creating a bucket (#15432) 
Currently, the code doesn't check if the user creating a bucket with
locking feature has bucket locking and versioning permissions enabled,
adding it in accordance with S3 spec.

https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateBucket.html

Object Lock - If ObjectLockEnabledForBucket is set to true in your CreateBucket request,
s3:PutBucketObjectLockConfiguration and s3:PutBucketVersioning permissions are required.
 2022-08-05 16:27:09 -07:00
Harshavardhana 8f7c739328
feat: add SpeedTest ResponseTimes and TTFB (#15479) 
Capture average, p50, p99, p999 response times
and ttfb values. These are needed for latency
measurements and overall understanding of our
speedtest results.
 2022-08-05 09:40:03 -07:00
Poorna 1beea3daba
fix: import bucket metadata import to return a summary (#15462) 2022-08-05 01:52:50 -07:00
Aditya Manthramurthy 3d94c38ec4
Add env variables to configuration APIs output (#15465) 
Config export and config get APIs now include environment 
variables set on the server
 2022-08-04 22:21:52 -07:00
Harshavardhana f4af2d3cdc
fix: decodeDirObject() in single drive DeleteObjects() call (#15477) 
Thanks to @bh4t for reproducing this issue.
 2022-08-04 18:57:43 -07:00
ebozduman b57e7321e7
Replaces 'disk'=>'drive' visible to end user (#15464) 2022-08-04 16:10:08 -07:00
Anis Elleuch e93867488b
actively cancel listIAMConfigItems to avoid goroutine leak (#15471) 
listConfigItems creates a goroutine but sometimes callers will
exit without properly asking listAllIAMConfigItems() to stop sending
results, hence a goroutine leak.

Create a new context and cancel it for each listAllIAMConfigItems
call.
 2022-08-04 13:20:43 -07:00
Harshavardhana 3bd9615d0e
fix: log if there is readDir() failure with ListBuckets (#15461) 
This is actionable and must be logged.

Bonus: also honor umask by using 0o666 for all Open() syscalls.
 2022-08-04 07:23:05 -07:00
Harshavardhana a6e0ec4e6f
Add support converting non-inlined to inlined (#15444) 
This is a feature to allow for inode compaction on
large clusters that use a lot of small files spread
across a large heirarchy.
 2022-08-02 23:10:22 -07:00
Andreas Auernhammer d774a3309b
kes: automatically reload KES client certificate (#15450) 
This commit adds support for automatically reloading
the MinIO client certificate for authentication to KES.

The client certificate will now be reloaded:
 - when the private key / certificate file changes
 - when a SIGHUP signal is received
 - every 15 minutes

Fixes #14869

Signed-off-by: Andreas Auernhammer <hi@aead.dev>
 2022-08-02 16:58:09 -07:00
Anis Elleuch b3edb25377
bloom: healObject to mark a path dirty only for dangling objects (#15458) 
The path is marked dirty automatically when healObject() is called, which is
wrong. HealObject() is called during self-healing and this will lead to
an increase in the false positive result of the bloom filter.

Also move NSUpdated() from renameData() and call it directly in
CompleteMultipart and PutObject, this is not a functional change but
it will make it less prone to errors in the future.
 2022-08-02 16:57:39 -07:00
Harshavardhana 53a816b17a
fix: readdir fallback on root of the drive (#15457) 
fixes #15452
 2022-08-02 14:57:36 -07:00
Harshavardhana 043aaa792d
fix: intrument os.OpenFile differently for Reads and Writes (#15449) 
allows us to trace latency for READs or WRITEs
 2022-08-01 13:22:43 -07:00
Shireesh Anjal e6eab2091f
fix: Incorrect ServersCount in cluster.info (#15431) 
The `ServersCount` field in cluster.info is expected to contain the
number of nodes, and not number of endpoints.
 2022-07-29 22:21:40 -07:00
Harshavardhana 3cdb609cca
allow root users to return appropriate policy in AccountInfo (#15437) 
fixes #15436

This fixes a regression caused after the removal of "consoleAdmin"
policy usage for 'root users' in PR #15402
 2022-07-29 20:58:03 -07:00
Harshavardhana aa874010e2
fix: regression in resolving the right versions (#15430) 
fix: regression in resolving right versions

commit d480022711 caused a regression in real
resolver, by picking up incorrect versionID.
 2022-07-29 10:03:53 -07:00
Cesar Celis Hernandez 8ec888d13d
feat: update binary once and push it to other servers (#15407) 2022-07-29 08:34:30 -07:00
Harshavardhana 916f274c83
choose starting concurrency based on number of local disks (#15428) 
smaller setups may have less drives per server choosing
the concurrency based on number of local drives, and let
the MinIO server change the overall concurrency as
necessary.
 2022-07-29 00:00:06 -07:00
Aditya Manthramurthy 7ac53c07af
fix: passing application configuration to console (#15409) 
This is an update to MinIO server after swagger codegen related build
fixes added after issues introduced in 39fd7b0b3b
 2022-07-28 18:30:24 -07:00
Harshavardhana bc72e4226e
do not allow filesystem fallback in server download (#15429) 
It is possible for anyone with admin access to relatively
to get any content of any random OS location by simply
providing the file with 'mc admin update alias/ /etc/passwd`.

Workaround is to disable 'admin:ServiceUpdate' action. Everyone
is advised to upgrade to this patch.

Thanks to @alevsk for finding this bug.
 2022-07-28 17:44:21 -07:00
Poorna 5e0776e96a
replication: Include replica object versions for resync (#15427) 2022-07-28 13:43:02 -07:00