MyFavMirrors
/
minio
mirror of https://github.com/minio/minio.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
11,342 Commits 5 Branches 503 Tags 186 MiB
Commit Graph

604 Commits

Author SHA1 Message Date
jiuker b28d391a22
fix: add correct worker count before startHTTPLogger() (#17091) 2023-04-27 10:51:16 -07:00
jiuker c8b92f6067
protect wg.Done from being called twice (#17075) 2023-04-27 07:55:36 -07:00
Anis Eleuch 31b5acc245
tcp: Increase user timeout to 10 minutes (#17087) 2023-04-26 17:48:31 -07:00
Anis Eleuch 0b7ca094e4
Remove Expect 100-continue in internode communications (#17061) 2023-04-26 09:33:45 -07:00
Praveen raj Mani 72802a5972
Use 'minio/pkg/sync/errgroup' and 'minio/pkg/workers' (#17069) 2023-04-25 22:57:40 -07:00
Harshavardhana 8fd07bcd51
simplify sort.Sort by using sort.Slice (#17066) 2023-04-24 13:28:18 -07:00
Harshavardhana 8a9b9832fd
add Dial timeout for Kafka broker pings (#17044) 2023-04-17 15:45:01 -07:00
jiuker e96c88e914
fix: DeleteBucketThrottle must delete ARN (#17034) 2023-04-15 02:14:26 -07:00
Klaus Post c133979b8e
Add part count to checksum (#17035) 2023-04-14 09:44:45 -07:00
Harshavardhana a5835cecbf
fix: regression in counting total requests (#17024) 2023-04-12 14:37:19 -07:00
Poorna d350654aee
config: fix duplication of replication priority key (#17014) 2023-04-11 19:22:10 -07:00
Harshavardhana 09a25ea7b7 lint: fix some lint issues on files 2023-04-06 22:42:10 -07:00
Harshavardhana 47b7469a60
add buffer pool for proxy forwarder (#16942) 2023-04-06 15:54:12 -07:00
Praveen raj Mani 51f7f9aaa3
Generalize the event store using go generics (#16910) 2023-04-04 10:52:24 -07:00
Anis Eleuch d90d0c8931
Use one http response recorder per external http call (#16938) 2023-03-31 09:37:29 -07:00
Anis Eleuch c259a8ea38
Set tcp user timeout to clean sockets with data in the buffer (#16887) 2023-03-24 08:10:58 -07:00
mstmdev 2d51e42305
Remove the redundant conditional in the validateParity function (#16866) 2023-03-23 14:06:22 -07:00
Anis Eleuch 1346561b9d
return quorum error instead of insufficient storage error (#16874) 2023-03-22 16:22:37 -07:00
Klaus Post 11d04279c8
Add lazy init of audit logger (#16842) 2023-03-21 10:50:40 -07:00
Harshavardhana 3b5dbf9046
allow bootstrapping to validate internode tokens (#16853) 2023-03-20 01:40:24 -07:00
Aditya Manthramurthy 09c733677a
Add test for fixed post policy exploit (#16855) 2023-03-20 01:06:45 -07:00
Harshavardhana 46f9049fb4
simplify error responses for KMS (#16793) 2023-03-16 11:59:42 -07:00
Aditya Manthramurthy 58266c9e2c
Add enable flag for LDAP IDP config (#16805) 2023-03-16 11:58:59 -07:00
Nitish Tiwari 50dbd2cacc
Update audit log flow to use new headers with unit (#16797) 2023-03-13 22:50:19 -07:00
Harshavardhana b984bf8d1a
allow expiration of all versions during Listing() (#16757) 2023-03-09 15:15:30 -08:00
Harshavardhana 901887e6bf
feat: add lambda transformation functions target (#16507) 2023-03-07 08:12:41 -08:00
ferhat elmas 714283fae2
cleanup ignored static analysis (#16767) 2023-03-06 08:56:10 -08:00
ferhat elmas 3423028713
cleanup Go linter settings (#16736) 2023-03-04 20:57:35 -08:00
Harshavardhana 0ff931dc76
fix: allow CORS to work by default (#16713) 2023-02-27 10:10:45 -08:00
Praveen raj Mani 4d708cebe9
Support adding service accounts with expiration (#16430) 
Co-authored-by: Harshavardhana <harsha@minio.io>
 2023-02-27 10:10:22 -08:00
Aditya Manthramurthy 8cde38404d
Add metrics for custom auth plugin (#16701) 2023-02-27 09:55:18 -08:00
Shubhendu 8b4eb2304b
Set logger webhook proxy on subnet proxy change (#16665) 
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
 2023-02-27 08:35:36 -08:00
Anis Elleuch 8da0f4c5bb
Better error message when TLS certs do not have proper permissions (#16703) 2023-02-24 06:34:55 -08:00
Harshavardhana b21d3f9b82
event target registration failures must be returned (#16700) 2023-02-23 21:59:14 +05:30
Harshavardhana 5c98223c89
add correct HostId instead of deploymentId for error responses (#16686) 2023-02-22 15:41:09 +05:30
Daniel Valdivia fb17f97cf3
move audit and logger message structure to minio/pkg (#16655) 
Signed-off-by: Daniel Valdivia <18384552+dvaldivia@users.noreply.github.com>
 2023-02-21 21:21:17 -08:00
Shubhendu 6b65ba1551
Added attribute proxy for `mc admin config set ALIAS logger_webhook` (#16657) 
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
 2023-02-21 21:19:46 -08:00
Allan Roger Reid 8bfe972bab
Set meaningful message from minio with env variable KMS_SECRET_KEY (#16584) 2023-02-22 07:13:01 +05:30
Klaus Post fd6622458b
Add detailed scanner trace output and notifications (#16668) 2023-02-21 09:33:33 -08:00
jiuker e470268c7c
fix: a possible closer leak in SelectObjectHandler (#16598) 2023-02-17 01:44:40 -08:00
Harshavardhana 6ea150fd68
fix: avoid printing certain errors under few locations (#16631) 2023-02-17 01:40:31 -08:00
Harshavardhana e1e9ddd4a4
use kes.Status() for Status() call (#16629) 2023-02-16 22:12:24 +05:30
Andreas Auernhammer 74887c7372
kms: add support for KES API keys and switch to KES Go SDK (#16617) 
Signed-off-by: Andreas Auernhammer <hi@aead.dev>
 2023-02-14 07:19:20 -08:00
Harshavardhana 31188e9327
add parallel workers in batch replication (#16609) 2023-02-13 12:07:58 -08:00
Harshavardhana d65debb6bc
fix: comply with RFC6750 UserInfo endpoint requirements (#16592) 2023-02-10 22:20:25 +05:30
Anis Elleuch c8ffa59d28
Periodically refresh buckets metadata from the backend disks (#16561) 
fixes #16553
 2023-02-09 10:29:20 -08:00
Anis Elleuch fadc46b906
Add the access key and parent user in the audit log (#16572) 2023-02-08 11:05:26 -08:00
Anis Elleuch 095b518802
Show a better error msg when internal data encryption key is incorrect (#16549) 2023-02-07 05:22:54 -08:00
Harshavardhana 14cf8f1b22
upgrade deps for minio/pkg v1.6.1 to include groups conditions (#16538) 2023-02-06 09:27:29 -08:00
Harshavardhana 5996c8c4d5
feat: allow offline disks on a fresh start (#16541) 2023-02-06 09:26:09 -08:00
Harshavardhana aa8b9572b9
remove double ENABLED help output (#16528) 2023-02-03 05:52:52 -08:00
Klaus Post 0b7989aa4b
Fix Kafka initialization crash (#16523) 2023-02-02 19:21:19 +05:30
Harshavardhana d19cbc81b5
fix: do not return IAM/Bucket metadata replication errors to client (#16486) 2023-01-26 11:11:54 -08:00
Harshavardhana 65c104a589
add x-amz-id-2 to indicate the node that received the request (#16474) 2023-01-25 09:14:10 -08:00
Harshavardhana 3683673fb0
add missing gorilla/mux migration, update credits (#16461) 2023-01-23 08:46:37 -08:00
Poorna 1b02e046c2
Fix bandwidth monitoring to be per remote target (#16360) 2023-01-19 18:52:16 +05:30
Aditya Manthramurthy 698862ec5d
Fix transports/timeouts related regressions (#16427) 2023-01-18 10:06:38 +05:30
Harshavardhana 3db658e51e
use correct xml package for custom MarshalXML() (#16421) 2023-01-17 05:08:33 +05:30
Allan Roger Reid 9815dac48f
fix: allow bind on ipv6 loopback failures (#16388) 2023-01-11 08:47:39 +05:30
Anis Elleuch 1ece3d1dfe
Add comment field to service accounts (#16380) 2023-01-10 21:57:52 +04:00
Anis Elleuch 2146ed4033
xl: Quit early when EC config is incorrect (#16390) 
Co-authored-by: Anis Elleuch <anis@min.io>
 2023-01-09 23:07:45 -08:00
Anis Elleuch 1c85652cff
lint: Fix in darwin environment (#16368) 2023-01-05 10:12:01 -08:00
Klaus Post 6a04067514
fix: tweak read buffer size to reduce over-reading (#16338) 2023-01-01 08:14:20 -08:00
Harshavardhana 75faef888e
disable builds for go1.18 (#16332) 2022-12-30 11:37:07 -08:00
Harshavardhana b67d97b1ba
add missing fields in audit logs for non-compressed handlers (#16328) 2022-12-30 10:20:19 -08:00
Anis Elleuch acc9c033ed
debug: Add X-Amz-Request-ID to lock/unlock calls (#16309) 2022-12-23 19:49:07 -08:00
Anis Elleuch e57e946206
Do not save credentials in config.json (#16275) 2022-12-19 12:27:06 -08:00
Harshavardhana 80fc3a8a52
use newDynamicTimeoutWithOpts() when appropriate (#16266) 2022-12-15 13:11:37 -08:00
Klaus Post 988a2e8fed
Faster startup of large distributed systems with latency (#16259) 2022-12-15 08:31:21 -08:00
Harshavardhana bad229e16e
fix: support event name s3:Restore:* (#16257) 2022-12-14 05:12:07 -08:00
Harshavardhana c73ea27ed7
do not log checksum mismatch error, client received the error (#16246) 2022-12-14 01:57:40 -08:00
Aditya Manthramurthy 9e6cc847f8
Add HTTP2 config option for policy plugin (#16225) 2022-12-13 14:28:48 -08:00
Anis Elleuch 939c0100a6
log: Do not interpret verbs in object names in console output (#16233) 2022-12-13 08:27:40 -08:00
Aditya Manthramurthy 2d60bf8c50
Refactor HTTP transports (#16222) 2022-12-12 20:31:21 -08:00
Aditya Manthramurthy a469e6768d
Add LDAP DNS SRV record lookup support (#16201) 2022-12-12 11:27:27 -08:00
Harshavardhana 2fc182d8e6
fix: iso8601TimeFormat padding issue for certain nanoseconds (#16207) 2022-12-12 10:28:30 -08:00
Aditya Manthramurthy e06127566d
Add IAM API to attach/detach policies for LDAP (#16182) 2022-12-09 13:08:33 -08:00
jiuker 8edc2faaa9
reuse sha256 in config GetSettings (#16188) 2022-12-08 03:03:24 -08:00
Javier Adriel 04ae9058ed
Populate end_session_endpoint (#16183) 2022-12-06 16:56:37 -08:00
Aditya Manthramurthy a30cfdd88f
Bump up madmin-go to v2 (#16162) 2022-12-06 13:46:50 -08:00
Anis Elleuch 932d2c3c62
Add X-Amz-Request-Id to internode calls (#16146) 2022-12-06 09:27:26 -08:00
Harshavardhana 419f351df3
avoid logging gzipped body in trace output (#16172) 2022-12-05 13:21:27 -08:00
Klaus Post a713aee3d5
Run staticcheck on CI (#16170) 2022-12-05 11:18:50 -08:00
Andreas Auernhammer d882ba2cb4
kms: add support for KES enclaves (#16139) 
Signed-off-by: Andreas Auernhammer <hi@aead.dev>
 2022-12-04 02:34:24 -08:00
Harshavardhana 9b35c72349
fix: a crash in KMS cert reload function (#16158) 2022-12-02 11:19:05 -08:00
Aditya Manthramurthy 87cbd41265
feat: Allow at most one claim based OpenID IDP (#16145) 2022-11-29 15:40:49 -08:00
Harshavardhana be92cf5959
change dependency from amqp -> amqp091 (RabbitMQ) official (#16142) 2022-11-28 16:05:06 -08:00
Anis Elleuch 1f1dcdce65
move HTTP recorder to an internal library (#16128) 2022-11-28 10:20:27 -08:00
Shireesh Anjal 98a67a3776
Improvements in logger and audit webhooks (#16102) 2022-11-28 08:03:26 -08:00
Andreas Auernhammer 9b1e70e4f9
kms: fix possible deadlock due to nested RLock calls. (#16136) 
Signed-off-by: Andreas Auernhammer <hi@aead.dev>
 2022-11-28 07:31:07 -08:00
Harshavardhana 09d4f8cd0f
avoid serializing decryptKey() every 15mins (#16135) 
if the certs are the same in an environment where the 
cert files are symlinks (e.g Kubernetes), then we resort
to reloading certs every 15mins - we can avoid reload
of the kes client instance. Ensure that the price to pay 
for contending with the lock must happen when necessary.
 2022-11-28 01:14:33 -08:00
Anis Elleuch 97eb7dbf5f
notify: Return detailed err msg when connecting to target fails (#16118) 2022-11-24 07:59:19 -08:00
Harshavardhana 08103870a5
update single drive setup error message (#16098) 2022-11-18 14:47:38 -08:00
Anis Elleuch 993e586855
config: return XMinioConfigNotFound code for non existing config (#16065) 2022-11-18 10:28:14 -08:00
Shireesh Anjal 5246e3be84
Send health diagnostics data as part of callhome (#16006) 2022-11-15 13:53:05 -08:00
Harshavardhana 962d1f1a71
choose default values upon incorrect storage_class value (#16058) 2022-11-12 10:18:21 -08:00
Harshavardhana 6d76db9d6c
improve server startup error when pools are incorrect (#16056) 2022-11-11 19:40:45 -08:00
Ray 66239f30ce
configuring the nats target to reconnect forever (#16050) 2022-11-11 12:42:41 -08:00
jiuker bf89f79694
save deploymentID to avoid mutating request entry in Audit (#16053) 2022-11-11 12:42:15 -08:00
Klaus Post 5b242f1d11
Add Audit target metrics (#16044) 2022-11-10 10:20:21 -08:00
Krishnan Parthasarathi 6eef9b4a23
lifecycle: simplify Eval and HasActiveRules (#16036) 2022-11-10 07:17:45 -08:00
Krishnan Parthasarathi 7ba281728f
ilm: fix x-amz-expiration header evaluation (#16029) 2022-11-09 04:20:34 -08:00
Klaus Post bbc312fce6
Add notification queue metrics (#16026) 2022-11-08 16:36:47 -08:00
Harshavardhana 1b0dfb0f58
remove printing map() checksums (#16028) 2022-11-08 13:29:24 -08:00
Aditya Manthramurthy 76d822bf1e
Add LDAP policy entities API (#15908) 2022-11-07 14:35:09 -08:00
Klaus Post ddeca9f12a
fix: filter rest errors and logs returned (#16019) 2022-11-07 10:38:08 -08:00
Harshavardhana 21251d8c22
initialize streaming events without lazy initialization (#16016) 2022-11-07 08:01:24 -08:00
Harshavardhana 944c62daf4
skip flaky tests on windows OS (#16015) 2022-11-07 00:11:21 -08:00
Harshavardhana 9547b7d0e9
add deadlineConnections on remoteTransport (#16010) 2022-11-05 11:09:21 -07:00
Harshavardhana 76c4ea7682
force all internal MinIO operations to be under UTC (#16009) 2022-11-04 16:44:38 -07:00
Harshavardhana 7be65f66b8
support HS256 series of JWT signature for OpenID connect (#15993) 2022-11-03 16:41:53 -07:00
Anis Elleuch 7721595aa9
config: Deprecated delay/max_wait/scanner and introduce speed (#15941) 2022-11-01 08:04:07 -07:00
Harshavardhana 4892a766a8
do not panic if webhook returns an error (#15970) 2022-10-30 16:45:53 -07:00
Poorna d765b89a63
improve validation for replication resync API (#15964) 2022-10-28 23:21:33 -07:00
Klaus Post 71954faa3a
mark pubsub type safe via generics (#15961) 2022-10-28 10:55:42 -07:00
Shireesh Anjal 6d22e74d11
mark SUBNET config keys as sensitive info (#15962) 2022-10-28 10:54:44 -07:00
Harshavardhana ec77d28e62
make subnet subsys dynamic and simplify callhome (#15927) 2022-10-27 00:20:01 -07:00
Harshavardhana 23b329b9df
remove gateway completely (#15929) 2022-10-24 17:44:15 -07:00
Poorna e4e90b53c1
fix: delete-marker replication check properly (#15923) 2022-10-21 14:45:06 -07:00
Anis Elleuch ac85c2af76
lifecycle: refactor rules filtering and tagging support (#15914) 2022-10-21 10:46:53 -07:00
Aditya Manthramurthy 2d16e74f38
Add LDAP IDP Configuration APIs (#15840) 2022-10-19 11:00:10 -07:00
Klaus Post bd3dfad8b9
Add concurrent Snowball extraction + options (#15836) 2022-10-18 13:50:21 -07:00
Aditya Manthramurthy 85fc7cea97
Pass role ARN for OIDC providers to console (#15862) 2022-10-15 12:57:03 -07:00
Harshavardhana 328d660106
support CRC32 Checksums on single drive setup (#15873) 2022-10-15 11:58:47 -07:00
Harshavardhana 59e33b3b21
validate setBucketTarget properly as per BucketExists() call (#15860) 2022-10-13 17:46:49 -07:00
Poorna 0e3c92c027 attempt delete marker replication after object is replicated (#15857) 
Ensure delete marker replication success, especially since the
recent optimizations to heal on HEAD, LIST and GET can force
replication attempts on delete marker before underlying object
version could have synced.
 2022-10-13 17:45:23 -07:00
hellivan b0b573052a
fix: unrecognized openid config parameters (#15847) 2022-10-12 12:19:44 -07:00
Javier Adriel 2939000342
Add metrics, version and apis handlers (#15839) 2022-10-12 12:08:03 -07:00
Anis Elleuch afd4279cd8
lock tests: Initialize different DRWMutex for each lock (#15833) 2022-10-10 15:14:32 -07:00
Harshavardhana 3c4ef4338f
marshal retention XML in expected format (#15821) 2022-10-08 13:11:00 -07:00
Aditya Manthramurthy 64cf887b28
use LDAP config from minio/pkg to share with console (#15810) 2022-10-07 22:12:36 -07:00
Krishnan Parthasarathi 6d6a731d6d
Handle overlapping and conflicting ILM rules (#15812) 2022-10-07 14:36:23 -07:00
Poorna b6b26dba87
fix: GetObjectRetention to parse in ISO8601 time format (#15809) 2022-10-06 13:53:56 -07:00
Harshavardhana 2a13cc28f2 feat: implement support batch replication (#15554) 2022-10-05 23:00:43 -07:00
Javier Adriel 3307aa1260
Implement KMS handlers (#15737) 2022-10-04 10:05:09 -07:00
Harshavardhana f696a221af
allow tagging policy condition for GetObject (#15777) 2022-10-02 12:29:29 -07:00
Anis Elleuch 6287e8c571
fix: race when accessing REST TCP dial values (#15770) 2022-09-29 09:27:58 -07:00
Anis Elleuch f69a98ce49
fix: loading Audit kafka configuration loading (#15766) 2022-09-29 08:35:08 -07:00
Daryl White d44f3526dc
Update links to documentation site (#15750) 2022-09-28 21:28:45 -07:00
Harshavardhana 41b633f5ea
support tagging based policy conditions (#15763) 2022-09-28 11:25:46 -07:00
Anis Elleuch 86bb48792c
non-blocking initialization of bucket target notifications (#15571) 2022-09-27 17:23:28 -07:00
Harshavardhana 94dbb4a427
fix: generalize SC config and also skip healing sub-sys under SD (#15757) 2022-09-26 09:04:54 -07:00
Anis Elleuch 048a46ec2a
Add RPC tcp timeout/errs and AVG duration to prometheus (#15747) 2022-09-26 09:04:26 -07:00
Harshavardhana 877bd95fa3
remove unused package internal/smart (#15758) 2022-09-25 08:44:53 -07:00
Poorna 8ea6fb368d
Add auto configuration of replication workers (#15636) 2022-09-24 16:20:28 -07:00
Poorna 5fd5ddea23
relax retention date validation on replication target (#15752) 2022-09-23 21:19:03 -07:00
Anis Elleuch 20c89ebbb3
freeze before exit when _MINIO_DEBUG_NO_EXIT is defined (#15709) 
this is to ensure keep k8s pods running, when they reach a "crashloop" stage
 2022-09-22 11:57:27 -07:00
Krishnan Parthasarathi 6f56ba80b3
lifecycle: Assign unique id to rules with empty id (#15731) 2022-09-22 10:51:54 -07:00
Klaus Post ff12080ff5
Remove deprecated io/ioutil (#15707) 2022-09-19 11:05:16 -07:00
Javier Adriel 0b6175b742
Implement KMS methods for keys, policies and identities (#15673) 2022-09-19 11:04:40 -07:00
Klaus Post c22f3ca7a8
fix: S3 Select CSV -> JSON with variable field count (#15677) 
When there are fewer fields than expected, output fewer fields.
 2022-09-12 17:00:59 -07:00
Anis Elleuch 4a92134235
prometheus: track errors during REST read/write calls (#15678) 
minio_inter_node_traffic_errors_total currently does not track
requests body write/read errors of internode REST communications.

This commit fixes this by wrapping resp.Body.
 2022-09-12 12:40:51 -07:00
Poorna 6b9fd256e1
Persist in-memory replication stats to disk (#15594) 
to avoid relying on scanner-calculated replication metrics.
This will improve the accuracy of the replication stats reported.

This PR also adds on to #15556 by handing replication
traffic that could not be queued by available workers to the 
MRF queue so that entries in `PENDING` status are healed faster.
 2022-09-12 12:40:02 -07:00
Harshavardhana 228c6686f8
allow non-standards fallback for all http.TimeFormats (#15662) 
fixes #15645
 2022-09-07 07:24:54 -07:00
Krishnan Parthasarathi 5ce1448049
audit: account for response headers separately (#15610) 2022-09-01 12:51:04 -07:00
Klaus Post 8e4a45ec41
fix: encrypt checksums in metadata (#15620) 2022-08-31 08:13:23 -07:00
Abirdcfly d4e0f13bb3
chore: remove duplicate word in comments (#15607) 
Signed-off-by: Abirdcfly <fp544037857@gmail.com>

Signed-off-by: Abirdcfly <fp544037857@gmail.com>
 2022-08-30 08:26:43 -07:00
Anis Elleuch 1f28a3bb80
Avoid messages from go test output (#15601) 
A lot of warning messages are printed in CI/CD failures generated by go
test. Avoid that by requiring at least Error level for logging when
doing go test.
 2022-08-30 08:23:40 -07:00
Klaus Post a9f1ad7924
Add extended checksum support (#15433) 2022-08-29 16:57:16 -07:00
Harshavardhana 97376f6e8f
improve performance for inlined data (#15603) 
inlined data often is bigger than the allowed
O_DIRECT alignment, so potentially we can write
'xl.meta' without O_DSYNC instead we can rely on
O_DIRECT + fdatasync() instead.

This PR allows O_DIRECT on inlined data that
would gain the benefits of performing O_DIRECT,
eventually performing an fdatasync() at the end.

Performance boost can be observed here for small
objects < 128KiB. The performance boost is mainly
seen on HDD, and marginal on NVMe setups.
 2022-08-29 11:19:29 -07:00
Harshavardhana 433b6fa8fe
upgrade golang-lint to the latest (#15600) 2022-08-26 12:52:29 -07:00
Krishnan Parthasarathi 99fbfe2421
Add concurrency to healing objects on a fresh disk (#15575) 2022-08-25 13:07:15 -07:00
Aditya Manthramurthy 18dffb26e7
Allow querying a single target in config get API (#15587) 2022-08-25 00:17:05 -07:00
Aditya Manthramurthy c54c13831a
fix: error types in internal config package (#15559) 2022-08-20 15:11:49 -07:00
Harshavardhana ae4ee95d25
change default lock retry interval to 50ms (#15560) 
competing calls on the same object on versioned bucket
mutating calls on the same object may unexpected have
higher delays.

This can be reproduced with a replicated bucket
overwriting the same object writes, deletes repeatedly.

For longer locks like scanner keep the 1sec interval
 2022-08-19 16:21:05 -07:00
Aditya Manthramurthy 21831b3fe2
fix: cleanup config KV parsing using madmin helpers (#15552) 2022-08-18 15:55:17 -07:00
Aditya Manthramurthy 701a82642b
Ensure subnet+callhome configs are single sub-sys (#15542) 2022-08-16 17:46:46 -07:00
Harshavardhana 48640b1de2
fix: trim arn:aws:kms from incoming SSE aws-kms-key-id (#15540) 2022-08-16 11:28:30 -07:00
Anis Elleuch 5682685c80
Introduce disk io stats metrics (#15512) 2022-08-16 07:13:49 -07:00
Aditya Manthramurthy ee2afcf70b
Move some config constants to madmin (#15529) 2022-08-13 14:50:16 -07:00
Aditya Manthramurthy 9986e103cf
Fix env var output in config get/export APIs (#15528) 
Fix a bug where env vars are not output when the config for the
subsystem is specified solely via env vars.
 2022-08-13 10:39:01 -07:00
Shireesh Anjal 316c492842
Upgrade madmin-go to latest version (v1.4.15) (#15510) 2022-08-10 07:36:13 -07:00
Harshavardhana a406bb0288
restrict number of disks used for scanning buckets upto GOMAXPROCS (#15492) 
control scanner parallelism to avoid higher CPU
usage on nodes that have more drives but an old CPU.
 2022-08-08 16:16:44 -07:00
Harshavardhana 1823ab6808
LDAP/OpenID must be initialized IAM Init() (#15491) 
This allows for LDAP/OpenID to be non-blocking,
allowing for unreachable Identity targets to be
initialized in IAM.
 2022-08-08 16:16:27 -07:00
Aditya Manthramurthy 3d94c38ec4
Add env variables to configuration APIs output (#15465) 
Config export and config get APIs now include environment 
variables set on the server
 2022-08-04 22:21:52 -07:00
ebozduman b57e7321e7
Replaces 'disk'=>'drive' visible to end user (#15464) 2022-08-04 16:10:08 -07:00
Andreas Auernhammer d774a3309b
kes: automatically reload KES client certificate (#15450) 
This commit adds support for automatically reloading
the MinIO client certificate for authentication to KES.

The client certificate will now be reloaded:
 - when the private key / certificate file changes
 - when a SIGHUP signal is received
 - every 15 minutes

Fixes #14869

Signed-off-by: Andreas Auernhammer <hi@aead.dev>
 2022-08-02 16:58:09 -07:00
Harshavardhana 026b87e39b
do not crash for unwrapErrs return `nil` (#15456) 
fixes #15454
 2022-08-02 15:10:11 -07:00
Harshavardhana fd349103e8
fix: allow P-384/P-512 constant time implementation (#15445) 
since go1.18.x P-384/P-512 are now constant time
implementations, enable them.
 2022-08-01 09:27:16 -07:00
Anis Elleuch 10b49eb4fb
Fix resetting a config with a non default target name (#15448) 
mc admin config reset <alias> notify_webhook:something was not working
properly.

The reason is that GetSubSys() was not calculating the target
name properly because it is quitting early when the number of config
inputs ('notify_webhook:something' in this case) is equal to 1.

This commit will make the code calculates always calculate the target
name if found.
 2022-08-01 07:52:23 -07:00
Anis Elleuch 3856d078d2
fix: set 20000 as maximum parallel event calls (#15435) 
This is needed to avoid consuming a lot of goroutines when a target is
very slow or there is a bug in a target library.
 2022-07-30 12:12:33 -07:00
Harshavardhana 5e763b71dc
use logger.LogOnce to reduce printing disconnection logs (#15408) 
fixes #15334

- re-use net/url parsed value for http.Request{}
- remove gosimple, structcheck and unusued due to https://github.com/golangci/golangci-lint/issues/2649
- unwrapErrs upto leafErr to ensure that we store exactly the correct errors
 2022-07-27 09:44:59 -07:00
jiuker 6b4f833a12
convert repeated error checks into single function in logger (#15387) 2022-07-25 17:53:03 -07:00
Eng Zer Jun 0a3b1ad4eb
test: use `T.TempDir` to create temporary test directory (#15400) 
This commit replaces `ioutil.TempDir` with `t.TempDir` in tests. The
directory created by `t.TempDir` is automatically removed when the test
and all its subtests complete.

Prior to this commit, temporary directory created using `ioutil.TempDir`
needs to be removed manually by calling `os.RemoveAll`, which is omitted
in some tests. The error handling boilerplate e.g.
	defer func() {
		if err := os.RemoveAll(dir); err != nil {
			t.Fatal(err)
		}
	}
is also tedious, but `t.TempDir` handles this for us nicely.

Reference: https://pkg.go.dev/testing#T.TempDir

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
 2022-07-25 12:37:26 -07:00
Klaus Post 3795b2c8ba
Add compression scheme to header (#15395) 
For easier debugging. We still do not return compressed size for security reasons.
 2022-07-24 07:15:49 -07:00
Taran Pelkey e83930333b
Allow DelKVS to delete specific sub-system fields. (#15354) 2022-07-22 14:48:23 -07:00
Harshavardhana b0d70a0e5e
support additional claim info in Auditing STS calls (#15381) 
Bonus: Adds a missing AuditLog from AssumeRoleWithCertificate API

Fixes #9529
 2022-07-22 11:12:03 -07:00
Aditya Manthramurthy 2393a13f86
Allow site replication config with multiple IDPs (#15361) 
Fixes a bug that did not let site replication be configured when
multiple IDPs are configured.
 2022-07-21 19:52:23 -07:00
Harshavardhana 8249cd4406
fix: allow payload verification error to be returned (#15364) 
without reading the reader the error is ignored
by the custom unmarshaller written by ObjectLegalHold
data structure.
 2022-07-21 01:24:03 -07:00
Andreas Auernhammer 242d06274a
kms: add `context.Context` to KMS API calls (#15327) 
This commit adds a `context.Context` to the
the KMS `{Stat, CreateKey, GenerateKey}` API
calls.

The context will be used to terminate external calls
as soon as the client requests gets canceled.

A follow-up PR will add a `context.Context` to
the remaining `DecryptKey` API call.

Signed-off-by: Andreas Auernhammer <hi@aead.dev>
 2022-07-18 18:54:27 -07:00
Harshavardhana b6eb8dff64
Add decommission compression+encryption enabled tests (#15322) 
update compression environment variables to follow
the expected sub-system style, however support fallback
mode.
 2022-07-17 08:43:14 -07:00
Harshavardhana 785b429737
add reconnect duration allows for verifying disconnect intervals (#15306) 2022-07-15 14:41:24 -07:00
Poorna 53cc561048
Default DeleteReplication rule status if unspecified. (#15301) 
Since this is a MinIO specific extension in the replication config,
default this to Disabled to allow other sdks to be used to configure
replication rules.

Co-authored-by: Poorna Krishnamoorthy <poorna@minio.io>
 2022-07-14 16:27:09 -07:00
LHHDZ df911c9b9e
correct `RefreshCall` & `UnlockCall` of `DefaultTimeouts` (#15288) 2022-07-14 07:20:48 -07:00
Klaus Post 0149382cdc
Add padding to compressed+encrypted files (#15282) 
Add up to 256 bytes of padding for compressed+encrypted files.

This will obscure the obvious cases of extremely compressible content 
and leave a similar output size for a very wide variety of inputs.

This does *not* mean the compression ratio doesn't leak information 
about the content, but the outcome space is much smaller, 
so often *less* information is leaked.
 2022-07-13 07:52:15 -07:00
Harshavardhana 0a8b78cb84
fix: simplify passing auditLog eventType (#15278) 
Rename Trigger -> Event to be a more appropriate
name for the audit event.

Bonus: fixes a bug in AddMRFWorker() it did not
cancel the waitgroup, leading to waitgroup leaks.
 2022-07-12 10:43:32 -07:00
Andreas Auernhammer f800cee4fa
metric: add KMS-related metrics (#15258) 
This commit adds a minimal set of KMS-related metrics:
```
 # HELP minio_cluster_kms_online Reports whether the KMS is online (1) or offline (0)
 # TYPE minio_cluster_kms_online gauge
 minio_cluster_kms_online{server="127.0.0.1:9000"} 1
 # HELP minio_cluster_kms_request_error Number of KMS requests that failed with a well-defined error
 # TYPE minio_cluster_kms_request_error counter
 minio_cluster_kms_request_error{server="127.0.0.1:9000"} 16790
 # HELP minio_cluster_kms_request_success Number of KMS requests that succeeded
 # TYPE minio_cluster_kms_request_success counter
 minio_cluster_kms_request_success{server="127.0.0.1:9000"} 348031
```

Currently, we report whether the KMS is available and how many requests
succeeded/failed. However, KES exposes much more metrics that can be
exposed if necessary. See: https://pkg.go.dev/github.com/minio/kes#Metric

Signed-off-by: Andreas Auernhammer <hi@aead.dev>
 2022-07-11 09:17:28 -07:00
Ray 3bdb92fcad
Adding error check for jetstream connection (#15252) 2022-07-07 23:14:47 -07:00
Harshavardhana 32b2f6117e
fix: do not pass around sync.Map (#15250) 
it is not safe to pass around sync.Map
through pointers, as it may be concurrently
updated by different callers.

this PR simplifies by avoiding sync.Map
altogether, we do not need sync.Map
to keep object->erasureMap association.

This PR fixes a crash when concurrently
using this value when audit logs are
configured.

```
fatal error: concurrent map iteration and map write

goroutine 247651580 [running]:
runtime.throw({0x277a6c1?, 0xc002381400?})
        runtime/panic.go:992 +0x71 fp=0xc004d29b20 sp=0xc004d29af0 pc=0x438671
runtime.mapiternext(0xc0d6e87f18?)
        runtime/map.go:871 +0x4eb fp=0xc004d29b90 sp=0xc004d29b20 pc=0x41002b
```
 2022-07-07 17:04:25 -07:00
Harshavardhana dd839bf295
add NATS JetStream support (#15201) 2022-07-06 13:29:08 -07:00