MyFavMirrors
/
minio
mirror of https://github.com/minio/minio.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
11,342 Commits 5 Branches 503 Tags 186 MiB
Commit Graph

604 Commits

Author SHA1 Message Date
Aditya Manthramurthy f3248a4b37
Redact all secrets from config viewing APIs (#17380) 
This change adds a `Secret` property to `HelpKV` to identify secrets
like passwords and auth tokens that should not be revealed by the server
in its configuration fetching APIs. Configuration reporting APIs now do
not return secrets.
 2023-06-23 07:45:27 -07:00
Harshavardhana 74759b05a5
make sure to set relevant config entries correctly (#17485) 
Bonus: also allow skipping keys properly.
 2023-06-22 10:04:02 -07:00
Praveen raj Mani 7c72b25ef0
Add an option to make bucket notifications synchronous (#17406) 
With the current asynchronous behaviour in sending notification events
to the targets, we can't provide guaranteed delivery as the systems
might go for restarts.

For such event-driven use-cases, we can provide an option to enable
synchronous events where the APIs wait until the event is successfully
sent or persisted.

This commit adds 'MINIO_API_SYNC_EVENTS' env which when set to 'on'
will enable sending/persisting events to targets synchronously.
 2023-06-20 17:38:59 -07:00
Aditya Manthramurthy 5a1612fe32
Bump up madmin-go and pkg deps (#17469) 2023-06-19 17:53:08 -07:00
Harshavardhana 22b7c8cd8a
upgrade pkg and dperf to latest packages (#17448) 
- dperf improvements in benchmarking read and write tests
- upgrade mimedb to use latest content-types
 2023-06-17 07:31:36 -07:00
jiuker 0474791cf8
fix: set time format right (#17402) 2023-06-14 07:49:13 -07:00
Harshavardhana f32efd5429
more compliance related fixes (#17408) 
- lifecycle must return InvalidArgument for rule errors
- do not return `null` versionId in HTTP header
- reject mixed SSE uploads with correct error message
 2023-06-13 13:52:33 -07:00
Anis Eleuch 0f0dcf0c5e
tar: Avoid storing snowball extraction header in extract objects (#17389) 2023-06-12 09:42:06 -07:00
Anis Eleuch bb24346e04
listen: Only error out if not able to bind any interface (#17353) 2023-06-12 09:09:28 -07:00
Harshavardhana b829e80ecb
do not disable root for invalid API config values (#17386) 2023-06-08 15:50:06 -07:00
Klaus Post 6e38d0f3ab
Add more bootstrap info in debug mode (#17362) 2023-06-08 08:39:47 -07:00
Harshavardhana dbd4c2425e
fix: kafka broker pings must not be greater than 1sec (#17376) 2023-06-07 11:47:00 -07:00
Anis Eleuch eba378e4a1
vrf: Fix testing for loopback coming from the address (#17372) 2023-06-07 09:53:05 -07:00
Harshavardhana 2f9e2147f5
allow quota enforcement to rely on older values (#17351) 
PUT calls cannot afford to have large latency build-ups due
to contentious usage.json, or worse letting them fail with
some unexpected error, this can happen when this file is
concurrently being updated via scanner or it is being
healed during a disk replacement heal.

However, these are fairly quick in theory, stressed clusters
can quickly show visible latency this can add up leading to
invalid errors returned during PUT.

It is perhaps okay for us to relax this error return requirement
instead, make sure that we log that we are proceeding to take in
the requests while the quota is using an older value for the quota
enforcement. These things will reconcile themselves eventually,
via scanner making sure to overwrite the usage.json.

Bonus: make sure that storage-rest-client sets ExpectTimeouts to
be 'true', such that DiskInfo() call with contextTimeout does
not prematurely disconnect the servers leading to a longer
healthCheck, back-off routine. This can easily pile up while also
causing active callers to disconnect, leading to quorum loss.

DiskInfo is actively used in the PUT, Multipart call path for
upgrading parity when disks are down, it in-turn shouldn't cause
more disks to go down.
 2023-06-05 16:56:35 -07:00
jiuker 8030e12ba5
fix: expMovingAvg is too small when startTime is zero (#17346) 2023-06-03 13:41:51 -07:00
jiuker fb5ce3b87a
record err time when remote node is offline (#17262) 2023-05-30 10:07:26 -07:00
Klaus Post 6fe028b7c5
Revert s3 select simdjson reuse (#17310) 2023-05-30 10:02:22 -07:00
Anis Eleuch 54c5c88fe6
Add number of offline disks in quorum errors (#16822) 2023-05-25 09:39:06 -07:00
jiuker 443250d135
fix: for Target isActive use net.Dial instead (#17251) 2023-05-25 09:24:11 -07:00
jiuker d749aaab69
fix: ignore existing target status when adding new targets (#17250) 2023-05-24 22:57:37 -07:00
Krishnan Parthasarathi 62df731006
Add updatedAt for GetBucketLifecycleConfig (#17271) 2023-05-24 22:52:39 -07:00
Harshavardhana d0a0eb9738
support fan-out objects via PostUpload() (#17233) 2023-05-24 22:51:07 -07:00
Klaus Post 5677f73794
Add PostObject Checksum (#17244) 2023-05-23 07:58:33 -07:00
Krishnan Parthasarathi 55a3310446
logger-http: Don't retry after a succesful send (#17266) 2023-05-22 14:53:18 -07:00
Harshavardhana fc03be7891
simplify bucket metadata lookups for versioning/object locking (#17253) 2023-05-22 12:05:14 -07:00
jiuker b1b00a5055
fix: Avoid Income globalStats twice upon error (#17263) 2023-05-22 07:42:27 -07:00
jiuker 41fa8fa2d2
fix: increment counter when entry be skipped (#17237) 2023-05-19 08:36:52 -07:00
jiuker e94e6adf91
fix: return proper error if OIDC Discoverydoc fails to respond (#17242) 2023-05-19 02:13:33 -07:00
Klaus Post b06d7bf834
fix: leaking connections in JSON SQL with limited return (#17239) 2023-05-18 11:26:46 -07:00
Aditya Manthramurthy 9d96b18df0
Add "name" and "description" params to service acc (#17172) 2023-05-17 17:05:36 -07:00
Praveen raj Mani 85912985b6
Check for only network errors in audit webhook for reachability (#17228) 2023-05-17 11:10:33 -07:00
Klaus Post aaf1abc993
simplify HardLimitReader by using LimitReader for internal usage (#17218) 2023-05-16 13:14:37 -07:00
Harshavardhana ef2fc0f99e
fix: reduce using memory and temporary files. (#17206) 2023-05-15 14:08:54 -07:00
Anis Eleuch 684399433b
lock: Retry locking with an increasing random interval (#17200) 2023-05-13 08:42:21 -07:00
Poorna e07c2ab868
Use hash.NewLimitReader for internal multipart calls (#17191) 2023-05-12 11:19:08 -07:00
Klaus Post 99c4ffa34f
fix: avoid audit log race protection deadlocks (#17168) 2023-05-09 08:11:32 -07:00
Harshavardhana a7f266c907
allow JWT parsing on large session policy based tokens (#17167) 2023-05-09 00:53:08 -07:00
Praveen raj Mani 57acacd5a7
Support persistent queue store for loggers (#17121) 2023-05-08 21:20:31 -07:00
Klaus Post 76913a9fd5
Signed trailers for signature v4 (#16484) 2023-05-05 19:53:12 -07:00
Harshavardhana 5569acd95c
disallow EC:0 if not set during server startup (#17141) 2023-05-04 14:44:30 -07:00
Alex 6e24dff26a
Added MINIO_BROWSER_LOGIN_ANIMATION env support for WebUI console (#17123) 
Signed-off-by: Benjamin Perez <benjamin@bexsoft.net>
 2023-05-03 15:32:50 -07:00
Harshavardhana 9571b0825e
add configurable VRF interface and user-timeout (#17108) 2023-05-03 14:12:25 -07:00
Krishnan Parthasarathi 0ec722bc54
Add tags to NewerNoncurrentVersions audit event (#17110) 2023-05-02 12:56:33 -07:00
Anis Eleuch 4640b13c66
Use expontential backoff algo for internode reconnections (#17052) 2023-05-02 12:35:52 -07:00
Praveen raj Mani 1704abaf6b
fix: store notification events immediately for persistent queues (#17112) 2023-05-02 07:53:13 -07:00
Klaus Post e8c0a50862
optimization use small blocks up to 64KB (#17107) 2023-05-01 09:47:49 -07:00
Harshavardhana 7ae69accc0
allow root user to be disabled via config settings (#17089) 2023-04-28 12:24:14 -07:00
Klaus Post 7fad0c8b41
Remove checksums from HTTP range request, add part checksums (#17105) 2023-04-28 08:26:32 -07:00
jiuker 6e27264c6b
update cleanupRoutine comment (#17102) 2023-04-28 01:11:51 -07:00
Anis Eleuch 5c83c9724f
audit: Add request path and host to audit event (#17099) 2023-04-27 22:18:24 -07:00
jiuker b28d391a22
fix: add correct worker count before startHTTPLogger() (#17091) 2023-04-27 10:51:16 -07:00
jiuker c8b92f6067
protect wg.Done from being called twice (#17075) 2023-04-27 07:55:36 -07:00
Anis Eleuch 31b5acc245
tcp: Increase user timeout to 10 minutes (#17087) 2023-04-26 17:48:31 -07:00
Anis Eleuch 0b7ca094e4
Remove Expect 100-continue in internode communications (#17061) 2023-04-26 09:33:45 -07:00
Praveen raj Mani 72802a5972
Use 'minio/pkg/sync/errgroup' and 'minio/pkg/workers' (#17069) 2023-04-25 22:57:40 -07:00
Harshavardhana 8fd07bcd51
simplify sort.Sort by using sort.Slice (#17066) 2023-04-24 13:28:18 -07:00
Harshavardhana 8a9b9832fd
add Dial timeout for Kafka broker pings (#17044) 2023-04-17 15:45:01 -07:00
jiuker e96c88e914
fix: DeleteBucketThrottle must delete ARN (#17034) 2023-04-15 02:14:26 -07:00
Klaus Post c133979b8e
Add part count to checksum (#17035) 2023-04-14 09:44:45 -07:00
Harshavardhana a5835cecbf
fix: regression in counting total requests (#17024) 2023-04-12 14:37:19 -07:00
Poorna d350654aee
config: fix duplication of replication priority key (#17014) 2023-04-11 19:22:10 -07:00
Harshavardhana 09a25ea7b7 lint: fix some lint issues on files 2023-04-06 22:42:10 -07:00
Harshavardhana 47b7469a60
add buffer pool for proxy forwarder (#16942) 2023-04-06 15:54:12 -07:00
Praveen raj Mani 51f7f9aaa3
Generalize the event store using go generics (#16910) 2023-04-04 10:52:24 -07:00
Anis Eleuch d90d0c8931
Use one http response recorder per external http call (#16938) 2023-03-31 09:37:29 -07:00
Anis Eleuch c259a8ea38
Set tcp user timeout to clean sockets with data in the buffer (#16887) 2023-03-24 08:10:58 -07:00
mstmdev 2d51e42305
Remove the redundant conditional in the validateParity function (#16866) 2023-03-23 14:06:22 -07:00
Anis Eleuch 1346561b9d
return quorum error instead of insufficient storage error (#16874) 2023-03-22 16:22:37 -07:00
Klaus Post 11d04279c8
Add lazy init of audit logger (#16842) 2023-03-21 10:50:40 -07:00
Harshavardhana 3b5dbf9046
allow bootstrapping to validate internode tokens (#16853) 2023-03-20 01:40:24 -07:00
Aditya Manthramurthy 09c733677a
Add test for fixed post policy exploit (#16855) 2023-03-20 01:06:45 -07:00
Harshavardhana 46f9049fb4
simplify error responses for KMS (#16793) 2023-03-16 11:59:42 -07:00
Aditya Manthramurthy 58266c9e2c
Add enable flag for LDAP IDP config (#16805) 2023-03-16 11:58:59 -07:00
Nitish Tiwari 50dbd2cacc
Update audit log flow to use new headers with unit (#16797) 2023-03-13 22:50:19 -07:00
Harshavardhana b984bf8d1a
allow expiration of all versions during Listing() (#16757) 2023-03-09 15:15:30 -08:00
Harshavardhana 901887e6bf
feat: add lambda transformation functions target (#16507) 2023-03-07 08:12:41 -08:00
ferhat elmas 714283fae2
cleanup ignored static analysis (#16767) 2023-03-06 08:56:10 -08:00
ferhat elmas 3423028713
cleanup Go linter settings (#16736) 2023-03-04 20:57:35 -08:00
Harshavardhana 0ff931dc76
fix: allow CORS to work by default (#16713) 2023-02-27 10:10:45 -08:00
Praveen raj Mani 4d708cebe9
Support adding service accounts with expiration (#16430) 
Co-authored-by: Harshavardhana <harsha@minio.io>
 2023-02-27 10:10:22 -08:00
Aditya Manthramurthy 8cde38404d
Add metrics for custom auth plugin (#16701) 2023-02-27 09:55:18 -08:00
Shubhendu 8b4eb2304b
Set logger webhook proxy on subnet proxy change (#16665) 
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
 2023-02-27 08:35:36 -08:00
Anis Elleuch 8da0f4c5bb
Better error message when TLS certs do not have proper permissions (#16703) 2023-02-24 06:34:55 -08:00
Harshavardhana b21d3f9b82
event target registration failures must be returned (#16700) 2023-02-23 21:59:14 +05:30
Harshavardhana 5c98223c89
add correct HostId instead of deploymentId for error responses (#16686) 2023-02-22 15:41:09 +05:30
Daniel Valdivia fb17f97cf3
move audit and logger message structure to minio/pkg (#16655) 
Signed-off-by: Daniel Valdivia <18384552+dvaldivia@users.noreply.github.com>
 2023-02-21 21:21:17 -08:00
Shubhendu 6b65ba1551
Added attribute proxy for `mc admin config set ALIAS logger_webhook` (#16657) 
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
 2023-02-21 21:19:46 -08:00
Allan Roger Reid 8bfe972bab
Set meaningful message from minio with env variable KMS_SECRET_KEY (#16584) 2023-02-22 07:13:01 +05:30
Klaus Post fd6622458b
Add detailed scanner trace output and notifications (#16668) 2023-02-21 09:33:33 -08:00
jiuker e470268c7c
fix: a possible closer leak in SelectObjectHandler (#16598) 2023-02-17 01:44:40 -08:00
Harshavardhana 6ea150fd68
fix: avoid printing certain errors under few locations (#16631) 2023-02-17 01:40:31 -08:00
Harshavardhana e1e9ddd4a4
use kes.Status() for Status() call (#16629) 2023-02-16 22:12:24 +05:30
Andreas Auernhammer 74887c7372
kms: add support for KES API keys and switch to KES Go SDK (#16617) 
Signed-off-by: Andreas Auernhammer <hi@aead.dev>
 2023-02-14 07:19:20 -08:00
Harshavardhana 31188e9327
add parallel workers in batch replication (#16609) 2023-02-13 12:07:58 -08:00
Harshavardhana d65debb6bc
fix: comply with RFC6750 UserInfo endpoint requirements (#16592) 2023-02-10 22:20:25 +05:30
Anis Elleuch c8ffa59d28
Periodically refresh buckets metadata from the backend disks (#16561) 
fixes #16553
 2023-02-09 10:29:20 -08:00
Anis Elleuch fadc46b906
Add the access key and parent user in the audit log (#16572) 2023-02-08 11:05:26 -08:00
Anis Elleuch 095b518802
Show a better error msg when internal data encryption key is incorrect (#16549) 2023-02-07 05:22:54 -08:00
Harshavardhana 14cf8f1b22
upgrade deps for minio/pkg v1.6.1 to include groups conditions (#16538) 2023-02-06 09:27:29 -08:00
Harshavardhana 5996c8c4d5
feat: allow offline disks on a fresh start (#16541) 2023-02-06 09:26:09 -08:00
Harshavardhana aa8b9572b9
remove double ENABLED help output (#16528) 2023-02-03 05:52:52 -08:00
Klaus Post 0b7989aa4b
Fix Kafka initialization crash (#16523) 2023-02-02 19:21:19 +05:30
Harshavardhana d19cbc81b5
fix: do not return IAM/Bucket metadata replication errors to client (#16486) 2023-01-26 11:11:54 -08:00
Harshavardhana 65c104a589
add x-amz-id-2 to indicate the node that received the request (#16474) 2023-01-25 09:14:10 -08:00
Harshavardhana 3683673fb0
add missing gorilla/mux migration, update credits (#16461) 2023-01-23 08:46:37 -08:00
Poorna 1b02e046c2
Fix bandwidth monitoring to be per remote target (#16360) 2023-01-19 18:52:16 +05:30
Aditya Manthramurthy 698862ec5d
Fix transports/timeouts related regressions (#16427) 2023-01-18 10:06:38 +05:30
Harshavardhana 3db658e51e
use correct xml package for custom MarshalXML() (#16421) 2023-01-17 05:08:33 +05:30
Allan Roger Reid 9815dac48f
fix: allow bind on ipv6 loopback failures (#16388) 2023-01-11 08:47:39 +05:30
Anis Elleuch 1ece3d1dfe
Add comment field to service accounts (#16380) 2023-01-10 21:57:52 +04:00
Anis Elleuch 2146ed4033
xl: Quit early when EC config is incorrect (#16390) 
Co-authored-by: Anis Elleuch <anis@min.io>
 2023-01-09 23:07:45 -08:00
Anis Elleuch 1c85652cff
lint: Fix in darwin environment (#16368) 2023-01-05 10:12:01 -08:00
Klaus Post 6a04067514
fix: tweak read buffer size to reduce over-reading (#16338) 2023-01-01 08:14:20 -08:00
Harshavardhana 75faef888e
disable builds for go1.18 (#16332) 2022-12-30 11:37:07 -08:00
Harshavardhana b67d97b1ba
add missing fields in audit logs for non-compressed handlers (#16328) 2022-12-30 10:20:19 -08:00
Anis Elleuch acc9c033ed
debug: Add X-Amz-Request-ID to lock/unlock calls (#16309) 2022-12-23 19:49:07 -08:00
Anis Elleuch e57e946206
Do not save credentials in config.json (#16275) 2022-12-19 12:27:06 -08:00
Harshavardhana 80fc3a8a52
use newDynamicTimeoutWithOpts() when appropriate (#16266) 2022-12-15 13:11:37 -08:00
Klaus Post 988a2e8fed
Faster startup of large distributed systems with latency (#16259) 2022-12-15 08:31:21 -08:00
Harshavardhana bad229e16e
fix: support event name s3:Restore:* (#16257) 2022-12-14 05:12:07 -08:00
Harshavardhana c73ea27ed7
do not log checksum mismatch error, client received the error (#16246) 2022-12-14 01:57:40 -08:00
Aditya Manthramurthy 9e6cc847f8
Add HTTP2 config option for policy plugin (#16225) 2022-12-13 14:28:48 -08:00
Anis Elleuch 939c0100a6
log: Do not interpret verbs in object names in console output (#16233) 2022-12-13 08:27:40 -08:00
Aditya Manthramurthy 2d60bf8c50
Refactor HTTP transports (#16222) 2022-12-12 20:31:21 -08:00
Aditya Manthramurthy a469e6768d
Add LDAP DNS SRV record lookup support (#16201) 2022-12-12 11:27:27 -08:00
Harshavardhana 2fc182d8e6
fix: iso8601TimeFormat padding issue for certain nanoseconds (#16207) 2022-12-12 10:28:30 -08:00
Aditya Manthramurthy e06127566d
Add IAM API to attach/detach policies for LDAP (#16182) 2022-12-09 13:08:33 -08:00
jiuker 8edc2faaa9
reuse sha256 in config GetSettings (#16188) 2022-12-08 03:03:24 -08:00
Javier Adriel 04ae9058ed
Populate end_session_endpoint (#16183) 2022-12-06 16:56:37 -08:00
Aditya Manthramurthy a30cfdd88f
Bump up madmin-go to v2 (#16162) 2022-12-06 13:46:50 -08:00
Anis Elleuch 932d2c3c62
Add X-Amz-Request-Id to internode calls (#16146) 2022-12-06 09:27:26 -08:00
Harshavardhana 419f351df3
avoid logging gzipped body in trace output (#16172) 2022-12-05 13:21:27 -08:00
Klaus Post a713aee3d5
Run staticcheck on CI (#16170) 2022-12-05 11:18:50 -08:00
Andreas Auernhammer d882ba2cb4
kms: add support for KES enclaves (#16139) 
Signed-off-by: Andreas Auernhammer <hi@aead.dev>
 2022-12-04 02:34:24 -08:00
Harshavardhana 9b35c72349
fix: a crash in KMS cert reload function (#16158) 2022-12-02 11:19:05 -08:00
Aditya Manthramurthy 87cbd41265
feat: Allow at most one claim based OpenID IDP (#16145) 2022-11-29 15:40:49 -08:00
Harshavardhana be92cf5959
change dependency from amqp -> amqp091 (RabbitMQ) official (#16142) 2022-11-28 16:05:06 -08:00
Anis Elleuch 1f1dcdce65
move HTTP recorder to an internal library (#16128) 2022-11-28 10:20:27 -08:00
Shireesh Anjal 98a67a3776
Improvements in logger and audit webhooks (#16102) 2022-11-28 08:03:26 -08:00
Andreas Auernhammer 9b1e70e4f9
kms: fix possible deadlock due to nested RLock calls. (#16136) 
Signed-off-by: Andreas Auernhammer <hi@aead.dev>
 2022-11-28 07:31:07 -08:00
Harshavardhana 09d4f8cd0f
avoid serializing decryptKey() every 15mins (#16135) 
if the certs are the same in an environment where the 
cert files are symlinks (e.g Kubernetes), then we resort
to reloading certs every 15mins - we can avoid reload
of the kes client instance. Ensure that the price to pay 
for contending with the lock must happen when necessary.
 2022-11-28 01:14:33 -08:00
Anis Elleuch 97eb7dbf5f
notify: Return detailed err msg when connecting to target fails (#16118) 2022-11-24 07:59:19 -08:00
Harshavardhana 08103870a5
update single drive setup error message (#16098) 2022-11-18 14:47:38 -08:00
Anis Elleuch 993e586855
config: return XMinioConfigNotFound code for non existing config (#16065) 2022-11-18 10:28:14 -08:00
Shireesh Anjal 5246e3be84
Send health diagnostics data as part of callhome (#16006) 2022-11-15 13:53:05 -08:00
Harshavardhana 962d1f1a71
choose default values upon incorrect storage_class value (#16058) 2022-11-12 10:18:21 -08:00
Harshavardhana 6d76db9d6c
improve server startup error when pools are incorrect (#16056) 2022-11-11 19:40:45 -08:00
Ray 66239f30ce
configuring the nats target to reconnect forever (#16050) 2022-11-11 12:42:41 -08:00
jiuker bf89f79694
save deploymentID to avoid mutating request entry in Audit (#16053) 2022-11-11 12:42:15 -08:00
Klaus Post 5b242f1d11
Add Audit target metrics (#16044) 2022-11-10 10:20:21 -08:00