c27d0583d4
Send kafka notification messages in batches when queue_dir is enabled ( #18164 )
...
Fixes #18124
2023-10-07 08:07:38 -07:00
603437e70f
Fix startup formatting ( #18156 )
...
Percentages in root user names are used for formatting.
Before:
```
S3-API: http://192.168.50.21:9000 http://172.31.96.1:9000 http://127.0.0.1:9000
RootUser: "U4B6Zi!b75DXSPm%!!(MISSING)a(MISSING)vZb"
RootPass: "Q4#Q6y8G%!P(MISSING)x#npP4dudUobU#NBcGB7RMKV4ajYb"
Console: http://192.168.50.21:51915 http://172.31.96.1:51915 http://127.0.0.1:51915
RootUser: "U4B6Zi!b75DXSPm%!!(MISSING)a(MISSING)vZb"
RootPass: "Q4#Q6y8G%!P(MISSING)x#npP4dudUobU#NBcGB7RMKV4ajYb"
Command-line: https://min.io/docs/minio/linux/reference/minio-mc.html#quickstart
FORMAT: %117s MESSAGE: $ mc alias set myminio http://192.168.50.21:9000 "U4B6Zi!b75DXSPm%avZb" "Q4#Q6y8G%%Px#npP4dudUobU#NBcGB7RMKV4ajYb"
$ mc alias set myminio http://192.168.50.21:9000 "U4B6Zi!b75DXSPm%!a(MISSING)vZb" "Q4#Q6y8G%Px#npP4dudUobU#NBcGB7RMKV4ajYb"
```
After:
```
Status: 1 Online, 0 Offline.
S3-API: http://192.168.50.21:9000 http://172.31.96.1:9000 http://127.0.0.1:9000
RootUser: "U4B6Zi!b75DXSPm%avZb"
RootPass: "Q4#Q6y8G%%Px#npP4dudUobU#NBcGB7RMKV4ajYb"
Console: http://192.168.50.21:52421 http://172.31.96.1:52421 http://127.0.0.1:52421
RootUser: "U4B6Zi!b75DXSPm%avZb"
RootPass: "Q4#Q6y8G%%Px#npP4dudUobU#NBcGB7RMKV4ajYb"
Command-line: https://min.io/docs/minio/linux/reference/minio-mc.html#quickstart
$ mc alias set myminio http://192.168.50.21:9000 "U4B6Zi!b75DXSPm%avZb" "Q4#Q6y8G%%Px#npP4dudUobU#NBcGB7RMKV4ajYb"
```
No need for special Windows case. `mc` works just fine.
2023-10-02 07:39:47 -06:00
10d5dd3a67
fix: a regression with audit log sending ( #18112 )
...
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
2023-09-26 12:23:02 -07:00
4eeb48f8e0
Return cached online/offline status for audit/http loggers ( #18083 )
...
To avoid having delays in prometheus scrape and in 'mc admin info' command.
2023-09-21 16:58:24 -07:00
1472875670
fix: failed messages counting in audit_http metrics ( #18075 )
...
all retries must not be counted as failed messages,
a failed message is a single counter not for all
retries, this PR fixes this.
Also we do not need to retry 10-times, instead we should
retry at max 3 times with some jitter to deliver the
messages.
2023-09-21 11:24:56 -07:00
1c99fb106c
Update to minio/pkg/v2 ( #17967 )
2023-09-04 12:57:37 -07:00
6a8d8f34a5
kafka: Do not require key when sending a message ( #17962 )
...
Keys are helpful to ensure the strict ordering of messages, however currently the
code uses a random request id for every log, hence using the request-id
as a Kafka key is not serve any purpose;
This commit removes the usage of the key, to also fix the audit issue from
internal subsystem that does not have a request ID.
2023-09-01 08:37:22 -07:00
07b1281046
add queue_dir to help message for logger/audit targets
2023-08-29 16:07:35 -07:00
adb8be069e
tune-kafka targets to ensure timeout triggers on hung brokers ( #17898 )
...
hung brokers can cause slowness to the entire system
when many callers are hung, leading to large goroutine
build-up.
2023-08-22 20:26:35 -07:00
3a0125fa1f
remove unexpected logging from peer calls ( #17888 )
...
also make sure RequestID is set for system logs
2023-08-21 14:25:24 -07:00
11dfc817f3
do not log client canceled events ( #17838 )
2023-08-17 14:53:43 -07:00
0285df5a02
fix: prioritize audit_webhook and logger_webhook ENVs over the config KVS ( #17783 )
2023-08-03 02:47:07 -07:00
9c0e8cd15b
logger: Avoid slow calls in http logger Send() function ( #17747 )
...
Send() is synchronous and can affect the latency of S3 requests when the
logger buffer is full.
Avoid checking if the HTTP target is online or not and increase the
workers anyway since the buffer is already full.
Also, avoid logs flooding when the audit target is down.
2023-07-29 12:49:18 -07:00
f3248a4b37
Redact all secrets from config viewing APIs ( #17380 )
...
This change adds a `Secret` property to `HelpKV` to identify secrets
like passwords and auth tokens that should not be revealed by the server
in its configuration fetching APIs. Configuration reporting APIs now do
not return secrets.
2023-06-23 07:45:27 -07:00
5a1612fe32
Bump up madmin-go and pkg deps ( #17469 )
2023-06-19 17:53:08 -07:00
dbd4c2425e
fix: kafka broker pings must not be greater than 1sec ( #17376 )
2023-06-07 11:47:00 -07:00
55a3310446
logger-http: Don't retry after a succesful send ( #17266 )
2023-05-22 14:53:18 -07:00
41fa8fa2d2
fix: increment counter when entry be skipped ( #17237 )
2023-05-19 08:36:52 -07:00
85912985b6
Check for only network errors in audit webhook for reachability ( #17228 )
2023-05-17 11:10:33 -07:00
99c4ffa34f
fix: avoid audit log race protection deadlocks ( #17168 )
2023-05-09 08:11:32 -07:00
57acacd5a7
Support persistent queue store for loggers ( #17121 )
2023-05-08 21:20:31 -07:00
6e27264c6b
update cleanupRoutine comment ( #17102 )
2023-04-28 01:11:51 -07:00
5c83c9724f
audit: Add request path and host to audit event ( #17099 )
2023-04-27 22:18:24 -07:00
b28d391a22
fix: add correct worker count before startHTTPLogger() ( #17091 )
2023-04-27 10:51:16 -07:00
8a9b9832fd
add Dial timeout for Kafka broker pings ( #17044 )
2023-04-17 15:45:01 -07:00
a5835cecbf
fix: regression in counting total requests ( #17024 )
2023-04-12 14:37:19 -07:00
d90d0c8931
Use one http response recorder per external http call ( #16938 )
2023-03-31 09:37:29 -07:00
11d04279c8
Add lazy init of audit logger ( #16842 )
2023-03-21 10:50:40 -07:00
3b5dbf9046
allow bootstrapping to validate internode tokens ( #16853 )
2023-03-20 01:40:24 -07:00
46f9049fb4
simplify error responses for KMS ( #16793 )
2023-03-16 11:59:42 -07:00
50dbd2cacc
Update audit log flow to use new headers with unit ( #16797 )
2023-03-13 22:50:19 -07:00
714283fae2
cleanup ignored static analysis ( #16767 )
2023-03-06 08:56:10 -08:00
fb17f97cf3
move audit and logger message structure to minio/pkg ( #16655 )
...
Signed-off-by: Daniel Valdivia <18384552+dvaldivia@users.noreply.github.com>
2023-02-21 21:21:17 -08:00
6b65ba1551
Added attribute proxy for `mc admin config set ALIAS logger_webhook` ( #16657 )
...
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
2023-02-21 21:19:46 -08:00
fadc46b906
Add the access key and parent user in the audit log ( #16572 )
2023-02-08 11:05:26 -08:00
aa8b9572b9
remove double ENABLED help output ( #16528 )
2023-02-03 05:52:52 -08:00
b67d97b1ba
add missing fields in audit logs for non-compressed handlers ( #16328 )
2022-12-30 10:20:19 -08:00
939c0100a6
log: Do not interpret verbs in object names in console output ( #16233 )
2022-12-13 08:27:40 -08:00
a30cfdd88f
Bump up madmin-go to v2 ( #16162 )
2022-12-06 13:46:50 -08:00
1f1dcdce65
move HTTP recorder to an internal library ( #16128 )
2022-11-28 10:20:27 -08:00
98a67a3776
Improvements in logger and audit webhooks ( #16102 )
2022-11-28 08:03:26 -08:00
bf89f79694
save deploymentID to avoid mutating request entry in Audit ( #16053 )
2022-11-11 12:42:15 -08:00
5b242f1d11
Add Audit target metrics ( #16044 )
2022-11-10 10:20:21 -08:00
ddeca9f12a
fix: filter rest errors and logs returned ( #16019 )
2022-11-07 10:38:08 -08:00
23b329b9df
remove gateway completely ( #15929 )
2022-10-24 17:44:15 -07:00
f696a221af
allow tagging policy condition for GetObject ( #15777 )
2022-10-02 12:29:29 -07:00
f69a98ce49
fix: loading Audit kafka configuration loading ( #15766 )
2022-09-29 08:35:08 -07:00
20c89ebbb3
freeze before exit when _MINIO_DEBUG_NO_EXIT is defined ( #15709 )
...
this is to ensure keep k8s pods running, when they reach a "crashloop" stage
2022-09-22 11:57:27 -07:00
5ce1448049
audit: account for response headers separately ( #15610 )
2022-09-01 12:51:04 -07:00
1f28a3bb80
Avoid messages from go test output ( #15601 )
...
A lot of warning messages are printed in CI/CD failures generated by go
test. Avoid that by requiring at least Error level for logging when
doing go test.
2022-08-30 08:23:40 -07:00
Praveen raj Mani
Sveinn
Shubhendu
Anis Eleuch
Harshavardhana
Aditya Manthramurthy
Krishnan Parthasarathi
jiuker
Klaus Post
Nitish Tiwari
ferhat elmas
Daniel Valdivia
Shubhendu
Shireesh Anjal