jiuker
0aae0180fb
feat: add userCredentials for nats ( #19139 )
2024-02-27 10:11:55 -08:00
Harshavardhana
cd419a35fe
simplify broker healthcheck by following kafka guidelines ( #19082 )
...
fixes #19081
2024-02-20 00:16:35 -08:00
Praveen raj Mani
ac8e9ce04f
Send a bucket notification event on DeleteObject() for non-existing object ( #19037 )
...
Send a bucket notification event on DeleteObject for non-existing objects
2024-02-13 07:34:17 -08:00
Praveen raj Mani
c905d3fe21
fix: Re-use TCP connections for Kafka dials ( #18860 )
...
Fixes #18857
2024-01-24 13:10:52 -08:00
Harshavardhana
dd2542e96c
add codespell action ( #18818 )
...
Original work here, #18474 , refixed and updated.
2024-01-17 23:03:17 -08:00
Sveinn
9b8ba97f9f
feat: add support for GetObjectAttributes API ( #18732 )
2024-01-05 10:43:06 -08:00
Praveen raj Mani
10ca0a6936
Label the notification target metrics by their target IDs ( #18633 )
...
This patch adds the targetID to the existing notification target metrics
and deprecates the current target metrics which points to the overall
event notification subsystem
2023-12-14 09:09:26 -08:00
jiuker
6ca6788bb7
feat: add events_errors_total metric ( #18610 )
2023-12-07 16:21:17 -08:00
Harshavardhana
73dde66dbe
stick to go1.19 go.mod ( #18600 )
2023-12-06 01:09:22 -08:00
Harshavardhana
fbb5e75e01
avoid run-away goroutine build-up in notification send, use channels ( #18533 )
...
use memory for async events when necessary and dequeue them as
needed, for all synchronous events customers must enable
```
MINIO_API_SYNC_EVENTS=on
```
Async events can be lost but is upto to the admin to
decide what they want, we will not create run-away number
of goroutines per event instead we will queue them properly.
Currently the max async workers is set to runtime.GOMAXPROCS(0)
which is more than sufficient in general, but it can be made
configurable in future but may not be needed.
2023-12-05 02:16:33 -08:00
jiuker
34187e047d
feat: support elasticsearch notification endpoint compression codec ( #18562 )
2023-11-30 00:25:03 -08:00
Praveen raj Mani
3369eeb920
Relax batch size limit for kafka events ( #18513 )
...
Fixes #18490
2023-11-24 09:07:38 -08:00
jiuker
41091d9472
fix: close http body for es action ( #18491 )
2023-11-20 22:22:10 -08:00
jiuker
f56a182b71
fix: close http body when webhook send ( #18487 )
2023-11-20 14:40:07 -08:00
Anis Eleuch
6ef8e87492
Support case insensitive kafka SASL mechanism config values ( #18398 )
2023-11-08 20:04:01 -08:00
Harshavardhana
754f7a8a39
replace io.Discard usage to fix some NUMA copy() latencies ( #18394 )
...
replace io.Discard usage to fix NUMA copy() latencies
On NUMA systems copying from 8K buffer allocated via
io.Discard leads to large latency build-up for every
```
copy(new8kbuf, largebuf)
```
can in-cur upto 1ms worth of latencies on NUMA systems
due to memory sharding across NUMA nodes.
2023-11-06 14:26:08 -08:00
Klaus Post
128256e3ab
Add event counters ( #18232 )
...
Export metric for global events sent and skipped for the lifetime of the server.
2023-10-12 15:39:22 -07:00
Praveen raj Mani
c27d0583d4
Send kafka notification messages in batches when queue_dir is enabled ( #18164 )
...
Fixes #18124
2023-10-07 08:07:38 -07:00
Anis Eleuch
419e5baf16
fix: webhook notify endpoint with standard ports ( #18016 )
2023-09-14 20:10:44 -07:00
Aditya Manthramurthy
1c99fb106c
Update to minio/pkg/v2 ( #17967 )
2023-09-04 12:57:37 -07:00
Harshavardhana
0d1fbef751
fix: a possible crash in event target Close() ( #17948 )
...
these are possible crashes when the configured
target is still in init() state and never finished
- however a delete config was initiated.
2023-08-30 07:27:45 -07:00
Harshavardhana
adb8be069e
tune-kafka targets to ensure timeout triggers on hung brokers ( #17898 )
...
hung brokers can cause slowness to the entire system
when many callers are hung, leading to large goroutine
build-up.
2023-08-22 20:26:35 -07:00
jiuker
fa2a8d7209
fix: drain the req.body into io.Discard correctly ( #17881 )
2023-08-21 01:09:07 -07:00
Praveen raj Mani
7c72b25ef0
Add an option to make bucket notifications synchronous ( #17406 )
...
With the current asynchronous behaviour in sending notification events
to the targets, we can't provide guaranteed delivery as the systems
might go for restarts.
For such event-driven use-cases, we can provide an option to enable
synchronous events where the APIs wait until the event is successfully
sent or persisted.
This commit adds 'MINIO_API_SYNC_EVENTS' env which when set to 'on'
will enable sending/persisting events to targets synchronously.
2023-06-20 17:38:59 -07:00
Aditya Manthramurthy
5a1612fe32
Bump up madmin-go and pkg deps ( #17469 )
2023-06-19 17:53:08 -07:00
Harshavardhana
dbd4c2425e
fix: kafka broker pings must not be greater than 1sec ( #17376 )
2023-06-07 11:47:00 -07:00
jiuker
443250d135
fix: for Target isActive use net.Dial instead ( #17251 )
2023-05-25 09:24:11 -07:00
Praveen raj Mani
57acacd5a7
Support persistent queue store for loggers ( #17121 )
2023-05-08 21:20:31 -07:00
Praveen raj Mani
1704abaf6b
fix: store notification events immediately for persistent queues ( #17112 )
2023-05-02 07:53:13 -07:00
Harshavardhana
8a9b9832fd
add Dial timeout for Kafka broker pings ( #17044 )
2023-04-17 15:45:01 -07:00
Praveen raj Mani
51f7f9aaa3
Generalize the event store using go generics ( #16910 )
2023-04-04 10:52:24 -07:00
Harshavardhana
901887e6bf
feat: add lambda transformation functions target ( #16507 )
2023-03-07 08:12:41 -08:00
Harshavardhana
b21d3f9b82
event target registration failures must be returned ( #16700 )
2023-02-23 21:59:14 +05:30
Klaus Post
fd6622458b
Add detailed scanner trace output and notifications ( #16668 )
2023-02-21 09:33:33 -08:00
Klaus Post
0b7989aa4b
Fix Kafka initialization crash ( #16523 )
2023-02-02 19:21:19 +05:30
Harshavardhana
bad229e16e
fix: support event name s3:Restore:* ( #16257 )
2022-12-14 05:12:07 -08:00
Aditya Manthramurthy
a30cfdd88f
Bump up madmin-go to v2 ( #16162 )
2022-12-06 13:46:50 -08:00
Harshavardhana
be92cf5959
change dependency from amqp -> amqp091 (RabbitMQ) official ( #16142 )
2022-11-28 16:05:06 -08:00
Ray
66239f30ce
configuring the nats target to reconnect forever ( #16050 )
2022-11-11 12:42:41 -08:00
Klaus Post
bbc312fce6
Add notification queue metrics ( #16026 )
2022-11-08 16:36:47 -08:00
Harshavardhana
21251d8c22
initialize streaming events without lazy initialization ( #16016 )
2022-11-07 08:01:24 -08:00
Harshavardhana
4892a766a8
do not panic if webhook returns an error ( #15970 )
2022-10-30 16:45:53 -07:00
Klaus Post
71954faa3a
mark pubsub type safe via generics ( #15961 )
2022-10-28 10:55:42 -07:00
Anis Elleuch
86bb48792c
non-blocking initialization of bucket target notifications ( #15571 )
2022-09-27 17:23:28 -07:00
Klaus Post
ff12080ff5
Remove deprecated io/ioutil ( #15707 )
2022-09-19 11:05:16 -07:00
Anis Elleuch
3856d078d2
fix: set 20000 as maximum parallel event calls ( #15435 )
...
This is needed to avoid consuming a lot of goroutines when a target is
very slow or there is a bug in a target library.
2022-07-30 12:12:33 -07:00
Harshavardhana
5e763b71dc
use logger.LogOnce to reduce printing disconnection logs ( #15408 )
...
fixes #15334
- re-use net/url parsed value for http.Request{}
- remove gosimple, structcheck and unusued due to https://github.com/golangci/golangci-lint/issues/2649
- unwrapErrs upto leafErr to ensure that we store exactly the correct errors
2022-07-27 09:44:59 -07:00
Ray
3bdb92fcad
Adding error check for jetstream connection ( #15252 )
2022-07-07 23:14:47 -07:00
Harshavardhana
dd839bf295
add NATS JetStream support ( #15201 )
2022-07-06 13:29:08 -07:00
Klaus Post
ac055b09e9
Add detailed scanner metrics ( #15161 )
2022-07-05 14:45:49 -07:00
