Commit Graph

10 Commits

Author SHA1 Message Date
Harshavardhana fbb5e75e01
avoid run-away goroutine build-up in notification send, use channels (#18533)
use memory for async events when necessary and dequeue them as
needed, for all synchronous events customers must enable

```
MINIO_API_SYNC_EVENTS=on
```

Async events can be lost but is upto to the admin to
decide what they want, we will not create run-away number
of goroutines per event instead we will queue them properly.

Currently the max async workers is set to runtime.GOMAXPROCS(0)
which is more than sufficient in general, but it can be made
configurable in future but may not be needed.
2023-12-05 02:16:33 -08:00
Klaus Post 128256e3ab
Add event counters (#18232)
Export metric for global events sent and skipped for the lifetime of the server.
2023-10-12 15:39:22 -07:00
Praveen raj Mani c27d0583d4
Send kafka notification messages in batches when queue_dir is enabled (#18164)
Fixes #18124
2023-10-07 08:07:38 -07:00
Praveen raj Mani 7c72b25ef0
Add an option to make bucket notifications synchronous (#17406)
With the current asynchronous behaviour in sending notification events
to the targets, we can't provide guaranteed delivery as the systems
might go for restarts.

For such event-driven use-cases, we can provide an option to enable
synchronous events where the APIs wait until the event is successfully
sent or persisted.

This commit adds 'MINIO_API_SYNC_EVENTS' env which when set to 'on'
will enable sending/persisting events to targets synchronously.
2023-06-20 17:38:59 -07:00
Praveen raj Mani 57acacd5a7
Support persistent queue store for loggers (#17121) 2023-05-08 21:20:31 -07:00
Harshavardhana 901887e6bf
feat: add lambda transformation functions target (#16507) 2023-03-07 08:12:41 -08:00
Klaus Post bbc312fce6
Add notification queue metrics (#16026) 2022-11-08 16:36:47 -08:00
Anis Elleuch 86bb48792c
non-blocking initialization of bucket target notifications (#15571) 2022-09-27 17:23:28 -07:00
Anis Elleuch 3856d078d2
fix: set 20000 as maximum parallel event calls (#15435)
This is needed to avoid consuming a lot of goroutines when a target is
very slow or there is a bug in a target library.
2022-07-30 12:12:33 -07:00
Harshavardhana 1f262daf6f
rename all remaining packages to internal/ (#12418)
This is to ensure that there are no projects
that try to import `minio/minio/pkg` into
their own repo. Any such common packages should
go to `https://github.com/minio/pkg`
2021-06-01 14:59:40 -07:00