Commit Graph

57 Commits

Author SHA1 Message Date
Anis Eleuch 2a75225569
kafka: _MINIO_KAFKA_DEBUG to enable sarama debug messages (#19849) 2024-06-01 08:02:59 -07:00
Aditya Manthramurthy 5f78691fcf
ldap: Add user DN attributes list config param (#19758)
This change uses the updated ldap library in minio/pkg (bumped
up to v3). A new config parameter is added for LDAP configuration to
specify extra user attributes to load from the LDAP server and to store
them as additional claims for the user.

A test is added in sts_handlers.go that shows how to access the LDAP
attributes as a claim.

This is in preparation for adding SSH pubkey authentication to MinIO's SFTP
integration.
2024-05-24 16:05:23 -07:00
Ramon de Klein 701da1282a
Validates PostgreSQL table name (#19602) 2024-04-24 10:51:07 -07:00
jiuker d7fb6fddf6
feat: add user specific redis auth (#19285) 2024-03-18 21:37:54 -07:00
jiuker 0aae0180fb
feat: add userCredentials for nats (#19139) 2024-02-27 10:11:55 -08:00
Harshavardhana cd419a35fe
simplify broker healthcheck by following kafka guidelines (#19082)
fixes #19081
2024-02-20 00:16:35 -08:00
Praveen raj Mani c905d3fe21
fix: Re-use TCP connections for Kafka dials (#18860)
Fixes #18857
2024-01-24 13:10:52 -08:00
Harshavardhana dd2542e96c
add codespell action (#18818)
Original work here, #18474,  refixed and updated.
2024-01-17 23:03:17 -08:00
jiuker 34187e047d
feat: support elasticsearch notification endpoint compression codec (#18562) 2023-11-30 00:25:03 -08:00
Praveen raj Mani 3369eeb920
Relax batch size limit for kafka events (#18513)
Fixes #18490
2023-11-24 09:07:38 -08:00
jiuker 41091d9472
fix: close http body for es action (#18491) 2023-11-20 22:22:10 -08:00
jiuker f56a182b71
fix: close http body when webhook send (#18487) 2023-11-20 14:40:07 -08:00
Anis Eleuch 6ef8e87492
Support case insensitive kafka SASL mechanism config values (#18398) 2023-11-08 20:04:01 -08:00
Harshavardhana 754f7a8a39
replace io.Discard usage to fix some NUMA copy() latencies (#18394)
replace io.Discard usage to fix NUMA copy() latencies

On NUMA systems copying from 8K buffer allocated via
io.Discard leads to large latency build-up for every

```
copy(new8kbuf, largebuf)
```

can in-cur upto 1ms worth of latencies on NUMA systems
due to memory sharding across NUMA nodes.
2023-11-06 14:26:08 -08:00
Praveen raj Mani c27d0583d4
Send kafka notification messages in batches when queue_dir is enabled (#18164)
Fixes #18124
2023-10-07 08:07:38 -07:00
Anis Eleuch 419e5baf16
fix: webhook notify endpoint with standard ports (#18016) 2023-09-14 20:10:44 -07:00
Aditya Manthramurthy 1c99fb106c
Update to minio/pkg/v2 (#17967) 2023-09-04 12:57:37 -07:00
Harshavardhana 0d1fbef751
fix: a possible crash in event target Close() (#17948)
these are possible crashes when the configured
target is still in init() state and never finished
- however a delete config was initiated.
2023-08-30 07:27:45 -07:00
Harshavardhana adb8be069e
tune-kafka targets to ensure timeout triggers on hung brokers (#17898)
hung brokers can cause slowness to the entire system
when many callers are hung, leading to large goroutine
build-up.
2023-08-22 20:26:35 -07:00
jiuker fa2a8d7209
fix: drain the req.body into io.Discard correctly (#17881) 2023-08-21 01:09:07 -07:00
Harshavardhana dbd4c2425e
fix: kafka broker pings must not be greater than 1sec (#17376) 2023-06-07 11:47:00 -07:00
jiuker 443250d135
fix: for Target isActive use net.Dial instead (#17251) 2023-05-25 09:24:11 -07:00
Praveen raj Mani 57acacd5a7
Support persistent queue store for loggers (#17121) 2023-05-08 21:20:31 -07:00
Praveen raj Mani 1704abaf6b
fix: store notification events immediately for persistent queues (#17112) 2023-05-02 07:53:13 -07:00
Harshavardhana 8a9b9832fd
add Dial timeout for Kafka broker pings (#17044) 2023-04-17 15:45:01 -07:00
Praveen raj Mani 51f7f9aaa3
Generalize the event store using go generics (#16910) 2023-04-04 10:52:24 -07:00
Harshavardhana b21d3f9b82
event target registration failures must be returned (#16700) 2023-02-23 21:59:14 +05:30
Klaus Post 0b7989aa4b
Fix Kafka initialization crash (#16523) 2023-02-02 19:21:19 +05:30
Harshavardhana be92cf5959
change dependency from amqp -> amqp091 (RabbitMQ) official (#16142) 2022-11-28 16:05:06 -08:00
Ray 66239f30ce
configuring the nats target to reconnect forever (#16050) 2022-11-11 12:42:41 -08:00
Klaus Post bbc312fce6
Add notification queue metrics (#16026) 2022-11-08 16:36:47 -08:00
Harshavardhana 21251d8c22
initialize streaming events without lazy initialization (#16016) 2022-11-07 08:01:24 -08:00
Harshavardhana 4892a766a8
do not panic if webhook returns an error (#15970) 2022-10-30 16:45:53 -07:00
Anis Elleuch 86bb48792c
non-blocking initialization of bucket target notifications (#15571) 2022-09-27 17:23:28 -07:00
Klaus Post ff12080ff5
Remove deprecated io/ioutil (#15707) 2022-09-19 11:05:16 -07:00
Harshavardhana 5e763b71dc
use logger.LogOnce to reduce printing disconnection logs (#15408)
fixes #15334

- re-use net/url parsed value for http.Request{}
- remove gosimple, structcheck and unusued due to https://github.com/golangci/golangci-lint/issues/2649
- unwrapErrs upto leafErr to ensure that we store exactly the correct errors
2022-07-27 09:44:59 -07:00
Ray 3bdb92fcad
Adding error check for jetstream connection (#15252) 2022-07-07 23:14:47 -07:00
Harshavardhana dd839bf295
add NATS JetStream support (#15201) 2022-07-06 13:29:08 -07:00
Harshavardhana 9d07cde385
use crypto/sha256 only for FIPS 140-2 compliance (#14983)
It would seem like the PR #11623 had chewed more
than it wanted to, non-fips build shouldn't really
be forced to use slower crypto/sha256 even for
presumed "non-performance" codepaths. In MinIO
there are really no "non-performance" codepaths.
This assumption seems to have had an adverse
effect in certain areas of CPU usage.

This PR ensures that we stick to sha256-simd
on all non-FIPS builds, our most common build
to ensure we get the best out of the CPU at
any given point in time.
2022-05-27 06:00:19 -07:00
Harshavardhana 43eb5a001c
re-use transport for AdminInfo() call (#14571)
avoids creating new transport for each `isServerResolvable`
request, instead re-use the available global transport and do
not try to forcibly close connections to avoid TIME_WAIT
build upon large clusters.

Never use httpClient.CloseIdleConnections() since that can have
a drastic effect on existing connections on the transport pool.

Remove it everywhere.
2022-03-17 16:20:10 -07:00
Klaus Post 7060c809c0
Add authorization header to HEAD requests (#14510)
Add Authorization to network check requests.

Fixes #14507
2022-03-09 10:48:56 -08:00
Harshavardhana 1bfbe354f5
fix: clientId must be unique for all servers (#14398)
This is a regression from #14037, distributed setups
with MQTT was not working anymore. According to MQTT
spec it is expected this is unique per server.

We shall proceed to use unix nano timestamp hex
value instead here.
2022-02-23 20:19:59 -08:00
Aditya Manthramurthy bc110d8055
fix: mysql notification target table creation (#14350)
Add a generated hash column as the primary key for the key name as 
MySQL does not allow indexes on long VARCHAR columns.
2022-02-18 12:13:49 -08:00
Harshavardhana 0d3ae3810f
make sure to comply with MQTT spec (#14037)
- keep-alive cannot be 0 by default anymore
- client_id cannot be empty

fixes #13993
2022-01-06 11:25:39 -08:00
Harshavardhana f527c708f2
run gofumpt cleanup across code-base (#14015) 2022-01-02 09:15:06 -08:00
Klaus Post ebd78e983f
Limit key size to 3K (#13974)
User is reporting `Error 1071 :Specified key was too long,max key 
length is 3072 bytes`.

Regression caused by #13414
2021-12-22 11:41:51 -08:00
Harshavardhana 24d904d194
reload certs from disk upon SIGHUP (#13792) 2021-12-01 00:38:32 -08:00
Harshavardhana 08f4a0a816
fix: make sure esClient is allocated before use (#13727) 2021-11-22 12:46:46 -08:00
Harshavardhana 661b263e77
add gocritic/ruleguard checks back again, cleanup code. (#13665)
- remove some duplicated code
- reported a bug, separately fixed in #13664
- using strings.ReplaceAll() when needed
- using filepath.ToSlash() use when needed
- remove all non-Go style comments from the codebase

Co-authored-by: Aditya Manthramurthy <donatello@users.noreply.github.com>
2021-11-16 09:28:29 -08:00
Aditya Manthramurthy 91a0e7bdaa
update mysql notification key length, character set and collation (#13414)
fixes #13227
2021-10-11 17:40:11 -07:00