Commit Graph

64 Commits

Author SHA1 Message Date
Hamid
5b8975bf4b
Simplify redis access event format to faciliate parsing (#9046) 2020-02-26 09:23:32 -08:00
Praveen raj Mani
1b427ddb69
Support for Kafka version in the config (#9001)
Add a field for the Kafka version in the config. The user can explicitly 
set the version of the Kafka cluster.

Fixes #8768
2020-02-17 07:56:33 +05:30
Anis Elleuch
7432b5c9b2
Use user CAs in checkEndpoint() call (#8911)
The server info handler makes a http connection to other
nodes to check if they are up but does not load the custom
CAs in ~/.minio/certs/CAs.

This commit fix it.

Co-authored-by: Harshavardhana <harsha@minio.io>
2020-02-02 07:15:29 +05:30
Harshavardhana
e2b3c083aa
fix: close and drain the response body always (#8847) 2020-01-21 02:46:58 -08:00
Praveen raj Mani
4cd1bbb50a This PR fixes two things (#8772)
- Stop spawning store replay routines when testing the notification targets
- Properly honor the target.Close() to clean the resources used

Fixes #8707

Co-authored-by: Harshavardhana <harsha@minio.io>
2020-01-09 19:45:44 +05:30
Harshavardhana
c364f0af6c Start using custom HTTP transport for webhook endpoints (#8630)
Use a more performant http transport for webhook
endpoints with proper connection pooling, appropriate
timeouts etc.
2019-12-12 06:53:50 -08:00
Ashish Kumar Sinha
24fb1bf258 New Admin Info (#8497) 2019-12-11 14:27:03 -08:00
Harshavardhana
476111968a Update help messages with new wording (#8616)
Final update to all messages across sub-systems
after final review, the only change here is that
NATS now has TLS and TLSSkipVerify to be consistent
for all other notification targets.
2019-12-06 13:53:51 -08:00
Aleksandr Petruhin
d2dc964cb5 Support TLS auth for Kafka notification target (#8609) 2019-12-05 15:31:46 -08:00
Harshavardhana
c9940d8c3f Final changes to config sub-system (#8600)
- Introduces changes such as certain types of
  errors that can be ignored or which need to 
  go into safe mode.
- Update help text as per the review
2019-12-04 15:32:37 -08:00
Harshavardhana
5d3d57c12a
Start using error wrapping with fmt.Errorf (#8588)
Use fatih/errwrap to fix all the code to use
error wrapping with fmt.Errorf()
2019-12-02 09:28:01 -08:00
poornas
ca96560d56 Add object retention at the per object (#8528)
level - this PR builds on #8120 which
added PutBucketObjectLockConfiguration and
GetBucketObjectLockConfiguration APIS

This PR implements PutObjectRetention,
GetObjectRetention API and enhances
PUT and GET API operations to display
governance metadata if permissions allow.
2019-11-20 13:18:09 -08:00
svistoi
c9be601988 NATS TLS specify CA and client TLS authentication (#8389)
- added ability to specify CA for self-signed certificates
- added option to authenticate using client certificates
- added unit tests for nats connections
2019-11-15 09:13:23 -08:00
Harshavardhana
26a866a202
Fix review comments and new changes in config (#8515)
- Migrate and save only settings which are enabled
- Rename logger_http to logger_webhook and
  logger_http_audit to audit_webhook
- No more pretty printing comments, comment
  is a key=value pair now.
- Avoid quotes on values which do not have space in them
- `state="on"` is implicit for all SetConfigKV unless
  specified explicitly as `state="off"`
- Disabled IAM users should be disabled always
2019-11-13 17:38:05 -08:00
Harshavardhana
9e7a3e6adc Extend further validation of config values (#8469)
- This PR allows config KVS to be validated properly
  without being affected by ENV overrides, rejects
  invalid values during set operation

- Expands unit tests and refactors the error handling
  for notification targets, returns error instead of
  ignoring targets for invalid KVS

- Does all the prep-work for implementing safe-mode
  style operation for MinIO server, introduces a new
  global variable to toggle safe mode based operations
  NOTE: this PR itself doesn't provide safe mode operations
2019-10-30 23:39:09 -07:00
Harshavardhana
ee4a6a823d Migrate config to KV data format (#8392)
- adding oauth support to MinIO browser (#8400) by @kanagaraj
- supports multi-line get/set/del for all config fields
- add support for comments, allow toggle
- add extensive validation of config before saving
- support MinIO browser to support proper claims, using STS tokens
- env support for all config parameters, legacy envs are also
  supported with all documentation now pointing to latest ENVs
- preserve accessKey/secretKey from FS mode setups
- add history support implements three APIs
  - ClearHistory
  - RestoreHistory
  - ListHistory
- add help command support for each config parameters
- all the bug fixes after migration to KV, and other bug
  fixes encountered during testing.
2019-10-22 22:59:13 -07:00
Harshavardhana
f2cc97a44c
Improve MQTT token registration retry (#8397) 2019-10-15 19:39:14 -07:00
Harshavardhana
d48fd6fde9
Remove unusued params and functions (#8399) 2019-10-15 18:35:41 -07:00
Harshavardhana
c33bae057f Fix build issues with event target logger 2019-10-11 19:07:09 -07:00
poornas
d7060c4c32 Allow logging targets to be configured to receive minio (#8347)
specific errors, `application` errors or `all` by default.

console logging on server by default lists all logs -
enhance admin console API to accept `type` as query parameter to
subscribe to application/minio logs.
2019-10-11 18:50:54 -07:00
Harshavardhana
175b07d6e4
Fix queueStore stops working with concurrent PUT/DELETE requests (#8381)
- This PR fixes situation to avoid underflow, this is possible
  because of disconnected operations in replay/sendEvents
- Hold right locks if Del() operation is performed in Get()
- Remove panic in the code and use loggerOnce
- Remove Timer and instead use Ticker instead for proper ticks
2019-10-11 17:46:03 -07:00
Harshavardhana
3b8adf7528 Move storageclass config handling into cmd/config/storageclass (#8360)
Continuation of the changes done in PR #8351 to refactor,
add tests and move global handling into a more idiomatic
style for Go as packages.
2019-10-07 11:20:24 +05:30
Praveen raj Mani
dc0dce9beb Check for connection errors during event transit in elasticsearch and webhook (#8273) 2019-09-19 08:23:43 -07:00
Praveen raj Mani
456ce4cc92 Add rootCAs support to Kafka & MQTT (#8236)
Fixes #8211
2019-09-18 23:43:04 +05:30
Praveen raj Mani
8700945cdf Handle connection failures on webhook/url pings (#8204)
Properly handle connection failures while replaying events

Fixes #8194
2019-09-12 16:44:51 -07:00
Praveen raj Mani
428836d4e1 Ignore "connection reset" errors while initializing the targets with queue store enabled (#8185)
Fixes #8178
2019-09-05 13:55:48 -07:00
Praveen raj Mani
e96f19c867 Use default ports for the schemes when dialing (#7960)
Fixes #7946
2019-08-13 16:43:11 +05:30
Praveen raj Mani
b0cea1c0f3 Enable event persistence in AMQP (#7565) 2019-07-25 11:20:24 -07:00
Praveen raj Mani
55d4eee6f1 Enable event persistence in MySQL and PostgreSQL (#7629) 2019-07-24 10:18:29 -07:00
Praveen raj Mani
c9349747ca Enable event-persistence in NATS and NATS-Streaming (#7612) 2019-07-23 10:37:25 -07:00
Praveen raj Mani
2b9b907f9c Enable event persistence in Redis (#7601) 2019-07-23 10:22:08 -07:00
Christian Muehlhaeuser
38bc3a45db Fixed tautological conditions (#7959)
We already check for err being equal to nil above, no need
to check again.
2019-07-22 17:06:08 -07:00
Praveen raj Mani
bf278ca36f Enable event persistence in NSQ (#7579) 2019-07-12 10:41:57 +01:00
Praveen raj Mani
bba562235b Enable persistent event store in elasticsearch (#7564) 2019-07-12 08:23:20 +05:30
Praveen raj Mani
bb871a7c31 Enable event persistence in webhook (#7614) 2019-07-05 15:21:41 +05:30
iliul
d3f9f8be88 golint: fix redundant code logic (#7842)
Signed-off-by: Lei Liu <liul.stone@gmail.com>
2019-06-27 15:18:33 +05:30
Harshavardhana
ff7799452b Fix build failure due to nats project migration 2019-06-03 17:49:49 -07:00
Praveen raj Mani
763fce909b Enable event persistence in kafka (#7633) 2019-05-29 13:19:48 -07:00
Praveen raj Mani
c4c79f61ce Notification: Changes to persistent event store (#7658)
This patch includes the following changes in event store interface
- Removes memory store. We will not persist events in memory anymore, if `queueDir` is not set.
- Orders the events before replaying to the broker.
2019-05-22 13:34:48 -07:00
Praveen raj Mani
998f01fadc Support IPv6 hosts for postgres connections (#7168)
Fixes #7145
2019-05-03 17:31:33 +05:30
Praveen raj Mani
47ca411163 Enhance the event store interface to support channeling (#7343)
- Avoids code duplication across the other targets. By having a
  centralized function call.

- Reduce the room for race.
2019-04-10 18:16:01 +05:30
kannappanr
5ecac91a55
Replace Minio refs in docs with MinIO and links (#7494) 2019-04-09 11:39:42 -07:00
poornas
dd1d69ab5c fix mysql config for native password authentication (#7436)
fixes #7430
2019-03-28 10:45:22 -07:00
Praveen raj Mani
78d116c487 Event persistence for MQTT (#7268)
- The events will be persisted in queueStore if `queueDir` is set.
- Else, if queueDir is not set events persist in memory.

The events are replayed back when the mqtt broker is back online.
2019-02-25 18:01:13 -08:00
Nitish Tiwari
13c3b8afe2
Update Redis client used for bucket notifications (#7213)
Current Redis client used https://github.com/garyburd/redigo/
has been archived and further development is done at
https://github.com/gomodule/redigo

This commit updates the client library accordingly.

Fixes #6392
2019-02-11 19:06:27 +05:30
Praveen raj Mani
fad59da29d clientID removed in the MQTT config (#7157)
More than one client can't use the same clientID for MQTT connection. 
This causes problem in distributed deployments where config is shared 
across nodes, as each Minio instance tries to connect to MQTT using the
same clientID.

This commit removes the clientID field in config, and allows
MQTT client to create random clientID for each node.
2019-01-29 15:00:15 +05:30
Harshavardhana
e8791ae274 Remove Minio server arch, version from Server: header (#7074) 2019-01-15 13:16:11 +05:30
Praveen raj Mani
6571641735 Persist offline mqtt events in the queueDir and replay (#7037) 2019-01-14 12:39:00 +05:30
Praveen raj Mani
e7af31c2ff Removed clientID from NATS-Streaming Config (#6391)
clientID must be a unique `UUID` for each connections. Now, the
server generates it, rather considering the config.

Removing it as it is non-beneficial right now.

Fixes #6364
2018-11-30 10:46:17 +05:30
Ashish Kumar Sinha
b0d04b9a81 Retry Connection for RabbitMQ (#6837)
Add retries to connect to RabbitMQ 5 times 
with 2s interval

Fixes #6807
2018-11-21 08:37:29 +05:30