Commit Graph

33 Commits

Author SHA1 Message Date
Klaus Post
a896125490
Add crawler delay config + dynamic config values (#11018) 2020-12-04 09:32:35 -08:00
Harshavardhana
2042d4873c
rename crawler config option to heal (#10678) 2020-10-14 13:51:51 -07:00
Harshavardhana
8b74a72b21
fix: rename READY deadline to CLUSTER deadline ENV (#10535) 2020-09-23 09:14:33 -07:00
Klaus Post
fa01e640f5
Continous healing: add optional bitrot check (#10417) 2020-09-12 00:08:12 -07:00
Klaus Post
86a3319d41
Ignore config values from unknown subsystems (#10432) 2020-09-08 08:57:04 -07:00
Anis Elleuch
778e9c864f
Move dependency from minio-go v6 to v7 (#10042) 2020-07-14 09:38:05 -07:00
Krishna Srinivas
7d19ab9f62
readiness returns error quickly if any of the set is down (#9662)
This PR adds a new configuration parameter which allows readiness
check to respond within 10secs, this can be reduced to a lower value
if necessary using 

```
mc admin config set api ready_deadline=5s
```

 or

```
export MINIO_API_READY_DEADLINE=5s
```
2020-05-23 17:38:39 -07:00
Harshavardhana
d92db198d1
Add target parsing code for config (#9375)
This code is helper for mcs project
2020-04-16 17:43:14 -07:00
Anis Elleuch
8a94aebdb8
config: Add api requests max & deadline configs (#9273)
Add two new configuration entries, api.requests-max and
api.requests-deadline which have the same role of
MINIO_API_REQUESTS_MAX and MINIO_API_REQUESTS_DEADLINE.
2020-04-14 12:46:37 -07:00
Harshavardhana
b412a222ae
Add missing comment key from key list (#9313)
Continuing from previous PR #9304, comment
is a special key is not present in the
default KV list. Add it explicitly when
tokenizing fields as it may be possible that
some clients might try to set comments.
2020-04-10 11:44:28 -07:00
Harshavardhana
3184205519
fix: config to support keys with special values (#9304)
This PR adds context-based `k=v` splits based
on the sub-system which was obtained, if the
keys are not provided an error will be thrown
during parsing, if keys are provided with wrong
values an error will be thrown. Keys can now
have values which are of a much more complex
form such as `k="v=v"` or `k=" v = v"`
and other variations.

additionally, deprecate unnecessary postgres/mysql
configuration styles, support only

- connection_string for Postgres
- dsn_string for MySQL

All other parameters are removed.
2020-04-09 21:45:17 -07:00
Harshavardhana
686d4656de
fix: set appropriate defaults when new keys added (#8795)
A new key was added in identity_openid recently
required explicitly for client to set the optional
value without that it would be empty, handle this
appropriately.

Fixes #8787
2020-01-10 16:57:18 -08:00
Harshavardhana
0b7bd024fb Fix dependencies graph for minio source compilation (#8717)
We had messy cyclical dependency problem with `mc`
due to dependencies in pkg/console, moved the pkg/console
to minio for more control and also to avoid any further
cyclical dependencies of `mc` clobbering up the
dependencies on server.

Fixes #8659
2019-12-31 09:36:13 +05:30
Harshavardhana
1dc5f2d0af
Remove safe mode for invalid entries in config (#8650)
The approach is that now safe mode is only invoked when
we cannot read the config or under some catastrophic
situations, but not under situations when config entries
are invalid or unreachable. This allows for maximum
availability for MinIO and not fail on our users unlike
most of our historical releases.
2019-12-14 17:27:57 -08:00
Andreas Auernhammer
c3d4c1f584 add minio/keys KMS integration (#8631)
This commit adds support for the minio/kes KMS.
See: https://github.com/minio/kes

In particular you can configure it as KMS by:
 - `export MINIO_KMS_KES_ENDPOINT=`  // Server URL
 - `export MINIO_KMS_KES_KEY_FILE=`  // TLS client private key
 - `export MINIO_KMS_KES_CERT_FILE=` // TLS client certificate
 - `export MINIO_KMS_KES_CA_PATH=`   // Root CAs issuing server cert
 - `export MINIO_KMS_KES_KEY_NAME=`  // The name of the (default)
master key
2019-12-13 12:57:11 -08:00
Harshavardhana
39e8e4f4aa
Allow empty target KVS for notification targets (#8644)
This is allowed with enable=off arg value
2019-12-12 17:02:14 -08:00
Harshavardhana
ca62ac65d4
Reject mandatory KVS if not set for any sub-sys (#8641) 2019-12-12 14:55:07 -08:00
Harshavardhana
476111968a Update help messages with new wording (#8616)
Final update to all messages across sub-systems
after final review, the only change here is that
NATS now has TLS and TLSSkipVerify to be consistent
for all other notification targets.
2019-12-06 13:53:51 -08:00
Harshavardhana
97deba2a7c GetKVS should add new keys automatically, preserve order (#8612) 2019-12-06 16:13:10 +05:30
Harshavardhana
d8e3de0cae Ensure comment is always a valid key (#8604)
Also fix LDAP leaky connection
2019-12-05 18:17:42 +05:30
Harshavardhana
c9940d8c3f Final changes to config sub-system (#8600)
- Introduces changes such as certain types of
  errors that can be ignored or which need to 
  go into safe mode.
- Update help text as per the review
2019-12-04 15:32:37 -08:00
Harshavardhana
794eb54da8 Export command prints turned-off sub-sys as comments (#8594)
This PR also tries to

- Preserve the order of keys printed in export command
- Fix cache to be enabled with _STATE env to keep
  backward compatibility
2019-12-03 10:50:20 -08:00
Harshavardhana
30e80d0a86
Add ReadFrom,WriteTo helpers for server config (#8580) 2019-11-27 09:36:08 -08:00
Harshavardhana
0a56e33ce1 Preserve client sent config appropriately (#8566) 2019-11-22 13:46:05 -08:00
Harshavardhana
5ac4b517c9
Order all keys in config (#8541)
New changes

- return default values when sub-sys is
  not configured.
- state is hidden parameter now
- remove worm mode to be saved in config
2019-11-20 15:10:24 -08:00
Harshavardhana
32c200fe12 Fix console logger crash in gateway mode (#8525)
This PR also fixes config migration only
for credentials and region which are valid
and set.

Also fix implicit `state="on"` behavior
2019-11-14 14:19:57 -08:00
Harshavardhana
26a866a202
Fix review comments and new changes in config (#8515)
- Migrate and save only settings which are enabled
- Rename logger_http to logger_webhook and
  logger_http_audit to audit_webhook
- No more pretty printing comments, comment
  is a key=value pair now.
- Avoid quotes on values which do not have space in them
- `state="on"` is implicit for all SetConfigKV unless
  specified explicitly as `state="off"`
- Disabled IAM users should be disabled always
2019-11-13 17:38:05 -08:00
Harshavardhana
d97d53bddc
Honor etcd legacy v/s new config settings properly (#8510)
This PR also fixes issues related to

- Add proper newline for `mc admin config get` output
  for more than one targets
- Fixes issue of temporary user credentials to have
  consistent output
- Fixes a crash when setting a key with empty values
- Fixes a parsing issue with `mc admin config history`
- Fixes gateway ENV handling for etcd server and gateway
2019-11-12 03:16:25 -08:00
Harshavardhana
822eb5ddc7 Bring in safe mode support (#8478)
This PR refactors object layer handling such
that upon failure in sub-system initialization
server reaches a stage of safe-mode operation
wherein only certain API operations are enabled
and available.

This allows for fixing many scenarios such as

 - incorrect configuration in vault, etcd,
   notification targets
 - missing files, incomplete config migrations
   unable to read encrypted content etc
 - any other issues related to notification,
   policies, lifecycle etc
2019-11-09 09:27:23 -08:00
Harshavardhana
1e457dda7e Enhance config restore to carry previous set content as well (#8483)
This PR brings support for `history` list to
list in the following agreed format

```
~ mc admin config history list -n 2 myminio
RestoreId: df0ebb1e-69b0-4043-b9dd-ab54508f2897
Date: Mon, 04 Nov 2019 17:27:27 GMT

region name="us-east-1" state="on"
region name="us-east-1" state="on"
region name="us-east-1" state="on"
region name="us-east-1" state="on"

RestoreId: ecc6873a-0ed3-41f9-b03e-a2a1bab48b5f
Date: Mon, 04 Nov 2019 17:28:23 GMT

region name=us-east-1 state=off
```

This PR also moves the help templating and coloring to
fully `mc` side instead than `madmin` API.
2019-11-05 06:18:26 -08:00
Harshavardhana
9e7a3e6adc Extend further validation of config values (#8469)
- This PR allows config KVS to be validated properly
  without being affected by ENV overrides, rejects
  invalid values during set operation

- Expands unit tests and refactors the error handling
  for notification targets, returns error instead of
  ignoring targets for invalid KVS

- Does all the prep-work for implementing safe-mode
  style operation for MinIO server, introduces a new
  global variable to toggle safe mode based operations
  NOTE: this PR itself doesn't provide safe mode operations
2019-10-30 23:39:09 -07:00
Harshavardhana
47b13cdb80 Add etcd part of config support, add noColor/json support (#8439)
- Add color/json mode support for get/help commands
- Support ENV help for all sub-systems
- Add support for etcd as part of config
2019-10-30 00:04:39 -07:00
Harshavardhana
ee4a6a823d Migrate config to KV data format (#8392)
- adding oauth support to MinIO browser (#8400) by @kanagaraj
- supports multi-line get/set/del for all config fields
- add support for comments, allow toggle
- add extensive validation of config before saving
- support MinIO browser to support proper claims, using STS tokens
- env support for all config parameters, legacy envs are also
  supported with all documentation now pointing to latest ENVs
- preserve accessKey/secretKey from FS mode setups
- add history support implements three APIs
  - ClearHistory
  - RestoreHistory
  - ListHistory
- add help command support for each config parameters
- all the bug fixes after migration to KV, and other bug
  fixes encountered during testing.
2019-10-22 22:59:13 -07:00