Commit Graph

100 Commits

Author SHA1 Message Date
Harshavardhana
9bbf5cb74f
fix: Avoid re-reading bucket names from etcd (#8924)
This helps improve performance when there are
1000+ bucket entries on etcd, improves the
startup time significantly.
2020-02-03 13:54:20 +05:30
Harshavardhana
e2b3c083aa
fix: close and drain the response body always (#8847) 2020-01-21 02:46:58 -08:00
Harshavardhana
686d4656de
fix: set appropriate defaults when new keys added (#8795)
A new key was added in identity_openid recently
required explicitly for client to set the optional
value without that it would be empty, handle this
appropriately.

Fixes #8787
2020-01-10 16:57:18 -08:00
Praveen raj Mani
4cd1bbb50a This PR fixes two things (#8772)
- Stop spawning store replay routines when testing the notification targets
- Properly honor the target.Close() to clean the resources used

Fixes #8707

Co-authored-by: Harshavardhana <harsha@minio.io>
2020-01-09 19:45:44 +05:30
Harshavardhana
abc1c1070a Add custom policy claim name (#8764)
In certain organizations policy claim names
can be not just 'policy' but also things like
'roles', the value of this field might also
be *string* or *[]string* support this as well

In this PR we are still not supporting multiple
policies per STS account which will require a
more comprehensive change.
2020-01-08 17:21:58 -08:00
Anis Elleuch
d861edfc00 xl: Print the correct err msg when access to the backend is forbidden (#8735)
minio server /data{1..4} shows an error about inability to bind a port, though
the real problem is /data{1..4} cannot be created because of the lack of
permissions.

This commit fix the behavior.
2020-01-03 21:15:26 +05:30
Harshavardhana
0b7bd024fb Fix dependencies graph for minio source compilation (#8717)
We had messy cyclical dependency problem with `mc`
due to dependencies in pkg/console, moved the pkg/console
to minio for more control and also to avoid any further
cyclical dependencies of `mc` clobbering up the
dependencies on server.

Fixes #8659
2019-12-31 09:36:13 +05:30
Harshavardhana
3af70b36fd Disallow creating buckets even with different domains (#8716)
If two distinct clusters are started with different domains
along with single common domain, this situation was leading
to conflicting buckets getting created on different clusters

To avoid this do not prematurely error out if the key has no
entries, let the caller decide on which entry matches and
which entry is valid. This allows support for MINIO_DOMAIN
with one common domain, but each cluster may have their own
domains.

Fixes #8705
2019-12-30 17:11:47 -08:00
Anis Elleuch
c31e67dcce Better error when the server is unable to write in the backend (#8697) 2019-12-25 22:05:54 -08:00
Harshavardhana
f68a7005c0 Improve disk formatting stage for large disk sets (#8690) 2019-12-23 16:31:03 -08:00
Harshavardhana
c9c0d5eec2 Allow CNAME records when specified as MINIO_PUBLIC_IPS (#8662)
This is necessary for `m3` global bucket support
2019-12-18 11:02:45 +05:30
Harshavardhana
1dc5f2d0af
Remove safe mode for invalid entries in config (#8650)
The approach is that now safe mode is only invoked when
we cannot read the config or under some catastrophic
situations, but not under situations when config entries
are invalid or unreachable. This allows for maximum
availability for MinIO and not fail on our users unlike
most of our historical releases.
2019-12-14 17:27:57 -08:00
Andreas Auernhammer
c3d4c1f584 add minio/keys KMS integration (#8631)
This commit adds support for the minio/kes KMS.
See: https://github.com/minio/kes

In particular you can configure it as KMS by:
 - `export MINIO_KMS_KES_ENDPOINT=`  // Server URL
 - `export MINIO_KMS_KES_KEY_FILE=`  // TLS client private key
 - `export MINIO_KMS_KES_CERT_FILE=` // TLS client certificate
 - `export MINIO_KMS_KES_CA_PATH=`   // Root CAs issuing server cert
 - `export MINIO_KMS_KES_KEY_NAME=`  // The name of the (default)
master key
2019-12-13 12:57:11 -08:00
Harshavardhana
471a3a650a
fix: Don't allow to set unconfigured notification ARNs (#8643)
Fixes #8642
2019-12-13 12:36:45 -08:00
Harshavardhana
39e8e4f4aa
Allow empty target KVS for notification targets (#8644)
This is allowed with enable=off arg value
2019-12-12 17:02:14 -08:00
Harshavardhana
ca62ac65d4
Reject mandatory KVS if not set for any sub-sys (#8641) 2019-12-12 14:55:07 -08:00
Harshavardhana
c364f0af6c Start using custom HTTP transport for webhook endpoints (#8630)
Use a more performant http transport for webhook
endpoints with proper connection pooling, appropriate
timeouts etc.
2019-12-12 06:53:50 -08:00
Harshavardhana
476111968a Update help messages with new wording (#8616)
Final update to all messages across sub-systems
after final review, the only change here is that
NATS now has TLS and TLSSkipVerify to be consistent
for all other notification targets.
2019-12-06 13:53:51 -08:00
Harshavardhana
97deba2a7c GetKVS should add new keys automatically, preserve order (#8612) 2019-12-06 16:13:10 +05:30
Aleksandr Petruhin
d2dc964cb5 Support TLS auth for Kafka notification target (#8609) 2019-12-05 15:31:46 -08:00
Harshavardhana
d8e3de0cae Ensure comment is always a valid key (#8604)
Also fix LDAP leaky connection
2019-12-05 18:17:42 +05:30
Harshavardhana
c9940d8c3f Final changes to config sub-system (#8600)
- Introduces changes such as certain types of
  errors that can be ignored or which need to 
  go into safe mode.
- Update help text as per the review
2019-12-04 15:32:37 -08:00
Harshavardhana
794eb54da8 Export command prints turned-off sub-sys as comments (#8594)
This PR also tries to

- Preserve the order of keys printed in export command
- Fix cache to be enabled with _STATE env to keep
  backward compatibility
2019-12-03 10:50:20 -08:00
Harshavardhana
0bfd20a8e3
Add client_id support for OpenID (#8579)
- One click OpenID authorization on Login page
- Add client_id help, config keys etc

Thanks to @egorkaru @ihostage for the
original work and testing.
2019-11-29 21:37:42 -08:00
Harshavardhana
b21835f195 Honor DurationSeconds properly for WebIdentity (#8581)
Also cleanup code to add various constants for
verbatim strings across the code base.

Fixes #8482
2019-11-29 18:57:54 +05:30
Harshavardhana
30e80d0a86
Add ReadFrom,WriteTo helpers for server config (#8580) 2019-11-27 09:36:08 -08:00
Harshavardhana
78eb3b78bb
Repurpose Get/SetConfig as import/export support (#8578) 2019-11-26 10:08:25 -08:00
Harshavardhana
e542084c37
Add etcd path prefix for all IAM assets (#8569)
Currently, we use the top-level prefix "config/"
for all our IAM assets, instead of to provide
tenant-level separation bring 'path_prefix'
to namespace the access properly.

Fixes #8567
2019-11-25 16:33:34 -08:00
Harshavardhana
0a56e33ce1 Preserve client sent config appropriately (#8566) 2019-11-22 13:46:05 -08:00
Harshavardhana
5ac4b517c9
Order all keys in config (#8541)
New changes

- return default values when sub-sys is
  not configured.
- state is hidden parameter now
- remove worm mode to be saved in config
2019-11-20 15:10:24 -08:00
Harshavardhana
7cdb67680e
Add help with order of keys (#8535) 2019-11-19 13:48:13 -08:00
Harshavardhana
13a3d17321
Do not add comments after migration (#8530)
Also filter out empty comments from being
printed.
2019-11-16 14:57:36 -08:00
svistoi
c9be601988 NATS TLS specify CA and client TLS authentication (#8389)
- added ability to specify CA for self-signed certificates
- added option to authenticate using client certificates
- added unit tests for nats connections
2019-11-15 09:13:23 -08:00
Ville Skyttä
95e5d7a9c3 Improve access and secret key validation error, sync with implementation (#8516) 2019-11-14 14:47:35 -08:00
Harshavardhana
32c200fe12 Fix console logger crash in gateway mode (#8525)
This PR also fixes config migration only
for credentials and region which are valid
and set.

Also fix implicit `state="on"` behavior
2019-11-14 14:19:57 -08:00
Harshavardhana
26a866a202
Fix review comments and new changes in config (#8515)
- Migrate and save only settings which are enabled
- Rename logger_http to logger_webhook and
  logger_http_audit to audit_webhook
- No more pretty printing comments, comment
  is a key=value pair now.
- Avoid quotes on values which do not have space in them
- `state="on"` is implicit for all SetConfigKV unless
  specified explicitly as `state="off"`
- Disabled IAM users should be disabled always
2019-11-13 17:38:05 -08:00
Harshavardhana
d97d53bddc
Honor etcd legacy v/s new config settings properly (#8510)
This PR also fixes issues related to

- Add proper newline for `mc admin config get` output
  for more than one targets
- Fixes issue of temporary user credentials to have
  consistent output
- Fixes a crash when setting a key with empty values
- Fixes a parsing issue with `mc admin config history`
- Fixes gateway ENV handling for etcd server and gateway
2019-11-12 03:16:25 -08:00
Harshavardhana
aa04f97f95 Config migration should handle plain-text (#8506)
This PR fixes issues found in config migration

 - StorageClass migration error when rrs is empty
 - Plain-text migration of older config
 - Do not run in safe mode with incorrect credentials
 - Update logger_http documentation for _STATE env

Refer more reported issues at #8434
2019-11-11 12:01:21 -08:00
Kanagaraj M
4082764d48 fix loading config from openid config_url (#8503) 2019-11-11 09:31:46 -08:00
Harshavardhana
822eb5ddc7 Bring in safe mode support (#8478)
This PR refactors object layer handling such
that upon failure in sub-system initialization
server reaches a stage of safe-mode operation
wherein only certain API operations are enabled
and available.

This allows for fixing many scenarios such as

 - incorrect configuration in vault, etcd,
   notification targets
 - missing files, incomplete config migrations
   unable to read encrypted content etc
 - any other issues related to notification,
   policies, lifecycle etc
2019-11-09 09:27:23 -08:00
Harshavardhana
26863009c0 Load certs even if they are symlinks (#8494) 2019-11-08 11:59:20 +05:30
Harshavardhana
1e457dda7e Enhance config restore to carry previous set content as well (#8483)
This PR brings support for `history` list to
list in the following agreed format

```
~ mc admin config history list -n 2 myminio
RestoreId: df0ebb1e-69b0-4043-b9dd-ab54508f2897
Date: Mon, 04 Nov 2019 17:27:27 GMT

region name="us-east-1" state="on"
region name="us-east-1" state="on"
region name="us-east-1" state="on"
region name="us-east-1" state="on"

RestoreId: ecc6873a-0ed3-41f9-b03e-a2a1bab48b5f
Date: Mon, 04 Nov 2019 17:28:23 GMT

region name=us-east-1 state=off
```

This PR also moves the help templating and coloring to
fully `mc` side instead than `madmin` API.
2019-11-05 06:18:26 -08:00
Harshavardhana
d28bcb4f84 Migrate all backend at .minio.sys/config to encrypted backend (#8474)
- Supports migrating only when the credential ENVs are set,
  so any FS mode deployments which do not have ENVs set will
  continue to remain as is.
- Credential ENVs can be rotated using MINIO_ACCESS_KEY_OLD
  and MINIO_SECRET_KEY_OLD envs, in such scenarios it allowed
  to rotate the encrypted content to a new admin key.
2019-11-01 15:53:16 -07:00
Harshavardhana
9e7a3e6adc Extend further validation of config values (#8469)
- This PR allows config KVS to be validated properly
  without being affected by ENV overrides, rejects
  invalid values during set operation

- Expands unit tests and refactors the error handling
  for notification targets, returns error instead of
  ignoring targets for invalid KVS

- Does all the prep-work for implementing safe-mode
  style operation for MinIO server, introduces a new
  global variable to toggle safe mode based operations
  NOTE: this PR itself doesn't provide safe mode operations
2019-10-30 23:39:09 -07:00
Harshavardhana
47b13cdb80 Add etcd part of config support, add noColor/json support (#8439)
- Add color/json mode support for get/help commands
- Support ENV help for all sub-systems
- Add support for etcd as part of config
2019-10-30 00:04:39 -07:00
Harshavardhana
ee4a6a823d Migrate config to KV data format (#8392)
- adding oauth support to MinIO browser (#8400) by @kanagaraj
- supports multi-line get/set/del for all config fields
- add support for comments, allow toggle
- add extensive validation of config before saving
- support MinIO browser to support proper claims, using STS tokens
- env support for all config parameters, legacy envs are also
  supported with all documentation now pointing to latest ENVs
- preserve accessKey/secretKey from FS mode setups
- add history support implements three APIs
  - ClearHistory
  - RestoreHistory
  - ListHistory
- add help command support for each config parameters
- all the bug fixes after migration to KV, and other bug
  fixes encountered during testing.
2019-10-22 22:59:13 -07:00
Harshavardhana
6a4ef2e48e Initialize configs correctly, move notification config (#8367)
This PR also removes deprecated tests, adds checks
to avoid races reproduced on CI/CD.
2019-10-09 11:41:15 +05:30
Harshavardhana
290ad0996f Move etcd, logger, crypto into their own packages (#8366)
- Deprecates _MINIO_PROFILER, `mc admin profile` does the job
- Move ENVs to common location in cmd/config/
2019-10-08 11:17:56 +05:30
Harshavardhana
3b8adf7528 Move storageclass config handling into cmd/config/storageclass (#8360)
Continuation of the changes done in PR #8351 to refactor,
add tests and move global handling into a more idiomatic
style for Go as packages.
2019-10-07 11:20:24 +05:30
Harshavardhana
589e32a4ed Refactor config and split them in packages (#8351)
This change is related to larger config migration PR
change, this is a first stage change to move our
configs to `cmd/config/` - divided into its subsystems
2019-10-04 23:05:33 +05:30