Commit Graph

4104 Commits

Author SHA1 Message Date
Anis Elleuch
cb0b36f8c2 svcacct: Fix updating service account and add missing check (#12251)
UpdateServiceAccount ignores updating fields when not passed from upper
layer, such as empty policy, empty account status, and empty secret key.

This PR will check for a secret key only if it is empty and add more
check on the value of the account status.

Signed-off-by: Anis Elleuch <anis@min.io>
2021-05-07 09:13:30 -07:00
Klaus Post
254698f126 fix: minor allocation improvements in xlMetaV2 (#12133) 2021-05-07 09:11:05 -07:00
Krishnan Parthasarathi
0bab1c1895 Heal restored object contents on disk (#12238) 2021-05-06 16:06:57 -07:00
Harshavardhana
d495cb68d3 fix: crash in prometherus metrics collector (#12244)
node_health metrics crashes in gateway mode,
in gateway mode ignore node health metrics.

fixes #12243
2021-05-06 15:43:34 -07:00
Andreas Auernhammer
af0c65be93 add SSE-KMS support and use SSE-KMS for auto encryption (#12237)
This commit adds basic SSE-KMS support.
Now, a client can specify the SSE-KMS headers
(algorithm, optional key-id, optional context)
such that the object gets encrypted using the
SSE-KMS method. Further, auto-encryption now
defaults to SSE-KMS.

This commit does not try to do any refactoring
and instead tries to implement SSE-KMS as a minimal
change to the code base. However, refactoring the entire
crypto-related code is planned - but needs a separate
effort.

Signed-off-by: Andreas Auernhammer <aead@mail.de>
2021-05-06 15:24:01 -07:00
Nitish Tiwari
776589f0da Add free inode metric for Prometheus (#12225) 2021-05-06 12:50:48 -07:00
Harshavardhana
361940706d fix: avoid races in NewMultipartUpload under multiple pools (#12233)
It is possible in some scenarios that in multiple pools,
two concurrent calls for the same object as a multipart operation
can lead to duplicate entries on two different pools.

This PR fixes this

- hold locks to serialize multiple callers so that we don't race.
- make sure to look for existing objects on the namespace as well
  not just for existing uploadIDs
2021-05-06 10:45:33 -07:00
Harshavardhana
1aa5858543 move madmin to github.com/minio/madmin-go (#12239) 2021-05-06 08:52:02 -07:00
Harshavardhana
f4623ea8dc fix: validate secret key before updating service accounts 2021-05-05 16:41:47 -07:00
Harshavardhana
b8833c2947 do not change targetUser after permission validation
for service accounts make sure that targetUser is
always the one that is presented/validated from
the incoming request, not the parentUser.
2021-05-05 16:13:52 -07:00
Anis Elleuch
af1b6e3458 iam: Do not create service accounts for non existant IAM users (#12236)
When running MinIO server without LDAP/OpenID, we should error out when
the code tries to create a service account for a non existant regular
user.

Bonus: refactor the check code to be show all cases more clearly

Signed-off-by: Anis Elleuch <anis@min.io>

Co-authored-by: Anis Elleuch <anis@min.io>
2021-05-05 16:04:50 -07:00
Harshavardhana
0eeb0a4e04 Revert "add SSE-KMS support and use SSE-KMS for auto encryption (#11767)"
This reverts commit 26f1fcab7d.
2021-05-05 15:20:46 -07:00
Andreas Auernhammer
26f1fcab7d add SSE-KMS support and use SSE-KMS for auto encryption (#11767)
This commit adds basic SSE-KMS support.
Now, a client can specify the SSE-KMS headers
(algorithm, optional key-id, optional context)
such that the object gets encrypted using the
SSE-KMS method. Further, auto-encryption now
defaults to SSE-KMS.

This commit does not try to do any refactoring
and instead tries to implement SSE-KMS as a minimal
change to the code base. However, refactoring the entire
crypto-related code is planned - but needs a separate
effort.

Signed-off-by: Andreas Auernhammer <aead@mail.de>
Co-authored-by: Klaus Post <klauspost@gmail.com>
2021-05-05 11:24:14 -07:00
Harshavardhana
3a0e7347ca support startTLS with serverName TLSConfig (#12219)
fixes #12216
2021-05-04 20:13:24 -07:00
Harshavardhana
67001e3ce9 fix: allow root credentials to generate STS, service accounts (#12210) 2021-05-04 11:58:19 -07:00
Harshavardhana
804a23a06d update docs to remove _OLD credential references
also update the docs about config, IAM on encryption.
2021-05-04 10:27:51 -07:00
Nitish Tiwari
c8aa56ccd7 Add node cpu & memory metrics to Prometheus cluster endpoint (#12214) 2021-05-04 10:17:10 -07:00
Harshavardhana
ff36baeaa7 fix: attempt to drain the ReadFileStream for connection pooling (#12208)
avoid time_wait build up with getObject requests if there are
pending callers and they timeout, can lead to time_wait states

Bonus share the same buffer pool with erasure healing logic,
additionally also fixes a race where parallel readers were
never cleanup during Encode() phase, because pipe.Reader end
was never closed().

Added closer right away upon an error during Encode to make
sure to avoid racy Close() while stream was still being
Read().
2021-05-04 10:12:08 -07:00
Krishnan Parthasarathi
860bf1bab2 Add IsRemote method on FileInfo, ObjectInfo (#12209)
Provides a convenient method to know if an object's contents are in its remote
tier.
2021-05-04 08:40:42 -07:00
Andreas Auernhammer
4815f92fa8 fix MINIO_KMS_SECRET_KEY env. variable parsing (#12200)
This commit fixes a bug when parsing the env. variable
`MINIO_KMS_SECRET_KEY`. Before, the env. variable
name - instead of its value - was parsed. This (obviously)
did not work properly.

This commit fixes this.

Signed-off-by: Andreas Auernhammer <aead@mail.de>
2021-04-30 18:47:30 -07:00
Harshavardhana
0d3ddf7286 fix: improve NewObjectReader implementation for careful cleanup usage (#12199)
cleanup functions should never be cleaned before the reader is
instantiated, this type of design leads to situations where order
of lockers and places for them to use becomes confusing.

Allow WithCleanupFuncs() if the caller wishes to add cleanupFns
to be run upon close() or an error during initialization of the
reader.

Also make sure streams are closed before we unlock the resources,
this allows for ordered cleanup of resources.
2021-04-30 18:37:58 -07:00
Harshavardhana
3524c00090 fix: nats testdata relocation fix 2021-04-30 17:22:56 -07:00
Harshavardhana
f7a87b30bf Revert "deprecate embedded browser (#12163)"
This reverts commit 736d8cbac4.

Bring contrib files for older contributions
2021-04-30 08:50:39 -07:00
Harshavardhana
64f6020854 fix: cleanup locking, cancel context upon lock timeout (#12183)
upon errors to acquire lock context would still leak,
since the cancel would never be called. since the lock
is never acquired - proactively clear it before returning.
2021-04-29 20:55:21 -07:00
Harshavardhana
0faa4e6187 fix: make sure failed requests only to failed queue (#12196)
failed queue should be used for retried requests to
avoid cascading the failures into incoming queue, this
would allow for a more fair retry for failed replicas.

Additionally also avoid taking context in queue task
to avoid confusion, simplifies its usage.
2021-04-29 18:20:39 -07:00
Poorna Krishnamoorthy
90112b5644 Update ReplicationStatus if metadata not updated correctly (#12191)
There can be situations where replication completed but the
`X-Amz-Replication-Status` metadata update failed such as
when the server returns 503 under high load. This object version will
continue to be picked up by the scanner and replicateObject would perform
no action since the versions match between source and target.
The metadata would never reflect that replication was successful
without this fix, leading to repeated re-queuing.
2021-04-29 16:46:26 -07:00
Harshavardhana
c4b21ac7fa fix: remove healthcheck routine for replication targets (#12192)
Bonus also fix a racy lookup on arnsMap() without a
read lock, hold read locks to avoid such race.

moving the healthcheck logic to minio-go
2021-04-29 16:41:28 -07:00
Andreas Auernhammer
e5ec1325fc docs: add QuickStart section to KMS encryption of IAM data (#12190)
This commit enhances the docs about IAM encryption.
It adds a quick-start section that explains how to
get started quickly with `MINIO_KMS_SECRET_KEY`
instead of setting up KES.

It also removes the startup message that gets printed
when the server migrates IAM data to plaintext.
We will point this out in the release notes.

Signed-off-by: Andreas Auernhammer <aead@mail.de>
2021-04-29 14:20:28 -07:00
Harshavardhana
c5a80ca5d5 support service accounts for OpenID connect properly (#12178)
OpenID connect generated service accounts do not work
properly after console logout, since the parentUser state
is lost - instead use sub+iss claims for parentUser, according
to OIDC spec both the claims provide the necessary stability
across logins etc.
2021-04-29 13:01:42 -07:00
Harshavardhana
8cd89e10ea Revert "fix: remove deprecated MINIO_ACCESS_KEY, MINIO_SECRET_KEY envs (#12173)"
This reverts commit b0baaeaa3d.
2021-04-29 10:56:53 -07:00
Harshavardhana
091845df39 fix: return quorum error upon decode failures (#12184) 2021-04-29 10:00:03 -07:00
Harshavardhana
336c8ac99f fix: do not heal when disks are down (#12186)
HeadObject() was erroneously attempting
a heal when disks are down, avoid it.
2021-04-29 09:54:16 -07:00
Harshavardhana
b3c8a1864f fix: optimize ListBuckets for anonymous users (#12182)
anonymous users are never allowed to listBuckets(),
we do not need to further validate the policy, we can
simply reject if credentials are empty.
2021-04-28 21:37:02 -07:00
Poorna Krishnamoorthy
632252ff1d fix: change SetRemoteTarget API to allow editing remote target granularly (#12175)
Currently, only credentials could be updated with
`mc admin bucket remote edit`. 

Allow updating synchronous replication flag, path, 
bandwidth and healthcheck duration on buckets, and
a flag to disable proxying in active-active replication.
2021-04-28 15:26:20 -07:00
Harshavardhana
77f9c71133 Revert "redirect to console project for browser (#12172)"
This reverts commit 301669cf7b.

fixes #12179
2021-04-28 12:22:15 -07:00
Krishnan Parthasarathi
0c9d095deb ilm: Close warmBackend GetObject reader (#12174) 2021-04-27 22:42:18 -07:00
Harshavardhana
b0baaeaa3d fix: remove deprecated MINIO_ACCESS_KEY, MINIO_SECRET_KEY envs (#12173) 2021-04-27 22:41:24 -07:00
Harshavardhana
301669cf7b redirect to console project for browser (#12172) 2021-04-27 16:39:41 -07:00
Anis Elleuch
9e797532dc lock: Always cancel the returned Get(R)Lock context (#12162)
* lock: Always cancel the returned Get(R)Lock context

There is a leak with cancel created inside the locking mechanism. The
cancel purpose was to cancel operations such erasure get/put that are
holding non-refreshable locks.

This PR will ensure the created context.Cancel is passed to the unlock
API so it will cleanup and avoid leaks.

* locks: Avoid returning nil cancel in local lockers

Since there is no Refresh mechanism in the local locking mechanism, we
do not generate a new context or cancel. Currently, a nil cancel
function is returned but this can cause a crash. Return a dummy function
instead.
2021-04-27 16:12:50 -07:00
Harshavardhana
736d8cbac4 deprecate embedded browser (#12163)
https://github.com/minio/console takes over the functionality for the
future object browser development

Signed-off-by: Harshavardhana <harsha@minio.io>
2021-04-27 10:52:12 -07:00
Harshavardhana
cf335f6c63 service accounts should use LDAP user DN to assign credentials (#12166)
LDAP DN should be used when allowing setting service accounts
for LDAP users instead of just simple user,

Bonus root owner should be allowed full access
to all service account APIs.

Signed-off-by: Harshavardhana <harsha@minio.io>
2021-04-27 10:04:08 -07:00
Harshavardhana
c8050bc079 fix: sleeper behavior in data scanner (#12164)
do not apply healReplication() for ILM
expired, transitioned objects
2021-04-27 08:24:44 -07:00
Harshavardhana
edda244066 move pkg/rpc, pkg/csvparser, pkg/argon2 to contrib
Signed-off-by: Harshavardhana <harsha@minio.io>
2021-04-26 18:24:40 -07:00
Poorna Krishnamoorthy
4be0f92067 Fix multipart restore to remove part match (#12161)
Part ETags are not available after multipart finalizes, removing this
check as not useful.

Signed-off-by: Poorna Krishnamoorthy <poorna@minio.io>
Co-authored-by: Harshavardhana <harsha@minio.io>
2021-04-26 18:24:06 -07:00
Harshavardhana
26544848ea remove legacy master_key support by June (#12153)
Signed-off-by: Harshavardhana <harsha@minio.io>
2021-04-26 16:02:05 -07:00
Harshavardhana
2966823818 use jsoniter for json marshal/unmarshal in KMS (#12146)
Signed-off-by: Harshavardhana <harsha@minio.io>
2021-04-26 16:01:52 -07:00
Harshavardhana
d501c5e38b add missing responseBody drain (#12147)
Signed-off-by: Harshavardhana <harsha@minio.io>
2021-04-26 08:59:54 -07:00
Harshavardhana
d825d92499 rename production to release directory, rebuild assets 2021-04-25 16:51:29 -07:00
Andreas Auernhammer
f7feff8665 avoid parsing MINIO_KMS_MASTER_KEY as base64 (#12149)
This commit reverts a change that added support for
parsing base64-encoded keys set via `MINIO_KMS_MASTER_KEY`.

The env. variable `MINIO_KMS_MASTER_KEY` is deprecated and
should ONLY support parsing existing keys - not the new format.

Any new deployment should use `MINIO_KMS_SECRET_KEY`. The legacy
env. variable `MINIO_KMS_MASTER_KEY` will be removed at some point
in time.

Signed-off-by: Andreas Auernhammer <aead@mail.de>
2021-04-25 11:04:31 -07:00
Harshavardhana
4eb9b6eaf8 preserve metadata multipart restore (#12139)
avoid re-read of xl.meta instead just use
the success criteria from PutObjectPart()
and check the ETag matches per Part, if
they match then the parts have been
successfully restored as is.

Signed-off-by: Harshavardhana <harsha@minio.io>
2021-04-24 19:07:27 -07:00