Commit Graph

8918 Commits

Author SHA1 Message Date
Poorna Krishnamoorthy 6c941122eb
cancel active goroutine when remote target is edited (#13243) 2021-09-17 20:05:38 -07:00
Harshavardhana 1a884cd8e1
fix: deleting objects was not working after upgrades (#13242)
DeleteObject() on existing objects before `xl.json` to
`xl.meta` change were not working, not sure when this
regression was added. This PR fixes this properly.

Also this PR ensures that we perform rename of xl.json
to xl.meta only during "write" phase of the call i.e
either during Healing or PutObject() overwrites.

Also handles few other scenarios during migration where
`backendEncryptedFile` was missing deleteConfig() will
fail with `configNotFound` this case was not ignored,
which can lead to failure during upgrades.
2021-09-17 19:34:48 -07:00
Poorna Krishnamoorthy 18f008f7c7
Fix retention enforcement check for deleted object versions (#13240)
if an object is pending version purge, it should be treated
as ErrNone in retention enforcement check
2021-09-17 15:21:24 -07:00
Harshavardhana 6d42569ade
remove ListBucketsMetadata instead add them to AccountInfo() (#13241) 2021-09-17 15:02:21 -07:00
Harshavardhana 5ed781a330
check for context canceled after competing for locks (#13239)
once we have competed for locks, verify if the
context is still valid - this is to ensure that
we do not start readdir() or read() calls on the
drives on canceled connections.
2021-09-17 14:11:01 -07:00
Harshavardhana 66fcd02aa2
de-couple walkMu and walkReadMu for some granularity (#13231)
This commit brings two locks instead of single lock for
WalkDir() calls on top of c25816eabc.

The main reason is to avoid contention between readMetadata()
and ListDir() calls, ListDir() can take time on prefixes that
are huge for readdir() but this shouldn't end up blocking
all readMetadata() operations, this allows for more room for
I/O while not overly penalizing all listing operations.
2021-09-17 12:14:12 -07:00
Andreas Auernhammer 1fc0e9a6aa
sts: allow clients to send certificate chain (#13235)
This commit fixes an issue in the `AssumeRoleWithCertificate`
handler.

Before clients received an error when they send
a chain of X.509 certificates (their client certificate as
well as intermediate / root CAs).

Now, client can send a certificate chain and the server
will only consider non-CA / leaf certificates as possible
client certificate candidates. However, the client still
can only send one certificate.

Signed-off-by: Andreas Auernhammer <hi@aead.dev>
2021-09-17 09:37:01 -07:00
Anis Elleuch 91567ba916
s3: Put bucket tagging to return an error when bucket is not found (#13232)
instead of creating new metadata in .minio.sys directory
2021-09-17 08:32:32 -07:00
Klaus Post d80826b05d
Clean up metacache saver (#13225)
Don't report success before the listing has actually finished. 
This will make stop conditions more clear.
2021-09-16 13:35:25 -07:00
Harshavardhana 45bcf73185
feat: Add ListBucketsWithMetadata extension API (#13219) 2021-09-16 09:52:41 -07:00
Poorna Krishnamoorthy 78dc08bdc2
remove s3:ReplicateDelete permission check from DeleteObject APIs (#13220) 2021-09-15 23:02:16 -07:00
Klaus Post f98f115ac2
fs: Fix non-progressing scanner (#13218)
Scanner would keep doing the same cycle in FS mode leading to missed updates.

Add a few sanity checks and handle errors better.
2021-09-15 09:24:41 -07:00
Minio Trusted bf409936e7 Update yaml files to latest version RELEASE.2021-09-15T04-54-25Z 2021-09-15 08:04:57 +00:00
Shireesh Anjal b4364723ef
Add config to store subnet license (#13194)
Command to set subnet license:

`mc admin config set {alias} subnet license={token}`

Signed-off-by: Shireesh Anjal <shireesh@minio.io>
Co-authored-by: Harshavardhana <harsha@minio.io>
2021-09-14 21:54:25 -07:00
Harshavardhana bcc6359dec support Console UI with userInfo claims for OpenID 2021-09-14 17:09:18 -07:00
Harshavardhana 787a72a993
make sure to ignore the rootDisk when healing drives (#13209)
fixes #13208
2021-09-14 15:10:00 -07:00
Harshavardhana d9eb962969
allow admin API to support UNSIGNED-PAYLOAD (#13207)
admin API requests do not support x-amz-content-sha256
set with UNSIGNED-PAYLOAD, keep this consistent and
support it properly.
2021-09-14 13:55:24 -07:00
Anis Elleuch f221153776
s3-gateway: Allow encryption S3 passthrough for SSE-S3 (#13204)
This reverts commit 35cbe43b6d.
2021-09-14 12:55:32 -07:00
Harshavardhana 67596ef0cc
fix sse-kms context unmarshal failure (#13206)
json.Unmarshal expects a pointer receiver, otherwise
kms.Context unmarshal fails with lack of pointer receiver,
this becomes complicated due to type aliasing over
map[string]string - fix it properly.
2021-09-14 12:52:46 -07:00
Klaus Post bf5bfe589f
xlmeta: Recover corrupted metadata (#13205)
When unable to load existing metadata new versions 
would not be written. This would leave objects in a 
permanently unrecoverable state

Instead, start with clean metadata and write the incoming data.
2021-09-14 11:34:25 -07:00
Harshavardhana af78c3925a
add userinfo support for OpenID (#12469)
Some identity providers like GitLab do not provide
information about group membership as part of the
identity token claims. They only expose it via OIDC compatible
'/oauth/userinfo' endpoint, as described in the OpenID
Connect 1.0 sepcification.

But this of course requires application to make sure to add
additional accessToken, since idToken cannot be re-used to
perform the same 'userinfo' call. This is why this is specialized
requirement. Gitlab seems to be the only OpenID vendor that requires
this support for the time being.

fixes #12367
2021-09-13 16:22:14 -07:00
Harshavardhana d144958669 update helm to v3.1.1 - saAccount fix 2021-09-13 09:43:10 -07:00
Klaus Post 5a64003f6f
select: Return null for non-exiting column indexes (#13196)
Fixes #13186
2021-09-13 09:13:25 -07:00
Kanagaraj M 311718309c
fix ServiceAccount creation in helm chart (#13197)
Fixed the variable name on the templated and added name.
2021-09-13 09:12:20 -07:00
Anis Elleuch 98479d7ffd
Fix deadlock when error during metacache generation (#13201)
A typo forgot to release a lock after acquiring it.
2021-09-13 09:11:39 -07:00
Harshavardhana 90e505e58f calculate API requests/error as increase() intervals not as rate() 2021-09-12 11:28:28 -07:00
Harshavardhana 410b8dd0fd add service monitor support and service account support 2021-09-12 11:19:27 -07:00
Anis Elleuch c2f25b6f62
gateway/s3: allow tracing requests to backend service (#13189)
fixes #13089
fixes #13133

Co-authored-by: Anis Elleuch <anis@min.io>
Co-authored-by: Harshavardhana <harsha@minio.io>
2021-09-11 09:20:01 -07:00
Krishna Srinivas 03a2a74697
Support speedtest autotune on the server side (#13086) 2021-09-10 17:43:34 -07:00
ArthurMa 2807c11410
http hook should accept more than 200 statusCode (#13180)
Co-authored-by: Klaus Post <klauspost@gmail.com>
2021-09-10 14:27:37 -07:00
Harshavardhana 39d51ce845 fix: add Dockerfile.release* /opt/bin writable 2021-09-09 22:27:33 -07:00
Minio Trusted a216583d95 Update yaml files to latest version RELEASE.2021-09-09T21-37-07Z 2021-09-09 23:50:26 +00:00
Harshavardhana 5c448b1b97
fix: update allowed max chunk payloadsize to 16MiB (#13169)
fixes #13163
2021-09-09 14:37:07 -07:00
Klaus Post 7f49c38e2d
Recover corrupted usage files if any (#13179) 2021-09-09 11:24:22 -07:00
Klaus Post 0e7fdcee30
Healing: Decide healing inlining based on metadata (#13178)
Don't perform an independent evaluation of inlining, but mirror the decision made when uploading the object.

Leads to some objects being inlined or not based on new metrics. Instead respect previous decision.
2021-09-09 08:55:43 -07:00
Poorna Krishnamoorthy 418f8bed6a
Detect multipart uploads correctly in unencrypted case (#13176)
This is a fix building on #13171 to ensure objects uploaded using multipart are 
replicated as multipart
2021-09-09 07:52:49 -07:00
soudhaf 950fe73c4f
fix: minio console nginx config (#13156) 2021-09-08 23:41:06 -07:00
Harshavardhana 0892f1e406
fix: multipart replication and encrypted etag for sse-s3 (#13171)
Replication was not working properly for encrypted
objects in single PUT object for preserving etag,

We need to make sure to preserve etag such that replication
works properly and not gets into infinite loops of copying
due to ETag mismatches.
2021-09-08 22:25:23 -07:00
Poorna Krishnamoorthy 9af4e7b1da
Add healthcheck back for replication targets (#13168)
This will allow objects to relinquish read lock held during
replication earlier if the target is known to be down
without waiting for connection timeout when replication 
is attempted.
2021-09-08 15:34:50 -07:00
Harshavardhana 198a838d00 update console tag to v0.9.8 2021-09-08 14:05:17 -07:00
Harshavardhana aaa3fc3805
fix: remove deprecated LDAP username format support (#13165) 2021-09-08 13:31:51 -07:00
Klaus Post 3c2efd9cf3
Stop async listing earlier (#13160)
Stop async listing if we have not heard back from the client for 3 minutes.

This will stop spending resources on async listings when they are unlikely to get used. 
If the client returns a new listing will be started on the second request.

Stop saving cache metadata to disk. It is cleared on restarts anyway. Removes all 
load/save functionality
2021-09-08 11:06:45 -07:00
Harshavardhana 951b1e6a7a
fix: Optimize listing calls for NFS mounts (#13159)
--no-compat should allow for some optimized
behavior for NFS mounts by removing Stat()
operations.
2021-09-08 08:15:42 -07:00
Harshavardhana 9c5fd6a776 update CREDITS file with latest 2021-09-07 19:17:18 -07:00
Andreas Auernhammer e438dccf19
sts: add support for certificate-based authentication (#12748)
This commit adds a new STS API for X.509 certificate
authentication.

A client can make an HTTP POST request over a TLS connection
and MinIO will verify the provided client certificate, map it to an 
S3 policy and return temp. S3 credentials to the client.

So, this STS API allows clients to authenticate with X.509
certificates over TLS and obtain temp. S3 credentials.

For more details and examples refer to the docs/sts/tls.md
documentation.

Signed-off-by: Andreas Auernhammer <hi@aead.dev>
2021-09-07 19:03:48 -07:00
Andreas Auernhammer 43d2655ee4
tls: add TLS 1.3 ciphers to the list of supported ciphers (#13158)
This commit adds the TLS 1.3 ciphers to the list of
supported ciphers. Now, clients can connect to MinIO
using TLS 1.3

Signed-off-by: Andreas Auernhammer <hi@aead.dev>
2021-09-07 09:57:32 -07:00
Klaus Post b2c92cdaaa
select: Add more compression formats (#13142)
Support Zstandard, LZ4, S2, and snappy as additional 
compression formats for S3 Select.
2021-09-06 09:09:53 -07:00
Anis Elleuch 42b1d92b2a
azure: Fix crash after using ParseForm() for query param lookups (#13147)
Azure storage SDK uses http.Request feature which panics when the
request contains r.Form popuplated.

Azure gateway code creates a new request, however it modifies the
transport to add our metrics code which sets Request.Form during
shouldMeterRequest() call.

This commit simplifies shouldMeterRequest() to avoid setting
request.Form and avoid the crash.
2021-09-04 09:07:24 -07:00
Harshavardhana 1250312287
fail ready/liveness if etcd is unhealthy in gateway mode (#13146) 2021-09-03 17:05:41 -07:00
Klaus Post 308371b434
Clean up ToFileInfo and avoid copy (#13144)
Simplify code and remove an iteration of all versions. Remove unneded copy.
2021-09-03 12:31:32 -07:00