Commit Graph

9035 Commits

Author SHA1 Message Date
Klaus Post
7f49c38e2d
Recover corrupted usage files if any (#13179) 2021-09-09 11:24:22 -07:00
Klaus Post
0e7fdcee30
Healing: Decide healing inlining based on metadata (#13178)
Don't perform an independent evaluation of inlining, but mirror the decision made when uploading the object.

Leads to some objects being inlined or not based on new metrics. Instead respect previous decision.
2021-09-09 08:55:43 -07:00
Poorna Krishnamoorthy
418f8bed6a
Detect multipart uploads correctly in unencrypted case (#13176)
This is a fix building on #13171 to ensure objects uploaded using multipart are 
replicated as multipart
2021-09-09 07:52:49 -07:00
soudhaf
950fe73c4f
fix: minio console nginx config (#13156) 2021-09-08 23:41:06 -07:00
Harshavardhana
0892f1e406
fix: multipart replication and encrypted etag for sse-s3 (#13171)
Replication was not working properly for encrypted
objects in single PUT object for preserving etag,

We need to make sure to preserve etag such that replication
works properly and not gets into infinite loops of copying
due to ETag mismatches.
2021-09-08 22:25:23 -07:00
Poorna Krishnamoorthy
9af4e7b1da
Add healthcheck back for replication targets (#13168)
This will allow objects to relinquish read lock held during
replication earlier if the target is known to be down
without waiting for connection timeout when replication 
is attempted.
2021-09-08 15:34:50 -07:00
Harshavardhana
198a838d00 update console tag to v0.9.8 2021-09-08 14:05:17 -07:00
Harshavardhana
aaa3fc3805
fix: remove deprecated LDAP username format support (#13165) 2021-09-08 13:31:51 -07:00
Klaus Post
3c2efd9cf3
Stop async listing earlier (#13160)
Stop async listing if we have not heard back from the client for 3 minutes.

This will stop spending resources on async listings when they are unlikely to get used. 
If the client returns a new listing will be started on the second request.

Stop saving cache metadata to disk. It is cleared on restarts anyway. Removes all 
load/save functionality
2021-09-08 11:06:45 -07:00
Harshavardhana
951b1e6a7a
fix: Optimize listing calls for NFS mounts (#13159)
--no-compat should allow for some optimized
behavior for NFS mounts by removing Stat()
operations.
2021-09-08 08:15:42 -07:00
Harshavardhana
9c5fd6a776 update CREDITS file with latest 2021-09-07 19:17:18 -07:00
Andreas Auernhammer
e438dccf19
sts: add support for certificate-based authentication (#12748)
This commit adds a new STS API for X.509 certificate
authentication.

A client can make an HTTP POST request over a TLS connection
and MinIO will verify the provided client certificate, map it to an 
S3 policy and return temp. S3 credentials to the client.

So, this STS API allows clients to authenticate with X.509
certificates over TLS and obtain temp. S3 credentials.

For more details and examples refer to the docs/sts/tls.md
documentation.

Signed-off-by: Andreas Auernhammer <hi@aead.dev>
2021-09-07 19:03:48 -07:00
Andreas Auernhammer
43d2655ee4
tls: add TLS 1.3 ciphers to the list of supported ciphers (#13158)
This commit adds the TLS 1.3 ciphers to the list of
supported ciphers. Now, clients can connect to MinIO
using TLS 1.3

Signed-off-by: Andreas Auernhammer <hi@aead.dev>
2021-09-07 09:57:32 -07:00
Klaus Post
b2c92cdaaa
select: Add more compression formats (#13142)
Support Zstandard, LZ4, S2, and snappy as additional 
compression formats for S3 Select.
2021-09-06 09:09:53 -07:00
Anis Elleuch
42b1d92b2a
azure: Fix crash after using ParseForm() for query param lookups (#13147)
Azure storage SDK uses http.Request feature which panics when the
request contains r.Form popuplated.

Azure gateway code creates a new request, however it modifies the
transport to add our metrics code which sets Request.Form during
shouldMeterRequest() call.

This commit simplifies shouldMeterRequest() to avoid setting
request.Form and avoid the crash.
2021-09-04 09:07:24 -07:00
Harshavardhana
1250312287
fail ready/liveness if etcd is unhealthy in gateway mode (#13146) 2021-09-03 17:05:41 -07:00
Klaus Post
308371b434
Clean up ToFileInfo and avoid copy (#13144)
Simplify code and remove an iteration of all versions. Remove unneded copy.
2021-09-03 12:31:32 -07:00
Anis Elleuch
88e6c11746
iam: Retry connection to etcd during initialization (#13143)
Retry connecting to etcd during IAM initialization when etcd is enabled.
2021-09-03 09:10:48 -07:00
Harshavardhana
2ee4ae88f9 update helm release to 3.0.2 2021-09-03 01:11:32 -07:00
Minio Trusted
e2b6fb0a6a Update yaml files to latest version RELEASE.2021-09-03T03-56-13Z 2021-09-03 06:45:54 +00:00
Harshavardhana
a19e3bc9d9
add more dangling heal related tests (#13140)
also make sure that HealObject() never returns
'ObjectNotFound' or 'VersionNotFound' errors,
as those are meaningless and not useful for the
caller.
2021-09-02 20:56:13 -07:00
Harshavardhana
495c55e6a5
fix: make sure to delete dangling objects during heal (#13138)
heal with --remove was not removing dangling versions
on versioned buckets, this PR fixes this properly.

this is a regression introduced in PR #12617
2021-09-02 17:45:30 -07:00
Poorna Krishnamoorthy
a366143c5b
Remove replication permission check (#13135)
Fixes #13105
2021-09-02 09:31:13 -07:00
Harshavardhana
b8f5a7db33 update console release to v0.9.6 2021-09-01 23:57:19 -07:00
Harshavardhana
f486cfae86 point all docs for containers to quay.io 2021-09-01 18:48:26 -07:00
Harshavardhana
0838732df3 point helm releases to quay.io 2021-09-01 18:47:43 -07:00
Harshavardhana
f422f09dff update console to latest master 2021-09-01 18:10:44 -07:00
Harshavardhana
aff196ae57 add helm chart support for gateway 2021-09-01 16:27:12 -07:00
Harshavardhana
27c9f8be7a ignore MINIO_CERT_PASSWD to be same on all instances 2021-09-01 16:02:35 -07:00
Harshavardhana
67b6c945e2 whitelist certain ENVs from bootstrap check 2021-09-01 15:10:17 -07:00
Harshavardhana
c89aee37b9
fix: log errors for incorrect environment inputs (#13121)
Invalid MINIO_ARGS, MINIO_ENDPOINTS would be silently
ignored when using remoteEnv style, make sure to log
errors to indicate invalid configuration.
2021-09-01 11:34:07 -07:00
Harshavardhana
03b7bebc96
fix: invalid quorum calculation in TransitionObject (#13125)
Quorum calculation should be based on the
existing metadata, custom quorum calculation
can lead to unreadable content.
2021-09-01 08:57:42 -07:00
Harshavardhana
f89d0f68d0
fix: missing cleanup of tmp folders in NAS gateway setup (#13124)
console service should be shutdown last once all shutdown
sequences are complete, this is to ensure that we do not
prematurely kill the server before it cleans up the

`.minio.sys/tmp/uuid` folder.

NOTE: this only applies to NAS gateway setup.
2021-08-31 18:52:48 -07:00
Harshavardhana
72a288f73f
fix: add x-amz-server-side-encryption header in ListObjectsM (#13122)
fixes #13046
2021-08-31 17:18:13 -07:00
Harshavardhana
0073aee1ed update to console release v0.9.5 2021-08-31 15:18:43 -07:00
Harshavardhana
0f7a51f461
fix: speedTest between peers keep the connection alive (#13120)
for longer durations keep the speedTest alive instead
of timing them out based on ResponseHeaderTimeout.
2021-08-31 14:08:23 -07:00
Klaus Post
556552340a
listing: Don't log errFileNotFound and friends (#13119) 2021-08-31 09:46:42 -07:00
Harshavardhana
50bce0130a release helm/minio v2.0.1 upgrade MinIO 2021-08-31 02:09:09 -07:00
Minio Trusted
5c6dc63577 Update yaml files to latest version RELEASE.2021-08-31T05-46-54Z 2021-08-31 07:27:43 +00:00
Harshavardhana
2077d27053 init() globalAPIConfig only once 2021-08-30 22:46:54 -07:00
Klaus Post
76b3d3c559
fix: missing close on error for keepAlive connections (#13109)
Add missing close when error is reported
before body is done.
2021-08-30 08:46:46 -07:00
Dominik Hassler
514b2d6f12
fix: build on illumos (Solaris) (#13097) 2021-08-30 08:40:16 -07:00
Klaus Post
470553ff5d
Tweak readall allocation and renameData buffer reuse (#13108)
Use a single allocation for reading the file, not the growing buffer of `io.ReadAll`.

Reuse the write buffer if we can when writing metadata in RenameData.
2021-08-30 08:38:11 -07:00
Harshavardhana
88e0aa1cb2
verify all nodes have same ENVs in bootstrap (#13096) 2021-08-30 08:27:39 -07:00
Harshavardhana
35f2552fc5
reduce extra getObjectInfo() calls during ILM transition (#13091)
* reduce extra getObjectInfo() calls during ILM transition

This PR also changes expiration logic to be non-blocking,
scanner is now free from additional costs incurred due
to slower object layer calls and hitting the drives.

* move verifying expiration inside locks
2021-08-27 17:06:47 -07:00
Anis Elleuch
e05886561d
lock: Fix Refresh logic with multi resources lock (#13092)
A multi resources lock is a single lock UID with multiple associated
resources. This is created for example by multi objects delete
operation. This commit changes the behavior of Refresh() to iterate over
all locks having the same UID and refresh them.

Bonus: Fix showing top locks for multi delete objects
2021-08-27 13:07:55 -07:00
Klaus Post
2451b9a75a
fix: hanging operations on PUT with slow IO (#13087)
#11878 added "keepHTTPResponseAlive" to CreateFile requests. 
The problem is that it will begin writing to the response before the 
body is read after 10 seconds. This will abort the writes on the 
client-side, since it assumes the server has received what it wants.

The proposed solution here is to monitor the completion of the body 
before beginning to send keepalive pings.

Fixes observed high number of goroutines stuck in `io.Copy` in 
`github.com/minio/minio/cmd.(*xlStorage).CreateFile` and 
`(*storageRESTClient).CreateFile` stuck in `http.DrainBody`.
2021-08-27 09:16:36 -07:00
Anis Elleuch
06b71c99ee
locks: Ensure local lock removal after a failed refresh (#12979)
In the event when a lock is not refreshed in the cluster, this latter
will be automatically removed in the subsequent cleanup of non 
refreshed locks routine, but it forgot to clean the local server, 
hence having the same weird stale locks present.

This commit will remove the lock locally also in remote nodes, if
removing a lock from a remote node will fail, it will be anyway 
removed later in the locks cleanup routine.
2021-08-27 08:59:36 -07:00
Harshavardhana
ae8f7f11d5
fix: svc accounts cannot have same name as parent/targetUser (#13082)
Currently in master this can cause existing
parent users to stop working and lead to
credentials getting overwritten.

```
~ mc admin user add alias/ minio123 minio123456
```

```
~ mc admin user svcacct add alias/ minio123 \
    --access-key minio123 --secret-key minio123456
```

This PR rejects all such scenarios.
2021-08-26 21:57:30 -07:00
Harshavardhana
ed16ce9b73
add healing workers support to parallelize healing (#13081)
Faster healing as well as making healing more
responsive for faster scanner times.

also fixes a bug introduced in #13079, newly replaced
disks were not healing automatically.
2021-08-26 20:32:58 -07:00