Commit Graph

7425 Commits

Author SHA1 Message Date
Harshavardhana f7f9517b6a fix: host extraction without port 2020-09-23 12:10:14 -07:00
Harshavardhana 90cff10e2b avoid crash if disks are not initialized 2020-09-23 12:00:29 -07:00
Harshavardhana 81caf35926
fix: reduce healthcheck interval for storage rest client (#10544) 2020-09-23 10:43:42 -07:00
poornas 5726cef3ca
validate bucket exists in ListRemoteTargets api (#10552) 2020-09-23 10:37:54 -07:00
飞雪无情 5fdf47b118
Add Chinese documentation for distributed design (#10549) 2020-09-23 09:26:47 -07:00
Harshavardhana 8b74a72b21
fix: rename READY deadline to CLUSTER deadline ENV (#10535) 2020-09-23 09:14:33 -07:00
Klaus Post eec69d6796
Fix stale context for bucket retrieval (#10551)
The provided context gets captured by the closure making all subsequent calls fail.
2020-09-23 08:30:31 -07:00
Harshavardhana 0537a21b79
avoid concurrenct use of rand.NewSource (#10543) 2020-09-22 15:34:27 -07:00
poornas 4c54ed8748
Close replica channel only once (#10542)
Also enforce s3:GetReplicationConfiguration permission check as a
bucket level resource.
2020-09-22 12:47:24 -07:00
poornas a4006e23a0
Update replication docs to clarify permissions (#10536)
Co-authored-by: Klaus Post <klauspost@gmail.com>
2020-09-22 11:58:04 -07:00
Shireesh Anjal b17dc81540
Change "disks" node to "drives" in OBD output (#10540) 2020-09-22 11:53:19 -07:00
Minio Trusted d73c4f09f3 Update yaml files to latest version RELEASE.2020-09-21T22-31-59Z 2020-09-21 22:58:40 +00:00
Anis Elleuch 4c81201f95
fix: healing delete marker on versioned buckets (#10530)
Healing was not working correctly in the distributed mode because
errFileVersionNotFound was not properly converted in storage rest
client.

Besides, fixing the healing delete marker is not working as expected.
2020-09-21 15:16:16 -07:00
Harshavardhana cd8d511d3d move versionsOrder struct to xl-storage-utils 2020-09-21 14:24:42 -07:00
Kaan Kabalak 899a2fa1c7
re-order right hand side dropdown options (#10534) 2020-09-21 13:59:42 -07:00
Harshavardhana 17e17da00d
add parallel workers to perform replication in parallel (#10525)
set the concurrency for replication be to runtime.NumCPU()/2
2020-09-21 13:43:29 -07:00
Harshavardhana a5da9120f3
fix: [fs] an error upon rwPool.Write() just attempt rwPool.Create() (#10533)
On some NFS clients looks like errno is incorrectly set,
which leads to incorrect errors thrown upwards.
2020-09-21 12:54:23 -07:00
poornas aa12d75d75
fix crawler to detect lifecycle on bucket even if filter nil (#10532) 2020-09-21 11:41:07 -07:00
飞雪无情 dd4a2d7419
update distributed zh_CN document. (#10531) 2020-09-21 10:54:56 -07:00
Harshavardhana 6fcbdd5607
remove unused putObjectDir code (#10528) 2020-09-21 09:41:39 -07:00
Harshavardhana 3831cc9e3b
fix: [fs] CompleteMultipart use trie structure for partMatch (#10522)
performance improves by around 100x or more

```
go test -v -run NONE -bench BenchmarkGetPartFile
goos: linux
goarch: amd64
pkg: github.com/minio/minio/cmd
BenchmarkGetPartFileWithTrie
BenchmarkGetPartFileWithTrie-4          1000000000               0.140 ns/op           0 B/op          0 allocs/op
PASS
ok      github.com/minio/minio/cmd      1.737s
```

fixes #10520
2020-09-21 01:18:13 -07:00
Krishna Srinivas 230fc0d186
Support for "directory" objects (#10499) 2020-09-19 08:39:41 -07:00
Harshavardhana 7f9498f43f
fix: ignore faulty drives and continue (#10511)
drives might return different types of errors
handle them individually, and for some errors
just log an error and continue
2020-09-18 12:09:05 -07:00
Harshavardhana 1cf322b7d4
change leader locker only for crawler (#10509) 2020-09-18 11:15:54 -07:00
Derek Bender 3168e93730
fix typo in healthcheck README.md (#10518) 2020-09-18 09:52:37 -07:00
Klaus Post 0b1c824618
Fix incorrect request start time (#10516)
Log request start time BEFORE starting processing the request
2020-09-18 09:30:52 -07:00
Klaus Post c851e022b7
Tweaks to dynamic locks (#10508)
* Fix cases where minimum timeout > default timeout.
* Add defensive code for too small/negative timeouts.
* Never set timeout below the maximum value of a request.
* Protect against (unlikely) int64 wraps.
* Decrease timeout slower.
* Don't re-lock before copying.
2020-09-18 09:18:18 -07:00
Aleksey Pogibelev 6f45e303f5
fix tls flag (#10487) 2020-09-18 15:35:54 +05:30
Klaus Post 5ad032826a
Add a reasonable if unable to get total RAM (#10506)
Though unlikely we shouldn't skip initializing the API if we cannot get RAM.

Add 16GiB as a default and log the error.
2020-09-18 02:03:02 -07:00
Harshavardhana 84bf4624a4
fix: make sure to preserve metadata during overwrite in FS mode (#10512)
This bug was introduced in 14f0047295
almost 3yrs ago, as a side affect of removing stale `fs.json`
but we in-fact end up removing existing good `fs.json` for an
existing object, leading to some form of a data loss.

fixes #10496
2020-09-18 00:16:16 -07:00
飞雪无情 dff37aa33d
update disk caching zh_CN document. (#10472)
add disk caching DESIGN zh_CN document.
2020-09-18 00:07:48 -07:00
saurabh29789 d12831eb07
Add support for searching objects (#10424) 2020-09-17 23:01:37 -07:00
Harshavardhana 4a36cd7035
fix: improve performance ListObjectParts in FS mode (#10510)
from 20s for 10000 parts to less than 1sec

Without the patch
```
~ time aws --endpoint-url=http://localhost:9000 --profile minio s3api \
       list-parts --bucket testbucket --key test \
       --upload-id c1cd1f50-ea9a-4824-881c-63b5de95315a

real    0m20.394s
user    0m0.589s
sys     0m0.174s
```

With the patch
```
~ time aws --endpoint-url=http://localhost:9000 --profile minio s3api \
       list-parts --bucket testbucket --key test \
       --upload-id c1cd1f50-ea9a-4824-881c-63b5de95315a

real    0m0.891s
user    0m0.624s
sys     0m0.182s
```

fixes #10503
2020-09-17 18:51:16 -07:00
poornas 00555c747e
Strip standard ports off remote target url (#10498) 2020-09-17 11:09:50 -07:00
Klaus Post 03490c811b
Fix obd goroutine leak (#10504)
The gouroutine collecting transfer stats never exits. Add missing channel close.
2020-09-17 10:10:20 -07:00
Minio Trusted 48d2c03250 Update yaml files to latest version RELEASE.2020-09-17T04-49-20Z 2020-09-17 05:07:58 +00:00
Harshavardhana ed78854cea fix: list across all drives to avoid stale disks 2020-09-16 21:17:10 -07:00
Harshavardhana e60834838f
fix: background disk heal, to reload format consistently (#10502)
It was observed in VMware vsphere environment during a
pod replacement, `mc admin info` might report incorrect
offline nodes for the replaced drive. This issue eventually
goes away but requires quite a lot of time for all servers
to be in sync.

This PR fixes this behavior properly.
2020-09-16 21:14:35 -07:00
Harshavardhana d616d8a857
serialize replication and feed it through task model (#10500)
this allows for eventually controlling the concurrency
of replication and overally control of throughput
2020-09-16 16:04:55 -07:00
Anis Elleuch 24cab7f9df
ilm: Remove a 'null' version if not latest (#10494)
If the ILM document requires removing noncurrent versions, the 
the server should be able to remove 'null' versions as well. 
'null' versions are created when versioning is not enabled 
or suspended.
2020-09-16 10:21:50 -07:00
Minio Trusted b2536476c9 Update yaml files to latest version RELEASE.2020-09-16T04-22-35Z 2020-09-16 04:41:00 +00:00
Harshavardhana 02c1a08a5b
fix: make sure to lock CopyObject for in-place updates (#10492) 2020-09-15 20:44:48 -07:00
Ritesh H Shukla 5c47ce456e
Run replication in the background (#10491) 2020-09-15 18:44:58 -07:00
Anis Elleuch 8ea55f9dba
obd: Add console log to OBD output (#10372) 2020-09-15 18:02:54 -07:00
poornas 80e3dce631
azure: update content-md5 to metadata after upload (#10482)
Fixes #10453
2020-09-15 16:31:47 -07:00
Harshavardhana 80fab03b63
fix: S3 gateway doesn't support full passthrough for encryption (#10484)
The entire encryption layer is dependent on the fact that
KMS should be configured for S3 encryption to work properly
and we only support passing the headers as is to the backend
for encryption only if KMS is configured.

Make sure that this predictability is maintained, currently
the code was allowing encryption to go through and fail
at later to indicate that KMS was not configured. We should
simply reply "NotImplemented" if KMS is not configured, this
allows clients to simply proceed with their tests.
2020-09-15 13:57:15 -07:00
Harshavardhana 730d2dc7be
fix: allow CopyObject/PutObjecTags on pre-existing content (#10485)
fixes #10475
2020-09-15 09:18:41 -07:00
Harshavardhana 0ee9678190
fix: add missing delete marker created filter (#10481) 2020-09-14 21:32:52 -07:00
Klaus Post 34859c6d4b
Preallocate (safe) slices when we know the size (#10459) 2020-09-14 20:44:18 -07:00
Klaus Post b1c99e88ac
reduce CPU usage upto 50% in readdir (#10466) 2020-09-14 17:19:54 -07:00