Commit Graph

2772 Commits

Author SHA1 Message Date
Harshavardhana
91817d0d1a
fix: implement generic Walk for gateway (#9938)
Walk() functionality was missing on gateway
implementations leading to missing functionality
for the browser UI such as remove multiple objects,
download as zip file etc.

This PR brings a generic implementation across
all gateway's, it is not required to repeat the
same code in all gateway's
2020-06-29 17:07:23 -07:00
poornas
55a3b071ea
Allow optionally to disable range caching. (#9908)
The default behavior is to cache each range requested
to cache drive. Add an environment variable
`MINIO_RANGE_CACHE` - when set to off, it disables
range caching and instead downloads entire object
in the background.

Fixes #9870
2020-06-29 13:25:29 -07:00
Harshavardhana
a38ce29137
fix: simplify background heal and trigger heal items early (#9928)
Bonus fix during versioning merge one of the PR was missing
the offline/online disk count fix from #9801 port it correctly
over to the master branch from release.

Additionally, add versionID support for MRF

Fixes #9910
Fixes #9931
2020-06-29 13:07:26 -07:00
Harshavardhana
4bba2cd034
fix: disallow versioning to be suspended with object lock (#9930) 2020-06-28 08:15:15 -07:00
Harshavardhana
f7f12b8604
fix: crash in storage rest client due to spurious query params (#9924)
regression got introduced in dee3cf2d7f
when the DeleteVersion API was changed, but the corresponding query
params were left in-tact.
2020-06-26 16:49:49 -07:00
Praveen raj Mani
cf5d051afc
update notification rulesMap when reloading bucketMetadata (#9917) 2020-06-26 13:17:31 -07:00
Harshavardhana
2f681bed57
fix: pop entries from each drives in parallel (#9918) 2020-06-25 23:20:12 -07:00
Praveen raj Mani
b1705599e1
Fix config leaks and deprecate file-based config setters in NAS gateway (#9884)
This PR has the following changes

- Removing duplicate lookupConfigs() calls.
- Deprecate admin config APIs for NAS gateways. This will avoid repeated reloads of the config from the disk.
- WatchConfigNASDisk will be removed
- Migration guide for NAS gateways users to migrate to ENV settings.

NOTE: THIS PR HAS A BREAKING CHANGE

Fixes #9875

Co-authored-by: Harshavardhana <harsha@minio.io>
2020-06-25 15:59:28 +05:30
Harshavardhana
f4b2ed2a92
fix: filter list buckets operation with ListObjects perm (#9907)
fix regression introduced in #9305
2020-06-23 23:21:11 -07:00
Harshavardhana
dee3cf2d7f
fix: preserve modTime for DeleteMarker on remote disks (#9905) 2020-06-23 10:20:31 -07:00
Harshavardhana
21058c34d0
add some description of xl.meta (#9901) 2020-06-22 17:27:54 -07:00
Harshavardhana
5b1e6c7dbc
Add check for object statTime non-negative (#9899) 2020-06-22 14:33:58 -07:00
Harshavardhana
e92434c2e7
fix: support client customized scopes for OpenID (#9880)
Fixes #9238
2020-06-22 12:08:50 -07:00
Klaus Post
cae09d8b84
crawler: Wait max 1 second (#9894)
Add 1-second timeout to crawler wait.

This will make the crawler able to run, albeit very, 
very slowly on high load servers.
2020-06-22 11:57:22 -07:00
Harshavardhana
c54e3b4ea3
Add support for minioreleaser a fork for goreleaser (#9890)
This is to support building containers for multiple
platforms, rpms and debs all in a single build process

https://github.com/harshavardhana/minioreleaser
2020-06-22 08:26:40 -07:00
Klaus Post
972d876ca9
Do not select zones with <5% free after upload (#9877)
Looking into full disk errors on zoned setup. We don't take the
5% space requirement into account when selecting a zone.

The interesting part is that even considering this we don't
know the size of the object the user wants to upload when
they do multipart uploads.

It seems quite defensive to always upload multiparts to
the zone where there is the most space since all load will
be directed to a part of the cluster.

In these cases we make sure it can at least hold a 1GiB file
and we disadvantage fuller zones more by subtracting the
expected size before weighing.
2020-06-20 06:36:44 -07:00
Harshavardhana
b8cb21c954
allow more than N number of locks in TopLocks (#9883) 2020-06-20 06:33:01 -07:00
Harshavardhana
67062840c1
fix: perform CopyObject under more conditions (#9879)
- x-amz-storage-class specified CopyObject
  should proceed regardless, its not a precondition
- sourceVersionID is specified CopyObject should
  proceed regardless, its not a precondition
2020-06-19 13:53:45 -07:00
Harshavardhana
9626a981bc
fix: Preserve old data appropriately (#9873)
This PR fixes all the below scenarios
and handles them correctly.

- existing data/bucket is replaced with
  new content, no versioning enabled old
  structure vanishes.

- existing data/bucket - enable versioning
  before uploading any data, once versioning
  enabled upload new content, old content
  is preserved.

- suspend versioning on the bucket again, now
  upload content again the old content is purged
  since that is the default "null" version.

Additionally sync data after xl.json -> xl.meta
rename(), to avoid any surprises if there is a
crash during this rename operation.
2020-06-19 10:58:17 -07:00
Harshavardhana
b912c8f035
fix: generate new version when replacing metadata in CopyObject (#9871) 2020-06-19 08:44:51 -07:00
Harshavardhana
fa13fe2184
allow loading some from config and some values from ENVs (#9872)
A regression perhaps introduced in #9851
2020-06-18 17:31:56 -07:00
Harshavardhana
85a1956e5c
Avoid duplicate object holding locks (#9867)
Fixes #9866
2020-06-18 10:25:07 -07:00
Harshavardhana
7ed1077879
Add a custom healthcheck function for online status (#9858)
- Add changes to ensure remote disks are not
  incorrectly taken online if their order has
  changed or are incorrect disks.
- Bring changes to peer to detect disconnection
  with separate Health handler, to avoid a
  rather expensive call GetLocakDiskIDs()
- Follow up on the same changes for Lockers
  as well
2020-06-17 14:49:26 -07:00
Harshavardhana
94424e14d7
fix: rename legacy xl.json to xl.meta properly in ListDir() (#9863) 2020-06-17 13:58:38 -07:00
Harshavardhana
e79874f58e
[feat] Preserve version supplied by client (#9854)
Just like GET/DELETE APIs it is possible to preserve
client supplied versionId's, of course the versionIds
have to be uuid, if an existing versionId is found
it is overwritten if no object locking policies
are found.

- PUT /bucketname/objectname?versionId=<id>
- POST /bucketname/objectname?uploads=&versionId=<id>
- PUT /bucketname/objectname?verisonId=<id> (with x-amz-copy-source)
2020-06-17 11:13:41 -07:00
Klaus Post
8aae8b1d27
Put an upper limit on walk pool sizes (#9848)
Fixes potentially infinite allocations, especially in FS mode, 
since lookups live up to 30 minutes. Limit walk pool sizes to 50 
max parameter entries and 4 concurrent operations with the same
parameters.

Fixes #9835
2020-06-17 09:52:07 -07:00
Klaus Post
1813ff9dfa
Re-add missing bucket bloom filters (#9861) 2020-06-17 08:54:41 -07:00
Harshavardhana
4ac31ea82b
fix: find current location of object multi-zones (#9840)
PutObject on multiple-zone with versioning would not
overwrite the correct location of the object if the
object has delete marker, leading to duplicate objects
on two zones.

This PR fixes by adding affinity towards delete marker
when GetObjectInfo() returns error, use the zone index
which has the delete marker.
2020-06-17 08:33:14 -07:00
Harshavardhana
67ca157329
fix: content-md5 is not mandatory for PutBucketVersioning (#9852) 2020-06-17 07:59:08 -07:00
Harshavardhana
f5e1b3d09e
fix: initialize config once per startup (#9851) 2020-06-16 20:15:21 -07:00
Klaus Post
3ba4804d6c
Move online status to REST client (#9808) 2020-06-16 18:59:32 -07:00
Harshavardhana
216de230e2
remove unnecessary log for setMaxResources (#9856)
fixes #9855
2020-06-16 18:57:29 -07:00
ebozduman
a91cfa03e7
extend the HINT on backend ownership and its contents (#9846) 2020-06-16 15:32:29 -07:00
Harshavardhana
087aaaf894
fix: save deleteMarker properly, precision upto UnixNano() (#9843) 2020-06-16 07:54:27 -07:00
Harshavardhana
cbb7a09376
Allow etcd, cache setup to exit when starting gateway mode (#9842)
- Initialize etcd once per call
- Fail etcd, cache setup pro-actively for gateway setups
- Support deleting/updating bucket notification,
  tagging, lifecycle, sse-encryption
2020-06-15 22:09:39 -07:00
Harshavardhana
1a956424e0 Add logs when quorum is lost during readiness checks (#9839) 2020-06-15 13:11:22 -07:00
Harshavardhana
f9aa239973
fix: export prometheus metrics for cache GC triggers (#9815)
Bonus change to use channel to serialize triggers,
instead of using atomic variables. More efficient
mechanism for synchronization.

Co-authored-by: Nitish Tiwari <nitish@minio.io>
2020-06-15 09:05:35 -07:00
Anis Elleuch
2073b79633
fix: Remove unnecessary debug log line (#9834) 2020-06-15 08:55:33 -07:00
Anis Elleuch
63e9005f01 fix: Avoid updating object tags on failed disks (#9819) 2020-06-14 10:53:07 -07:00
Harshavardhana
d55f4336ae
preserve context per request for local locks (#9828)
In the Current bug we were re-using the context
from previously granted lockers, this would
lead to lock timeouts for existing valid
read or write locks, leading to premature
timeout of locks.

This bug affects only local lockers in FS
or standalone erasure coded mode. This issue
is rather historical as well and was present
in lsync for some time but we were lucky to
not see it.

Similar changes are done in dsync as well
to keep the code more familiar

Fixes #9827
2020-06-14 07:43:10 -07:00
ethan ho
535efd34a0
Fix peer server update failure (#9824)
When updating all servers following the constructions of mc update,
only the endpoint server will be updated successfully.
All the other peer servers' updating failed due to the error below:
--------------------------------------------------------------------------
parsing time "2006-01-02T15:04:05Z07:00" as "<release version>": cannot parse "-01-02T15:04:05Z07:00" as "0-" 
--------------------------------------------------------------------------
2020-06-13 07:12:49 -07:00
Harshavardhana
4915433bd2
Support bucket versioning (#9377)
- Implement a new xl.json 2.0.0 format to support,
  this moves the entire marshaling logic to POSIX
  layer, top layer always consumes a common FileInfo
  construct which simplifies the metadata reads.
- Implement list object versions
- Migrate to siphash from crchash for new deployments
  for object placements.

Fixes #2111
2020-06-12 20:04:01 -07:00
Klaus Post
43d6e3ae06
merge object lifecycle checks into usage crawler (#9579) 2020-06-12 10:28:21 -07:00
kannappanr
225b812b5e
Update minio-go library to latest (#9813) 2020-06-12 10:18:42 -07:00
Harshavardhana
96ed0991b5
fix: optimize IAM users load, add fallback (#9809)
Bonus fix, load service accounts properly
when service accounts were generated with
LDAP
2020-06-11 14:11:30 -07:00
Harshavardhana
a42df3d364
Allow idiomatic usage of middlewares in gorilla/mux (#9802)
Historically due to lack of support for middlewares
we ended up writing wrapped handlers for all
middlewares on top of the gorilla/mux, this causes
multiple issues when we want to let's say

- Overload r.Body with some custom implementation
  to track the incoming Reads()
- Add other sort of top level checks to avoid
  DDOSing the server with large incoming HTTP
  bodies.

Since 1.7.x release gorilla/mux provides proper
use of middlewares, which are honored by the muxer
directly. This makes sure that Go can honor its
own internal ServeHTTP(w, r) implementation where
Go net/http can wrap into its own customer readers.

This PR as a side-affect fixes rare issues of client
hangs which were reported in the wild but never really
understood or fixed in our codebase.

Fixes #9759
Fixes #7266
Fixes #6540
Fixes #5455
Fixes #5150

Refer https://github.com/boto/botocore/pull/1328 for
one variation of the same issue in #9759
2020-06-11 08:19:55 -07:00
Harshavardhana
ff94b1b0a9
isEndpointConnected should take local disk inputs (#9803)
PR #9801 while it is correct, the loop isEndpointConnected()
was changed to rely on endpoint.String() which has the host
information as well, which is not correct value as input to
detect if the disk is down or up, if endpoint is local use
its local path value instead.
2020-06-11 08:05:25 -07:00
Andreas Auernhammer
b1845c6c83
kes: try to auto. create master key if not present (#9790)
This commit changes the data key generation such that
if a MinIO server/nodes tries to generate a new DEK
but the particular master key does not exist - then
MinIO asks KES to create a new master key and then
requests the DEK again.

From now on, a SSE-S3 master key must not be created
explicitly via: `kes key create <key-name>`.
Instead, it is sufficient to just set the env. var.
```
export MINIO_KMS_KES_KEY_NAME=<key-name>
```

However, the MinIO identity (mTLS client certificate)
must have the permission to access the `/v1/key/create/`
API. Therefore, KES policy for MinIO must look similar to:
```
[
  /v1/key/create/<key-name-pattern>
  /v1/key/generate/<key-name-pattern>
  /v1/key/decrypt/<key-name-pattern>
]
```
However, in our guides we already suggest that.
See e.g.: https://github.com/minio/kes/wiki/MinIO-Object-Storage#kes-server-setup

***

The ability to create master keys on request may also be
necessary / useful in case of SSE-KMS.
2020-06-11 02:00:47 -07:00
Harshavardhana
62b1da3e2c
fix offline disk calculation (#9801)
Current code was relying on globalEndpoints as
the source of secondary truth to obtain
the missing endpoints list when the disk
is offline, this is problematic

- there is no way to know if the getDisks()
  returned endpoints total is same as the
  ones list of globalEndpoints and it
  belongs to a particular set.
- there is no order guarantee as getDisks()
  is ordered as per format.json, globalEndpoints
  may not be, so potentially end up including
  incorrect endpoints.

To fix this bring getEndpoints() just like getDisks()
to ensure that consistently ordered endpoints are
always available for us to ensure that returned values
are consistent with what each erasure set would observe.
2020-06-10 17:10:31 -07:00
poornas
d26b24f670
avoid storing X-Amz-Tagging-Directive in metadata (#9800) 2020-06-10 14:29:24 -07:00
kannappanr
2c372a9894
Send Partscount only when partnumber is specified (#9793)
Fixes #9789
2020-06-10 09:22:15 -07:00
poornas
3d3b75fb8d
Avoid overwriting object tags when changing lock (#9794) 2020-06-10 08:16:30 -07:00
Klaus Post
142b057be8
Check object names on windows (#9798)
Uploading files with names that could not be written to disk 
would result in "reduce your request" errors returned.

Instead check explicitly for disallowed characters and reject 
files with `Object name contains unsupported characters.`
2020-06-10 08:14:22 -07:00
Harshavardhana
4790868878
allow background IAM load to speed up startup (#9796)
Also fix healthcheck handler to run success
only if object layer has initialized fully
for S3 API access call.
2020-06-09 19:19:03 -07:00
Harshavardhana
342ade03f6
deprecate listDir usage for healing (#9792)
listDir was incorrectly used for healing which
is slower, instead use Walk() to heal the entire
set.
2020-06-09 17:09:19 -07:00
P R
9407dbf387
display proper used space based on disk usage (#9551)
Fixes #9346
2020-06-09 15:05:39 -07:00
Harshavardhana
423aeb0d81
allow large buffer to list more entries per directory (#9785) 2020-06-09 09:44:50 -07:00
Anis Elleuch
790323ac37
lifecycle: Fix object expiration date (#9791)
re-use PredictExpiryTime() in ComputeAction()
2020-06-09 09:40:53 -07:00
Harshavardhana
febe9cc26a
fix: avoid timer leaks in dsync/lsync (#9781)
At a customer setup with lots of concurrent calls
it can be observed that in newRetryTimer there
were lots of tiny alloations which are not
relinquished upon retries, in this codepath
we were only interested in re-using the timer
and use it wisely for each locker.

```
(pprof) top
Showing nodes accounting for 8.68TB, 97.02% of 8.95TB total
Dropped 1198 nodes (cum <= 0.04TB)
Showing top 10 nodes out of 79
      flat  flat%   sum%        cum   cum%
    5.95TB 66.50% 66.50%     5.95TB 66.50%  time.NewTimer
    1.16TB 13.02% 79.51%     1.16TB 13.02%  github.com/ncw/directio.AlignedBlock
    0.67TB  7.53% 87.04%     0.70TB  7.78%  github.com/minio/minio/cmd.xlObjects.putObject
    0.21TB  2.36% 89.40%     0.21TB  2.36%  github.com/minio/minio/cmd.(*posix).Walk
    0.19TB  2.08% 91.49%     0.27TB  2.99%  os.statNolog
    0.14TB  1.59% 93.08%     0.14TB  1.60%  os.(*File).readdirnames
    0.10TB  1.09% 94.17%     0.11TB  1.25%  github.com/minio/minio/cmd.readDirN
    0.10TB  1.07% 95.23%     0.10TB  1.07%  syscall.ByteSliceFromString
    0.09TB  1.03% 96.27%     0.09TB  1.03%  strings.(*Builder).grow
    0.07TB  0.75% 97.02%     0.07TB  0.75%  path.(*lazybuf).append
```
2020-06-08 11:28:40 -07:00
Praveen raj Mani
2ce2e88adf
Support mTLS Authentication in Webhooks (#9777) 2020-06-08 05:55:44 -07:00
Harshavardhana
c7599d323b
fix: throw error if symmetry cannot be obtained (#9780)
For example `{1...17}/{1...52}` symmetrical
distribution of drives cannot be obtained

- Because 17 is a prime number
- Is not divisible by any pre-defined setCounts i.e
  from 1 to 16
2020-06-06 22:13:48 -07:00
Harshavardhana
d93bdea433
fix remove LDAPPassword from audit logs (#9773)
the previous fix for #9707 was not correct,
fix this properly passing the right filter
keys to be filtered from the audit
log output.

Fixes #9767
2020-06-04 22:07:55 -07:00
Harshavardhana
5e529a1c96
simplify context timeout for readiness (#9772)
additionally also add CORS support to restrict
for specific origin, adds a new config and
updated the documentation as well
2020-06-04 14:58:34 -07:00
Harshavardhana
5686a7e273
fix NAS gateway support for policy/notification (#9765)
Fixes #9764
2020-06-03 13:18:54 -07:00
Harshavardhana
566e0e2048
allow deleting of dropped multiparts (#9753)
bonus change trigger MRF heal when single
offline disk is found, break out early.
2020-06-02 15:27:03 -07:00
Anis Elleuch
3aad09be28
heal: Fix passing healing opts (#9756)
Manual healing (as background healing) creates a heal task with a
possiblity to override healing options, such as deep or normal mode.

Use a pointer type in heal opts so nil would mean use the default
healing options.
2020-06-02 09:07:16 -07:00
Harshavardhana
f0358acb32
concurrently load bucket metadata (#9749) 2020-06-01 22:32:53 -07:00
Anis Elleuch
fd0de4ab32
azure: Show better message when credentials are wrong (#9748) 2020-06-01 18:23:48 -07:00
Anis Elleuch
73a308502f
Relax content-md5 requirement in set encryption handler (#9750)
aws cli fails to set a bucket encryption configuration to MinIO server.
The reason is that aws cli does not send MD5-Content header. It seems
that MD5-Content is not required anymore.

This commit also returns Not Implemented header early to help mint tests
to ignore testing this API in gateway modes.
2020-06-01 18:08:19 -07:00
Anis Elleuch
bd59f150b8
azure: Implement CopyPart API (#9747) 2020-06-01 11:12:18 -07:00
Harshavardhana
f90422a890
fix prometheus calculation of offline disks per instance (#9744)
This was a regression introduced in 9baeda7 for prometheus
calculation of offline disks which should be local to
an instance.

fixes #9742
2020-06-01 07:35:40 -07:00
Harshavardhana
8befedef14
simplify FS multipart cleanup (#9740)
fixes #9671
2020-05-30 13:56:31 -07:00
Nathan Brown
2af3004409
Use registry to check Atime support on Windows (#9741) 2020-05-30 09:47:42 -07:00
Harshavardhana
38ee40d59c
move to upstream code colinmarc/hdfs (#9738)
- supports SASL based authentication now
- upgrades to new changes in gokrb library
- implement force delete feature

Fixes #8206
2020-05-29 18:38:50 -07:00
kannappanr
d583f1ac0e
check if container is empty before invoking DeleteContainer (#9733) 2020-05-29 13:24:39 -07:00
Harshavardhana
2bcb02f628
Avoid '\n' from constant strings (#9737)
Fixes #9736
2020-05-29 11:40:57 -07:00
Klaus Post
167ddf9c9c
Workaround for Windows Docker Engine 19.03.8 (#9735)
Add workaround for issue preventing servers from starting on 
Windows Docker Engine 19.03.8

Fixes #9726
2020-05-29 07:05:19 -07:00
Anton Huck
f833e41e69
IAM: Fix nil panic due to uninit. iamGroupPolicyMap. Fixes #9730 (#9734) 2020-05-29 06:13:54 -07:00
Harshavardhana
41688a936b
fix: CopyObject behavior on expanded zones (#9729)
CopyObject was not correctly figuring out the correct
destination object location and would end up creating
duplicate objects on two different zones, reproduced
by doing encryption based key rotation.
2020-05-28 14:36:38 -07:00
Harshavardhana
b2db8123ec
Preserve errors returned by diskInfo to detect disk errors (#9727)
This PR basically reverts #9720 and re-implements it differently
2020-05-28 13:03:04 -07:00
Harshavardhana
b330c2c57e
Introduce simpler GetMultipartInfo call for performance (#9722)
Advantages avoids 100's of stats which are needed for each
upload operation in FS/NAS gateway mode when uploading a large
multipart object, dramatically increases performance for
multipart uploads by avoiding recursive calls.

For other gateway's simplifies the approach since
azure, gcs, hdfs gateway's don't capture any specific
metadata during upload which needs handler validation
for encryption/compression.

Erasure coding was already optimized, additionally
just avoids small allocations of large data structure.

Fixes #7206
2020-05-28 12:36:20 -07:00
kannappanr
7214a0160a
allow bucket policy to set/removed in NAS gateway (#9706) 2020-05-28 08:31:16 -07:00
Anis Elleuch
375b79f11b
storage: Implement GetDiskID request in REST server side (#9720)
GetDiskID() in storage rest client does not really issue a REST request
to the remote disk, but returns an in-memory value instead.

However, GetDiskID() should return an error when format.json is not
found or for other similar issues (unmounted disks, etc..)

GetDiskID() is only called when formatting disks and getting storage
informatio, hence this commit should not have a performance degradation.
2020-05-28 08:17:42 -07:00
Harshavardhana
3da1869d5e
Avoid double reads on metadata during GetObject() (#9719)
Overall TTFB can see a dramatic improvement with
this change - did not do any benchmark as such
but the change itself is self-explanatory
2020-05-27 16:14:26 -07:00
Harshavardhana
7cedc5369d
fix: send valid claims in AuditLogs for browser requests (#9713)
Additionally also fix STS logs to filter out LDAP
password to be sent out in audit logs.

Bonus fix handle the reload of users properly by
making sure to preserve the newer users during the
reload to be not invalidated.

Fixes #9707
Fixes #9644
Fixes #9651
2020-05-27 12:38:44 -07:00
Harshavardhana
53aaa5d2a5
Export bucket usage counts as part of bucket metrics (#9710)
Bonus fixes in quota enforcement to use the
new datastructure and use timedValue to cache
a value/reload automatically avoids one less
global variable.
2020-05-27 06:45:43 -07:00
P R
9d39fb3604
add copyobject tagging replace directive for gateway (#9711) 2020-05-26 17:32:53 -07:00
Klaus Post
4a007e3767
Prefer local disks when fetching data blocks (#9563)
If the requested server is part of the set this will always read 
from the local disk, even if the disk contains a parity shard. 
In default setup there is a 50% chance that at least 
one shard that otherwise would have been fetched remotely 
will be read locally instead.

It basically trades RPC call overhead for reed-solomon. 
On distributed localhost this seems to be fairly break-even, 
with a very small gain in throughput and latency. 
However on networked servers this should be a bigger

1MB objects, before:

```
Operation: GET. Concurrency: 32. Hosts: 4.

Requests considered: 76257:
 * Avg: 25ms 50%: 24ms 90%: 32ms 99%: 42ms Fastest: 7ms Slowest: 67ms
 * First Byte: Average: 23ms, Median: 22ms, Best: 5ms, Worst: 65ms

Throughput:
* Average: 1213.68 MiB/s, 1272.63 obj/s (59.948s, starting 14:45:44 CEST)
```

After:
```
Operation: GET. Concurrency: 32. Hosts: 4.

Requests considered: 78845:
 * Avg: 24ms 50%: 24ms 90%: 31ms 99%: 39ms Fastest: 8ms Slowest: 62ms
 * First Byte: Average: 22ms, Median: 21ms, Best: 6ms, Worst: 57ms

Throughput:
* Average: 1255.11 MiB/s, 1316.08 obj/s (59.938s, starting 14:43:58 CEST)
```

Bonus fix: Only ask for heal once on an object.
2020-05-26 16:47:23 -07:00
Klaus Post
95814359bd
cache disk info to avoid repeated calls (#9682)
This value is requested on every upload when there are multiple zones.

Since this will result in an RPC call to every remote disk this scales 
quite badly in a distributed setup. Load every 1second interval.

2 servers, localhost only. In large distributed setups much bigger 
gains can be expected.

```
Operations: 21743 -> 22454
* Average: +3.28% (+0.0 MiB/s) throughput, +3.28% (+11.9) obj/s
* Fastest: +3.37% (+0.0 MiB/s) throughput, +3.37% (+13.0) obj/s
* 50% Median: +3.03% (+0.0 MiB/s) throughput, +3.03% (+11.2) obj/s
* Slowest: +8.03% (+0.0 MiB/s) throughput, +8.03% (+22.8) obj/s
```

For easy management of this a generic helper has been added.
2020-05-26 12:52:24 -07:00
Harshavardhana
d0ae69087c
fix: add proper errors for disks with preexisting content (#9703) 2020-05-26 09:32:33 -07:00
Harshavardhana
7ea026ff1d
fix: reply back user-metadata in lower case form (#9697)
some clients such as veeam expect the x-amz-meta to
be sent in lower cased form, while this does indeed
defeats the HTTP protocol contract it is harder to
change these applications, while these applications
get fixed appropriately in future.

x-amz-meta is usually sent in lowercased form
by AWS S3 and some applications like veeam
incorrectly end up relying on the case sensitivity
of the HTTP headers.

Bonus fixes

 - Fix the iso8601 time format to keep it same as
   AWS S3 response
 - Increase maxObjectList to 50,000 and use
   maxDeleteList as 10,000 whenever multi-object
   deletes are needed.
2020-05-25 16:51:32 -07:00
Harshavardhana
6e0575a53d
Revert "Disable crawler in FS/NAS gateway mode (#9695)" (#9702)
This reverts commit eba423bb9d.

Additionally also address the FS crawler to properly
calculate the sizes for encrypted/compressed content.
2020-05-25 11:32:53 -07:00
Harshavardhana
eba423bb9d
Disable crawler in FS/NAS gateway mode (#9695)
No one really uses FS for large scale accounting
usage, neither we crawl in NAS gateway mode. It is
worthwhile to simply disable this feature as its
not useful for anyone.

Bonus disable bucket quota ops as well in, FS
and gateway mode
2020-05-25 00:17:52 -07:00
Erkki Eilonen
301de169e9
in cache build ranges metadata as needed (#9698) 2020-05-25 00:17:03 -07:00
Harshavardhana
0c71ce3398
fix size accounting for encrypted/compressed objects (#9690)
size calculation in crawler was using the real size
of the object instead of its actual size i.e either
a decrypted or uncompressed size.

this is needed to make sure all other accounting
such as bucket quota and mcs UI to display the
correct values.
2020-05-24 11:19:17 -07:00
Krishna Srinivas
7d19ab9f62
readiness returns error quickly if any of the set is down (#9662)
This PR adds a new configuration parameter which allows readiness
check to respond within 10secs, this can be reduced to a lower value
if necessary using 

```
mc admin config set api ready_deadline=5s
```

 or

```
export MINIO_API_READY_DEADLINE=5s
```
2020-05-23 17:38:39 -07:00
P R
3f6d624c7b
add gateway object tagging support (#9124) 2020-05-23 11:09:35 -07:00
Harshavardhana
c138272d63
reject object lock requests on existing buckets (#9684)
a regression was introduced fix it to ensure that we
do not allow object locking settings on existing buckets
without object locking
2020-05-23 10:01:01 -07:00
Harshavardhana
7dbfea1353
avoid net/http ErrorLog for consistent logging experience (#9672)
net/http exposes ErrorLog but it is log.Logger
instance not an interface which can be overridden,
because of this reason the logging is interleaved
sometimes with TLS with messages like this on the
server

```
http: TLS handshake error from 139.178.70.188:63760: EOF
```

This is bit problematic for us as we need to have
consistent logging view for allow --json or --quiet
flags.

With this PR we ensure that this format is adhered to.
2020-05-22 21:59:18 -07:00
Sidhartha Mani
c121d27f31
progressively report obd results (#9639) 2020-05-22 17:56:45 -07:00
Anis Elleuch
43c19a6b82
nas: ensure loading of bucket notifications during startup (#9681) 2020-05-22 11:55:30 -07:00
Harshavardhana
e45c90060f
remove references for deprecated dockerfiles and deployment styles (#9675) 2020-05-22 08:40:59 -07:00
Harshavardhana
d15042470e
add missing signature v2 query params (#9670) 2020-05-21 18:51:23 -07:00
Anis Elleuch
cdf4815a6b
Add x-amz-expiration header in some S3 responses (#9667)
x-amz-expiration is described in the S3 specification as a header which
indicates if the object in question will expire any time in the future.
2020-05-21 14:12:52 -07:00
kannappanr
fade056244
filter all encryption headers in gateway (#9661)
fixes #9655
2020-05-21 11:07:50 -07:00
Harshavardhana
a546047c95
keep bucket metadata fields to be consistent (#9660)
added bonus reload bucket metadata always after
a successful MakeBucket, current we were only
doing it with object locking enabled.
2020-05-21 11:03:59 -07:00
ebozduman
2896e780ae
fixes misleading assume role error msgs (#9642) 2020-05-21 09:09:18 -07:00
Harshavardhana
baa30f4289
reload bucket metadata outside the locker (#9659) 2020-05-20 14:11:13 -07:00
Harshavardhana
189c861835
fix: remove LDAP groups claim and store them on server (#9637)
Groups information shall be now stored as part of the
credential data structure, this is a more idiomatic
way to support large LDAP groups.

Avoids the complication of setups where LDAP groups
can be in the range of 150+ which may lead to excess
HTTP header size > 8KiB, to reduce such an occurrence
we shall save the group information on the server as
part of the credential data structure.

Bonus change support multiple mapped policies, across
all types of users.
2020-05-20 11:33:35 -07:00
Harshavardhana
6656fa3066
simplify further bucket configuration properly (#9650)
This PR is a continuation from #9586, now the
entire parsing logic is fully merged into
bucket metadata sub-system, simplify the
quota API further by reducing the remove
quota handler implementation.
2020-05-20 10:18:15 -07:00
Praveen raj Mani
0cc2ed04f5
humanize timeToFirstByte and timeToResponse upto nanoseconds (#9641) 2020-05-19 18:34:02 -07:00
Anis Elleuch
9baeda781a
fix storage info output with unordered endpoints arguments (#9610)
Shuffling arguments that we pass to MinIO server are supported. However,
when that happens, Prometheus returns wrong information about disks usage
and online/offline status.

The commit fixes the issue by avoiding relying on xl.endpoints since
it is not ordered.
2020-05-19 14:27:20 -07:00
Harshavardhana
bd032d13ff
migrate all bucket metadata into a single file (#9586)
this is a major overhaul by migrating off all
bucket metadata related configs into a single
object '.metadata.bin' this allows us for faster
bootups across 1000's of buckets and as well
as keeps the code simple enough for future
work and additions.

Additionally also fixes #9396, #9394
2020-05-19 13:53:54 -07:00
Harshavardhana
d31eaddba3
fix: avoid double body reads in SelectObject call (#9638)
Bonus fix handle encryption headers in response
properly for both notification and response to
the client.
2020-05-19 02:01:08 -07:00
poornas
3202f78f0f
Fix cache metadata update for range GET (#9636)
This was inadvertently deleting cached ranges
because HTTPRangeSpec was not being passed down

fixes #9597
2020-05-18 18:33:43 -07:00
Harshavardhana
6de410a0aa
fix: possiblity of double write lockers on same resource (#9616)
To avoid this issue with refCounter refactor the code
such that

- locker() always increases refCount upon success
- unlocker() always decrements refCount upon success
  (as a special case removes the resource if the
  refCount is zero)

By these two assumptions we are able to see that we
are never granted two write lockers in any situation.

Thanks to @vcabbage for writing a nice reproducer.
2020-05-18 17:33:35 -07:00
Klaus Post
1847f17f50
Set Deployment ID before starting handlers (#9635)
Global handler ID is added to response headers, so initialize it before the server starts.

Fixes #9634
2020-05-18 11:35:05 -07:00
Harshavardhana
1bc32215b9
enable full linter across the codebase (#9620)
enable linter using golangci-lint across
codebase to run a bunch of linters together,
we shall enable new linters as we fix more
things the codebase.

This PR fixes the first stage of this
cleanup.
2020-05-18 09:59:45 -07:00
Anis Elleuch
96009975d6
relax validation when loading lifecycle document from the backend (#9612) 2020-05-18 08:33:43 -07:00
Harshavardhana
de9b391db3
fix: Disable presigned without appropriate policy (#9621)
Fixes #9590
2020-05-17 23:38:52 -07:00
kannappanr
a62572fb86
Check for address flags in all positions (#9615)
Fixes #9599
2020-05-17 08:46:23 -07:00
poornas
011a2c0b78
Add docs for bucket quota feature (#9503)
This PR also adds a check to not enforce
bucket quota for server-side metadata copy
of an object onto itself.
2020-05-16 19:27:33 -07:00
Harshavardhana
814ddc0923
add missing admin actions, enhance AccountUsageInfo (#9607) 2020-05-15 18:16:45 -07:00
Harshavardhana
d348ec0f6c
avoid double listObjectParts calls improves performance (#9606)
this PR is to avoid double calls across multiple calls
in APIs

- CopyObjectPart
- PutObjectPart
2020-05-15 08:06:45 -07:00
Harshavardhana
b730bd1396
fix: possible race in FS local lockMap (#9598) 2020-05-14 23:59:07 -07:00
Klaus Post
56e0c6adf8
Track if bloom filter is dirty (#9601)
Only save bloom filter on cycles and updates.

Fixes #9600
2020-05-14 21:46:36 -07:00
Anis Elleuch
f44a960dcd
tests: Fix one multi-delete test failure in Windows CI (#9602)
There is a disparency of behavior under Linux & Windows about
the returned error when trying to rename a non existant path.

err := os.Rename("/path/does/not/exist", "/tmp/copy")

Linux:
  isSysErrNotDir(err) = false
  os.IsNotExist(err) = true

Windows:
  isSysErrNotDir(err) = true
  os.IsNotExist(err) = true

ENOTDIR in Linux is returned when the destination path
of the rename call contains a file in one of the middle
segments of the path (e.g. /tmp/file/dst, where /tmp/file
is an actual file not a directory)

However, as shown above, Windows has more scenarios when
it returns ENOTDIR. For example, when the source path contains
an inexistant directory in its path.

In that case, we want errFileNotFound returned and not
errFileAccessDenied, so this commit will add a further check to close
the disparency between Windows & Linux.
2020-05-14 18:09:30 -07:00
kannappanr
6c1bbf918d
do not add quotes around etag, if already present (#9603) 2020-05-14 17:43:54 -07:00
Anis Elleuch
48e614b167
honor lifecycle expiration with tag rule (#9604) 2020-05-14 16:21:03 -07:00
poornas
fe8d33452b
Allow writes for bucket exceeding FIFO quota (#9575)
the quota will be enforced while
deleting oldest entries in FIFO manner.
2020-05-14 15:18:24 -07:00
Klaus Post
216fa57b88
merge nested hash readers (#9582)
The `ioutil.NopCloser(reader)` was hiding nested hash readers.

We make it an `io.Closer` so it can be attached without wrapping 
and allows for nesting, by merging the requests.
2020-05-14 14:01:31 -07:00
Klaus Post
ee9077db7d
fix: windows tests for all cases (#9594)
Replaces #9299
2020-05-13 23:55:38 -07:00
Harshavardhana
9c85928740
add formatting message for zones in ordinals (#9596)
Unlike the message
> Formatting 2 zone, 1 set(s), 6 drives per set.

It is more readable as ordinal
> Formatting 2nd zone, 1 set(s), 6 drives per set.
2020-05-13 20:25:29 -07:00
Harshavardhana
6ac48a65cb
fix: use unused cacheMetrics code in prometheus (#9588)
remove all other unusued/deadcode
2020-05-13 08:15:26 -07:00
Krishna Srinivas
94f1a1dea3
add option for O_SYNC writes for standalone FS backend (#9581) 2020-05-12 19:24:59 -07:00
Anis Elleuch
c045ae15e7
fix: avoid undoing bucket creation and return the first err instead (#9578) 2020-05-12 15:20:42 -07:00
Harshavardhana
1756b7c6ff
fix: LDAP derivative accounts parentUser validation is not needed (#9573)
* fix: LDAP derivative accounts parentUser validation is not needed

fixes #9435

* Update cmd/iam.go

Co-authored-by: Lenin Alevski <alevsk.8772@gmail.com>

Co-authored-by: Lenin Alevski <alevsk.8772@gmail.com>
2020-05-12 09:21:08 -07:00
Klaus Post
e25ace2151
Forward RPC errors from crawler (#9569)
The `keepHTTPResponseAlive` would cause errors to be 
returned with status OK.

- Add '32' as a filler byte until a response is ready
- '0' to indicate the response is ready to be consumed
- '1' to indicate response has an error which needs
to be returned to the caller

Clear out 'file not found' errors from dir walker, since it may be 
in a folder that has been deleted since it was scanned.
2020-05-11 20:41:38 -07:00
poornas
a8e5a86fa0
Remove brittle tests for cache (#9570) 2020-05-11 15:41:10 -07:00
Harshavardhana
f8edc233ab
support multiple policies for temporary users (#9550) 2020-05-11 13:04:11 -07:00
Harshavardhana
337c2a7cb4
add audit logging for all admin calls (#9568)
- add ServiceRestart/ServiceStop actions
- audit log appropriately in all admin handlers

fixes #9522
2020-05-11 10:34:08 -07:00
Harshavardhana
b5ed42c845
ignore policy/group missing errors appropriately (#9559) 2020-05-09 13:59:12 -07:00
Klaus Post
d9e7cadacf
Update reed+solomon (#9562)
Only create encoder when strictly needed.
2020-05-09 09:54:20 -07:00
Anis Elleuch
6d76efb9bb
Add support of TCP fast open in internode calls (#9486) 2020-05-08 14:33:23 -07:00
Harshavardhana
a1de9cec58
cleanup object-lock/bucket tagging for gateways (#9548)
This PR is to ensure that we call the relevant object
layer APIs for necessary S3 API level functionalities
allowing gateway implementations to return proper
errors as NotImplemented{}

This allows for all our tests in mint to behave
appropriately and can be handled appropriately as
well.
2020-05-08 13:44:44 -07:00
Anis Elleuch
6885c72f32
disable check for DirectIO in standalone FS mode (#9558) 2020-05-08 12:07:51 -07:00
poornas
0f1389e992
Fix azure gateway handling of ETag for CopyObject (#9544)
fixes #9428
2020-05-08 11:30:35 -07:00
Harshavardhana
9dda1fd624
Remove B2 gateway implementation (#9547)
S3 is now natively supported by B2 cloud storage provider
there is no reason to use specialized gateway for B2 anymore,
our current S3 gateway with caching would work with B2.

Resolves #8584
2020-05-07 19:00:30 -07:00
Harshavardhana
2dc46cb153
Report correct error when O_DIRECT is not supported (#9545)
fixes #9537
2020-05-07 16:12:16 -07:00
remche
0674c0075e
add LDAP StartTLS support (#9472) 2020-05-07 15:08:33 -07:00