Commit Graph

5886 Commits

Author SHA1 Message Date
Praveen raj Mani a73da7755e Remove senstive encryption entries from event data (#7719)
Fixes #7716
2019-05-29 22:29:37 -07:00
Harshavardhana 2c0b3cadfc Update go mod with sem versions of our libraries (#7687) 2019-05-29 16:35:12 -07:00
Praveen raj Mani 763fce909b Enable event persistence in kafka (#7633) 2019-05-29 13:19:48 -07:00
Kanagaraj M da8214845a allow users to change password through browser (#7683)
Allow IAM users to change the password using
browser UI.
2019-05-29 13:18:46 -07:00
Krishna Srinivas 74e2fe0879 Return "SlowDown" to S3 clients for network related errors (#7610)
Consider errors returned by httpClient.Do() as network errors. This is because
the http clients returns different types of errors and it is hard to catch
all the error types.
2019-05-29 10:21:47 -07:00
Harshavardhana cb7f9ba286 Add disaggregated architecture document (#7691)
This document describes how to configure Spark,
Hive to use MinIO as replacement for HDFS.
2019-05-29 08:04:51 +05:30
Harshavardhana 7906f464fd
Skip non-bucket dns entry (#7699)
Fixes #7697
2019-05-28 19:13:49 -07:00
Harshavardhana 7e4c9a9e1e Properly watch for users, policies, temp users (#7701)
Users were not reloaded properly when etcd was
configured in gateway, server modes.

This PR fixes this issue.
2019-05-28 11:18:53 +05:30
Scott Edlund f6fd407e47 Add limitation to b2.md (#7680)
The B2 gateway does not implement reporting md5sum as an etag response from PutObject.  Add to Known Limitations.
2019-05-27 21:04:43 -07:00
Nitish Tiwari 46ced81f41
Fix Gateway startup sequence to populate etcd (if set) with bucket info (#7686) 2019-05-24 08:41:52 +05:30
Minio Trusted 7a02faab72 Update yaml files to latest version RELEASE.2019-05-23T00-29-34Z 2019-05-23 00:35:13 +00:00
Dee Koder e252114f06 Revert "cache: Rewrite to cache only on download (#7575)" (#7684)
This reverts commit a13b58f630.
2019-05-22 14:54:15 -07:00
Harshavardhana 39b3e4f9b3 Avoid using io.ReadFull() for WriteAll and CreateFile (#7676)
With these changes we are now able to peak performances
for all Write() operations across disks HDD and NVMe.

Also adds readahead for disk reads, which also increases
performance for reads by 3x.
2019-05-22 13:47:15 -07:00
Anis Elleuch 158b8c2e86 sets: Correctly set IsTruncated in listing (#7675)
IsTruncated should not be set to true if there is no further
possible entries beyond maxKeys.

This commit will also move wide testing on object API from xl
to xl sets.
2019-05-22 13:36:16 -07:00
Praveen raj Mani c4c79f61ce Notification: Changes to persistent event store (#7658)
This patch includes the following changes in event store interface
- Removes memory store. We will not persist events in memory anymore, if `queueDir` is not set.
- Orders the events before replaying to the broker.
2019-05-22 13:34:48 -07:00
Harshavardhana 59e1d94770 Remove stale entry spurious logging (#7663)
The problem in current code was we were removing
an entry from a lock lockerMap without considering
the fact that different entry for same resource is
a possibility due the nature of locks that can be
acquired in parallel before we decide if the lock
is considered stale

A sequence of events is as follows

 - Lock("resource")
 - lockMaintenance(finds a long lived lock in this "resource")
 - Owner node rebooted which now retruns Expired() as true for
   this "resource"
 - Unlock("resource") which succeeded in quorum
 - Now by this time application retried and acquired a new
   Lock() on the same "resource"
 - Now that we have Expired() true from the previous call,
   we proceed to purge the entry from the local lockMap()
   local lockMap reports a different entry for the expired
   UID which results in a spurious log entry.

This PR removes this logging as this situation is an
expected scenario.
2019-05-22 12:21:36 -07:00
Andrei Mikhalenia 59e847aebe Signature v4: Allow signed headers from GET parameters 2019-05-21 21:00:02 -07:00
poornas a13b58f630 cache: Rewrite to cache only on download (#7575)
This will allow cache to consistently work for
server and gateways. Range GET requests will
be cached in the background after the request
is served from the backend.

Fixes: #7458, #7573, #6265, #6630
2019-05-22 08:30:27 +05:30
Kaan Kabalak 9e7a19d6b9 Fix object share link expiry always being 7 days (#7669)
The expiry value wasn't being sent to the server on modal value change

Fixes #7668
2019-05-19 17:45:28 -07:00
Harshavardhana 16c648b109 Remove "Connection" close instead reduce MaxConns per host (#7654)
This is necessary to avoid connection build up between servers
unexpectedly for example in a situation where 16 servers are
talking to each other and one server now allows a maximum of

15*4096 = 61440 idle connections

Will be kept in pool. Such a large pool is perhaps inefficient for
many reasons and also affects overall system resources.

This PR also reduces idleConnection timeout from 120 secs to 60 secs.
2019-05-17 12:52:25 +05:30
Krishnan Parthasarathi c871456269 File must be sync'd before closing (#7657)
- group sync and close action into a single defer statement to avoid
  evaluation order related bugs in future.
2019-05-16 18:30:51 -07:00
Harshavardhana 55aa20595f
Remove an empty entry being added into XML marshal (#7656) 2019-05-16 13:09:21 -07:00
poornas 707ed2b302 gcs: use MD5Sum as ETag if present in object attrs (#7643)
Fixes: 7642
2019-05-16 12:00:12 -07:00
ebozduman 78be3f8947 Removes the incorrect coverage badge from the docs (#7651) 2019-05-16 12:11:49 +05:30
ebozduman 67d508b214 Adjusts help content dynamically according to OS (#7646) 2019-05-15 14:02:44 +05:30
Harshavardhana 0022c9d210 Add connection close proactively for Walk() http/rpc (#7645) 2019-05-14 16:10:51 -07:00
Anis Elleuch 9b4a81ee60 xl: Avoid possible race during bulk Multi Delete (#7644)
errs was passed to many goroutines but they are all allowed
to update errs if any error happens during deletion, which
can cause a data race.

This commit will avoid issuing bulk delete operations in parallel
to avoid the warning race.
2019-05-14 14:43:22 -07:00
Harshavardhana b3f22eac56 Offload listing to posix layer (#7611)
This PR adds one API WalkCh which
sorts and sends list over the network

Each disk walks independently in a sorted manner.
2019-05-14 13:49:10 -07:00
Krishna Srinivas a343d14f19 Simplify putObject by not breaking the stream into parts (#7199)
We broke into parts previously as we had checksum for the entire file
to tax less on memory and to have better TTFB. We dont need to now,
after the streaming-bitrot change.
2019-05-14 12:33:18 -07:00
Anis Elleuch 9c90a28546 Implement bulk delete (#7607)
Bulk delete at storage level in Multiple Delete Objects API

In order to accelerate bulk delete in Multiple Delete objects API,
a new bulk delete is introduced in storage layer, which will accept
a list of objects to delete rather than only one. Consequently,
a new API is also need to be added to Object API.
2019-05-13 12:25:49 -07:00
Praveen raj Mani d9a7f80f68 Remove duplicate checkPutObjectArgs in PutObject and (#7396)
Fixes #7384
2019-05-13 10:12:06 -07:00
Krishna Srinivas 502456db91 Accept simpler response from OPA (#7637) 2019-05-12 18:40:49 +05:30
Harshavardhana 3eb7a8bde8
Sync before Close() to avoid random I/O (#7638) 2019-05-11 15:03:10 -07:00
Harshavardhana ac3b59645e
Add API retry functionality in mc admin (#7602) 2019-05-09 17:41:54 -07:00
Harshavardhana 72929ec05b
Turn off md5sum optionally if content-md5 is not set (#7609)
This PR also brings --compat option to run MinIO in strict
S3 compatibility mode, MinIO by default will now try to run
high performance mode.
2019-05-08 18:35:40 -07:00
Harshavardhana 757c639044 Separate issue templates for bug report and feature requests (#7616) 2019-05-07 13:49:54 -07:00
Aditya Manthramurthy 589df3d5e7 Deadcode removal (#7627) 2019-05-07 13:49:15 -07:00
Harshavardhana a71e08efb4
On macOS avoid using GNU specific options (#7621)
Fix Makefile from `cp -uf` to `cp -f`. We are only
interested in `-f` anyways.
2019-05-07 10:39:26 -07:00
kannappanr c422f7f412 Fix: Handle regression caused by gorilla mux v1.7.0 (#7625)
PR #7595 fixed part of the regression, but did not handle the
scenario, where in docker, the internal port is different from
the port on the host.

This PR modifies the regular expression such that all the
scenarios are handled.

Fixes #7619
2019-05-07 10:36:00 -07:00
Anis Elleuch 08b9244c48 Fix listing empty directory in recursive mode (#7613)
After recent listing refactor, recursive list doesn't return empty
directories, this commit will fix the behavior and add unit tests
so it won't happen again.
2019-05-06 07:52:42 -07:00
kannappanr efbc665ad3
Delete checked in binary file (#7615)
healthcheck binary was checked in by mistake.
2019-05-03 13:16:35 -07:00
Praveen raj Mani 998f01fadc Support IPv6 hosts for postgres connections (#7168)
Fixes #7145
2019-05-03 17:31:33 +05:30
kannappanr 4b858b562a
Compression: Handle auto encryption when size is unknown (#7600)
When size is unknown and auto encryption is enabled,
and compression is set to true, putobject API is failing.

Moving adding the SSE-S3 header as part of the request to before
checking if compression can be done, otherwise the size is set to -1
and that seems to cause problems.
2019-05-02 08:28:18 -07:00
poornas 033f3a4d51 gcs: check error on object writer close (#7606)
Fixes #7605. Object metadata should be written to storage
only when the object was written successfully
2019-05-02 08:27:10 -07:00
poornas cf2a436bc8 Show SlowDown error message if backend is busy (#7521)
or if there are too many open file descriptors.
2019-05-02 07:09:57 -07:00
Harshavardhana 64998fc4ab Remove delayIsLeaf requirement simplify ListObjects further (#7593) 2019-05-02 10:36:57 +05:30
Krishna Srinivas 43be00ea1b Remove logs from bitrot-streaming.go as erasure layer is already logging (#7603) 2019-05-01 21:46:00 -07:00
Eco 3517f64d20 Clarify server maximums via Federation (#7608)
To eliminate confusion about 32 server limit, clarify that the limit is only per cluster, but that unlimited clusters can be Federated
2019-05-01 16:20:25 -07:00
Harshavardhana c5f26d5cdd Fix hdfsReader fd leak upon GetObject() (#7596)
Also migrate to minio/hdfs/v3@v3.0.0
2019-05-01 14:43:21 -07:00
Praveen raj Mani c113d4e49c Posix CreateFile should work for compressed lengths (#7584) 2019-04-30 16:27:31 -07:00