Commit Graph

1760 Commits

Author SHA1 Message Date
kannappanr
39ddb78c75
CORS: Expose all headers on response (#7331)
Fixes #7289
2019-03-06 11:58:53 -08:00
Harshavardhana
12eb71828b Fix posix tests for SimpleCI (#7328) 2019-03-05 19:53:01 -08:00
Praveen raj Mani
c0a1369b73 Construct dynamic XML error responses for postpolicyform validation (#7321)
Fixes #7314
2019-03-05 12:10:47 -08:00
kannappanr
c57159a0fe
fs mode: List already existing buckets with capital letters (#7244)
if a bucket with `Captialized letters` is created, `InvalidBucketName` error
will be returned. 
In the case of pre-existing buckets, it will be listed.

Fixes #6938
2019-03-05 10:42:32 -08:00
Kale Blankenship
ef132c5714 Replace snappy.Writer/io.Pipe with snappyCompressReader. (#7316)
Prevents deferred close functions from being called while still
  attempting to copy reader to snappyWriter.
 Reduces code duplication when compressing objects.
2019-03-05 08:35:37 -08:00
Aditya Manthramurthy
c54b0c0ca1 Fix a race in tests (#7326) 2019-03-05 21:34:17 +05:30
Aditya Manthramurthy
e8e9cd3e74 Close GlobalServiceDoneCh when quitting (#7322)
This change allows indefinitely running go-routines to cleanup
gracefully.

This channel is now closed at the beginning of each test so that
long-running go-routines quit and a new one is assigned.
2019-03-04 14:33:14 -08:00
poornas
2564147ab4 Filter Expires header from user metadata (#7269)
Instead save it as a struct field in ObjectInfo as it is
a standard HTTP header - Fixes minio/mc#2690
2019-02-28 11:01:25 -08:00
Harshavardhana
c3ca954684 Implement AssumeRole API for Minio users (#7267)
For actual API reference read here

https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html

Documentation is added and updated as well at docs/sts/assume-role.md

Fixes #6381
2019-02-27 17:46:55 -08:00
Harshavardhana
ce588d1489 Improve ListObjects performance by listing in parallel (#7270)
The side affect of this change memory
increase, but this is a trade-off between
performance and actual memory usage.

For all practical scenarios this should be
an adequate change.
2019-02-27 14:39:22 -08:00
poornas
8022a6efd9 Return ETag for 0-byte object prefixes (#7291)
Fixes: #7290
2019-02-26 15:09:14 -08:00
Praveen raj Mani
78d116c487 Event persistence for MQTT (#7268)
- The events will be persisted in queueStore if `queueDir` is set.
- Else, if queueDir is not set events persist in memory.

The events are replayed back when the mqtt broker is back online.
2019-02-25 18:01:13 -08:00
Anis Elleuch
6584c7ea2b s3: Encode StartAfter when encoding type is passed (#7281)
In ObjectList V2, StartAfter needs to be encoded when encoding-type
is specified.
2019-02-24 18:50:28 -08:00
Anis Elleuch
5efbe8a1b3 s3: Add support of encodingType parameter (#7265)
This commit honors encoding-type parameter in object listing,
parts listing and multipart uploads listing.
2019-02-24 11:44:24 +05:30
Harshavardhana
7923b83953 Support multiple-domains in MINIO_DOMAIN (#7274)
Fixes #7173
2019-02-23 08:48:01 +05:30
Harshavardhana
bedcb7442a Write xml.Header first instead of spaces to handle XML parsers (#7253)
Clients like AWS SDK Java and AWS cli XML parsers are
unable to handle on `\r\n` characters to avoid these
errors send XML header first and write white space characters
instead.

Also handle cases to avoid double WriteHeader calls
2019-02-21 11:50:15 +05:30
Harshavardhana
91576d416d Fix GetLocalPeer usage in perf handlers (#7249)
GetLocalPeer usage should be fixed and used only
once per call for not all local endpoints.
2019-02-20 16:04:55 -08:00
Krishna Srinivas
6dd26b8231 Detect change in underlying mounted disks (#7229) 2019-02-20 13:32:29 -08:00
poornas
e098852a80 Revert PR #7241 to fix vault renewal (#7259)
- Current implementation was spawning renewer goroutines
without waiting for the lease duration to end. Remove vault renewer
and call vault.RenewToken directly and manage reauthentication if
lease expired.
2019-02-20 12:23:59 -08:00
Krishna Srinivas
ce960565b1 Validate and reject unusual requests (#7258) 2019-02-19 21:02:41 -08:00
poornas
755e675d5c Fix: send decrypted size to notification event (#7248) 2019-02-19 14:14:26 +05:30
Harshavardhana
b6c00405ec Do not pro-actively return false in isObjectDir() (#7246)
We should change the logic for both isObject()
and isObjectDir() leaf detection to be done
with quorum, due to how our directory navigation
works - this allows for properly deleting all
the dangling directories or objects if any.
2019-02-15 16:21:19 -08:00
Harshavardhana
8f62935448 Add proper requestID for STS errors (#7245) 2019-02-14 17:54:33 -08:00
Harshavardhana
396d78352d Support HTTP/2.0 (#7204)
Fixes #6704
2019-02-14 17:53:46 -08:00
Harshavardhana
a51781e5cf Use context to fill in more details about error XML (#7232) 2019-02-13 16:07:21 -08:00
Krishna Srinivas
90213ff1b2 Detect peer reboots to invalidate current storage REST clients (#7227) 2019-02-13 15:29:46 -08:00
Andreas Auernhammer
6f764a8efd crypto: fix nil pointer dereference of vault secret (#7241)
This commit fixes a nil pointer dereference issue
that can occur when the Vault KMS returns e.g. a 404
with an empty HTTP response. The Vault client SDK
does not treat that as error and returns nil for
the error and the secret.

Further it simplifies the token renewal and
re-authentication mechanism by using a single
background go-routine.

The control-flow of Vault authentications looks
like this:
1. `authenticate()`: Initial login and start of background job
2. Background job starts a `vault.Renewer` to renew the token
3. a) If this succeeds the token gets updated
   b) If this fails the background job tries to login again
4. If the login in 3b. succeeded goto 2. If it fails
   goto 3b.
2019-02-13 15:25:32 -08:00
Harshavardhana
df35d7db9d Introduce staticcheck for stricter builds (#7035) 2019-02-13 18:29:36 +05:30
Harshavardhana
4ba77a916d Select should return early errors as XML (#7230)
Currently, we were sending errors in Select binary format,
which is incompatible with AWS S3 behavior, errors in binary
are  sent after HTTP status code is already 200 OK - i.e it
happens during the evaluation of the record reader.
2019-02-13 13:18:11 +05:30
Anis Elleuch
f9fecf0e76 storage: Increase the timeout of storage REST requests (#7218)
This commit increases storage REST requests to 5 minutes, this includes
the opening TCP connection, and sending/receiving data. This will reduce
clients receiving errors when the server is under high load.
2019-02-12 23:27:33 -08:00
Krishna Srinivas
14544d8d84 Validate incoming requests (#7234) 2019-02-12 13:24:14 -08:00
Harshavardhana
fef5416b3c Support unknown gateway errors and convert at handler layer (#7219)
Different gateway implementations due to different backend
API errors, might return different unsupported errors at
our handler layer. Current code posed a problem for us because
this information was lost and we would convert it to InternalError
in this situation all S3 clients end up retrying the request.

To avoid this unexpected situation implement a way to support
this cleanly such that the underlying information is not lost
which is returned by gateway.
2019-02-12 14:55:52 +05:30
Harshavardhana
9f87283cd5 Revert and bring back B2 gateway implementation (#7224)
This PR is simply a revert of 3265112d04
just for B2 gateway.
2019-02-12 12:44:22 +05:30
Harshavardhana
b8955fe577 Fix DummyHandlers to authorize and send/validate correct XMLs (#7223) 2019-02-11 17:58:26 -08:00
Harshavardhana
082f777281 Revamp bucket metadata healing (#7208)
Bucket metadata healing in the current code was executed multiple
times each time for a given set. Bucket metadata just like
objects are hashed in accordance with its name on any given set,
to allow hashing to play a role we should let the top level
code decide where to navigate.

Current code also had 3 bucket metadata files hardcoded, whereas
we should make it generic by listing and navigating the .minio.sys
to heal such objects.

We also had another bug where due to isObjectDangling changes
without pre-existing bucket metadata files, we were erroneously
reporting it as grey/corrupted objects.

This PR fixes all of the above items.
2019-02-11 09:23:13 +05:30
John Liu
9600e2b35e Comment Typo: Changed 'jason' to 'json` (#7216) 2019-02-10 05:49:00 -08:00
poornas
40b8d11209 Move metadata into ObjectOptions for NewMultipart and PutObject (#7060) 2019-02-09 11:01:06 +05:30
Sidhartha Mani
c1b3f1994b remove unnecessary buffer while discarding stream (#7214) 2019-02-08 19:29:51 -08:00
ebozduman
dd52e5ebe9 Implements dummy tagging handlers for Terraform (#7040) 2019-02-08 16:18:13 -08:00
Praveen raj Mani
8af1f0cc7b Improved error message for user and access key conflict (#7190) 2019-02-07 17:25:58 -08:00
Harshavardhana
85e939636f Fix JSON parser handling for certain objects (#7162)
This PR also adds some comments and simplifies
the code. Primary handling is done to ensure
that we make sure to honor cached buffer.

Added unit tests as well

Fixes #7141
2019-02-07 08:04:42 +05:30
poornas
d203e7e1cc azure gateway: return MD5Sum as ETag for S3 API compatibility (#6884)
Fixes #6872.

This PR refactors multipart upload implementation to use a per
part metadata file which is cleaned up at the end of the upload
2019-02-06 16:58:43 -08:00
Harshavardhana
817269475f Make sure to drain body upon an error (#7197)
Also cleanup redundant code and use it at a common place
2019-02-06 12:07:03 -08:00
Krishna Srinivas
2d168b532b Allow format.json healing on dev/test setup (single node XL, all root disks) (#7170) 2019-02-06 11:44:19 -08:00
Krishna Srinivas
3dfbe0f68c Send white spaces to client till completeMultipart() process completes (#7198) 2019-02-05 20:58:09 -08:00
Harshavardhana
30135eed86 Redo how to handle stale dangling files (#7171)
foo.CORRUPTED should never be created because when
multiple sets are involved we would hash the file
to wrong a location, this PR removes the code.

But allows DeleteBucket() to work properly to delete
dangling buckets/objects. Also adds another option
to Healing where a user needs to specify `--remove`
such that all dangling objects will be deleted with
user confirmation.
2019-02-05 17:58:48 -08:00
Harshavardhana
e4081aee62 Added support for reading body in STS API (#7188)
STS API supports both URL query params and reading
from a body.
2019-02-05 15:47:11 -08:00
kannappanr
df418a2783
Create Cors handler with permissive configuration (#7186)
Create new Cors handler allowing all origins with all standard
methods with any header and credentials.

Fixes #7181
2019-02-05 14:06:52 -08:00
kannappanr
9a65f6dc97 Remove duplicate code in object-handlers.go (#7176)
removed duplicate code in CompleteMultipartUploadHandler
and CopyObjectPartHandler.
2019-02-05 13:36:38 -08:00
Harshavardhana
ea6d61ab1f Use loadCachedConfigs appropriately to load ENVs (#7187) 2019-02-04 10:31:11 +05:30