Commit Graph

85 Commits

Author SHA1 Message Date
Harshavardhana 2f7358a8a6 XL: erasure Index should have its corresponding distribution order. (#2300) 2016-07-27 11:57:08 -07:00
Harshavardhana 77248bd6e8 api: Notify events only if bucket notifications are set. (#2293)
While the existing code worked, it went to an entire cycle
of constructing event structure and end up not sending it.

Avoid this in the first place, but returning quickly if
notifications are not set on the bucket.
2016-07-26 19:10:02 -07:00
Harshavardhana f248089523 api: Implement bucket notification. (#2271)
* Implement basic S3 notifications through queues

Supports multiple queues and three basic queue types:

1. NilQueue -- messages don't get sent anywhere
2. LogQueue -- messages get logged
3. AmqpQueue -- messages are sent to an AMQP queue

* api: Implement bucket notification.

Supports two different queue types

- AMQP
- ElasticSearch.

* Add support for redis
2016-07-23 22:51:12 -07:00
Harshavardhana f85d94288d api: extract http headers with some supported header list. (#2268) 2016-07-22 20:31:45 -07:00
Harshavardhana c1e953b368 api: Set content-encoding properly if set. (#2245)
Additionally don't set content-type if not present, golang http
server automaticaly handles this and sets it automatically.
2016-07-20 12:40:20 -07:00
utsl42 e5cd35aad0 XL: GetObjectInfo() store and retrieve user-defined object metadata. (#2189) 2016-07-12 12:45:17 -07:00
Harshavardhana e9647b5f12 API/CopyObject: Refactor the code and handle if-modified-since as well. (#2183)
This completes the S3 spec behavior for CopyObject API as reported
by `s3verify`.
2016-07-11 19:24:34 -07:00
Harshavardhana d676e660c9 API/CopyObject: If-None-Match should return Precondition failed. (#2164) 2016-07-10 17:32:59 -07:00
Krishna Srinivas aa7079fc7b XL/GetObject: If quorum not available during GetObject appropriate error should be returned. (#2135) 2016-07-10 17:12:22 -07:00
Harshavardhana 5102a5877e API/handler: CopyObject make it behave in accordance with S3 spec. (#2155)
Fixes bugs found while running s3verify tool - fixes #2152
2016-07-09 12:13:40 -07:00
Harshavardhana ec35330ebb XL/fs: GetObject should validate all its inputs. (#2142)
Fixes #2141
Fixes #2139
2016-07-08 07:46:49 -07:00
Nick Craig-Wood 8c767218a4 URL Encode X-Amz-Copy-Source as per the spec (#2114)
The documents for COPY state that the X-Amz-Copy-Source must be URL encoded.

http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectCOPY.html
2016-07-06 15:42:17 -07:00
Bala FA a51bb1d728 http: handle request range like Amazon S3. (#2112)
Fixes #2098
2016-07-06 12:50:24 -07:00
Harshavardhana 8a028a9efb handler/PUT: Handle signature verification through a custom reader. (#2066)
Change brings in a new signVerifyReader which provides a io.Reader
compatible reader, additionally implements Verify() function.

Verify() function validates the signature present in the incoming
request. This approach is choosen to avoid complexities involved
in using io.Pipe().

Thanks to Krishna for his inputs on this.

Fixes #2058
Fixes #2054
Fixes #2087
2016-07-05 01:04:50 -07:00
Harshavardhana d2557bb538 XL: GetObject caching implemented for XL. (#2017)
The object cache implementation is XL cache, which defaults
to 8GB worth of read cache. Currently GetObject() transparently
writes to this cache upon first client read and then subsequently
serves reads from the same cache.

Currently expiration is not implemented.
2016-07-03 17:15:38 -07:00
Krishna Srinivas eb5f782c74 object-handler: skip sha256 calculation if x-amz-content-sha256=="UNSIGNED-PAYLOAD" (#2038)
fixes #2024 #2056
2016-07-01 14:34:40 -07:00
Krishnan Parthasarathi bcb822c390 Send XML header before the first of whitespace chars (#2046)
* Sent XML header before the first of whitespace chars

XML parsing fails in aws cli due to unexpected whitespace character. To
fix this, we send the xml header before we send the first whitespace
character, if any.

* Fix race between sendWhiteSpaceChars and completeMultiUploadpart
2016-06-30 18:48:50 -07:00
Harshavardhana 748dc80047 API: add writePartTooSmallErrorResponse to extend standard error responses. (#2005)
This function is added to extend the standard error responses.
Which is needed in some cases for example CompleteMultipartUpload
should respond with ErrPartTooSmall error when parts uploaded are
lesser than 5MB (i.e minimum allowed size per part).

Fixes #1536
2016-06-28 14:51:49 -07:00
Krishnan Parthasarathi a854e8cc5c api: Sent ErrPreconditionFailed on If-Match failure (#2009)
* api: Sent ErrPreconditionFailed on If-Match failure

ref:
http://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html#ErrorCodeList

* tests: Added functional tests for GetObject w/ If-Match headers set

* tests: Used verifyError to simplify errorCode and description matching on error
2016-06-28 01:18:18 -07:00
Krishnan Parthasarathi d0be09fdd3 object: checkETag compares quoted ETags properly (#1997)
Previously, checkETag didn't handle ETags with leading and trailing
double quotes. e.g "abcdef1234" == "\"abcdef1234\"" would return false.
Now, checkETag function canonicalizes the ETags passed as arguments by
removing one leading/trailing double quote.
2016-06-26 18:10:08 -07:00
Krishnan Parthasarathi 129ebbd685 object layer: Send 200 OK and whitespace chars (#1897) 2016-06-16 09:01:06 +05:30
Harshavardhana feb337098d XL: bring in new storage API. (#1780)
Fixes #1771
2016-05-28 16:12:51 -07:00
Harshavardhana b2293c2bf4 XL: Rename, cleanup and add more comments. (#1769)
- xl-v1-bucket.go - removes a whole bunch of code.
- {xl-v1,fs-v1}-metadata.go - add a lot of comments and rename functions
   appropriately.
2016-05-28 15:15:53 -07:00
Krishnan Parthasarathi 3550660163 Return error for empty parts in multipartupload complete (#1758) 2016-05-28 15:15:53 -07:00
Harshavardhana ee6645f421 XL: Add additional PartNumber variable as part of `xl.json` (#1750)
This is needed for verification of incoming parts and to
support variadic part uploads. Which should be sorted
properly.

Fixes #1740
2016-05-28 15:15:53 -07:00
Aakash Muttineni b48b2e7f7c Part ID check (#1730)
* Added check in PutObjectPartHandler to make sure part ID does not exceed 10000. ErrInvalidMaxParts written to response if part ID exceeds the maximum value.
2016-05-24 01:52:47 -07:00
Harshavardhana 9fdb69563d handler: CopyObject should save metadata. (#1698)
- Content-Type
- Content-Encoding
- ETag

Fixes #1682
2016-05-19 17:10:08 -07:00
Harshavardhana 7d6ed50fc2 objects: Save all the incoming metadata properly. (#1688)
For both multipart and single put operation
2016-05-18 19:54:25 -07:00
Harshavardhana 9472299308 logging: Log only for unhandled errors, remove all the debug logging. (#1652)
This patch brings in the removal of debug logging altogether, instead
we bring in the functionality of being able to trace the errors properly
pointing back to the origination of the problem.

To enable tracing you need to enable "MINIO_TRACE" set to "1" or "true"
environment variable which would print back traces whenever there is an
error which is unhandled or at the handler layer.

By default this tracing is turned off and only user level logging is
provided.
2016-05-16 14:31:28 -07:00
Harshavardhana 498ce1e9bb handler: Add a waitgroup to avoid expect100Continue crash. (#1623)
This waitgroup allows for safe blocking operation where we can cleanly
control the flow of the writes and the underlying pipe altogether.

Fixes #1553
2016-05-14 17:18:00 -07:00
Harshavardhana 5b29cefd40 api: DeleteObject should always return 204. (#1645)
Fixes #1643
2016-05-14 15:47:19 -07:00
Harshavardhana d4745c7d6a object: PutObjectHandler should set the md5Sum properly. (#1604)
Additionally add a test case as well for validating for us
to reply BadDigest properly.

Fixes #1603
2016-05-11 16:13:37 -07:00
karthic rao 26e2c4bf4d Replacing fastsha256 with crypto/sha256 package from golang standard package (#1584) 2016-05-10 14:20:11 -07:00
Harshavardhana 091c1e8456 copyObject: No need to verify md5sum. (#1520)
Multipart objects are kept in non hex md5sum format.
This format doesn't comply with hex, so decoding
would fail invariably.

This is not necessary to validate and its not expected
error during a CopyObject operation.

Fixes #1519
2016-05-07 03:43:08 -07:00
Harshavardhana 751fa972f5 xl/fs: Multipart re-org introduce "uploads.json" (#1505)
Fixes #1457
2016-05-07 02:08:03 -07:00
Harshavardhana 17868ccd7f handlers: overhaul entire writErrorResponse, simplify. (#1472) 2016-05-05 20:24:29 -07:00
Harshavardhana 4e34e03dd4 xl/fs: Split object layer into interface. (#1415) 2016-04-29 14:24:10 -07:00
Harshavardhana a98a7fb1ad Implement XL layer - preliminary work. 2016-04-25 12:47:31 -07:00
Harshavardhana e0f8fed011 object: handle Error responses and handle errDiskFull. (#1331) 2016-04-19 02:42:10 -07:00
Harshavardhana be002ac01e fs/object: Fix issues from review comments. 2016-04-16 17:57:14 -07:00
Harshavardhana 8457af5708 fs: Add proper volume and path validation. 2016-04-16 16:25:53 -07:00
Krishna Srinivas caa35f68fa listMultipart: implement support for marker. (#1313) 2016-04-16 16:25:53 -07:00
Harshavardhana 30b0b4deba storage/server/client: Enable storage server, enable client storage. 2016-04-16 16:25:53 -07:00
Krishna Srinivas 01a439f95b refactor: add multipart code to the object layer. 2016-04-16 16:25:53 -07:00
Krishna Srinivas 3c48537f20 refactor: refactor code to separate fs into object-layer and fs layer. (#1305) 2016-04-16 16:25:53 -07:00
Harshavardhana b182e94acc signature: Handle presigned payload if set.
Validate payload with incoming content.



Fixes #1288
2016-04-07 03:04:18 -07:00
Donald Guy e8cd1aad8d accessPolicy: prevent backdoor ListBucket via brute-force 404s, per docs + small fixes
* accessPolicy: copy object should require PutObject

* accessPolicy: cite mpu perms doc only for relevant operations

* accessPolicy: prevent backdoor ListBucket via brute-force 404s, per docs
2016-04-06 18:31:40 -07:00
Donald Guy 8b4a5f07b4 accessPolicy: allow anonymous HEAD for Getable objects
* accessPolicy: allow anonymous HEAD for Getable objects

* accessPolicy: allow anonymous HEAD of Listable Buckets
2016-04-06 16:40:54 -07:00
Harshavardhana c69fdf0cf2 listObjects: Cleanup and naming conventions.
- Marker should be escaped outside in handlers.

- Delimiter should be handled outside in handlers.

- Add missing comments and change the function names.

- Handle case of 'maxKeys' when its set to '0', its a valid

  case and should be treated as such.
2016-04-04 19:55:07 -07:00
Harshavardhana 0479d4976b objectAPI: Fix object API interface, remove unnecessary structs.
ObjectAPI changes.
```
ListObjects(bucket, prefix, marker, delimiter string, maxKeys int) (ListObjectsInfo, *probe.Error)
ListMultipartUploads(bucket, objectPrefix, keyMarker, uploadIDMarker, delimiter string, maxUploads int) (ListMultipartsInfo, *probe.Error)
ListObjectParts(bucket, object, uploadID string, partNumberMarker, maxParts int) (ListPartsInfo, *probe.Error)
CompleteMultipartUpload(bucket string, object string, uploadID string, parts []completePart) (ObjectInfo, *probe.Error)
```
2016-04-03 15:25:01 -07:00