MyFavMirrors/minio
1
0
Fork 0
mirror of https://github.com/minio/minio.git synced 2024-12-26 07:05:55 -05:00
Code Issues Packages Projects Releases Wiki Activity
3,085 Commits 4 Branches 506 Tags 189 MiB
2a95eabb8a
Commit Graph

76 Commits

Author SHA1 Message Date
Harshavardhana
5102a5877e API/handler: CopyObject make it behave in accordance with S3 spec. (#2155) 
Fixes bugs found while running s3verify tool - fixes #2152
 2016-07-09 12:13:40 -07:00
Harshavardhana
ec35330ebb XL/fs: GetObject should validate all its inputs. (#2142) 
Fixes #2141
Fixes #2139
 2016-07-08 07:46:49 -07:00
Nick Craig-Wood
8c767218a4 URL Encode X-Amz-Copy-Source as per the spec (#2114) 
The documents for COPY state that the X-Amz-Copy-Source must be URL encoded.

http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectCOPY.html
 2016-07-06 15:42:17 -07:00
Bala FA
a51bb1d728 http: handle request range like Amazon S3. (#2112) 
Fixes #2098
 2016-07-06 12:50:24 -07:00
Harshavardhana
8a028a9efb handler/PUT: Handle signature verification through a custom reader. (#2066) 
Change brings in a new signVerifyReader which provides a io.Reader
compatible reader, additionally implements Verify() function.

Verify() function validates the signature present in the incoming
request. This approach is choosen to avoid complexities involved
in using io.Pipe().

Thanks to Krishna for his inputs on this.

Fixes #2058
Fixes #2054
Fixes #2087
 2016-07-05 01:04:50 -07:00
Harshavardhana
d2557bb538 XL: GetObject caching implemented for XL. (#2017) 
The object cache implementation is XL cache, which defaults
to 8GB worth of read cache. Currently GetObject() transparently
writes to this cache upon first client read and then subsequently
serves reads from the same cache.

Currently expiration is not implemented.
 2016-07-03 17:15:38 -07:00
Krishna Srinivas
eb5f782c74 object-handler: skip sha256 calculation if x-amz-content-sha256=="UNSIGNED-PAYLOAD" (#2038) 
fixes #2024 #2056
 2016-07-01 14:34:40 -07:00
Krishnan Parthasarathi
bcb822c390 Send XML header before the first of whitespace chars (#2046) 
* Sent XML header before the first of whitespace chars

XML parsing fails in aws cli due to unexpected whitespace character. To
fix this, we send the xml header before we send the first whitespace
character, if any.

* Fix race between sendWhiteSpaceChars and completeMultiUploadpart
 2016-06-30 18:48:50 -07:00
Harshavardhana
748dc80047 API: add writePartTooSmallErrorResponse to extend standard error responses. (#2005) 
This function is added to extend the standard error responses.
Which is needed in some cases for example CompleteMultipartUpload
should respond with ErrPartTooSmall error when parts uploaded are
lesser than 5MB (i.e minimum allowed size per part).

Fixes #1536
 2016-06-28 14:51:49 -07:00
Krishnan Parthasarathi
a854e8cc5c api: Sent ErrPreconditionFailed on If-Match failure (#2009) 
* api: Sent ErrPreconditionFailed on If-Match failure

ref:
http://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html#ErrorCodeList

* tests: Added functional tests for GetObject w/ If-Match headers set

* tests: Used verifyError to simplify errorCode and description matching on error
 2016-06-28 01:18:18 -07:00
Krishnan Parthasarathi
d0be09fdd3 object: checkETag compares quoted ETags properly (#1997) 
Previously, checkETag didn't handle ETags with leading and trailing
double quotes. e.g "abcdef1234" == "\"abcdef1234\"" would return false.
Now, checkETag function canonicalizes the ETags passed as arguments by
removing one leading/trailing double quote.
 2016-06-26 18:10:08 -07:00
Krishnan Parthasarathi
129ebbd685 object layer: Send 200 OK and whitespace chars (#1897) 2016-06-16 09:01:06 +05:30
Harshavardhana
feb337098d XL: bring in new storage API. (#1780) 
Fixes #1771
 2016-05-28 16:12:51 -07:00
Harshavardhana
b2293c2bf4 XL: Rename, cleanup and add more comments. (#1769) 
- xl-v1-bucket.go - removes a whole bunch of code.
- {xl-v1,fs-v1}-metadata.go - add a lot of comments and rename functions
   appropriately.
 2016-05-28 15:15:53 -07:00
Krishnan Parthasarathi
3550660163 Return error for empty parts in multipartupload complete (#1758) 2016-05-28 15:15:53 -07:00
Harshavardhana
ee6645f421 XL: Add additional PartNumber variable as part of xl.json (#1750) 
This is needed for verification of incoming parts and to
support variadic part uploads. Which should be sorted
properly.

Fixes #1740
 2016-05-28 15:15:53 -07:00
Aakash Muttineni
b48b2e7f7c Part ID check (#1730) 
* Added check in PutObjectPartHandler to make sure part ID does not exceed 10000. ErrInvalidMaxParts written to response if part ID exceeds the maximum value.
 2016-05-24 01:52:47 -07:00
Harshavardhana
9fdb69563d handler: CopyObject should save metadata. (#1698) 
- Content-Type
- Content-Encoding
- ETag

Fixes #1682
 2016-05-19 17:10:08 -07:00
Harshavardhana
7d6ed50fc2 objects: Save all the incoming metadata properly. (#1688) 
For both multipart and single put operation
 2016-05-18 19:54:25 -07:00
Harshavardhana
9472299308 logging: Log only for unhandled errors, remove all the debug logging. (#1652) 
This patch brings in the removal of debug logging altogether, instead
we bring in the functionality of being able to trace the errors properly
pointing back to the origination of the problem.

To enable tracing you need to enable "MINIO_TRACE" set to "1" or "true"
environment variable which would print back traces whenever there is an
error which is unhandled or at the handler layer.

By default this tracing is turned off and only user level logging is
provided.
 2016-05-16 14:31:28 -07:00
Harshavardhana
498ce1e9bb handler: Add a waitgroup to avoid expect100Continue crash. (#1623) 
This waitgroup allows for safe blocking operation where we can cleanly
control the flow of the writes and the underlying pipe altogether.

Fixes #1553
 2016-05-14 17:18:00 -07:00
Harshavardhana
5b29cefd40 api: DeleteObject should always return 204. (#1645) 
Fixes #1643
 2016-05-14 15:47:19 -07:00
Harshavardhana
d4745c7d6a object: PutObjectHandler should set the md5Sum properly. (#1604) 
Additionally add a test case as well for validating for us
to reply BadDigest properly.

Fixes #1603
 2016-05-11 16:13:37 -07:00
karthic rao
26e2c4bf4d Replacing fastsha256 with crypto/sha256 package from golang standard package (#1584) 2016-05-10 14:20:11 -07:00
Harshavardhana
091c1e8456 copyObject: No need to verify md5sum. (#1520) 
Multipart objects are kept in non hex md5sum format.
This format doesn't comply with hex, so decoding
would fail invariably.

This is not necessary to validate and its not expected
error during a CopyObject operation.

Fixes #1519
 2016-05-07 03:43:08 -07:00
Harshavardhana
751fa972f5 xl/fs: Multipart re-org introduce "uploads.json" (#1505) 
Fixes #1457
 2016-05-07 02:08:03 -07:00
Harshavardhana
17868ccd7f handlers: overhaul entire writErrorResponse, simplify. (#1472) 2016-05-05 20:24:29 -07:00
Harshavardhana
4e34e03dd4 xl/fs: Split object layer into interface. (#1415) 2016-04-29 14:24:10 -07:00
Harshavardhana
a98a7fb1ad Implement XL layer - preliminary work. 2016-04-25 12:47:31 -07:00
Harshavardhana
e0f8fed011 object: handle Error responses and handle errDiskFull. (#1331) 2016-04-19 02:42:10 -07:00
Harshavardhana
be002ac01e fs/object: Fix issues from review comments. 2016-04-16 17:57:14 -07:00
Harshavardhana
8457af5708 fs: Add proper volume and path validation. 2016-04-16 16:25:53 -07:00
Krishna Srinivas
caa35f68fa listMultipart: implement support for marker. (#1313) 2016-04-16 16:25:53 -07:00
Harshavardhana
30b0b4deba storage/server/client: Enable storage server, enable client storage. 2016-04-16 16:25:53 -07:00
Krishna Srinivas
01a439f95b refactor: add multipart code to the object layer. 2016-04-16 16:25:53 -07:00
Krishna Srinivas
3c48537f20 refactor: refactor code to separate fs into object-layer and fs layer. (#1305) 2016-04-16 16:25:53 -07:00
Harshavardhana
b182e94acc signature: Handle presigned payload if set. 
Validate payload with incoming content.



Fixes #1288
 2016-04-07 03:04:18 -07:00
Donald Guy
e8cd1aad8d accessPolicy: prevent backdoor ListBucket via brute-force 404s, per docs + small fixes 
* accessPolicy: copy object should require PutObject

* accessPolicy: cite mpu perms doc only for relevant operations

* accessPolicy: prevent backdoor ListBucket via brute-force 404s, per docs
 2016-04-06 18:31:40 -07:00
Donald Guy
8b4a5f07b4 accessPolicy: allow anonymous HEAD for Getable objects 
* accessPolicy: allow anonymous HEAD for Getable objects

* accessPolicy: allow anonymous HEAD of Listable Buckets
 2016-04-06 16:40:54 -07:00
Harshavardhana
c69fdf0cf2 listObjects: Cleanup and naming conventions. 
- Marker should be escaped outside in handlers.

- Delimiter should be handled outside in handlers.

- Add missing comments and change the function names.

- Handle case of 'maxKeys' when its set to '0', its a valid

  case and should be treated as such.
 2016-04-04 19:55:07 -07:00
Harshavardhana
0479d4976b objectAPI: Fix object API interface, remove unnecessary structs. 
ObjectAPI changes.
```
ListObjects(bucket, prefix, marker, delimiter string, maxKeys int) (ListObjectsInfo, *probe.Error)
ListMultipartUploads(bucket, objectPrefix, keyMarker, uploadIDMarker, delimiter string, maxUploads int) (ListMultipartsInfo, *probe.Error)
ListObjectParts(bucket, object, uploadID string, partNumberMarker, maxParts int) (ListPartsInfo, *probe.Error)
CompleteMultipartUpload(bucket string, object string, uploadID string, parts []completePart) (ObjectInfo, *probe.Error)
```
 2016-04-03 15:25:01 -07:00
Harshavardhana
a6a4e7e297 signature: No need to validate region for getBucketLocation and listBuckets. 
This type of check is added for making sure that we can support
custom regions.

ListBuckets and GetBucketLocation are always "us-east-1" rest
should look for the configured region.

Fixes #1278
 2016-04-02 18:42:32 -07:00
Harshavardhana
efc80343e3 fs: Break fs package to top-level and introduce ObjectAPI interface. 
ObjectAPI interface brings in changes needed for XL ObjectAPI layer.

The new interface for any ObjectAPI layer is as below

```
// ObjectAPI interface.
type ObjectAPI interface {
        // Bucket resource API.
        DeleteBucket(bucket string) *probe.Error
        ListBuckets() ([]BucketInfo, *probe.Error)
        MakeBucket(bucket string) *probe.Error
        GetBucketInfo(bucket string) (BucketInfo, *probe.Error)

        // Bucket query API.
        ListObjects(bucket, prefix, marker, delimiter string, maxKeys int) (ListObjectsResult, *probe.Error)
        ListMultipartUploads(bucket string, resources BucketMultipartResourcesMetadata) (BucketMultipartResourcesMetadata, *probe.Error)

        // Object resource API.
        GetObject(bucket, object string, startOffset int64) (io.ReadCloser, *probe.Error)
        GetObjectInfo(bucket, object string) (ObjectInfo, *probe.Error)
        PutObject(bucket string, object string, size int64, data io.Reader, metadata map[string]string) (ObjectInfo, *probe.Error)
        DeleteObject(bucket, object string) *probe.Error

        // Object query API.
        NewMultipartUpload(bucket, object string) (string, *probe.Error)
        PutObjectPart(bucket, object, uploadID string, partID int, size int64, data io.Reader, md5Hex string) (string, *probe.Error)
        ListObjectParts(bucket, object string, resources ObjectResourcesMetadata) (ObjectResourcesMetadata, *probe.Error)
        CompleteMultipartUpload(bucket string, object string, uploadID string, parts []CompletePart) (ObjectInfo, *probe.Error)
        AbortMultipartUpload(bucket, object, uploadID string) *probe.Error
}
```
 2016-04-01 15:58:39 -07:00
Harshavardhana
02ad48466d error: Signature errors should be returned with APIErrorCode. 
The reasoning is that we can reply back with wide range of
S3 error responses, which would provide more richer context
to S3 client.

Fixes #1267
 2016-03-31 23:28:40 -07:00
Harshavardhana
9dca46e156 signature: Use a layered approach for signature verification. 
Signature calculation has now moved out from being a package to
top-level as a layered mechanism.

In case of payload calculation with body, go-routines are initiated
to simultaneously write and calculate shasum. Errors are sent
over the writer so that the lower layer removes the temporary files
properly.
 2016-03-26 15:21:05 -07:00
awwalker
34f2c5bcdf verify before writing 
merge

verify headers before writing
 2016-03-16 18:03:23 -07:00
awwalker
9a5e3299fc api/object: Add CopyObject to support match/modified copy headers 
Adds support for the following request headers:

- x-amz-copy-source-if-match
- x-amz-copy-source-if-none-match
- x-amz-copy-source-if-unmodified-since
- x-amz-copy-source-if-modified-since

Fixes #1176
 2016-03-12 10:54:23 -08:00
Harshavardhana
52751d81cb cleanup: Rename ObjectMetadata as ObjectInfo. 
Fixes #1215
 2016-03-11 16:58:08 -08:00
Harshavardhana
fdf3d64793 error: Add proper prefixes for s3Error codes. 
This patch adds 'Err' prefix for all standard API
error codes and also adds a proper type for them.
 2016-03-10 18:38:46 -08:00
Harshavardhana
d5057b3c51 accessPolicy: Implement Put, Get, Delete access policy. 
This patch implements Get,Put,Delete bucket policies

Supporting - http://docs.aws.amazon.com/AmazonS3/latest/dev/access-policy-language-overview.html

Currently supports following actions.

   "*":                             true,
   "s3:*":                          true,
   "s3:GetObject":                  true,
   "s3:ListBucket":                 true,
   "s3:PutObject":                  true,
   "s3:CreateBucket":               true,
   "s3:GetBucketLocation":          true,
   "s3:DeleteBucket":               true,
   "s3:DeleteObject":               true,
   "s3:AbortMultipartUpload":       true,
   "s3:ListBucketMultipartUploads": true,
   "s3:ListMultipartUploadParts":   true,

following conditions for "StringEquals" and "StringNotEquals"

   "s3:prefix", "s3:max-keys"
 2016-03-08 17:44:50 -08:00