Commit Graph

92 Commits

Author SHA1 Message Date
Krishna Srinivas 01a439f95b refactor: add multipart code to the object layer. 2016-04-16 16:25:53 -07:00
Krishna Srinivas 3c48537f20 refactor: refactor code to separate fs into object-layer and fs layer. (#1305) 2016-04-16 16:25:53 -07:00
Harshavardhana b182e94acc signature: Handle presigned payload if set.
Validate payload with incoming content.



Fixes #1288
2016-04-07 03:04:18 -07:00
Donald Guy e8cd1aad8d accessPolicy: prevent backdoor ListBucket via brute-force 404s, per docs + small fixes
* accessPolicy: copy object should require PutObject

* accessPolicy: cite mpu perms doc only for relevant operations

* accessPolicy: prevent backdoor ListBucket via brute-force 404s, per docs
2016-04-06 18:31:40 -07:00
Donald Guy 8b4a5f07b4 accessPolicy: allow anonymous HEAD for Getable objects
* accessPolicy: allow anonymous HEAD for Getable objects

* accessPolicy: allow anonymous HEAD of Listable Buckets
2016-04-06 16:40:54 -07:00
Harshavardhana c69fdf0cf2 listObjects: Cleanup and naming conventions.
- Marker should be escaped outside in handlers.

- Delimiter should be handled outside in handlers.

- Add missing comments and change the function names.

- Handle case of 'maxKeys' when its set to '0', its a valid

  case and should be treated as such.
2016-04-04 19:55:07 -07:00
Harshavardhana 0479d4976b objectAPI: Fix object API interface, remove unnecessary structs.
ObjectAPI changes.
```
ListObjects(bucket, prefix, marker, delimiter string, maxKeys int) (ListObjectsInfo, *probe.Error)
ListMultipartUploads(bucket, objectPrefix, keyMarker, uploadIDMarker, delimiter string, maxUploads int) (ListMultipartsInfo, *probe.Error)
ListObjectParts(bucket, object, uploadID string, partNumberMarker, maxParts int) (ListPartsInfo, *probe.Error)
CompleteMultipartUpload(bucket string, object string, uploadID string, parts []completePart) (ObjectInfo, *probe.Error)
```
2016-04-03 15:25:01 -07:00
Harshavardhana a6a4e7e297 signature: No need to validate region for getBucketLocation and listBuckets.
This type of check is added for making sure that we can support
custom regions.

ListBuckets and GetBucketLocation are always "us-east-1" rest
should look for the configured region.

Fixes #1278
2016-04-02 18:42:32 -07:00
Harshavardhana efc80343e3 fs: Break fs package to top-level and introduce ObjectAPI interface.
ObjectAPI interface brings in changes needed for XL ObjectAPI layer.

The new interface for any ObjectAPI layer is as below

```
// ObjectAPI interface.
type ObjectAPI interface {
        // Bucket resource API.
        DeleteBucket(bucket string) *probe.Error
        ListBuckets() ([]BucketInfo, *probe.Error)
        MakeBucket(bucket string) *probe.Error
        GetBucketInfo(bucket string) (BucketInfo, *probe.Error)

        // Bucket query API.
        ListObjects(bucket, prefix, marker, delimiter string, maxKeys int) (ListObjectsResult, *probe.Error)
        ListMultipartUploads(bucket string, resources BucketMultipartResourcesMetadata) (BucketMultipartResourcesMetadata, *probe.Error)

        // Object resource API.
        GetObject(bucket, object string, startOffset int64) (io.ReadCloser, *probe.Error)
        GetObjectInfo(bucket, object string) (ObjectInfo, *probe.Error)
        PutObject(bucket string, object string, size int64, data io.Reader, metadata map[string]string) (ObjectInfo, *probe.Error)
        DeleteObject(bucket, object string) *probe.Error

        // Object query API.
        NewMultipartUpload(bucket, object string) (string, *probe.Error)
        PutObjectPart(bucket, object, uploadID string, partID int, size int64, data io.Reader, md5Hex string) (string, *probe.Error)
        ListObjectParts(bucket, object string, resources ObjectResourcesMetadata) (ObjectResourcesMetadata, *probe.Error)
        CompleteMultipartUpload(bucket string, object string, uploadID string, parts []CompletePart) (ObjectInfo, *probe.Error)
        AbortMultipartUpload(bucket, object, uploadID string) *probe.Error
}
```
2016-04-01 15:58:39 -07:00
Harshavardhana 02ad48466d error: Signature errors should be returned with APIErrorCode.
The reasoning is that we can reply back with wide range of
S3 error responses, which would provide more richer context
to S3 client.

Fixes #1267
2016-03-31 23:28:40 -07:00
Harshavardhana 9dca46e156 signature: Use a layered approach for signature verification.
Signature calculation has now moved out from being a package to
top-level as a layered mechanism.

In case of payload calculation with body, go-routines are initiated
to simultaneously write and calculate shasum. Errors are sent
over the writer so that the lower layer removes the temporary files
properly.
2016-03-26 15:21:05 -07:00
awwalker 34f2c5bcdf verify before writing
merge

verify headers before writing
2016-03-16 18:03:23 -07:00
awwalker 9a5e3299fc api/object: Add CopyObject to support match/modified copy headers
Adds support for the following request headers:

- x-amz-copy-source-if-match
- x-amz-copy-source-if-none-match
- x-amz-copy-source-if-unmodified-since
- x-amz-copy-source-if-modified-since

Fixes #1176
2016-03-12 10:54:23 -08:00
Harshavardhana 52751d81cb cleanup: Rename ObjectMetadata as ObjectInfo.
Fixes #1215
2016-03-11 16:58:08 -08:00
Harshavardhana fdf3d64793 error: Add proper prefixes for s3Error codes.
This patch adds 'Err' prefix for all standard API
error codes and also adds a proper type for them.
2016-03-10 18:38:46 -08:00
Harshavardhana d5057b3c51 accessPolicy: Implement Put, Get, Delete access policy.
This patch implements Get,Put,Delete bucket policies

Supporting - http://docs.aws.amazon.com/AmazonS3/latest/dev/access-policy-language-overview.html

Currently supports following actions.

   "*":                             true,
   "s3:*":                          true,
   "s3:GetObject":                  true,
   "s3:ListBucket":                 true,
   "s3:PutObject":                  true,
   "s3:CreateBucket":               true,
   "s3:GetBucketLocation":          true,
   "s3:DeleteBucket":               true,
   "s3:DeleteObject":               true,
   "s3:AbortMultipartUpload":       true,
   "s3:ListBucketMultipartUploads": true,
   "s3:ListMultipartUploadParts":   true,

following conditions for "StringEquals" and "StringNotEquals"

   "s3:prefix", "s3:max-keys"
2016-03-08 17:44:50 -08:00
Harshavardhana aed62788d9 api: Implement multiple objects Delete api - fixes #956
This API takes input XML input in following form.

```
<?xml version="1.0" encoding="UTF-8"?>
<Delete>
    <Quiet>true</Quiet>
    <Object>
         <Key>Key</Key>
    </Object>
    <Object>
         <Key>Key</Key>
    </Object>
    ...
</Delete>
```

and responds the list of successful deletes, list of errors
for all the deleted objects.

```
<?xml version="1.0" encoding="UTF-8"?>
<DeleteResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  <Deleted>
    <Key>sample1.txt</Key>
  </Deleted>
  <Error>
    <Key>sample2.txt</Key>
    <Code>AccessDenied</Code>
    <Message>Access Denied</Message>
  </Error>
</DeleteResult>
```
2016-03-06 18:31:50 -08:00
Brendan Ashworth adf74ffdb0 api: DRY code and add new test
This commit makes code cleaner and reduces the repetitions in the code
base. Specifically, it reduces the clutter in setObjectHeaders. It also
merges encodeSuccessResponse and encodeErrorResponse together because
they served no purpose differently. Finally, it adds a simple test for
generateRequestID.
2016-03-06 13:26:27 -08:00
Harshavardhana f111997184 multipart: remove proper MD5, rather create MD5 based on parts to be s3 compatible.
This increases the performance phenominally.
2016-03-02 14:20:49 -08:00
Harshavardhana 17d145df3a signature: Fix signature handling of parallel requests.
Signature struct should be immutable, this fixes an issue
with AWS cli not being able to do multipart put operations.
2016-03-02 11:49:50 -08:00
Harshavardhana ee1b86e517 api: Implement support for additional request headers.
Now GetObject and HeadObject both support

  - If-Modified-Since, If-Unmodified-Since
  - If-Match, If-None-Match

request headers.

These headers are used to further handle the responses for GetObject
and HeadObject API.

Fixes #1098
2016-02-28 19:34:20 -08:00
Harshavardhana 3ff8a1b719 api: Implement CopyObject s3 API, doing server side copy.
Fixes #1172
2016-02-27 19:51:59 -08:00
Harshavardhana 91a092792a presigned: Fix a bug in presigned request verification.
Additionally add Docker proxy configuration.
2016-02-18 02:23:12 -08:00
Harshavardhana dd9aaa855c web/rpc: Merge ports with API server.
Fixes #1081 and #1130
2016-02-17 20:28:15 -08:00
Harshavardhana 5a9333a67b signature: Rewrite signature handling and move it into a library. 2016-02-16 17:28:16 -08:00
Harshavardhana 62f6ffb6db xl: Moved to minio/minio - fixes #1112 2016-02-11 15:43:36 -08:00
Harshavardhana 99fbc0fcb3 getObject: Add support for special response headers.
Supports now response-content-type, response-content-disposition,
response-cache-control, response-expires.
2016-02-07 03:55:16 -08:00
Harshavardhana a4c005ce30 multipart: Code cleanup
- More locking cleanup. Fix naming convention.
- Simplify concatenation and blocking calls.
2016-02-05 14:42:09 -08:00
Harshavardhana 7a3409c309 fs: Cleanup Golang errors to be called 'e' and probe to be called as 'err'
- Replace the ACL checks back, remove them when bucket
  policy is implemented.
- Move FTW (File Tree Walk) into ioutils package.
2016-02-04 13:43:52 -08:00
Harshavardhana 835b297ba7 fs: Fail createObject with appropriate message.
Fail createObject() if a file already exists and one attempts
to create a prefix/directory by same name.

Send an approriate error back to the client as 409 Conflict.
2016-02-03 21:49:36 -08:00
Harshavardhana df91661ec6 flags: Remove anonymous, ratelimit, json and web-address flags.
- Web address now uses the port + 1 from the API address port directly.
- Remove ratelimiting, ratelimiting will be achieved if necessary through
  iptables.
- Remove json flag, not needed anymore.
- Remove anonymous flag, server will be no more anonymous for play.minio.io
  we will use demo credentials.
2016-02-02 18:37:09 -08:00
Harshavardhana 0aedb67de0 contentType: Reply back proper contentTypes based on the file extension.
Currently the server would set 'application/octet-stream' for all
objects, set this value based on the file extension transparently.

This is useful in case of minio browser to facilitate displaying
proper icons for the different mime data types.
2016-02-01 12:19:58 -08:00
Harshavardhana cb7b2762f9 serverConfig: Add a new region config entry.
To change default region from 'us-east-1' to 'custom'.
Add a region value in your 'config.json'.

	"version": "2",
	"credentials": {
		"accessKeyId": "****************",
		"secretAccessKey": "***************",
	        "region": "my-region"
	},
2016-01-17 01:39:11 -08:00
Harshavardhana 0c6a6dc380 http: Enable Transfer-Encoding chunked transfer
Fixes #1020
2016-01-08 00:47:20 -08:00
Harshavardhana e7474bed43 location: Return a set location properly on complete multipart upload request. 2015-12-28 15:20:06 -08:00
Harshavardhana 2f67559684 handlers: read ContentLength value directly from http.Request.
Do not look for Content-Length in headers and try to convert them into
integer representations use ContentLength field from *http.Request*.

If Content-Length is understood to be as '-1' then treat it as an error
condition, since it could be a malformed body to crash the server.

Fixes #1011
2015-12-27 23:03:32 -08:00
Harshavardhana 4fc161ddb1 bucketName: relax bucket names, now allow numbers as starting characters. 2015-12-07 13:58:55 -08:00
Harshavardhana e1a33deabf acl: Handle readonly buckets properly 2015-11-13 20:07:39 -08:00
Harshavardhana 5b2fa33bdb Implementing min-free-disk 2015-10-18 00:23:14 -07:00
Harshavardhana 704fa420a3 Reply back proper statuses for DeleteBucket/DeleteObject 2015-10-16 20:03:44 -07:00
Harshavardhana 0eb7f078f9 Implement Bucket ACL support 2015-10-16 19:47:30 -07:00
Harshavardhana 762b798767 Migrate this project to minio micro services code 2015-10-16 11:26:08 -07:00