Commit Graph

899 Commits

Author SHA1 Message Date
Bala.FA
c70bc2209e api: refactor list object handling in fs backend
When list object is invoked, it creates a goroutine if not available
for given parameters else uses existing goroutine.  These goroutines
are alive for 15 seconds for further continuation list object request
else they exit.

Fixes #1076
2016-03-11 02:20:51 +05:30
Harshavardhana
d5057b3c51 accessPolicy: Implement Put, Get, Delete access policy.
This patch implements Get,Put,Delete bucket policies

Supporting - http://docs.aws.amazon.com/AmazonS3/latest/dev/access-policy-language-overview.html

Currently supports following actions.

   "*":                             true,
   "s3:*":                          true,
   "s3:GetObject":                  true,
   "s3:ListBucket":                 true,
   "s3:PutObject":                  true,
   "s3:CreateBucket":               true,
   "s3:GetBucketLocation":          true,
   "s3:DeleteBucket":               true,
   "s3:DeleteObject":               true,
   "s3:AbortMultipartUpload":       true,
   "s3:ListBucketMultipartUploads": true,
   "s3:ListMultipartUploadParts":   true,

following conditions for "StringEquals" and "StringNotEquals"

   "s3:prefix", "s3:max-keys"
2016-03-08 17:44:50 -08:00
Brendan Ashworth
fab45aae40 pkg/fs: add bucket test and benchmarks
Lots of useful benchmarks and a simple test addition!
2016-03-07 00:07:11 -08:00
Brendan Ashworth
7399d8ceaa pkg/fs: skip unnecessary os.Stat system call 2016-03-07 00:07:11 -08:00
Brendan Ashworth
0a0451a0fb pkg/fs: DRY SetBucketMetadata
It had a lot of code that was the same as GetBucketMetadata, so instead
call GBM from SBM so as to reduce doing the same thing in two different
spots. Theoretically this will induce a small overhead as now at least
two calls of denormalizeBucket are made, although this shouldn't be
noticeable.
2016-03-07 00:07:11 -08:00
Brendan Ashworth
294ea814bf pkg/fs: for locks, prefer defer and read-only ops
This commit prefers the use of 'defer' for fs.Unlock (and fs.RUnlock)
because it is more idiomatic Go and reduces repetition in the code,
lending to a cleaner code base.

It also switches a few uses of the lock to read-only locks, which should
improve performance of those functions dramatically in certain contexts.
2016-03-07 00:07:11 -08:00
Harshavardhana
aed62788d9 api: Implement multiple objects Delete api - fixes #956
This API takes input XML input in following form.

```
<?xml version="1.0" encoding="UTF-8"?>
<Delete>
    <Quiet>true</Quiet>
    <Object>
         <Key>Key</Key>
    </Object>
    <Object>
         <Key>Key</Key>
    </Object>
    ...
</Delete>
```

and responds the list of successful deletes, list of errors
for all the deleted objects.

```
<?xml version="1.0" encoding="UTF-8"?>
<DeleteResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  <Deleted>
    <Key>sample1.txt</Key>
  </Deleted>
  <Error>
    <Key>sample2.txt</Key>
    <Code>AccessDenied</Code>
    <Message>Access Denied</Message>
  </Error>
</DeleteResult>
```
2016-03-06 18:31:50 -08:00
Harshavardhana
f111997184 multipart: remove proper MD5, rather create MD5 based on parts to be s3 compatible.
This increases the performance phenominally.
2016-03-02 14:20:49 -08:00
Harshavardhana
17d145df3a signature: Fix signature handling of parallel requests.
Signature struct should be immutable, this fixes an issue
with AWS cli not being able to do multipart put operations.
2016-03-02 11:49:50 -08:00
Karthic Rao
6651f5b368 go vet shadow error patch 2016-03-02 09:55:00 +05:30
Harshavardhana
c7021f6a95 list: Fix handling of maxKeys and prefixes.
This fixes a problem of requeuing the same request
and also fixes a major problem of sending truncated
for full key prefixes.

Fixes #1177
2016-03-01 17:34:31 -08:00
Harshavardhana
3ff8a1b719 api: Implement CopyObject s3 API, doing server side copy.
Fixes #1172
2016-02-27 19:51:59 -08:00
Harshavardhana
408aa72146 build/vet: Fix all the shadowing reports with go1.6
Golang 1.6 is default version for the build now.

Additionally set 'GODEBUG=cgocheck=0' for now, until
we fix the erasure coding package.

Readmore here https://tip.golang.org/doc/go1.6#cgo
2016-02-23 14:34:39 -08:00
Anand Babu (AB) Periasamy
07da31f8b8 Merge pull request #1150 from harshavardhana/signature
signV4: Move pkg/signature to pkg/s3/signature4
2016-02-23 12:39:28 +05:30
Harshavardhana
653ceee9ee signV4: Move pkg/signature to pkg/s3/signature4
Cleanup and move this to relevant path.
2016-02-22 22:47:09 -08:00
Harshavardhana
800b19d8e5 cleanup: Remove definitions and move them to its relative places accordingly
- Move fs-definitions.go and break them into fs-datatypes.go, fs-bucket-acl.go
  and fs-utils.go
- Move api-definitions.go to api-response.go, where they should be.
- Move web-definitions to its related handlers.
2016-02-22 10:41:27 -08:00
Harshavardhana
91a092792a presigned: Fix a bug in presigned request verification.
Additionally add Docker proxy configuration.
2016-02-18 02:23:12 -08:00
Harshavardhana
dd9aaa855c web/rpc: Merge ports with API server.
Fixes #1081 and #1130
2016-02-17 20:28:15 -08:00
Harshavardhana
5a9333a67b signature: Rewrite signature handling and move it into a library. 2016-02-16 17:28:16 -08:00
Harshavardhana
9e10ee7e47 cpu: Remove pkg/cpu in favor of better klauspost/cpuid.
Fixes #1128
2016-02-15 13:50:33 -08:00
Anand Babu (AB) Periasamy
bbca70e13b rewrite minio runtime checks 2016-02-15 17:56:56 +05:30
Harshavardhana
fbab7128d5 fs/multipart: Handle un-ordered creation of multiparts.
Fixes #1125
2016-02-14 00:39:15 -08:00
Harshavardhana
62f6ffb6db xl: Moved to minio/minio - fixes #1112 2016-02-11 15:43:36 -08:00
Harshavardhana
6e9d73426b pkg/ioutils: True should be true 2016-02-10 13:33:36 -08:00
Bala.FA
5e4b13f4bd remove unused functions 2016-02-10 13:32:53 -08:00
Bala.FA
255505a83b pkg/ioutils: remove usage of os.Lstat() in FTW()
As os.Readdir() is used get file entries where statinfo is already
present.  This patch fixes to use statinfo provided by os.Readdir().
2016-02-10 13:32:53 -08:00
Harshavardhana
9b29af8bbe listObjects: list objects minor optimization.
Minor optimization.

- Add 1000 entries buffered channel for walkerCh.
- Reset marker after the lexical order has reached and
  compare only if the marker is set.
2016-02-09 21:45:19 -08:00
Harshavardhana
42fcb27308 pkg/user: Support 32bit darwin in user package. 2016-02-08 01:34:25 -08:00
Harshavardhana
99fbc0fcb3 getObject: Add support for special response headers.
Supports now response-content-type, response-content-disposition,
response-cache-control, response-expires.
2016-02-07 03:55:16 -08:00
Harshavardhana
f4c8120cf9 server: Remove max-buckets option and now max buckets is unlimited.
minio server max-buckets option removed. min-free-disk option is
now a flag.
2016-02-06 18:25:47 -08:00
Harshavardhana
4e6e78598f multipart: Increase locked critical for CompleteMultipart. 2016-02-06 01:46:05 -08:00
Harshavardhana
8df201ef30 multipart: Multipart session map now is based on uploadID.
- Fixes initiating parallel uploads, and configs being quickly
  re-written by another incoming request.
- Parallel uploads work smoothly now and return expected behavior.
2016-02-05 23:32:30 -08:00
Harshavardhana
69bd001c8b multipart: Multipart resume simplify further. 2016-02-05 17:40:08 -08:00
Harshavardhana
8bf1045645 setBucketMetadata: Fix a deadlock. 2016-02-05 15:48:08 -08:00
Harshavardhana
6f80380497 fs: Use mimedb now. 2016-02-05 15:09:23 -08:00
Harshavardhana
35dcccb4cd Merge remote-tracking branch 'abperiasamy/mimedb' into mimedb 2016-02-05 15:03:47 -08:00
Harshavardhana
a4c005ce30 multipart: Code cleanup
- More locking cleanup. Fix naming convention.
- Simplify concatenation and blocking calls.
2016-02-05 14:42:09 -08:00
Anand Babu (AB) Periasamy
d8abb36653 contentdb replaced by new mimedb 2016-02-05 03:49:24 -08:00
Harshavardhana
8557cbc9b7 fs: Add granular locking. 2016-02-04 20:40:58 -08:00
Harshavardhana
7a3409c309 fs: Cleanup Golang errors to be called 'e' and probe to be called as 'err'
- Replace the ACL checks back, remove them when bucket
  policy is implemented.
- Move FTW (File Tree Walk) into ioutils package.
2016-02-04 13:43:52 -08:00
Harshavardhana
835b297ba7 fs: Fail createObject with appropriate message.
Fail createObject() if a file already exists and one attempts
to create a prefix/directory by same name.

Send an approriate error back to the client as 409 Conflict.
2016-02-03 21:49:36 -08:00
Harshavardhana
454d71cafa expiry: Remove auto-expiry.
Move the logic outside and use scripting, cronjob to delete files.

Fixes #1019
2016-02-02 19:35:51 -08:00
Harshavardhana
81fcbd2a54 statObject: Make sure to lowercase file extensions. 2016-02-02 17:54:59 -08:00
Harshavardhana
0aedb67de0 contentType: Reply back proper contentTypes based on the file extension.
Currently the server would set 'application/octet-stream' for all
objects, set this value based on the file extension transparently.

This is useful in case of minio browser to facilitate displaying
proper icons for the different mime data types.
2016-02-01 12:19:58 -08:00
Harshavardhana
2469c9c591 presign: Verify query params for presign individually
Incoming request params in presigned can come in different order
for different implementations. Rather than verifying a full string
we should verify individual params instead.

This patch fixes an incompatibility issue with AWS SDK Java.

Fixes #1059 - Thanks to @notnoopci for reporting this problem.
2016-01-28 12:16:56 -08:00
Harshavardhana
5934a00058 listObjects: ListObjects should have idempotent behavior.
listObjects was returning inconsistent results, i.e missing
entries during recursive and non-recursive listing. This led
to 'mc mirror' copying contents repeatedly consisdering
these files to be missing on the destination.

This patch addresses this problem - fixes #1056
2016-01-28 03:17:40 -08:00
Harshavardhana
db387912f2 jwt: Deprecate RSA usage, use HMAC instead.
HMAC is a much simpler implementation, providing the same
benefits as RSA, avoids additional steps and keeps the code
simpler.

This patch also additionally

- Implements PutObjectURL API.
- GetObjectURL, PutObjectURL take TargetHost as another
  argument for generating URL's for proper target destination.
- Adds experimental TLS support for JSON RPC calls.
2016-01-27 03:38:33 -08:00
Harshavardhana
9ca3372870 listObjects: Marker should be unescaped before being used internally.
Without this change listObjects() goes into an infinite loop for
files which have special characters i.e "++" encoded with "%2B%2B".

We have to unescape and convert them to their native representation
before being used internally.

Fixes #1052
2016-01-26 23:32:59 -08:00
Harshavardhana
5d87fdb35c Merge pull request #1051 from harshavardhana/fix-multipart
multipart: NewMultipartUpload shouldn't return empty UploadID
2016-01-26 15:15:44 -08:00
Harshavardhana
2e311168ee multipart: NewMultipartUpload shouldn't return empty UploadID
Existing code
```
{
  if os.IsNotExist(e) {
       e = os.MkdirAll(objectDir, 0700)
       if e != nil {
            return "", probe.NewError(e)
       }
  }
  return "", probe.NewError(e)  ---> Error was here.
}
```
For a successful 'MkdirAll' it would still return an empty uploadID,
but the 'error' would be nil. This would succeed the request but
client would fail.

Fix is to re-arrange the logic. Thanks to Alexander Neumann @fd0, for
reporting this problem.
2016-01-26 15:00:34 -08:00