Commit Graph

860 Commits

Author SHA1 Message Date
Harshavardhana
7a3409c309 fs: Cleanup Golang errors to be called 'e' and probe to be called as 'err'
- Replace the ACL checks back, remove them when bucket
  policy is implemented.
- Move FTW (File Tree Walk) into ioutils package.
2016-02-04 13:43:52 -08:00
Harshavardhana
835b297ba7 fs: Fail createObject with appropriate message.
Fail createObject() if a file already exists and one attempts
to create a prefix/directory by same name.

Send an approriate error back to the client as 409 Conflict.
2016-02-03 21:49:36 -08:00
Harshavardhana
454d71cafa expiry: Remove auto-expiry.
Move the logic outside and use scripting, cronjob to delete files.

Fixes #1019
2016-02-02 19:35:51 -08:00
Harshavardhana
81fcbd2a54 statObject: Make sure to lowercase file extensions. 2016-02-02 17:54:59 -08:00
Harshavardhana
0aedb67de0 contentType: Reply back proper contentTypes based on the file extension.
Currently the server would set 'application/octet-stream' for all
objects, set this value based on the file extension transparently.

This is useful in case of minio browser to facilitate displaying
proper icons for the different mime data types.
2016-02-01 12:19:58 -08:00
Harshavardhana
2469c9c591 presign: Verify query params for presign individually
Incoming request params in presigned can come in different order
for different implementations. Rather than verifying a full string
we should verify individual params instead.

This patch fixes an incompatibility issue with AWS SDK Java.

Fixes #1059 - Thanks to @notnoopci for reporting this problem.
2016-01-28 12:16:56 -08:00
Harshavardhana
5934a00058 listObjects: ListObjects should have idempotent behavior.
listObjects was returning inconsistent results, i.e missing
entries during recursive and non-recursive listing. This led
to 'mc mirror' copying contents repeatedly consisdering
these files to be missing on the destination.

This patch addresses this problem - fixes #1056
2016-01-28 03:17:40 -08:00
Harshavardhana
db387912f2 jwt: Deprecate RSA usage, use HMAC instead.
HMAC is a much simpler implementation, providing the same
benefits as RSA, avoids additional steps and keeps the code
simpler.

This patch also additionally

- Implements PutObjectURL API.
- GetObjectURL, PutObjectURL take TargetHost as another
  argument for generating URL's for proper target destination.
- Adds experimental TLS support for JSON RPC calls.
2016-01-27 03:38:33 -08:00
Harshavardhana
9ca3372870 listObjects: Marker should be unescaped before being used internally.
Without this change listObjects() goes into an infinite loop for
files which have special characters i.e "++" encoded with "%2B%2B".

We have to unescape and convert them to their native representation
before being used internally.

Fixes #1052
2016-01-26 23:32:59 -08:00
Harshavardhana
5d87fdb35c Merge pull request #1051 from harshavardhana/fix-multipart
multipart: NewMultipartUpload shouldn't return empty UploadID
2016-01-26 15:15:44 -08:00
Harshavardhana
2e311168ee multipart: NewMultipartUpload shouldn't return empty UploadID
Existing code
```
{
  if os.IsNotExist(e) {
       e = os.MkdirAll(objectDir, 0700)
       if e != nil {
            return "", probe.NewError(e)
       }
  }
  return "", probe.NewError(e)  ---> Error was here.
}
```
For a successful 'MkdirAll' it would still return an empty uploadID,
but the 'error' would be nil. This would succeed the request but
client would fail.

Fix is to re-arrange the logic. Thanks to Alexander Neumann @fd0, for
reporting this problem.
2016-01-26 15:00:34 -08:00
Harshavardhana
18375b7794 ioutils: Add tests 2016-01-26 12:34:04 -08:00
Harshavardhana
1341fb79c3 listBuckets: Bump up the limit of max buckets to 1000. 2016-01-26 11:49:17 -08:00
Harshavardhana
f5d6be158e listObjects: Simplify channel based changes. 2016-01-26 02:19:55 -08:00
Harshavardhana
682020ef2f listObjects: Channel based changes.
Supports:
 - prefixes
 - marker
2016-01-25 20:39:38 -08:00
Krishna Srinivas
9e18bfa60e listObjects: Channel based ftw - initial implementation. 2016-01-25 18:58:07 -08:00
Harshavardhana
2ec9b16667 deleteBucket: Directory not empty error on windows is "directory is not empty" 2016-01-25 17:58:43 -08:00
Harshavardhana
ae2f15c6d0 api: More cleanups at WebAPI.
- Fixes a bug where bucketName was not denormalized.
- Remove unneeded functions from jwt.go
2016-01-25 17:30:08 -08:00
Harshavardhana
497f13d733 api: Various fixes.
- limit list buckets to limit only 100 buckets, all uppercase buckets
  are now lowercase and work transparently with all calls.
- Change disk.Stat to disk.GetInfo and return back disk.Info{} struct.
- Introduce new ioutils package which implements ReadDirN(path, n),
  ReadDirNamesN(path, n)
2016-01-25 16:08:27 -08:00
Harshavardhana
8cdaf87c8f signature: Add aws-cli work-around for now.
Golang http server strips off 'Expect' header, if the
client sent this as part of signed headers we need to
handle otherwise we would see a signature mismatch.
`aws-cli` sets this as part of signed headers which is
a bad idea since servers trying to implement AWS

Signature version '4' will all encounter this issue.
According to
 http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.20
Expect header is always of form:

   Expect       =  "Expect" ":" 1#expectation
   expectation  =  "100-continue" | expectation-extension

So it safe to assume that '100-continue' is what would
be sent, for the time being keep this work around.
2016-01-09 10:47:59 -08:00
Harshavardhana
d955ce4123 s3cmd: Fix signature issues related to s3cmd.
Support regions both 'us-east-1' and 'US' (short hand for US Standard)
honored by S3.
2015-12-28 18:05:28 -08:00
Harshavardhana
1bfb490f90 pkg/user: Add pending tests. 2015-12-07 14:13:54 -08:00
Harshavardhana
4fc161ddb1 bucketName: relax bucket names, now allow numbers as starting characters. 2015-12-07 13:58:55 -08:00
Harshavardhana
836f5204af minio: Add config-folder option.
Fixes #997
2015-12-07 12:34:09 -08:00
Anand Babu (AB) Periasamy
8e8538175b fixes race in Init 2015-12-03 01:08:05 -08:00
Anand Babu (AB) Periasamy
25df427383 contentdb file.ext to content-type lookups 2015-12-02 16:55:26 -08:00
Harshavardhana
7c91a8495f signature/region: Remove 'milkyway' and use 'us-east-1' as default.
Fixes #980
2015-11-23 13:40:23 -08:00
Harshavardhana
a328120e4d fs: Filter out $multiparts properly.
Relax md5 requirement during complete multipart upload - ref #977
2015-11-22 01:49:57 -08:00
Harshavardhana
35b9f965f1 fs: use new atomic package - use FileCreateWithPrefix() API 2015-11-17 16:32:20 -08:00
Harshavardhana
f8e59e8399 386: Support minio server on 32bit linux. 2015-11-14 00:29:18 -08:00
Harshavardhana
d668117a99 fs: Improve upon proper lexical ordering for ListObjects()
Handle sorting properly making sure that we treat fs like a
flat namespace.
2015-11-11 14:52:03 -08:00
Krishna Srinivas
440bec28d9 docker: the docker image will now contain just the static binary 2015-11-06 20:44:58 -08:00
Harshavardhana
ab15f56a61 fs/bucket: Move bucket metadata into buckets.json 2015-11-01 21:25:01 -08:00
Harshavardhana
baf66988e9 Simplify bucket delete - remove only bucket directory, no need to recursively traverse 2015-10-30 16:03:18 -07:00
Krishna Srinivas
0010f0ee10 Remove empty directories while removing an oobject 2015-10-29 23:30:16 -07:00
Harshavardhana
53adfb38f4 Leverage sort Interface to provide sortUnique function 2015-10-23 15:55:41 -07:00
Harshavardhana
dbaa4d8643 Move ListObjects into its own file 2015-10-22 15:39:04 -07:00
Harshavardhana
1f66f4869b Fix all remaining windows path issues. 2015-10-22 00:05:10 -07:00
Harshavardhana
afa27b9847 On windows translate Prefix, Marker and Delimiter for paths 2015-10-21 22:00:03 -07:00
Harshavardhana
b74852116a Fix portability issues for arm on raspberry pi 2015-10-20 11:22:00 -07:00
Harshavardhana
dddb1650de Add 5% cumulative reduction in total size of the disk
This is done due to filesystem holding additional metadata and inode space
which is unaccounted for during min-free-disk calculation.
2015-10-19 10:33:34 -07:00
Harshavardhana
179d2d7dac Add initial cut of auto expiry of objects 2015-10-19 01:34:31 -07:00
Harshavardhana
c065be656c Implement min-free-disk as a subcommand, deprecate flag 2015-10-19 00:59:20 -07:00
Harshavardhana
5b2fa33bdb Implementing min-free-disk 2015-10-18 00:23:14 -07:00
Harshavardhana
a8a935f5fd Improve disk code to return back disk StatFS{} structure
```
StatFS {
Total int64
Free int64
FSType string
}
```

Provides more information in a cross platform way.
2015-10-17 20:19:26 -07:00
Harshavardhana
aee0845b2e Add disk package
Implements

   - Stat returns total and free disk space supported across platforms
   - Type returns type of the filesystem underneath
2015-10-17 16:48:24 -07:00
Harshavardhana
1256ca86d0 Add fs separator 2015-10-17 12:05:12 -07:00
Harshavardhana
2ec679a089 If directory already removed, return nil and move on 2015-10-17 00:13:46 -07:00
Harshavardhana
d534fc5a4f Implement delete bucket properly with proper error handlings 2015-10-17 00:01:12 -07:00
Harshavardhana
c9af01d807 Enhance listing further, this time handle cases related to common prefixes 2015-10-16 23:11:41 -07:00