Commit Graph

935 Commits

Author SHA1 Message Date
Harshavardhana
653ceee9ee signV4: Move pkg/signature to pkg/s3/signature4
Cleanup and move this to relevant path.
2016-02-22 22:47:09 -08:00
Harshavardhana
800b19d8e5 cleanup: Remove definitions and move them to its relative places accordingly
- Move fs-definitions.go and break them into fs-datatypes.go, fs-bucket-acl.go
  and fs-utils.go
- Move api-definitions.go to api-response.go, where they should be.
- Move web-definitions to its related handlers.
2016-02-22 10:41:27 -08:00
Harshavardhana
91a092792a presigned: Fix a bug in presigned request verification.
Additionally add Docker proxy configuration.
2016-02-18 02:23:12 -08:00
Harshavardhana
dd9aaa855c web/rpc: Merge ports with API server.
Fixes #1081 and #1130
2016-02-17 20:28:15 -08:00
Harshavardhana
5a9333a67b signature: Rewrite signature handling and move it into a library. 2016-02-16 17:28:16 -08:00
Harshavardhana
9e10ee7e47 cpu: Remove pkg/cpu in favor of better klauspost/cpuid.
Fixes #1128
2016-02-15 13:50:33 -08:00
Anand Babu (AB) Periasamy
bbca70e13b rewrite minio runtime checks 2016-02-15 17:56:56 +05:30
Harshavardhana
fbab7128d5 fs/multipart: Handle un-ordered creation of multiparts.
Fixes #1125
2016-02-14 00:39:15 -08:00
Harshavardhana
62f6ffb6db xl: Moved to minio/minio - fixes #1112 2016-02-11 15:43:36 -08:00
Harshavardhana
6e9d73426b pkg/ioutils: True should be true 2016-02-10 13:33:36 -08:00
Bala.FA
5e4b13f4bd remove unused functions 2016-02-10 13:32:53 -08:00
Bala.FA
255505a83b pkg/ioutils: remove usage of os.Lstat() in FTW()
As os.Readdir() is used get file entries where statinfo is already
present.  This patch fixes to use statinfo provided by os.Readdir().
2016-02-10 13:32:53 -08:00
Harshavardhana
9b29af8bbe listObjects: list objects minor optimization.
Minor optimization.

- Add 1000 entries buffered channel for walkerCh.
- Reset marker after the lexical order has reached and
  compare only if the marker is set.
2016-02-09 21:45:19 -08:00
Harshavardhana
42fcb27308 pkg/user: Support 32bit darwin in user package. 2016-02-08 01:34:25 -08:00
Harshavardhana
99fbc0fcb3 getObject: Add support for special response headers.
Supports now response-content-type, response-content-disposition,
response-cache-control, response-expires.
2016-02-07 03:55:16 -08:00
Harshavardhana
f4c8120cf9 server: Remove max-buckets option and now max buckets is unlimited.
minio server max-buckets option removed. min-free-disk option is
now a flag.
2016-02-06 18:25:47 -08:00
Harshavardhana
4e6e78598f multipart: Increase locked critical for CompleteMultipart. 2016-02-06 01:46:05 -08:00
Harshavardhana
8df201ef30 multipart: Multipart session map now is based on uploadID.
- Fixes initiating parallel uploads, and configs being quickly
  re-written by another incoming request.
- Parallel uploads work smoothly now and return expected behavior.
2016-02-05 23:32:30 -08:00
Harshavardhana
69bd001c8b multipart: Multipart resume simplify further. 2016-02-05 17:40:08 -08:00
Harshavardhana
8bf1045645 setBucketMetadata: Fix a deadlock. 2016-02-05 15:48:08 -08:00
Harshavardhana
6f80380497 fs: Use mimedb now. 2016-02-05 15:09:23 -08:00
Harshavardhana
35dcccb4cd Merge remote-tracking branch 'abperiasamy/mimedb' into mimedb 2016-02-05 15:03:47 -08:00
Harshavardhana
a4c005ce30 multipart: Code cleanup
- More locking cleanup. Fix naming convention.
- Simplify concatenation and blocking calls.
2016-02-05 14:42:09 -08:00
Anand Babu (AB) Periasamy
d8abb36653 contentdb replaced by new mimedb 2016-02-05 03:49:24 -08:00
Harshavardhana
8557cbc9b7 fs: Add granular locking. 2016-02-04 20:40:58 -08:00
Harshavardhana
7a3409c309 fs: Cleanup Golang errors to be called 'e' and probe to be called as 'err'
- Replace the ACL checks back, remove them when bucket
  policy is implemented.
- Move FTW (File Tree Walk) into ioutils package.
2016-02-04 13:43:52 -08:00
Harshavardhana
835b297ba7 fs: Fail createObject with appropriate message.
Fail createObject() if a file already exists and one attempts
to create a prefix/directory by same name.

Send an approriate error back to the client as 409 Conflict.
2016-02-03 21:49:36 -08:00
Harshavardhana
454d71cafa expiry: Remove auto-expiry.
Move the logic outside and use scripting, cronjob to delete files.

Fixes #1019
2016-02-02 19:35:51 -08:00
Harshavardhana
81fcbd2a54 statObject: Make sure to lowercase file extensions. 2016-02-02 17:54:59 -08:00
Harshavardhana
0aedb67de0 contentType: Reply back proper contentTypes based on the file extension.
Currently the server would set 'application/octet-stream' for all
objects, set this value based on the file extension transparently.

This is useful in case of minio browser to facilitate displaying
proper icons for the different mime data types.
2016-02-01 12:19:58 -08:00
Harshavardhana
2469c9c591 presign: Verify query params for presign individually
Incoming request params in presigned can come in different order
for different implementations. Rather than verifying a full string
we should verify individual params instead.

This patch fixes an incompatibility issue with AWS SDK Java.

Fixes #1059 - Thanks to @notnoopci for reporting this problem.
2016-01-28 12:16:56 -08:00
Harshavardhana
5934a00058 listObjects: ListObjects should have idempotent behavior.
listObjects was returning inconsistent results, i.e missing
entries during recursive and non-recursive listing. This led
to 'mc mirror' copying contents repeatedly consisdering
these files to be missing on the destination.

This patch addresses this problem - fixes #1056
2016-01-28 03:17:40 -08:00
Harshavardhana
db387912f2 jwt: Deprecate RSA usage, use HMAC instead.
HMAC is a much simpler implementation, providing the same
benefits as RSA, avoids additional steps and keeps the code
simpler.

This patch also additionally

- Implements PutObjectURL API.
- GetObjectURL, PutObjectURL take TargetHost as another
  argument for generating URL's for proper target destination.
- Adds experimental TLS support for JSON RPC calls.
2016-01-27 03:38:33 -08:00
Harshavardhana
9ca3372870 listObjects: Marker should be unescaped before being used internally.
Without this change listObjects() goes into an infinite loop for
files which have special characters i.e "++" encoded with "%2B%2B".

We have to unescape and convert them to their native representation
before being used internally.

Fixes #1052
2016-01-26 23:32:59 -08:00
Harshavardhana
5d87fdb35c Merge pull request #1051 from harshavardhana/fix-multipart
multipart: NewMultipartUpload shouldn't return empty UploadID
2016-01-26 15:15:44 -08:00
Harshavardhana
2e311168ee multipart: NewMultipartUpload shouldn't return empty UploadID
Existing code
```
{
  if os.IsNotExist(e) {
       e = os.MkdirAll(objectDir, 0700)
       if e != nil {
            return "", probe.NewError(e)
       }
  }
  return "", probe.NewError(e)  ---> Error was here.
}
```
For a successful 'MkdirAll' it would still return an empty uploadID,
but the 'error' would be nil. This would succeed the request but
client would fail.

Fix is to re-arrange the logic. Thanks to Alexander Neumann @fd0, for
reporting this problem.
2016-01-26 15:00:34 -08:00
Harshavardhana
18375b7794 ioutils: Add tests 2016-01-26 12:34:04 -08:00
Harshavardhana
1341fb79c3 listBuckets: Bump up the limit of max buckets to 1000. 2016-01-26 11:49:17 -08:00
Harshavardhana
f5d6be158e listObjects: Simplify channel based changes. 2016-01-26 02:19:55 -08:00
Harshavardhana
682020ef2f listObjects: Channel based changes.
Supports:
 - prefixes
 - marker
2016-01-25 20:39:38 -08:00
Krishna Srinivas
9e18bfa60e listObjects: Channel based ftw - initial implementation. 2016-01-25 18:58:07 -08:00
Harshavardhana
2ec9b16667 deleteBucket: Directory not empty error on windows is "directory is not empty" 2016-01-25 17:58:43 -08:00
Harshavardhana
ae2f15c6d0 api: More cleanups at WebAPI.
- Fixes a bug where bucketName was not denormalized.
- Remove unneeded functions from jwt.go
2016-01-25 17:30:08 -08:00
Harshavardhana
497f13d733 api: Various fixes.
- limit list buckets to limit only 100 buckets, all uppercase buckets
  are now lowercase and work transparently with all calls.
- Change disk.Stat to disk.GetInfo and return back disk.Info{} struct.
- Introduce new ioutils package which implements ReadDirN(path, n),
  ReadDirNamesN(path, n)
2016-01-25 16:08:27 -08:00
Harshavardhana
8cdaf87c8f signature: Add aws-cli work-around for now.
Golang http server strips off 'Expect' header, if the
client sent this as part of signed headers we need to
handle otherwise we would see a signature mismatch.
`aws-cli` sets this as part of signed headers which is
a bad idea since servers trying to implement AWS

Signature version '4' will all encounter this issue.
According to
 http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.20
Expect header is always of form:

   Expect       =  "Expect" ":" 1#expectation
   expectation  =  "100-continue" | expectation-extension

So it safe to assume that '100-continue' is what would
be sent, for the time being keep this work around.
2016-01-09 10:47:59 -08:00
Harshavardhana
d955ce4123 s3cmd: Fix signature issues related to s3cmd.
Support regions both 'us-east-1' and 'US' (short hand for US Standard)
honored by S3.
2015-12-28 18:05:28 -08:00
Harshavardhana
1bfb490f90 pkg/user: Add pending tests. 2015-12-07 14:13:54 -08:00
Harshavardhana
4fc161ddb1 bucketName: relax bucket names, now allow numbers as starting characters. 2015-12-07 13:58:55 -08:00
Harshavardhana
836f5204af minio: Add config-folder option.
Fixes #997
2015-12-07 12:34:09 -08:00
Anand Babu (AB) Periasamy
8e8538175b fixes race in Init 2015-12-03 01:08:05 -08:00
Anand Babu (AB) Periasamy
25df427383 contentdb file.ext to content-type lookups 2015-12-02 16:55:26 -08:00
Harshavardhana
7c91a8495f signature/region: Remove 'milkyway' and use 'us-east-1' as default.
Fixes #980
2015-11-23 13:40:23 -08:00
Harshavardhana
a328120e4d fs: Filter out $multiparts properly.
Relax md5 requirement during complete multipart upload - ref #977
2015-11-22 01:49:57 -08:00
Harshavardhana
35b9f965f1 fs: use new atomic package - use FileCreateWithPrefix() API 2015-11-17 16:32:20 -08:00
Harshavardhana
f8e59e8399 386: Support minio server on 32bit linux. 2015-11-14 00:29:18 -08:00
Harshavardhana
d668117a99 fs: Improve upon proper lexical ordering for ListObjects()
Handle sorting properly making sure that we treat fs like a
flat namespace.
2015-11-11 14:52:03 -08:00
Krishna Srinivas
440bec28d9 docker: the docker image will now contain just the static binary 2015-11-06 20:44:58 -08:00
Harshavardhana
ab15f56a61 fs/bucket: Move bucket metadata into buckets.json 2015-11-01 21:25:01 -08:00
Harshavardhana
baf66988e9 Simplify bucket delete - remove only bucket directory, no need to recursively traverse 2015-10-30 16:03:18 -07:00
Krishna Srinivas
0010f0ee10 Remove empty directories while removing an oobject 2015-10-29 23:30:16 -07:00
Harshavardhana
53adfb38f4 Leverage sort Interface to provide sortUnique function 2015-10-23 15:55:41 -07:00
Harshavardhana
dbaa4d8643 Move ListObjects into its own file 2015-10-22 15:39:04 -07:00
Harshavardhana
1f66f4869b Fix all remaining windows path issues. 2015-10-22 00:05:10 -07:00
Harshavardhana
afa27b9847 On windows translate Prefix, Marker and Delimiter for paths 2015-10-21 22:00:03 -07:00
Harshavardhana
b74852116a Fix portability issues for arm on raspberry pi 2015-10-20 11:22:00 -07:00
Harshavardhana
dddb1650de Add 5% cumulative reduction in total size of the disk
This is done due to filesystem holding additional metadata and inode space
which is unaccounted for during min-free-disk calculation.
2015-10-19 10:33:34 -07:00
Harshavardhana
179d2d7dac Add initial cut of auto expiry of objects 2015-10-19 01:34:31 -07:00
Harshavardhana
c065be656c Implement min-free-disk as a subcommand, deprecate flag 2015-10-19 00:59:20 -07:00
Harshavardhana
5b2fa33bdb Implementing min-free-disk 2015-10-18 00:23:14 -07:00
Harshavardhana
a8a935f5fd Improve disk code to return back disk StatFS{} structure
```
StatFS {
Total int64
Free int64
FSType string
}
```

Provides more information in a cross platform way.
2015-10-17 20:19:26 -07:00
Harshavardhana
aee0845b2e Add disk package
Implements

   - Stat returns total and free disk space supported across platforms
   - Type returns type of the filesystem underneath
2015-10-17 16:48:24 -07:00
Harshavardhana
1256ca86d0 Add fs separator 2015-10-17 12:05:12 -07:00
Harshavardhana
2ec679a089 If directory already removed, return nil and move on 2015-10-17 00:13:46 -07:00
Harshavardhana
d534fc5a4f Implement delete bucket properly with proper error handlings 2015-10-17 00:01:12 -07:00
Harshavardhana
c9af01d807 Enhance listing further, this time handle cases related to common prefixes 2015-10-16 23:11:41 -07:00
Harshavardhana
0eb7f078f9 Implement Bucket ACL support 2015-10-16 19:47:30 -07:00
Harshavardhana
762b798767 Migrate this project to minio micro services code 2015-10-16 11:26:08 -07:00
Harshavardhana
3318cba132 Add windows support for minhttp library 2015-10-11 01:08:16 -07:00
Harshavardhana
7fea9cb550 Add rpc signature handler 2015-10-08 22:28:11 -07:00
Harshavardhana
11048708bb Implement GetBucketACL - fixes #893 2015-10-08 11:12:44 -07:00
Anand Babu (AB) Periasamy
bf901d3b9a Merge pull request #895 from abperiasamy/tasker
new task model minio server
2015-10-08 02:23:29 -07:00
Anand Babu (AB) Periasamy
b52697e6ad new task model minio server 2015-10-08 02:20:24 -07:00
Harshavardhana
a060b158c8 Add quick.CheckVersion() to verify config version quickly before unmarshalling the full struct
This is needed during migration where we would need to verify the underlying version number
in a quick way.
2015-10-07 17:44:33 -07:00
Harshavardhana
ab5ea997ab Make erasure Encode and Decode atomic to avoid races 2015-10-06 23:05:01 -07:00
Harshavardhana
1b42398e8b Canonicalize all the incoming input values, now PresignedPostPolicy works with minio-go 2015-10-06 10:21:28 -07:00
Harshavardhana
d5ce2f6944 Make erasure matrix type not optional choose automatically
Remove option of providing Technique and handling errors based on that
choose a matrix type automatically based on number of data blocks.

INTEL recommends on using cauchy for consistent invertible matrices,
while vandermonde is faster we should default to cauchy for large
data blocks.
2015-10-05 22:38:02 -07:00
Harshavardhana
4ed50a8004 Fix encoding bug in donut during encoding phase
Stream reading needs to check for length parameter being non zero,
after Reading() a predefined set of buffer length an EOF might be returned
with length == 0.

Erasure taking this zeroed data in might wrongly encode it as part of existing
data blocks which leads to errors while decoding even when the other contents
are intact.
2015-10-05 22:12:53 -07:00
Harshavardhana
f0a8dbecae Add --json output formatter for server 2015-10-05 00:20:49 -07:00
Harshavardhana
c4faf47e64 First time mode for controller
- Upon first time invocation ``minio controller`` would create access keys and secret id
- Upon request passing 'keys' arg ``minio controller`` would provide the keys
- Add colorized notification
2015-10-04 16:42:16 -07:00
Harshavardhana
cfdb29cac0 Simplify signature handling
This change brings a new SignatureHandler where Presigned.
Requests without Payload are handled very early before even
going through the call.

This change simplifies Donut codebase to not have signature related
logic for all API's.

Simplification is still needed for Payload based signature eg. PUT/POST calls
, which are still part of the donut codebase, which will be done subsequently
after donut re-write.
2015-10-04 13:15:33 -07:00
Harshavardhana
c8de5bad2f Implement presigned policy 2015-10-04 00:01:34 -07:00
Anis ELLEUCH
b5ea05d839 A better way to print prettified json syntax error msg 2015-10-03 12:25:44 -07:00
Harshavardhana
3b070dee16 Fix an important metadata getObject bug in donut 2015-10-01 10:18:03 -07:00
Harshavardhana
81cc017f91 Implement presigned signature v4 support 2015-10-01 10:17:47 -07:00
Harshavardhana
50750efb52 Reduce memory usage for memory multipart write by doing io.Pipe() streaming copy 2015-09-30 20:53:30 -07:00
Harshavardhana
8c7c5df770 Prober stringer should avoid frivolous newlines 2015-09-29 10:13:11 -07:00
Harshavardhana
cd52d7a11c Fix Linux/Mac OS X erasure decoding failure with new Golang version 1.5.1
Fixes #871
2015-09-24 21:42:59 -07:00
Harshavardhana
3785489153 Disable multipart for donut backend from being used
Will enable it later after cleanup
2015-09-24 18:54:30 -07:00
Harshavardhana
b938e40fb5 Move atomic package to the top and simplify its tests 2015-09-20 13:51:38 -07:00
Harshavardhana
d54488f144 Move all server and controller packages into top-level 2015-09-19 01:07:42 -07:00