MyFavMirrors
/
minio
mirror of https://github.com/minio/minio.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
2,733 Commits 5 Branches 503 Tags 186 MiB
Commit Graph

925 Commits

Author SHA1 Message Date
Bala.FA 5e4b13f4bd remove unused functions 2016-02-10 13:32:53 -08:00
Bala.FA 255505a83b pkg/ioutils: remove usage of os.Lstat() in FTW() 
As os.Readdir() is used get file entries where statinfo is already
present.  This patch fixes to use statinfo provided by os.Readdir().
 2016-02-10 13:32:53 -08:00
Harshavardhana 9b29af8bbe listObjects: list objects minor optimization. 
Minor optimization.

- Add 1000 entries buffered channel for walkerCh.
- Reset marker after the lexical order has reached and
  compare only if the marker is set.
 2016-02-09 21:45:19 -08:00
Harshavardhana 42fcb27308 pkg/user: Support 32bit darwin in user package. 2016-02-08 01:34:25 -08:00
Harshavardhana 99fbc0fcb3 getObject: Add support for special response headers. 
Supports now response-content-type, response-content-disposition,
response-cache-control, response-expires.
 2016-02-07 03:55:16 -08:00
Harshavardhana f4c8120cf9 server: Remove max-buckets option and now max buckets is unlimited. 
minio server max-buckets option removed. min-free-disk option is
now a flag.
 2016-02-06 18:25:47 -08:00
Harshavardhana 4e6e78598f multipart: Increase locked critical for CompleteMultipart. 2016-02-06 01:46:05 -08:00
Harshavardhana 8df201ef30 multipart: Multipart session map now is based on uploadID. 
- Fixes initiating parallel uploads, and configs being quickly
  re-written by another incoming request.
- Parallel uploads work smoothly now and return expected behavior.
 2016-02-05 23:32:30 -08:00
Harshavardhana 69bd001c8b multipart: Multipart resume simplify further. 2016-02-05 17:40:08 -08:00
Harshavardhana 8bf1045645 setBucketMetadata: Fix a deadlock. 2016-02-05 15:48:08 -08:00
Harshavardhana 6f80380497 fs: Use mimedb now. 2016-02-05 15:09:23 -08:00
Harshavardhana 35dcccb4cd Merge remote-tracking branch 'abperiasamy/mimedb' into mimedb 2016-02-05 15:03:47 -08:00
Harshavardhana a4c005ce30 multipart: Code cleanup 
- More locking cleanup. Fix naming convention.
- Simplify concatenation and blocking calls.
 2016-02-05 14:42:09 -08:00
Anand Babu (AB) Periasamy d8abb36653 contentdb replaced by new mimedb 2016-02-05 03:49:24 -08:00
Harshavardhana 8557cbc9b7 fs: Add granular locking. 2016-02-04 20:40:58 -08:00
Harshavardhana 7a3409c309 fs: Cleanup Golang errors to be called 'e' and probe to be called as 'err' 
- Replace the ACL checks back, remove them when bucket
  policy is implemented.
- Move FTW (File Tree Walk) into ioutils package.
 2016-02-04 13:43:52 -08:00
Harshavardhana 835b297ba7 fs: Fail createObject with appropriate message. 
Fail createObject() if a file already exists and one attempts
to create a prefix/directory by same name.

Send an approriate error back to the client as 409 Conflict.
 2016-02-03 21:49:36 -08:00
Harshavardhana 454d71cafa expiry: Remove auto-expiry. 
Move the logic outside and use scripting, cronjob to delete files.

Fixes #1019
 2016-02-02 19:35:51 -08:00
Harshavardhana 81fcbd2a54 statObject: Make sure to lowercase file extensions. 2016-02-02 17:54:59 -08:00
Harshavardhana 0aedb67de0 contentType: Reply back proper contentTypes based on the file extension. 
Currently the server would set 'application/octet-stream' for all
objects, set this value based on the file extension transparently.

This is useful in case of minio browser to facilitate displaying
proper icons for the different mime data types.
 2016-02-01 12:19:58 -08:00
Harshavardhana 2469c9c591 presign: Verify query params for presign individually 
Incoming request params in presigned can come in different order
for different implementations. Rather than verifying a full string
we should verify individual params instead.

This patch fixes an incompatibility issue with AWS SDK Java.

Fixes #1059 - Thanks to @notnoopci for reporting this problem.
 2016-01-28 12:16:56 -08:00
Harshavardhana 5934a00058 listObjects: ListObjects should have idempotent behavior. 
listObjects was returning inconsistent results, i.e missing
entries during recursive and non-recursive listing. This led
to 'mc mirror' copying contents repeatedly consisdering
these files to be missing on the destination.

This patch addresses this problem - fixes #1056
 2016-01-28 03:17:40 -08:00
Harshavardhana db387912f2 jwt: Deprecate RSA usage, use HMAC instead. 
HMAC is a much simpler implementation, providing the same
benefits as RSA, avoids additional steps and keeps the code
simpler.

This patch also additionally

- Implements PutObjectURL API.
- GetObjectURL, PutObjectURL take TargetHost as another
  argument for generating URL's for proper target destination.
- Adds experimental TLS support for JSON RPC calls.
 2016-01-27 03:38:33 -08:00
Harshavardhana 9ca3372870 listObjects: Marker should be unescaped before being used internally. 
Without this change listObjects() goes into an infinite loop for
files which have special characters i.e "++" encoded with "%2B%2B".

We have to unescape and convert them to their native representation
before being used internally.

Fixes #1052
 2016-01-26 23:32:59 -08:00
Harshavardhana 5d87fdb35c Merge pull request #1051 from harshavardhana/fix-multipart 
multipart: NewMultipartUpload shouldn't return empty UploadID
 2016-01-26 15:15:44 -08:00
Harshavardhana 2e311168ee multipart: NewMultipartUpload shouldn't return empty UploadID 
Existing code
```
{
  if os.IsNotExist(e) {
       e = os.MkdirAll(objectDir, 0700)
       if e != nil {
            return "", probe.NewError(e)
       }
  }
  return "", probe.NewError(e)  ---> Error was here.
}
```
For a successful 'MkdirAll' it would still return an empty uploadID,
but the 'error' would be nil. This would succeed the request but
client would fail.

Fix is to re-arrange the logic. Thanks to Alexander Neumann @fd0, for
reporting this problem.
 2016-01-26 15:00:34 -08:00
Harshavardhana 18375b7794 ioutils: Add tests 2016-01-26 12:34:04 -08:00
Harshavardhana 1341fb79c3 listBuckets: Bump up the limit of max buckets to 1000. 2016-01-26 11:49:17 -08:00
Harshavardhana f5d6be158e listObjects: Simplify channel based changes. 2016-01-26 02:19:55 -08:00
Harshavardhana 682020ef2f listObjects: Channel based changes. 
Supports:
 - prefixes
 - marker
 2016-01-25 20:39:38 -08:00
Krishna Srinivas 9e18bfa60e listObjects: Channel based ftw - initial implementation. 2016-01-25 18:58:07 -08:00
Harshavardhana 2ec9b16667 deleteBucket: Directory not empty error on windows is "directory is not empty" 2016-01-25 17:58:43 -08:00
Harshavardhana ae2f15c6d0 api: More cleanups at WebAPI. 
- Fixes a bug where bucketName was not denormalized.
- Remove unneeded functions from jwt.go
 2016-01-25 17:30:08 -08:00
Harshavardhana 497f13d733 api: Various fixes. 
- limit list buckets to limit only 100 buckets, all uppercase buckets
  are now lowercase and work transparently with all calls.
- Change disk.Stat to disk.GetInfo and return back disk.Info{} struct.
- Introduce new ioutils package which implements ReadDirN(path, n),
  ReadDirNamesN(path, n)
 2016-01-25 16:08:27 -08:00
Harshavardhana 8cdaf87c8f signature: Add aws-cli work-around for now. 
Golang http server strips off 'Expect' header, if the
client sent this as part of signed headers we need to
handle otherwise we would see a signature mismatch.
`aws-cli` sets this as part of signed headers which is
a bad idea since servers trying to implement AWS

Signature version '4' will all encounter this issue.
According to
 http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.20
Expect header is always of form:

   Expect       =  "Expect" ":" 1#expectation
   expectation  =  "100-continue" | expectation-extension

So it safe to assume that '100-continue' is what would
be sent, for the time being keep this work around.
 2016-01-09 10:47:59 -08:00
Harshavardhana d955ce4123 s3cmd: Fix signature issues related to s3cmd. 
Support regions both 'us-east-1' and 'US' (short hand for US Standard)
honored by S3.
 2015-12-28 18:05:28 -08:00
Harshavardhana 1bfb490f90 pkg/user: Add pending tests. 2015-12-07 14:13:54 -08:00
Harshavardhana 4fc161ddb1 bucketName: relax bucket names, now allow numbers as starting characters. 2015-12-07 13:58:55 -08:00
Harshavardhana 836f5204af minio: Add config-folder option. 
Fixes #997
 2015-12-07 12:34:09 -08:00
Anand Babu (AB) Periasamy 8e8538175b fixes race in Init 2015-12-03 01:08:05 -08:00
Anand Babu (AB) Periasamy 25df427383 contentdb file.ext to content-type lookups 2015-12-02 16:55:26 -08:00
Harshavardhana 7c91a8495f signature/region: Remove 'milkyway' and use 'us-east-1' as default. 
Fixes #980
 2015-11-23 13:40:23 -08:00
Harshavardhana a328120e4d fs: Filter out $multiparts properly. 
Relax md5 requirement during complete multipart upload - ref #977
 2015-11-22 01:49:57 -08:00
Harshavardhana 35b9f965f1 fs: use new atomic package - use FileCreateWithPrefix() API 2015-11-17 16:32:20 -08:00
Harshavardhana f8e59e8399 386: Support minio server on 32bit linux. 2015-11-14 00:29:18 -08:00
Harshavardhana d668117a99 fs: Improve upon proper lexical ordering for ListObjects() 
Handle sorting properly making sure that we treat fs like a
flat namespace.
 2015-11-11 14:52:03 -08:00
Krishna Srinivas 440bec28d9 docker: the docker image will now contain just the static binary 2015-11-06 20:44:58 -08:00
Harshavardhana ab15f56a61 fs/bucket: Move bucket metadata into buckets.json 2015-11-01 21:25:01 -08:00
Harshavardhana baf66988e9 Simplify bucket delete - remove only bucket directory, no need to recursively traverse 2015-10-30 16:03:18 -07:00
Krishna Srinivas 0010f0ee10 Remove empty directories while removing an oobject 2015-10-29 23:30:16 -07:00
Harshavardhana 53adfb38f4 Leverage sort Interface to provide sortUnique function 2015-10-23 15:55:41 -07:00
Harshavardhana dbaa4d8643 Move ListObjects into its own file 2015-10-22 15:39:04 -07:00
Harshavardhana 1f66f4869b Fix all remaining windows path issues. 2015-10-22 00:05:10 -07:00
Harshavardhana afa27b9847 On windows translate Prefix, Marker and Delimiter for paths 2015-10-21 22:00:03 -07:00
Harshavardhana b74852116a Fix portability issues for arm on raspberry pi 2015-10-20 11:22:00 -07:00
Harshavardhana dddb1650de Add 5% cumulative reduction in total size of the disk 
This is done due to filesystem holding additional metadata and inode space
which is unaccounted for during min-free-disk calculation.
 2015-10-19 10:33:34 -07:00
Harshavardhana 179d2d7dac Add initial cut of auto expiry of objects 2015-10-19 01:34:31 -07:00
Harshavardhana c065be656c Implement min-free-disk as a subcommand, deprecate flag 2015-10-19 00:59:20 -07:00
Harshavardhana 5b2fa33bdb Implementing min-free-disk 2015-10-18 00:23:14 -07:00
Harshavardhana a8a935f5fd Improve disk code to return back disk StatFS{} structure 
```
StatFS {
Total int64
Free int64
FSType string
}
```

Provides more information in a cross platform way.
 2015-10-17 20:19:26 -07:00
Harshavardhana aee0845b2e Add disk package 
Implements

   - Stat returns total and free disk space supported across platforms
   - Type returns type of the filesystem underneath
 2015-10-17 16:48:24 -07:00
Harshavardhana 1256ca86d0 Add fs separator 2015-10-17 12:05:12 -07:00
Harshavardhana 2ec679a089 If directory already removed, return nil and move on 2015-10-17 00:13:46 -07:00
Harshavardhana d534fc5a4f Implement delete bucket properly with proper error handlings 2015-10-17 00:01:12 -07:00
Harshavardhana c9af01d807 Enhance listing further, this time handle cases related to common prefixes 2015-10-16 23:11:41 -07:00
Harshavardhana 0eb7f078f9 Implement Bucket ACL support 2015-10-16 19:47:30 -07:00
Harshavardhana 762b798767 Migrate this project to minio micro services code 2015-10-16 11:26:08 -07:00
Harshavardhana 3318cba132 Add windows support for minhttp library 2015-10-11 01:08:16 -07:00
Harshavardhana 7fea9cb550 Add rpc signature handler 2015-10-08 22:28:11 -07:00
Harshavardhana 11048708bb Implement GetBucketACL - fixes #893 2015-10-08 11:12:44 -07:00
Anand Babu (AB) Periasamy bf901d3b9a Merge pull request #895 from abperiasamy/tasker 
new task model minio server
 2015-10-08 02:23:29 -07:00
Anand Babu (AB) Periasamy b52697e6ad new task model minio server 2015-10-08 02:20:24 -07:00
Harshavardhana a060b158c8 Add quick.CheckVersion() to verify config version quickly before unmarshalling the full struct 
This is needed during migration where we would need to verify the underlying version number
in a quick way.
 2015-10-07 17:44:33 -07:00
Harshavardhana ab5ea997ab Make erasure Encode and Decode atomic to avoid races 2015-10-06 23:05:01 -07:00
Harshavardhana 1b42398e8b Canonicalize all the incoming input values, now PresignedPostPolicy works with minio-go 2015-10-06 10:21:28 -07:00
Harshavardhana d5ce2f6944 Make erasure matrix type not optional choose automatically 
Remove option of providing Technique and handling errors based on that
choose a matrix type automatically based on number of data blocks.

INTEL recommends on using cauchy for consistent invertible matrices,
while vandermonde is faster we should default to cauchy for large
data blocks.
 2015-10-05 22:38:02 -07:00
Harshavardhana 4ed50a8004 Fix encoding bug in donut during encoding phase 
Stream reading needs to check for length parameter being non zero,
after Reading() a predefined set of buffer length an EOF might be returned
with length == 0.

Erasure taking this zeroed data in might wrongly encode it as part of existing
data blocks which leads to errors while decoding even when the other contents
are intact.
 2015-10-05 22:12:53 -07:00
Harshavardhana f0a8dbecae Add --json output formatter for server 2015-10-05 00:20:49 -07:00
Harshavardhana c4faf47e64 First time mode for controller 
- Upon first time invocation ``minio controller`` would create access keys and secret id
- Upon request passing 'keys' arg ``minio controller`` would provide the keys
- Add colorized notification
 2015-10-04 16:42:16 -07:00
Harshavardhana cfdb29cac0 Simplify signature handling 
This change brings a new SignatureHandler where Presigned.
Requests without Payload are handled very early before even
going through the call.

This change simplifies Donut codebase to not have signature related
logic for all API's.

Simplification is still needed for Payload based signature eg. PUT/POST calls
, which are still part of the donut codebase, which will be done subsequently
after donut re-write.
 2015-10-04 13:15:33 -07:00
Harshavardhana c8de5bad2f Implement presigned policy 2015-10-04 00:01:34 -07:00
Anis ELLEUCH b5ea05d839 A better way to print prettified json syntax error msg 2015-10-03 12:25:44 -07:00
Harshavardhana 3b070dee16 Fix an important metadata getObject bug in donut 2015-10-01 10:18:03 -07:00
Harshavardhana 81cc017f91 Implement presigned signature v4 support 2015-10-01 10:17:47 -07:00
Harshavardhana 50750efb52 Reduce memory usage for memory multipart write by doing io.Pipe() streaming copy 2015-09-30 20:53:30 -07:00
Harshavardhana 8c7c5df770 Prober stringer should avoid frivolous newlines 2015-09-29 10:13:11 -07:00
Harshavardhana cd52d7a11c Fix Linux/Mac OS X erasure decoding failure with new Golang version 1.5.1 
Fixes #871
 2015-09-24 21:42:59 -07:00
Harshavardhana 3785489153 Disable multipart for donut backend from being used 
Will enable it later after cleanup
 2015-09-24 18:54:30 -07:00
Harshavardhana b938e40fb5 Move atomic package to the top and simplify its tests 2015-09-20 13:51:38 -07:00
Harshavardhana d54488f144 Move all server and controller packages into top-level 2015-09-19 01:07:42 -07:00
Anand Babu (AB) Periasamy 89a86948b5 new version format and some cleanup 2015-09-18 23:33:28 -07:00
Anand Babu (AB) Periasamy d1f1b7ac31 new version format and some cleanup 2015-09-18 23:27:04 -07:00
Harshavardhana 778f8cd222 Add new rpc tests for Server.Add and Server.List, improve Version.Get RPC to provide more details 2015-09-18 17:44:46 -07:00
Harshavardhana 2a15dd5eab Enhance signature handler - throw back valid error messages 2015-09-18 15:14:55 -07:00
Harshavardhana ac93bbb41d Merge pull request #846 from harshavardhana/new-changes 
With new auth config changes, restructure the API code to use the new style
 2015-09-18 03:45:03 -07:00
Harshavardhana 6a5e5c1826 With new auth config changes, restructure the API code to use the new style 2015-09-18 03:41:05 -07:00
Harshavardhana b4ce1e8c1d Add missing reply.Name and add possible failure tests 2015-09-18 03:15:19 -07:00
Harshavardhana f8bb85aeb7 Enhance auth JSONRPC, now provides persistent output 
Implements

   - Auth.Generate("user")
   - Auth.Fetch("user")
   - Auth.Reset("user")

This patch also adds testing for each of these cases
 2015-09-18 03:02:39 -07:00
Harshavardhana 4bcd86408b Fix minio header in accordance with rfc2616.txt 2015-09-17 23:46:10 -07:00
Harshavardhana 7093a05ab1 Version is a package now, will be re-used across codebase. 2015-09-17 20:17:33 -07:00