MyFavMirrors
/
minio
mirror of https://github.com/minio/minio.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
7,529 Commits 5 Branches 503 Tags 186 MiB
Commit Graph

1646 Commits

Author SHA1 Message Date
Brendan Ashworth 7399d8ceaa pkg/fs: skip unnecessary os.Stat system call 2016-03-07 00:07:11 -08:00
Brendan Ashworth 0a0451a0fb pkg/fs: DRY SetBucketMetadata 
It had a lot of code that was the same as GetBucketMetadata, so instead
call GBM from SBM so as to reduce doing the same thing in two different
spots. Theoretically this will induce a small overhead as now at least
two calls of denormalizeBucket are made, although this shouldn't be
noticeable.
 2016-03-07 00:07:11 -08:00
Brendan Ashworth 294ea814bf pkg/fs: for locks, prefer defer and read-only ops 
This commit prefers the use of 'defer' for fs.Unlock (and fs.RUnlock)
because it is more idiomatic Go and reduces repetition in the code,
lending to a cleaner code base.

It also switches a few uses of the lock to read-only locks, which should
improve performance of those functions dramatically in certain contexts.
 2016-03-07 00:07:11 -08:00
Harshavardhana aed62788d9 api: Implement multiple objects Delete api - fixes #956 
This API takes input XML input in following form.

```
<?xml version="1.0" encoding="UTF-8"?>
<Delete>
    <Quiet>true</Quiet>
    <Object>
         <Key>Key</Key>
    </Object>
    <Object>
         <Key>Key</Key>
    </Object>
    ...
</Delete>
```

and responds the list of successful deletes, list of errors
for all the deleted objects.

```
<?xml version="1.0" encoding="UTF-8"?>
<DeleteResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  <Deleted>
    <Key>sample1.txt</Key>
  </Deleted>
  <Error>
    <Key>sample2.txt</Key>
    <Code>AccessDenied</Code>
    <Message>Access Denied</Message>
  </Error>
</DeleteResult>
```
 2016-03-06 18:31:50 -08:00
Harshavardhana f111997184 multipart: remove proper MD5, rather create MD5 based on parts to be s3 compatible. 
This increases the performance phenominally.
 2016-03-02 14:20:49 -08:00
Harshavardhana 17d145df3a signature: Fix signature handling of parallel requests. 
Signature struct should be immutable, this fixes an issue
with AWS cli not being able to do multipart put operations.
 2016-03-02 11:49:50 -08:00
Karthic Rao 6651f5b368 go vet shadow error patch 2016-03-02 09:55:00 +05:30
Harshavardhana c7021f6a95 list: Fix handling of maxKeys and prefixes. 
This fixes a problem of requeuing the same request
and also fixes a major problem of sending truncated
for full key prefixes.

Fixes #1177
 2016-03-01 17:34:31 -08:00
Harshavardhana 3ff8a1b719 api: Implement CopyObject s3 API, doing server side copy. 
Fixes #1172
 2016-02-27 19:51:59 -08:00
Harshavardhana 408aa72146 build/vet: Fix all the shadowing reports with go1.6 
Golang 1.6 is default version for the build now.

Additionally set 'GODEBUG=cgocheck=0' for now, until
we fix the erasure coding package.

Readmore here https://tip.golang.org/doc/go1.6#cgo
 2016-02-23 14:34:39 -08:00
Anand Babu (AB) Periasamy 07da31f8b8 Merge pull request #1150 from harshavardhana/signature 
signV4: Move pkg/signature to pkg/s3/signature4
 2016-02-23 12:39:28 +05:30
Harshavardhana 653ceee9ee signV4: Move pkg/signature to pkg/s3/signature4 
Cleanup and move this to relevant path.
 2016-02-22 22:47:09 -08:00
Harshavardhana 800b19d8e5 cleanup: Remove definitions and move them to its relative places accordingly 
- Move fs-definitions.go and break them into fs-datatypes.go, fs-bucket-acl.go
  and fs-utils.go
- Move api-definitions.go to api-response.go, where they should be.
- Move web-definitions to its related handlers.
 2016-02-22 10:41:27 -08:00
Harshavardhana 91a092792a presigned: Fix a bug in presigned request verification. 
Additionally add Docker proxy configuration.
 2016-02-18 02:23:12 -08:00
Harshavardhana dd9aaa855c web/rpc: Merge ports with API server. 
Fixes #1081 and #1130
 2016-02-17 20:28:15 -08:00
Harshavardhana 5a9333a67b signature: Rewrite signature handling and move it into a library. 2016-02-16 17:28:16 -08:00
Harshavardhana 9e10ee7e47 cpu: Remove pkg/cpu in favor of better klauspost/cpuid. 
Fixes #1128
 2016-02-15 13:50:33 -08:00
Anand Babu (AB) Periasamy bbca70e13b rewrite minio runtime checks 2016-02-15 17:56:56 +05:30
Harshavardhana fbab7128d5 fs/multipart: Handle un-ordered creation of multiparts. 
Fixes #1125
 2016-02-14 00:39:15 -08:00
Harshavardhana 62f6ffb6db xl: Moved to minio/minio - fixes #1112 2016-02-11 15:43:36 -08:00
Harshavardhana 6e9d73426b pkg/ioutils: True should be true 2016-02-10 13:33:36 -08:00
Bala.FA 5e4b13f4bd remove unused functions 2016-02-10 13:32:53 -08:00
Bala.FA 255505a83b pkg/ioutils: remove usage of os.Lstat() in FTW() 
As os.Readdir() is used get file entries where statinfo is already
present.  This patch fixes to use statinfo provided by os.Readdir().
 2016-02-10 13:32:53 -08:00
Harshavardhana 9b29af8bbe listObjects: list objects minor optimization. 
Minor optimization.

- Add 1000 entries buffered channel for walkerCh.
- Reset marker after the lexical order has reached and
  compare only if the marker is set.
 2016-02-09 21:45:19 -08:00
Harshavardhana 42fcb27308 pkg/user: Support 32bit darwin in user package. 2016-02-08 01:34:25 -08:00
Harshavardhana 99fbc0fcb3 getObject: Add support for special response headers. 
Supports now response-content-type, response-content-disposition,
response-cache-control, response-expires.
 2016-02-07 03:55:16 -08:00
Harshavardhana f4c8120cf9 server: Remove max-buckets option and now max buckets is unlimited. 
minio server max-buckets option removed. min-free-disk option is
now a flag.
 2016-02-06 18:25:47 -08:00
Harshavardhana 4e6e78598f multipart: Increase locked critical for CompleteMultipart. 2016-02-06 01:46:05 -08:00
Harshavardhana 8df201ef30 multipart: Multipart session map now is based on uploadID. 
- Fixes initiating parallel uploads, and configs being quickly
  re-written by another incoming request.
- Parallel uploads work smoothly now and return expected behavior.
 2016-02-05 23:32:30 -08:00
Harshavardhana 69bd001c8b multipart: Multipart resume simplify further. 2016-02-05 17:40:08 -08:00
Harshavardhana 8bf1045645 setBucketMetadata: Fix a deadlock. 2016-02-05 15:48:08 -08:00
Harshavardhana 6f80380497 fs: Use mimedb now. 2016-02-05 15:09:23 -08:00
Harshavardhana 35dcccb4cd Merge remote-tracking branch 'abperiasamy/mimedb' into mimedb 2016-02-05 15:03:47 -08:00
Harshavardhana a4c005ce30 multipart: Code cleanup 
- More locking cleanup. Fix naming convention.
- Simplify concatenation and blocking calls.
 2016-02-05 14:42:09 -08:00
Anand Babu (AB) Periasamy d8abb36653 contentdb replaced by new mimedb 2016-02-05 03:49:24 -08:00
Harshavardhana 8557cbc9b7 fs: Add granular locking. 2016-02-04 20:40:58 -08:00
Harshavardhana 7a3409c309 fs: Cleanup Golang errors to be called 'e' and probe to be called as 'err' 
- Replace the ACL checks back, remove them when bucket
  policy is implemented.
- Move FTW (File Tree Walk) into ioutils package.
 2016-02-04 13:43:52 -08:00
Harshavardhana 835b297ba7 fs: Fail createObject with appropriate message. 
Fail createObject() if a file already exists and one attempts
to create a prefix/directory by same name.

Send an approriate error back to the client as 409 Conflict.
 2016-02-03 21:49:36 -08:00
Harshavardhana 454d71cafa expiry: Remove auto-expiry. 
Move the logic outside and use scripting, cronjob to delete files.

Fixes #1019
 2016-02-02 19:35:51 -08:00
Harshavardhana 81fcbd2a54 statObject: Make sure to lowercase file extensions. 2016-02-02 17:54:59 -08:00
Harshavardhana 0aedb67de0 contentType: Reply back proper contentTypes based on the file extension. 
Currently the server would set 'application/octet-stream' for all
objects, set this value based on the file extension transparently.

This is useful in case of minio browser to facilitate displaying
proper icons for the different mime data types.
 2016-02-01 12:19:58 -08:00
Harshavardhana 2469c9c591 presign: Verify query params for presign individually 
Incoming request params in presigned can come in different order
for different implementations. Rather than verifying a full string
we should verify individual params instead.

This patch fixes an incompatibility issue with AWS SDK Java.

Fixes #1059 - Thanks to @notnoopci for reporting this problem.
 2016-01-28 12:16:56 -08:00
Harshavardhana 5934a00058 listObjects: ListObjects should have idempotent behavior. 
listObjects was returning inconsistent results, i.e missing
entries during recursive and non-recursive listing. This led
to 'mc mirror' copying contents repeatedly consisdering
these files to be missing on the destination.

This patch addresses this problem - fixes #1056
 2016-01-28 03:17:40 -08:00
Harshavardhana db387912f2 jwt: Deprecate RSA usage, use HMAC instead. 
HMAC is a much simpler implementation, providing the same
benefits as RSA, avoids additional steps and keeps the code
simpler.

This patch also additionally

- Implements PutObjectURL API.
- GetObjectURL, PutObjectURL take TargetHost as another
  argument for generating URL's for proper target destination.
- Adds experimental TLS support for JSON RPC calls.
 2016-01-27 03:38:33 -08:00
Harshavardhana 9ca3372870 listObjects: Marker should be unescaped before being used internally. 
Without this change listObjects() goes into an infinite loop for
files which have special characters i.e "++" encoded with "%2B%2B".

We have to unescape and convert them to their native representation
before being used internally.

Fixes #1052
 2016-01-26 23:32:59 -08:00
Harshavardhana 5d87fdb35c Merge pull request #1051 from harshavardhana/fix-multipart 
multipart: NewMultipartUpload shouldn't return empty UploadID
 2016-01-26 15:15:44 -08:00
Harshavardhana 2e311168ee multipart: NewMultipartUpload shouldn't return empty UploadID 
Existing code
```
{
  if os.IsNotExist(e) {
       e = os.MkdirAll(objectDir, 0700)
       if e != nil {
            return "", probe.NewError(e)
       }
  }
  return "", probe.NewError(e)  ---> Error was here.
}
```
For a successful 'MkdirAll' it would still return an empty uploadID,
but the 'error' would be nil. This would succeed the request but
client would fail.

Fix is to re-arrange the logic. Thanks to Alexander Neumann @fd0, for
reporting this problem.
 2016-01-26 15:00:34 -08:00
Harshavardhana 18375b7794 ioutils: Add tests 2016-01-26 12:34:04 -08:00
Harshavardhana 1341fb79c3 listBuckets: Bump up the limit of max buckets to 1000. 2016-01-26 11:49:17 -08:00
Harshavardhana f5d6be158e listObjects: Simplify channel based changes. 2016-01-26 02:19:55 -08:00
Harshavardhana 682020ef2f listObjects: Channel based changes. 
Supports:
 - prefixes
 - marker
 2016-01-25 20:39:38 -08:00
Krishna Srinivas 9e18bfa60e listObjects: Channel based ftw - initial implementation. 2016-01-25 18:58:07 -08:00
Harshavardhana 2ec9b16667 deleteBucket: Directory not empty error on windows is "directory is not empty" 2016-01-25 17:58:43 -08:00
Harshavardhana ae2f15c6d0 api: More cleanups at WebAPI. 
- Fixes a bug where bucketName was not denormalized.
- Remove unneeded functions from jwt.go
 2016-01-25 17:30:08 -08:00
Harshavardhana 497f13d733 api: Various fixes. 
- limit list buckets to limit only 100 buckets, all uppercase buckets
  are now lowercase and work transparently with all calls.
- Change disk.Stat to disk.GetInfo and return back disk.Info{} struct.
- Introduce new ioutils package which implements ReadDirN(path, n),
  ReadDirNamesN(path, n)
 2016-01-25 16:08:27 -08:00
Harshavardhana 8cdaf87c8f signature: Add aws-cli work-around for now. 
Golang http server strips off 'Expect' header, if the
client sent this as part of signed headers we need to
handle otherwise we would see a signature mismatch.
`aws-cli` sets this as part of signed headers which is
a bad idea since servers trying to implement AWS

Signature version '4' will all encounter this issue.
According to
 http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.20
Expect header is always of form:

   Expect       =  "Expect" ":" 1#expectation
   expectation  =  "100-continue" | expectation-extension

So it safe to assume that '100-continue' is what would
be sent, for the time being keep this work around.
 2016-01-09 10:47:59 -08:00
Harshavardhana d955ce4123 s3cmd: Fix signature issues related to s3cmd. 
Support regions both 'us-east-1' and 'US' (short hand for US Standard)
honored by S3.
 2015-12-28 18:05:28 -08:00
Harshavardhana 1bfb490f90 pkg/user: Add pending tests. 2015-12-07 14:13:54 -08:00
Harshavardhana 4fc161ddb1 bucketName: relax bucket names, now allow numbers as starting characters. 2015-12-07 13:58:55 -08:00
Harshavardhana 836f5204af minio: Add config-folder option. 
Fixes #997
 2015-12-07 12:34:09 -08:00
Anand Babu (AB) Periasamy 8e8538175b fixes race in Init 2015-12-03 01:08:05 -08:00
Anand Babu (AB) Periasamy 25df427383 contentdb file.ext to content-type lookups 2015-12-02 16:55:26 -08:00
Harshavardhana 7c91a8495f signature/region: Remove 'milkyway' and use 'us-east-1' as default. 
Fixes #980
 2015-11-23 13:40:23 -08:00
Harshavardhana a328120e4d fs: Filter out $multiparts properly. 
Relax md5 requirement during complete multipart upload - ref #977
 2015-11-22 01:49:57 -08:00
Harshavardhana 35b9f965f1 fs: use new atomic package - use FileCreateWithPrefix() API 2015-11-17 16:32:20 -08:00
Harshavardhana f8e59e8399 386: Support minio server on 32bit linux. 2015-11-14 00:29:18 -08:00
Harshavardhana d668117a99 fs: Improve upon proper lexical ordering for ListObjects() 
Handle sorting properly making sure that we treat fs like a
flat namespace.
 2015-11-11 14:52:03 -08:00
Krishna Srinivas 440bec28d9 docker: the docker image will now contain just the static binary 2015-11-06 20:44:58 -08:00
Harshavardhana ab15f56a61 fs/bucket: Move bucket metadata into buckets.json 2015-11-01 21:25:01 -08:00
Harshavardhana baf66988e9 Simplify bucket delete - remove only bucket directory, no need to recursively traverse 2015-10-30 16:03:18 -07:00
Krishna Srinivas 0010f0ee10 Remove empty directories while removing an oobject 2015-10-29 23:30:16 -07:00
Harshavardhana 53adfb38f4 Leverage sort Interface to provide sortUnique function 2015-10-23 15:55:41 -07:00
Harshavardhana dbaa4d8643 Move ListObjects into its own file 2015-10-22 15:39:04 -07:00
Harshavardhana 1f66f4869b Fix all remaining windows path issues. 2015-10-22 00:05:10 -07:00
Harshavardhana afa27b9847 On windows translate Prefix, Marker and Delimiter for paths 2015-10-21 22:00:03 -07:00
Harshavardhana b74852116a Fix portability issues for arm on raspberry pi 2015-10-20 11:22:00 -07:00
Harshavardhana dddb1650de Add 5% cumulative reduction in total size of the disk 
This is done due to filesystem holding additional metadata and inode space
which is unaccounted for during min-free-disk calculation.
 2015-10-19 10:33:34 -07:00
Harshavardhana 179d2d7dac Add initial cut of auto expiry of objects 2015-10-19 01:34:31 -07:00
Harshavardhana c065be656c Implement min-free-disk as a subcommand, deprecate flag 2015-10-19 00:59:20 -07:00
Harshavardhana 5b2fa33bdb Implementing min-free-disk 2015-10-18 00:23:14 -07:00
Harshavardhana a8a935f5fd Improve disk code to return back disk StatFS{} structure 
```
StatFS {
Total int64
Free int64
FSType string
}
```

Provides more information in a cross platform way.
 2015-10-17 20:19:26 -07:00
Harshavardhana aee0845b2e Add disk package 
Implements

   - Stat returns total and free disk space supported across platforms
   - Type returns type of the filesystem underneath
 2015-10-17 16:48:24 -07:00
Harshavardhana 1256ca86d0 Add fs separator 2015-10-17 12:05:12 -07:00
Harshavardhana 2ec679a089 If directory already removed, return nil and move on 2015-10-17 00:13:46 -07:00
Harshavardhana d534fc5a4f Implement delete bucket properly with proper error handlings 2015-10-17 00:01:12 -07:00
Harshavardhana c9af01d807 Enhance listing further, this time handle cases related to common prefixes 2015-10-16 23:11:41 -07:00
Harshavardhana 0eb7f078f9 Implement Bucket ACL support 2015-10-16 19:47:30 -07:00
Harshavardhana 762b798767 Migrate this project to minio micro services code 2015-10-16 11:26:08 -07:00
Harshavardhana 3318cba132 Add windows support for minhttp library 2015-10-11 01:08:16 -07:00
Harshavardhana 7fea9cb550 Add rpc signature handler 2015-10-08 22:28:11 -07:00
Harshavardhana 11048708bb Implement GetBucketACL - fixes #893 2015-10-08 11:12:44 -07:00
Anand Babu (AB) Periasamy bf901d3b9a Merge pull request #895 from abperiasamy/tasker 
new task model minio server
 2015-10-08 02:23:29 -07:00
Anand Babu (AB) Periasamy b52697e6ad new task model minio server 2015-10-08 02:20:24 -07:00
Harshavardhana a060b158c8 Add quick.CheckVersion() to verify config version quickly before unmarshalling the full struct 
This is needed during migration where we would need to verify the underlying version number
in a quick way.
 2015-10-07 17:44:33 -07:00
Harshavardhana ab5ea997ab Make erasure Encode and Decode atomic to avoid races 2015-10-06 23:05:01 -07:00
Harshavardhana 1b42398e8b Canonicalize all the incoming input values, now PresignedPostPolicy works with minio-go 2015-10-06 10:21:28 -07:00
Harshavardhana d5ce2f6944 Make erasure matrix type not optional choose automatically 
Remove option of providing Technique and handling errors based on that
choose a matrix type automatically based on number of data blocks.

INTEL recommends on using cauchy for consistent invertible matrices,
while vandermonde is faster we should default to cauchy for large
data blocks.
 2015-10-05 22:38:02 -07:00
Harshavardhana 4ed50a8004 Fix encoding bug in donut during encoding phase 
Stream reading needs to check for length parameter being non zero,
after Reading() a predefined set of buffer length an EOF might be returned
with length == 0.

Erasure taking this zeroed data in might wrongly encode it as part of existing
data blocks which leads to errors while decoding even when the other contents
are intact.
 2015-10-05 22:12:53 -07:00
Harshavardhana f0a8dbecae Add --json output formatter for server 2015-10-05 00:20:49 -07:00
Harshavardhana c4faf47e64 First time mode for controller 
- Upon first time invocation ``minio controller`` would create access keys and secret id
- Upon request passing 'keys' arg ``minio controller`` would provide the keys
- Add colorized notification
 2015-10-04 16:42:16 -07:00
Harshavardhana cfdb29cac0 Simplify signature handling 
This change brings a new SignatureHandler where Presigned.
Requests without Payload are handled very early before even
going through the call.

This change simplifies Donut codebase to not have signature related
logic for all API's.

Simplification is still needed for Payload based signature eg. PUT/POST calls
, which are still part of the donut codebase, which will be done subsequently
after donut re-write.
 2015-10-04 13:15:33 -07:00
Harshavardhana c8de5bad2f Implement presigned policy 2015-10-04 00:01:34 -07:00
Anis ELLEUCH b5ea05d839 A better way to print prettified json syntax error msg 2015-10-03 12:25:44 -07:00
Harshavardhana 3b070dee16 Fix an important metadata getObject bug in donut 2015-10-01 10:18:03 -07:00
Harshavardhana 81cc017f91 Implement presigned signature v4 support 2015-10-01 10:17:47 -07:00
Harshavardhana 50750efb52 Reduce memory usage for memory multipart write by doing io.Pipe() streaming copy 2015-09-30 20:53:30 -07:00
Harshavardhana 8c7c5df770 Prober stringer should avoid frivolous newlines 2015-09-29 10:13:11 -07:00
Harshavardhana cd52d7a11c Fix Linux/Mac OS X erasure decoding failure with new Golang version 1.5.1 
Fixes #871
 2015-09-24 21:42:59 -07:00
Harshavardhana 3785489153 Disable multipart for donut backend from being used 
Will enable it later after cleanup
 2015-09-24 18:54:30 -07:00
Harshavardhana b938e40fb5 Move atomic package to the top and simplify its tests 2015-09-20 13:51:38 -07:00
Harshavardhana d54488f144 Move all server and controller packages into top-level 2015-09-19 01:07:42 -07:00
Anand Babu (AB) Periasamy 89a86948b5 new version format and some cleanup 2015-09-18 23:33:28 -07:00
Anand Babu (AB) Periasamy d1f1b7ac31 new version format and some cleanup 2015-09-18 23:27:04 -07:00
Harshavardhana 778f8cd222 Add new rpc tests for Server.Add and Server.List, improve Version.Get RPC to provide more details 2015-09-18 17:44:46 -07:00
Harshavardhana 2a15dd5eab Enhance signature handler - throw back valid error messages 2015-09-18 15:14:55 -07:00
Harshavardhana ac93bbb41d Merge pull request #846 from harshavardhana/new-changes 
With new auth config changes, restructure the API code to use the new style
 2015-09-18 03:45:03 -07:00
Harshavardhana 6a5e5c1826 With new auth config changes, restructure the API code to use the new style 2015-09-18 03:41:05 -07:00
Harshavardhana b4ce1e8c1d Add missing reply.Name and add possible failure tests 2015-09-18 03:15:19 -07:00
Harshavardhana f8bb85aeb7 Enhance auth JSONRPC, now provides persistent output 
Implements

   - Auth.Generate("user")
   - Auth.Fetch("user")
   - Auth.Reset("user")

This patch also adds testing for each of these cases
 2015-09-18 03:02:39 -07:00
Harshavardhana 4bcd86408b Fix minio header in accordance with rfc2616.txt 2015-09-17 23:46:10 -07:00
Harshavardhana 7093a05ab1 Version is a package now, will be re-used across codebase. 2015-09-17 20:17:33 -07:00
Harshavardhana 1887114444 Fix all the golint complaints about newly added changes 
Do not use func(this *server), such generic names should not be used
for writing struct methods.
 2015-09-17 18:53:42 -07:00
Harshavardhana 03ef6533c8 Add more documentation for probe 2015-09-17 18:10:42 -07:00
Krishna Srinivas c49407ced4 rename rpc/server.go -> rpc/rpc.go. rpc/server.go will accomodate ServerService 2015-09-17 15:46:51 -07:00
Harshavardhana 3f4b98ca4c Consolidate controller, move rpc package into controller - remove dangling code in pkg/server 2015-09-15 19:38:36 -07:00
Harshavardhana 8d5f6e0b96 Merge pull request #826 from krishnasrinivas/middleware-cleanup 
Remove unneeded functions in middleware init
 2015-09-15 18:11:45 -07:00
Krishna Srinivas b1b387b157 Remove unneeded functions in middleware init 2015-09-15 18:09:09 -07:00
Harshavardhana 45d8898019 Merge branch 'parallel-read2' of https://github.com/krishnasrinivas/minio into krishnasrinivas-parallel-read2 
Make few more changes and rebased with current master
 2015-09-15 17:33:33 -07:00
Anand Babu (AB) Periasamy 6930e4d668 logrus logger 2015-09-09 20:28:09 -07:00
Harshavardhana 1e2c010174 Avoid shadowing variables and enable checks to avoid them during build 2015-09-09 15:14:55 -07:00
Harshavardhana b649eff3fb Disable logging for now 2015-09-05 21:28:35 -07:00
Harshavardhana d0f945f8e7 Simplify erasure package for OSX 2015-09-05 20:19:43 -07:00
Harshavardhana afff3f8885 Revert "Enable controller to have CORS" 
This reverts commit f39ac24e99.
 2015-08-31 17:15:49 -07:00
Krishna Srinivas 1e82ee1192 Add CORS support to minio s3 server 2015-08-31 16:59:52 -07:00
Harshavardhana f39ac24e99 Enable controller to have CORS 2015-08-31 01:47:05 -07:00
Harshavardhana dcf0c71ca3 Merge pull request #810 from harshavardhana/restructure 
Restructure server code, controller now runs in silo
 2015-08-27 17:10:54 -07:00
Harshavardhana 025f95b1d6 Restructure server code, controller now runs in silo 2015-08-27 17:07:32 -07:00
Anand Babu (AB) Periasamy c11aa1c892 return *probe.Error for Untrace() as well. 2015-08-24 03:35:24 -07:00
Harshavardhana 988d39a5b6 Migrate to golang1.5 release with GO15VENDOREXPERIMENT=1 enabled 2015-08-22 18:35:37 -07:00
Harshavardhana 0e416ea699 Tests were running 4 times due to multiple times the TestingT{} was being called 
Calling multiple times TestingT{} will hook up runner for Suites for that many times
which would lead to repeated running tests.

Fix it by only initializing it once for all the Suites
 2015-08-20 22:32:50 -07:00
Harshavardhana 068d1d1ba9 Add a new quick.Load() function to load directly any config file provided a quick compatible struct{} is also provided 2015-08-20 20:33:49 -07:00
Anand Babu (AB) Periasamy b49b8cdbe8 return call stack in reverse 2015-08-19 22:40:27 -07:00
Anand Babu (AB) Periasamy 826202716f skip Env in json printinf if empty 2015-08-19 01:21:28 -07:00
Anand Babu (AB) Periasamy 76c40e075a make tracePoint public as well 2015-08-19 01:02:39 -07:00
Harshavardhana ac928b5092 Fix typo in probe 2015-08-18 23:42:41 -07:00
Anand Babu (AB) Periasamy cdf93e534c simplify probe APIs 2015-08-18 19:30:17 -07:00
Harshavardhana e9c5a51bc6 Migrate pkg/quick from mc 2015-08-13 16:29:55 -07:00
Harshavardhana 61175ef091 Migrate to govendor to avoid limitations of godep 
- over the course of a project history every maintainer needs to update
  its dependency packages, the problem essentially with godep is manipulating
  GOPATH - this manipulation leads to static objects created at different locations
  which end up conflicting with the overall functionality of golang.

  This also leads to broken builds. There is no easier way out of this other than
  asking developers to do 'godep restore' all the time. Which perhaps as a practice
  doesn't sound like a clean solution. On the other hand 'godep restore' has its own
  set of problems.

- govendor is a right tool but a stop gap tool until we wait for golangs official
  1.5 version which fixes this vendoring issue once and for all.

- govendor provides consistency in terms of how import paths should be handled unlike
  manipulation GOPATH.

  This has advantages
    - no more compiled objects being referenced in GOPATH and build time GOPATH
      manging which leads to conflicts.
    - proper import paths referencing the exact package a project is dependent on.

 govendor is simple and provides the minimal necessary tooling to achieve this.

 For now this is the right solution.
 2015-08-12 19:24:57 -07:00
Harshavardhana 45b59b8456 Probe revamped to provide for a new WrappedError struct to wrap probes as error interface 
This convenience was necessary to be used for golang library functions like io.Copy and io.Pipe
where we shouldn't be writing proxies and alternatives returning *probe.Error

This change also brings more changes across code base for clear separation regarding where an error
interface should be passed encapsulating *probe.Error and where it should be used as is.
 2015-08-08 00:16:38 -07:00
Harshavardhana f8141493bd use command not found helper 2015-08-03 18:06:19 -07:00
Harshavardhana d09fd8b0a1 Migrate from iodine to probe 2015-08-03 16:33:44 -07:00
Harshavardhana 884e9771b2 More changes to probe to avoid nil dereferences 2015-08-03 01:47:37 -07:00
Harshavardhana 65e4aede82 Minor changes to probe 2015-08-02 20:33:49 -07:00
Anand Babu (AB) Periasamy 697009c0a1 trace on New and add read locks 2015-08-02 11:58:28 -07:00
Harshavardhana d9493909d8 Merge pull request #788 from krishnasrinivas/open-and-openfile 
rename disk.OpenFile to Open which will do os.Open (which will be rea…
 2015-08-02 11:05:04 -07:00
Krishna Srinivas ee4432bc40 rename disk.OpenFile to Open which will do os.Open (which will be read-only). disk.OpenFile will do os.OpenFile (which can be rw, append) 2015-08-02 17:34:29 +05:30
Anand Babu (AB) Periasamy a728ddc027 probe package to trace & return errors 2015-08-02 02:38:08 -07:00
Harshavardhana aabfd541e1 Merge cmd/donut into minio cmd, deprecate controller RPC request 2015-07-31 12:57:15 -07:00
Harshavardhana 2671b2dbf4 crypto/cleanup: remove unused functions 2015-07-29 13:09:55 -07:00
Krishna Srinivas fdd2c22fa5 Do go fmt on bucket.go 2015-07-29 23:49:25 +05:30
Krishna Srinivas ae8089c9b6 when signature is not available there is no need to compute sha256 2015-07-29 21:09:02 +05:30
Krishna Srinivas bdc00624fd get erros from buffered channel. Return error during object read only if we have readers < k 2015-07-29 19:36:41 +05:30
Harshavardhana d346250f1c Collapse GetPartialObject() into GetObject() 2015-07-28 19:33:56 -07:00
Krishna Srinivas e1280779ed Read from the disks in parallel during object read 2015-07-28 18:04:17 +05:30
Krishna Srinivas 1ea91d2fa2 Use missingEncodedBlocksCount directly instead of "-1" workaround in missingEncodedBlocks[] 
Makes Code more readable
 2015-07-28 16:56:06 +05:30
Harshavardhana b0ea64a04f Strip off quotes from ETag for verifying complete multipart upload 2015-07-25 16:10:05 -07:00
Harshavardhana e082f26e10 Improving EncoderStream to return error only upon non io.EOF. 
io.EOF is okay since io.ReadFull will not have read any bytes at all.

Also making error channel receive only for go routine.
 2015-07-25 15:57:30 -07:00
Krishna Srinivas bcfaa12a4d Encoder now directly reads from the object stream. Using split.Stream() was causing lot of redundant memory operations. 2015-07-26 03:54:39 +05:30
Harshavardhana d6a0e0cc55 Rename more 2015-07-24 18:09:53 -07:00
Harshavardhana 63c9cf0c4b Move from Minimalist Object Storage to Minio Cloud Storage 2015-07-24 17:51:40 -07:00
Harshavardhana 43c908d5b9 ListObjects now considers multipart objects, also move to upstream check.v1 2015-07-18 15:49:41 -07:00
Harshavardhana 966786c78e remove unnecessary updateAt() 2015-07-17 19:48:09 -07:00
Harshavardhana c1da10a207 Donut multipart support no get, listobjects() support yet 2015-07-17 16:23:51 -07:00
Harshavardhana 86a887f9d4 Add corresponding tests 2015-07-16 17:40:19 -07:00
Harshavardhana 5eae32f2b0 Return proper InvalidArgument messages like s3 for invalid data for ListObjects(), ListObjectParts(), ListMultipartUploads() 2015-07-16 17:22:45 -07:00
Harshavardhana e605787e65 Method not allowed is right response for DELETE() operations and add tests 2015-07-16 14:15:24 -07:00
Harshavardhana 1f2e6a40a0 Fix an issue with reusing closed network connetion, changing the way rateLimitedListener is initialized 2015-07-16 10:56:14 -07:00
Harshavardhana 5507a39840 Heal buckets upon init if needed, adding new disks and HUP works 2015-07-16 09:59:05 -07:00
Harshavardhana e4574c7d6f Add File() method to Ratelimitlistener for extracting underlying fd() 2015-07-16 00:11:17 -07:00
Harshavardhana 2e8f154f34 Iodine should indent with EmitJSON() 2015-07-16 00:10:55 -07:00
Harshavardhana 4498662c16 Remove scsi non portable code, instead "donut make" implements functionality to instantiate a donut 2015-07-15 11:55:57 -07:00
Harshavardhana 6baf45e360 Fix a crash during listObjects() populating NextMarker 2015-07-15 10:04:15 -07:00
Harshavardhana 2553654e24 Add initial version of heal, remove rebalance 2015-07-14 22:56:41 -07:00
Harshavardhana e37c5315ec Live multiple disk removal works properly 2015-07-14 20:46:14 -07:00
Harshavardhana e1e4908515 Wire up sha512 matching inside donut along with md5sum 2015-07-14 19:47:50 -07:00
Harshavardhana e885259584 Handle removal of disks - getObject() now reads if disks are missing underneath, add initial stub healing code 2015-07-14 18:53:00 -07:00
Harshavardhana efbf3eabb7 Return x-amz-request-id for all replies 2015-07-14 14:44:03 -07:00
Harshavardhana da8b9fd112 Remove global custom config path variables, use get/set methods instead 2015-07-14 11:56:15 -07:00
Harshavardhana c4cf7635bf Read req.Body for PutBucket() if any 2015-07-14 09:30:10 -07:00
Harshavardhana 45ddec925c Fix an ugly multipart bug 2015-07-13 21:38:01 -07:00
Harshavardhana 7ae60a6d10 For missing parts reply back as InvalidPart{} 2015-07-13 19:31:01 -07:00
Harshavardhana 634f70f3b0 WriteObject() encoded data in go routine, fix another multipart issue 2015-07-13 15:56:54 -07:00
Harshavardhana 1bad92356d Add a ratelimited listener than a ratelimited handler - more precise 2015-07-13 11:12:17 -07:00
Anand Babu (AB) Periasamy 8af5933b07 Merge pull request #743 from flandr/fix-osx 
Fix OS X build
 2015-07-13 10:15:53 -07:00
Nate Rosenblum ec347f96fd Fix OS X build 
- Explicitly cast statfs_t members to int64 (this structure is
   platform-specific)
 - Add pass-through New methods to Darwin SHA package
 - Move scsi pkg types to common translation unit (package was empty)
 - Add stub implementations mount/disk ops for OS X
 2015-07-13 10:06:55 -07:00
Harshavardhana 55e4d0c6a5 mkdonut now creates a donut processing cli args 2015-07-12 21:21:31 -07:00
Harshavardhana 58a1d865a9 Renaming nimble to minhttp 2015-07-12 18:01:52 -07:00
Harshavardhana 847440196e Add signature v4 tests 2015-07-12 13:36:02 -07:00
Harshavardhana 7615a6bfe5 HEAD shouldn't have any body, handle it in writeErrorResponse() 2015-07-11 10:34:55 -07:00
Harshavardhana 8977f9a524 cached api test should also have a custom config path, would conflict with your .minio local path 2015-07-10 20:39:30 -07:00
Harshavardhana 97d4a27c7e Do not reply on ignoredHeaders for server, rely on SignedHeaders sent as part of Authorization header 2015-07-10 19:37:22 -07:00
Harshavardhana 53f5d2c32b nodejs http library sends Connection header during HTTP request, this clobbers up the signature handling ignore it 2015-07-10 15:44:47 -07:00
Harshavardhana 29838bb851 Cleanup temporary writers upon errors during putObject(), all metadata() write operations 2015-07-10 14:11:04 -07:00
Harshavardhana 15dd0df187 Support signature v4 at rest 2015-07-10 11:49:27 -07:00
Harshavardhana 7fa514351c Rename definitions to log.go, add valid prefixes 2015-07-10 11:11:20 -07:00
Harshavardhana d5ffc16f25 Add abbreviated close response, to avoid any leaks 2015-07-10 10:20:00 -07:00
Harshavardhana e5006c738d Handle both space and non-space characters, in signature v4 - add errors for all API's 2015-07-09 21:49:58 -07:00
Harshavardhana 84f427f14a All other API's now support signature v4 2015-07-09 19:45:56 -07:00
Harshavardhana 00890c254e CompleteMultipartUpload and CreateObjectPart now fully support signature v4 2015-07-09 19:01:15 -07:00
Harshavardhana 89c1215194 PutObject handler gets initial support for signature v4, working 2015-07-09 16:44:38 -07:00
Harshavardhana d461fa5ab1 Add mountinfo functions for detecting mount disks, and other rpc changes 2015-07-09 12:25:29 -07:00
Harshavardhana 8654ddb566 Generate auth now saves in ${HOME}/.minio/users.json, also authHandler verifies request validity 2015-07-08 21:53:13 -07:00
Harshavardhana 375860077d HTTP header Content-Length signifies body length of the request, if its smaller reply appropriately 
This patch also handles large individual part sizes > 5MB by using less memory copies.
 2015-07-08 20:56:41 -07:00
Harshavardhana ec33d79d57 Add server side signaturev4 check, not wired up to the readers yet. 2015-07-08 16:57:03 -07:00
Harshavardhana 396b728031 Add auth rpc service to generate access keys, add corresponding test 2015-07-08 14:40:39 -07:00
Harshavardhana 770fd23afa Renaming keys as auth, working towards signature v4 support for all put objects 2015-07-08 14:17:16 -07:00
Harshavardhana 2413a110e6 Fix SSL support, pointer indirection caused nil buffers 2015-07-08 11:02:15 -07:00
Harshavardhana d1deda3a96 Add API tests for both donut on disk and donut cache 2015-07-07 19:39:46 -07:00
Harshavardhana ece797c16e Add rpc tests 2015-07-07 17:27:34 -07:00
Harshavardhana 676b9058de Separate out memory statistics and system information into two different services 2015-07-07 16:59:20 -07:00
Harshavardhana 8abb96c030 If NodeDisks are not empty do not impose cache maxSize restriction 2015-07-07 16:41:40 -07:00
Harshavardhana a50a44b0ca Add nimbleNet tests 2015-07-07 16:15:47 -07:00
Harshavardhana 317096a0c4 Add net.Addr wrapper with IsEqual() and use it. 2015-07-07 15:33:08 -07:00
Harshavardhana 11b893804c Moving os.MkdirAll() inside atomic for auto parent directory creates 2015-07-07 12:35:57 -07:00
Harshavardhana 52cd23ad9f Move atomic file writes into its own package, use them inside quick and disk packages 2015-07-07 12:29:14 -07:00
Harshavardhana 3622fbc87d Across donut, split, nimble some code cleanup 2015-07-06 21:55:21 -07:00
Harshavardhana bbb89b5776 Add multi-thread protection and also allow atomic file creates, rename upon Close() 2015-07-06 18:54:32 -07:00
Harshavardhana c2c7bdf0cd Cleanup nimble http 2015-07-06 18:51:20 -07:00
Harshavardhana b029d0a5f0 Avoid config reload all the time, reload is manually triggerred from outside 2015-07-06 17:26:35 -07:00
Harshavardhana 8b94c53345 Fix issues with multipart upload 2015-07-06 16:22:27 -07:00
Harshavardhana 474954022e Add modified grace library from facebookgo, rename it as nimble 2015-07-06 15:40:12 -07:00
Harshavardhana 1d64e4b6c1 Add Donut rpc service for sending changes to configuration files 2015-07-06 11:10:06 -07:00
Harshavardhana 10b082144e Add updateConfig code to load config changes if possible for every function 2015-07-05 22:46:42 -07:00
Harshavardhana ba0a5ed416 Add sighup, sigusr2 into trapping code, to trap signals for reloading configuration. 
Need to still figure out a way of graceful restarts - gave facebookgo/httpdown a shot,
but it is not suitable.
 2015-07-05 21:40:53 -07:00
Harshavardhana 4a27ab0e58 Fix another deadlock inside CreateObjectPart() code, premature return without Unlocking() 
Also this patch changes the cache key element to be interface{} type not string.
 2015-07-05 20:26:32 -07:00
Harshavardhana 75788c7a1d Fix go installation check on amazon instance 2015-07-05 18:12:58 -07:00
Harshavardhana 7f0c14f2b7 Add basic controller code, initiating json rpc connection getting list of disks and memstats for now. 2015-07-05 17:17:41 -07:00
Harshavardhana a3ccb9d405 Rename stuttered service names and make them appropriate 2015-07-05 10:19:23 -07:00
Harshavardhana e66a84242a Add disk detection for Linux, add new RPC service GetDiskInfoService(), remove dummy HelloService() 2015-07-05 02:08:33 -07:00
Harshavardhana bab4a47525 Move to container/list datastructure from map[string][]byte 2015-07-04 17:08:23 -07:00
Harshavardhana aa67a19e99 Implement new CPU detection using cpuid, cpuidex plan9 instructions from klauspost/cpuid project, remove C code 2015-07-04 14:28:16 -07:00
Harshavardhana 0a827305ad Implement metadata cache, metadata cache is used by top level donut right now. Rename trove as data cache 
We should use it internally everywhere.
 2015-07-03 21:09:57 -07:00
Harshavardhana 14844f48dd Make donut do everything as an atomic operation, this avoids all the deadlocks and races 2015-07-03 17:16:58 -07:00
Harshavardhana 30fc14e703 Restructure codebase move crypto, checksum to top-level, move ``split`` into donut, move crypto/keys into api/auth 2015-07-03 15:24:51 -07:00
Harshavardhana 8a4e7bcdcf Add full API tests, move storage/donut to donut, add disk tests as well 2015-07-03 14:36:29 -07:00
Harshavardhana 7c37e9d06a Make donut fully integrated back into API handlers 2015-07-02 21:04:04 -07:00
Harshavardhana 12bde7df30 Add simple Ticket Master which pro-actively sends messages on proceedChannel 
Handlers are going to wait on proceedChannel, this the initial step towards
providing priority for different set of API operations
 2015-07-02 21:04:04 -07:00
Harshavardhana 5cfb05465e Add cache, donut tests separately - fix behavior differences 
Remove priority queue, implement it using a simpler channels
 2015-07-02 21:04:04 -07:00
Harshavardhana ebe61d99d9 Use cache Append() for saving objects in memory, GetObject() caches un-cached entries while reading 2015-07-02 21:04:04 -07:00
Harshavardhana bce93c1b3a Integrate cache with donut, add tests 2015-07-02 21:04:04 -07:00
Harshavardhana 0533abf6a8 Make priority queue lambda function return error over a channel 2015-07-02 21:04:04 -07:00
Harshavardhana 38a6ce36e5 Remove slow AppendUniq code, rolling through over a slice is in-efficient 
Remove it and use map instead
 2015-07-02 21:04:04 -07:00
Harshavardhana 84810162f5 Add simple Version and GetSysInfo services 2015-07-02 21:04:04 -07:00
Harshavardhana 14ec42d646 Add initial implementation of priority queue, uses container/heap 2015-07-02 21:04:04 -07:00
Harshavardhana eb5aa19dfa Remove custom Config, will use quick Config instead for user access keys 2015-07-02 21:04:04 -07:00
Harshavardhana 701c3e5242 Add new RPC helpers wrapping over regular rpc packages, add middleware chaining ability 2015-07-02 21:04:04 -07:00
Harshavardhana 4addf7a996 Restructure API handlers, add JSON RPC simple HelloService right now. 2015-07-02 21:04:04 -07:00
Harshavardhana 335c7827eb More donut, cache, api cleanup 2015-07-02 21:04:04 -07:00
Harshavardhana dc0df3dc0e Breakaway from driver model, move cache into donut 2015-07-02 21:04:03 -07:00
Harshavardhana 72572d6c71 Remove some api server code bringing in new cleanup 2015-07-02 21:04:03 -07:00
Harshavardhana c2031ca066 Add server and control command 2015-07-02 21:04:03 -07:00
Frederick F. Kautz IV cfbc169034 Fixing API definitions 2015-07-02 13:14:21 -07:00
Harshavardhana 8f61d6b6be Move memory code out, add it as layer on top of existing donut code 
Just like how http.Handlers can be overlayed on top of each other
with each implementing ServeHTTP().

drivers.Driver can be overlayed on top of each other in similar manner
which would implement the drivers.Driver interface.

   API <----> cache <----> donut <----> donut(format)
 2015-06-30 10:09:12 -07:00
Harshavardhana ab6e16bb41 Add dummy driver for community to submit new drivers 2015-06-29 16:43:50 -07:00
Harshavardhana 12de98fb62 Rename memory driver as cache 2015-06-29 16:43:50 -07:00
Harshavardhana 2571342451 Filesystem goes the high road *again* 2015-06-29 16:43:42 -07:00
Harshavardhana f74d6138da IsValidBucket() is sufficient we don't need to verify for "." 2015-06-29 15:15:54 -07:00
Harshavardhana 3109909355 Handle couple of cases of OOM conditions, move caching to GetObject() rather than PutObject() 2015-06-29 12:28:50 -07:00
Harshavardhana d07d0c670a Return back proper errors in writeObjectData(), rename few functions 2015-06-29 11:46:35 -07:00
Harshavardhana 10c807f233 Put object on successful write returns full metadata, to avoid subsequent GetObjectMetadata() calls in driver 2015-06-29 11:15:46 -07:00
Harshavardhana 6921328b93 Avoid frivolous GetObjectMetadata() calls at driver level, return back all the information in donut ListObjects() 2015-06-29 11:14:58 -07:00
Harshavardhana d8f7896a43 Expand http server struct to store more values 2015-06-29 00:12:28 -07:00
Harshavardhana 42c0287943 Use errorChannels only for services not for drivers, reduce them to use simple functions 2015-06-28 23:59:47 -07:00
Harshavardhana 91e5f648cb Move to Set() not Append() due to large memory reference copy 2015-06-28 10:13:12 -07:00
Harshavardhana ac4d8fe478 Add free() method for proxyReader to aggressively de-allocate Read data, to handle certain out of memory conditions 
There are still some more out there
 2015-06-27 20:43:25 -07:00
Harshavardhana 05f8654e3d Add Append() method to trove cache for appending data to an existing key 
This largely avoids a large buffer copy which would accumulate inside proxyReader{}

This patch also implements "initialize()" function to init and populate data
on all the existing buckets, avoiding the redundant ListBuckets() invoked by
every API call.
 2015-06-27 20:25:24 -07:00
Harshavardhana 367772b988 Make sure to populate on disk data into memory upon first API requests 2015-06-27 18:25:21 -07:00
Harshavardhana 45a7eab804 An attempt to bring in memory layer into donut driver 2015-06-27 17:23:34 -07:00
Harshavardhana f3c25bcfc4 Keeping the lexical order same add optimizations, provide a comprehensive response from ListObjects() 2015-06-27 13:12:44 -07:00
Harshavardhana ae66ae42c4 Rename functions for their purpose 2015-06-27 12:39:11 -07:00
Harshavardhana 9a4680475f Object metadata was wrongly misconstrued to be mutable, handle it 2015-06-26 23:22:53 -07:00
Harshavardhana 3aa6d90c5e Avoid racy maps, read from disk on success return quickly. Many more optimizations 2015-06-26 19:49:37 -07:00
Harshavardhana 68974918ac Donut moves to typed metadata, removing the necessity for strconv, parsing and string converstions 2015-06-26 16:23:12 -07:00
Harshavardhana 767d3743ee Add new metadata definitions and use them 2015-06-26 13:34:09 -07:00
Harshavardhana 9958e34772 Import quick key value store from Minio Client for persistent state files, primarily for donut 2015-06-25 19:57:31 -07:00
Harshavardhana fb9adb5524 Donut cleanup another set 
- Make sure to close all readers
- Fix errors in api_testsuite c.Assert(err, IsNil) should be done right after each function call
 2015-06-25 18:54:34 -07:00
Harshavardhana eec66f195a Take all the ListObjects into bucket handlers 
Earlier the listing would wait for all the objects to be processed
this is essentially very time consuming considering even for 100,000
files.
 2015-06-25 18:04:29 -07:00
Harshavardhana 45e9d25931 Remove more bloated code - simplify 2015-06-25 13:02:08 -07:00
Harshavardhana 82dcbf262d Add simple locking for donut API for now - fixes #671 2015-06-25 11:29:11 -07:00
Harshavardhana 03b4d3b308 Go vet fixes for donut 2015-06-24 21:07:03 -07:00
Harshavardhana 57a2b53178 Removing further bloated code simplifying 2015-06-24 21:03:39 -07:00
Harshavardhana a2c205ff2e Use external package disk for donut. 2015-06-24 21:03:02 -07:00
Harshavardhana 841ff01412 Move disk into its own package, remove bloated code 2015-06-24 20:13:47 -07:00
Harshavardhana 1682c748ac Remove unnecessary interfaces from donut, cleanup 2015-06-24 19:43:38 -07:00
Harshavardhana 3498872467 Add sha256 and sha512 windows compatibility layer 2015-06-24 14:39:41 -07:00
Harshavardhana 77d35b87d4 Fix a bug on windows regarding blockSSE3 calculation 2015-06-24 14:24:33 -07:00
Harshavardhana f1410731db Add windows code for sha1 and crc32c 2015-06-24 14:16:32 -07:00
Harshavardhana ba2d3dea74 Fix wrong TMPFS listing in donut filesystem map 2015-06-23 23:16:06 -07:00
Harshavardhana e818bc7187 Trim iodine path properly, so that now errors have github.com/minio/minio prefixed 2015-06-23 13:36:54 -07:00
Harshavardhana 2fd52ca284 Add missing strongly typed errors for Donut 2015-06-23 11:54:44 -07:00
Harshavardhana 7a060110ff Fix BUILDDEPS paths for golang installation 2015-06-22 12:07:23 -07:00
Harshavardhana 3bf64f5669 Verify d.donut to be non-nil, usually happens when multiple go versions compilations are linked - possible cause for #659 2015-06-20 11:03:17 -07:00
Harshavardhana 641f07cecf Use filepath everywhere instead of path.{} functions for portability - fixes #656 2015-06-18 16:02:45 -07:00
Harshavardhana 573a6134b2 Remove redundant !ok for map 2015-06-17 22:36:46 -07:00
Harshavardhana e9a3fd677a Hold lock on GetGlobalStateKey() - fixes #652 2015-06-17 20:37:49 -07:00
Harshavardhana b793f53d48 Minior change filter() to filterObjects() 2015-06-17 20:35:44 -07:00
Harshavardhana d4e71beb54 Add time.Since on buckets created time to avoid deleting buckets spuriously during eviction of objects 2015-06-13 11:28:04 -07:00
Harshavardhana 636788997b Minor optimization avoiding one unnecessary copy 2015-06-11 14:23:05 -07:00
Harshavardhana 6d44274db6 Make sure to reply back ETag with quotes for s3 compliance 2015-06-09 01:13:02 -07:00
Harshavardhana a947767339 Remove unnecessary tlsHelpers, SSL works with domain certificate 2015-06-08 20:54:41 -07:00
Harshavardhana c6ecd94420 Enable TLS and use it 2015-06-08 20:10:59 -07:00
Harshavardhana 64b014369c Return back entity too large for createObject requests bigger than cache size 2015-06-08 17:10:07 -07:00
Harshavardhana ab1976bc14 Load from persisted multipart session, now multipart uploads can resume even after restarting minio fs server 2015-06-08 15:15:37 -07:00
Harshavardhana 2976f109df More multipart fixes, filter out parts and multipart metadata from listObjects 2015-06-08 14:08:36 -07:00
Harshavardhana e08d59be00 Make sure to delete uploadid's from active session file properly 2015-06-08 12:25:51 -07:00
Harshavardhana 0d3710cb9b Store in json file instead of gob for fs metadata 2015-06-08 12:08:01 -07:00
Harshavardhana 8efc842b59 Add delete handlers and reply back as 'NotImplemented' instead of 404 2015-06-08 11:06:06 -07:00
Harshavardhana 8a320c4447 Enhance logging handler to print in humanize form for ContentLength 2015-06-08 02:14:09 -07:00
Harshavardhana cefd740002 Put back the older conn_limit 2015-06-06 18:11:38 -07:00
Harshavardhana 371651bde6 Add rate limiter instead of connection limit for now 2015-06-06 18:04:55 -07:00
Harshavardhana 21916dc0a1 Fix a deadlock in trove when evicting objects when we reach the maxSize used 2015-06-06 16:02:52 -07:00
Harshavardhana 8b43655d9f Make connection limit configurable 2015-06-06 14:31:50 -07:00
Harshavardhana a344f5b34d Misc changes before capturing proper video 
- Disable management UI command option
- Remove featureflags option, filesystem and memory already support multipart
- Print informative messages after starting minio server
 2015-06-03 18:27:15 -07:00
Harshavardhana 097f70a3c5 Enable filesystem for API tests, this patch also implements ListMultipartUploads() 2015-06-01 20:57:13 -07:00
Harshavardhana bfb9209689 Skip files and weird directories from listBuckets(), allowing this so that minio fs mode can be used as ftp service 2015-06-01 16:43:43 -07:00
Harshavardhana d86ba5f336 Implement multipart support 
- implemented NewMultipartUpload()
 - implemented CreateObjectPart()
 - implemented ListObjectParts()
 - implemented AbortMultipartUpload()
 - implemented CompleteMultipartUpload()
 2015-06-01 16:25:11 -07:00
Harshavardhana c07d48e7e8 ErrorResponse from minio server needed proper xml translations 2015-06-01 12:19:27 -07:00
Harshavardhana a91e519692 Make caching a package trove, and use it inside memory driver 2015-05-31 18:41:56 -07:00
Anis Elleuch 7e5a26f42c Protect the exported method Delete() 2015-05-31 20:29:42 +01:00
Harshavardhana 82a0eac659 Add filesystem factory functions and add related cli options 2015-05-30 03:23:06 -07:00
Harshavardhana 0cc63706bb Resurrect back fileystem driver 2015-05-30 03:12:01 -07:00
Anis Elleuch d360d0fec3 Forbid the upload of files bigger than the memory backend capacity 2015-05-28 22:07:02 +01:00
Harshavardhana 554a244b47 Add donut OSX support 2015-05-26 02:14:34 -07:00
Harshavardhana e46660a966 Use http.TimeFormat to return, S3 expects us to return in GMT not UTC 2015-05-25 20:17:04 -07:00
Harshavardhana 19250296c6 Handle authorization header better 2015-05-24 20:59:53 -07:00
Harshavardhana 9666f2e5bf Relax minium object size limit, one can really upload a 0byte file 
For example
~~~
$ mc ls https://s3.amazonaws.com/ferenginar/test.go
[2015-05-23 12:02:06 PDT]     0B test.go

$ curl -i -X HEAD https://s3.amazonaws.com/ferenginar/test.go
HTTP/1.1 200 OK
x-amz-id-2: ZMWLriPH+uQJ8IsaMHmuNOU/FXvdSq+s6O7ugUI9hZ695XJTNAZ1utKxh03w5Jcf
x-amz-request-id: F5A11F533B74DD8F
Date: Sat, 23 May 2015 19:19:29 GMT
Last-Modified: Sat, 23 May 2015 19:02:06 GMT
ETag: "d41d8cd98f00b204e9800998ecf8427e"
Accept-Ranges: bytes
Content-Type: binary/octet-stream
Content-Length: 0
Server: AmazonS3
~~~
 2015-05-23 12:19:16 -07:00
Harshavardhana e11f9110b6 add missing validate access keys after being extracted from AuthHeader 2015-05-21 19:23:47 -07:00
Harshavardhana ad60b97d52 Fix some donut TODO's regarding bucket created date 2015-05-21 11:13:49 -07:00
Harshavardhana cd817d34df Finish last bits of api testsuite on server 2015-05-21 02:57:03 -07:00
Harshavardhana f00bfd38da Bump gocyclo back to 16 2015-05-19 18:32:50 -07:00
Harshavardhana 3d61ba5930 Fix duplicate map allocation 2015-05-18 13:14:01 -07:00
Harshavardhana e4c0d574dc Update totalnumber of multiparts inside CreateObjectPart(), also introduce support for UploadIDMarker and Prefix for listMultipartUploads 2015-05-18 12:24:36 -07:00
Harshavardhana 7ce47130fd HEAD request should have Content-Length for only successful response, there is no response body for errors, just header is sufficient - fixes #603 
http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.13

"""  in the case of the HEAD method, the size of the entity-body that would have been sent had the request been a GET. """
 2015-05-15 21:55:59 -07:00
Harshavardhana 317637fcb8 Remove redundant WriteHeader calls 2015-05-15 20:58:28 -07:00
Harshavardhana 5c5761ae16 Remove unnecessary GetObjectMetadata prefix argument 2015-05-15 19:25:17 -07:00
Harshavardhana 6275238a47 For multiparts use its own cache avoids deadlocks during object expiration 
Using same object cache as regular objects for multiparts causes deadlocks during object expiration routine.

As a side affect this also fixes the bug of listing multiparts during regular 'listing' of objects.
 2015-05-14 20:26:59 -07:00
Harshavardhana c41ebf7535 Fixing a race in NewMultipartUpload() session for simultaneous requests, observed at play.minio.io 2015-05-14 19:11:19 -07:00
Harshavardhana 8fb427a892 Add more tests for multipart now in driver api_testsuite 2015-05-14 17:28:39 -07:00
Harshavardhana dba7c94571 Add tests for abortMultipartUpload, listobjectparts, listmultipartuploads 2015-05-14 17:07:30 -07:00
Harshavardhana 1bd94ec6ab An attempt to implement ListMultipartUploads() 2015-05-14 16:22:47 -07:00
Harshavardhana 82b043005a Avoid one last memory copy 2015-05-14 11:26:55 -07:00
Harshavardhana 5498c90788 Add a new validateContentTypeHandler{}, verify Accept header earlier 2015-05-13 12:22:15 -07:00
Harshavardhana de56909d55 More scripts and code from Minio-io to minio 2015-05-11 16:43:38 -07:00
Harshavardhana 8d06504068 Change all minio-io path to minio 2015-05-11 16:23:10 -07:00
Harshavardhana e39de4f603 Fix typo introduced in previous commit 2015-05-11 14:24:31 -07:00
Harshavardhana f45635524f Remove SignatureV2 support, bring in SignatureV4 header only validation for now 2015-05-11 13:38:40 -07:00
Harshavardhana b48b7f51ef Use convenience functions for url.Values over looping through keys, fix a bug in listObjectParts() 2015-05-10 11:32:41 -07:00
Harshavardhana 14b5828505 Handle partNumberMarker with listObjectParts now and other fixes 2015-05-09 19:39:00 -07:00
Frederick F. Kautz IV cf559fa48e Limiting upload id size 2015-05-09 18:28:50 -07:00
Frederick F. Kautz IV 61a9ea0f4b Adding support for memory driver test 2015-05-09 18:15:27 -07:00
Frederick F. Kautz IV 058604aa21 Adding tests and fixes for multipart uploads uncovered from tests 2015-05-09 17:42:14 -07:00
Harshavardhana 8860aa0c8c Implement AbortMultipart 2015-05-09 16:09:50 -07:00
Harshavardhana 18c7f620cc Add listparts support 2015-05-09 13:55:55 -07:00
Harshavardhana 04e388d620 Handle two different styles of time format, s3cmd now compatible 2015-05-08 16:01:59 -07:00
Harshavardhana 01b474fca3 Move expiration code into its own and add ticker select case 2015-05-08 12:37:49 -07:00
Harshavardhana 23a545673d Verify incoming md5sum for invidual parts in complete multipart request and some simplification of the code 
Add two new functions for session cleanup
 2015-05-08 03:59:05 -07:00
Harshavardhana 616241a2c1 Re-enable bucket deletion, this time with uploadID sitting there 2015-05-08 02:28:38 -07:00
Harshavardhana 2ea10c798b Implement proper errors for Multipart 2015-05-08 02:02:51 -07:00
Harshavardhana 82c3656f79 Reply back CompleteMultipartUploadResult properly with final ETag computed 
- Now s3 libraries and also objectstorage-go work properly
 2015-05-07 23:01:20 -07:00
Frederick F. Kautz IV ae5c99d7bb Router should work with ?uploads unescaped now 2015-05-07 21:37:34 -07:00
Frederick F. Kautz IV 9dc7b82b39 Merge pull request #576 from fkautz/pr_out_adding_multipart_support 2015-05-07 20:00:20 -07:00
Frederick F. Kautz IV f050d5e974 Adding multipart support 2015-05-07 19:55:30 -07:00
Frederick F. Kautz IV ed71c28c37 Fixing panic in intelligent, attempted to delete nonexistent key 2015-05-07 19:12:49 -07:00
Frederick F. Kautz IV d0c4334834 Adding feature flags 2015-05-07 13:56:54 -07:00
Harshavardhana 1df28f3060 Get rid of sub-domain code and some other formatting changes 2015-05-07 00:21:40 -07:00
Harshavardhana 6d3d92694f Add Marker support for donut, also add a testing code 2015-05-05 22:18:24 -07:00
Harshavardhana 75028c2ad1 Add lexicographic Marker/NextMarker support for recursive listing of objects. 
Also update times when an object is accessed logic
 2015-05-05 18:05:34 -07:00
Frederick F. Kautz IV 24af2222c9 Adding xmlns 2015-05-05 12:54:34 -07:00
Harshavardhana 193a6606db Fix a potential race between ExpireObjects() and Set() over items map 2015-05-04 22:36:47 -07:00
Harshavardhana ed1259d6f0 Deprecate LRU use just map[string]interface{} and call it intelligent 'cache' 2015-05-04 22:01:56 -07:00
Harshavardhana 63edb1e9a0 Nullify list memory to nil as we remove the element 2015-05-04 17:22:08 -07:00
Harshavardhana d63064b8af Optimize memory usage in GetPartialObject() for memory driver 2015-05-04 15:26:56 -07:00
Harshavardhana c8f31d97a8 Modify LRU further to add object expiration 2015-05-04 04:32:44 -07:00
Harshavardhana 670f997b07 Bring blockingWriter from client and use it here 2015-05-04 03:55:43 -07:00
Harshavardhana d0df548eb5 Use new LRU inside memory driver 2015-05-04 03:55:43 -07:00
Harshavardhana f7caef2d26 Change CreateObject() to take size argument from content-length 2015-05-04 03:55:43 -07:00
Harshavardhana c342ce1588 New LRU based on GroupCache which keeps track of memory rather than entries 2015-05-04 03:55:19 -07:00
Frederick F. Kautz IV 3cf045fd1b Deleting key from lastAccessedObjects on eviction 2015-05-01 22:01:07 -07:00
Harshavardhana 9ec8925f1c Rename IsValidObject() to IsValidObjectName() 2015-05-01 20:31:26 -07:00
Frederick F. Kautz IV 33ecba8e49 Reimagining memory settings cli input 2015-05-01 18:12:58 -07:00
Frederick F. Kautz IV 25c268ac88 Attempt to fix memory lock on Get preventing write 2015-05-01 17:45:40 -07:00
Frederick F. Kautz IV 85f1b46492 Removing log message from test 2015-05-01 17:32:18 -07:00
Frederick F. Kautz IV 7cf86e6114 Fix docker support 2015-05-01 15:23:39 -07:00
Harshavardhana 2c7b30e5e1 Even mux vars don't help, handle it by looking at req.URL.Path 2015-05-01 12:41:17 -07:00
Frederick F. Kautz IV b86b26e7e5 Exposing expiration for memory driver 2015-04-30 21:20:54 -07:00
Harshavardhana c01fb92eca Merge pull request #557 from harshavardhana/pr_out_make_builddate_universal 2015-04-30 21:19:04 -07:00
Harshavardhana bbc32d6ac0 Disable domain based routing for now 2015-04-30 21:15:53 -07:00
Harshavardhana f3078d047c Get bucketName from mux.Vars instead of heuristic code, resolves subdomain issues 2015-04-30 21:11:32 -07:00
Frederick F. Kautz IV e911cc332e Merge pull request #554 from fkautz/pr_out_disable_eviction_on_0_expiration_time 2015-04-30 19:40:46 -07:00
Harshavardhana 8e3d48bf35 Use UTC() everywhere 
ref - https://github.com/golang/go/issues/1988, no standard function to set timeZone
 2015-04-30 19:38:32 -07:00
Frederick F. Kautz IV 94e24f1a05 Disable eviction on 0 expiration time 2015-04-30 19:34:35 -07:00
Harshavardhana 72395798c8 Merge pull request #552 from harshavardhana/pr_out_minio_config_directory_should_be_0700_not_os_modedir 2015-04-30 17:40:15 -07:00
Harshavardhana 2de81c8a3c Minio config directory should be 0700 not os.ModeDir 2015-04-30 15:42:49 -07:00
Frederick F. Kautz IV 8e2f7f9493 Wrapping functions that require gc so that references are released before running 2015-04-30 12:38:12 -07:00
Harshavardhana d815e6adfd Changes to CreateObject() now returns back md5 along with any error 
- This change is necessary to avoid the racy calls to GetObjectMetadata()
- This change is also necessary since one has to reply back md5sum with
  PUT object response header
 2015-04-30 03:38:14 -07:00
Harshavardhana fd2203b1b7 Some more improvements to connection limit 2015-04-30 01:23:44 -07:00
Harshavardhana f8a16dd22b Memory now evicts bucket if no more objects in memory struct 
- To avoid race in expiration while accessing memory driver structs with
  in two competing write locks. Use lru Len() instead to know exact length
  to schedule for eviction.

- squash both bucket and object structs, instead use a separate map to keep
  mutable info of lastAccessTime which can be independently used inside the
  expiration routine.
 2015-04-30 00:31:52 -07:00
Frederick F. Kautz IV ed0a9a9e06 Removing entity too large errors from memory driver 2015-04-30 00:09:54 -07:00
Frederick F. Kautz IV aff2b3b06a Adding debug statements for testing 2015-04-29 21:13:32 -07:00
Frederick F. Kautz IV 82a16b8848 Make minio work with curl and browsers again 2015-04-29 20:17:26 -07:00
Frederick F. Kautz IV 3a1386165f Fixing race condition 2015-04-29 18:54:59 -07:00
Harshavardhana 03bca9a001 Fix out of memory issue 2015-04-29 18:33:19 -07:00
Harshavardhana 25d4c0f6fa A small logical change in messaging and logging 2015-04-29 17:30:54 -07:00
Frederick F. Kautz IV f596c744ed Adding error message on connection limit met 2015-04-29 17:21:23 -07:00
Harshavardhana a89c4a57d3 Return resources as its available, not a new value 2015-04-29 16:47:14 -07:00
Harshavardhana 251c99176a Merge pull request #537 from harshavardhana/pr_out_add_proper_content_length_for_error_and_success_responses 2015-04-29 15:32:12 -07:00
Frederick F. Kautz IV 0da04c6f17 Expirations are now based on last accessed instead of time created 2015-04-29 15:28:45 -07:00
Harshavardhana 92e4301414 Add proper content-length for error and success responses 
- All compliance issues with S3 API for Put,Get,List (Bucket,Object) respectively
 - Encodes and returns back proper HTTP headers
 2015-04-29 15:28:28 -07:00
Frederick F. Kautz IV 5fc5f79ab7 Adding lru 2015-04-29 15:08:26 -07:00
Frederick F. Kautz IV 8fcb697844 Simplifying memory sleep logic 2015-04-29 13:53:09 -07:00
Frederick F. Kautz IV 5cc9418ca7 Adding expiration 2015-04-29 13:02:21 -07:00
Frederick F. Kautz IV a608524ebf api logging is now in its own package 2015-04-29 12:24:53 -07:00
Harshavardhana d1d5f7a78d Collate success response into writeSuccessResponse(), add docs 2015-04-29 10:52:04 -07:00
Harshavardhana b18bb230da maxObjectSize and minObjectSize limitation added at putObjectHandler() 
Put() replies back with

   - EntityTooLarge with > 5GB in single PUT operation
   - EntityTooSmall with < 1B in single PUT operation
   - IncompleteBody with ho Content-Length found in HTTP request header
 2015-04-29 02:19:53 -07:00
Frederick F. Kautz IV 172cd18455 Adding initial logging framework to api_router 2015-04-28 20:37:08 -07:00
Frederick F. Kautz IV 88cca647ea Adding json logging to stderr 2015-04-28 18:11:53 -07:00
Frederick F. Kautz IV de091068e8 Removing quotas, will handle with iptables 2015-04-28 12:14:34 -07:00
Frederick F. Kautz IV f2257a643c Critical sections are now protected by lock 2015-04-27 19:59:58 -07:00
Harshavardhana 605edbc307 Add some logging with debug 2015-04-27 19:26:24 -07:00
Harshavardhana 7bdcf4bbf3 Merge pull request #515 from fkautz/pr_out_fix_multiple_response_writeheader_calls 
Fix multiple response.WriteHeader calls
 2015-04-27 19:21:47 -07:00
Frederick F. Kautz IV db5870be28 Fix multiple response.WriteHeader calls 2015-04-27 18:30:21 -07:00
Frederick F. Kautz IV fc2537eef9 Merge pull request #513 from fkautz/pr_out_adding_software_fallback 2015-04-27 14:41:41 -07:00
Frederick F. Kautz IV 289c859675 Adding software fallback 2015-04-27 13:55:40 -07:00
Harshavardhana ade803a923 Verify if request date is 5minutes late, reject such a request as 
it could be a possible replay attack.

This commit also fixes #505, by returning MethodNotAllowed instead of NotImplemented
 2015-04-27 03:55:10 -07:00
Harshavardhana 9232ce3b4e Now client requests for ACL changes are honored through PutBucketACL API 2015-04-27 03:04:29 -07:00
Harshavardhana c99d96dbc2 Now donut supports bucket ACL's, bringing in this support for dl.minio.io 2015-04-27 02:09:32 -07:00
Harshavardhana 8beca83090 Limit memory mode at 100 buckets and return error appropriately 
~~~
mc: <ERROR> Failed to create bucket for URL [http://localhost:9000/newbucket-101]. Reason: [You have attempted to create more buckets than allowed.].
~~~

Response from server
 2015-04-26 22:15:39 -07:00
Harshavardhana 2d96d5ad57 Enhance error responses for request limit and bring some code from api errors 
~~~
mc: <ERROR> Failed to create bucket for URL [http://localhost:9000/newbucket-101]. Reason: [Reduce your request rate.].
~~~

Client sees proper errors now.
 2015-04-26 22:10:59 -07:00
Frederick F. Kautz IV 8a57006b36 Merge pull request #504 from fkautz/pr_out_making_lock_much_less_aggressive_fixes_multiple_uploads_blocking 2015-04-26 20:11:50 -07:00
Frederick F. Kautz IV a974e4d7c7 Making lock much less aggressive, fixes multiple uploads blocking 2015-04-26 20:08:37 -07:00
Frederick F. Kautz IV 258bf55e31 Bandwidth quota now supports 100-Continue 2015-04-26 20:07:05 -07:00
Frederick F. Kautz IV 9431a54a27 finish me 2015-04-26 19:25:48 -07:00
Frederick F. Kautz IV 202fc26719 Revert "Exporting WriteErrorResponse" 
This reverts commit e05d46a0b5.
 2015-04-26 17:02:49 -07:00
Frederick F. Kautz IV e05d46a0b5 Exporting WriteErrorResponse 2015-04-26 16:28:54 -07:00
Frederick F. Kautz IV 64f7d04cd1 Adding connection limits 2015-04-26 16:14:51 -07:00
Frederick F. Kautz IV 7867ee89fa Bandwidth quotas now work on data returned from server 2015-04-26 15:49:55 -07:00
Frederick F. Kautz IV e2403cbc56 Quotas are more accurate, occur on Read() 2015-04-26 15:07:57 -07:00
Frederick F. Kautz IV bbb493080f Switching back to BigEndian 2015-04-25 18:16:21 -07:00
Frederick F. Kautz IV 3a48f9fe4d Adding quotas based upon type 2015-04-25 18:07:02 -07:00
Harshavardhana b75f46aaa0 Use standard functions to achieve byteorder converstion 2015-04-25 13:18:15 -07:00
Frederick F. Kautz IV e2475925ea quota handlers now log against ip properly against a duration 2015-04-24 20:49:33 -07:00
Harshavardhana b3a9d8b058 Merge pull request #483 from harshavardhana/pr_out_memory_now_handles_submitting_large_files_fixes_482 2015-04-24 15:25:38 -07:00
Harshavardhana c5d8ca245d Memory now handles submitting large files - fixes #482 2015-04-24 15:22:22 -07:00
Frederick F. Kautz IV fda50d52c8 Add initial framework for quota 2015-04-24 00:16:29 -07:00
Frederick F. Kautz IV 5a607a6762 Moving factories and start minio method to server package 2015-04-23 16:13:50 -07:00
Frederick F. Kautz IV 5c515d3138 Simplifying server config handling 2015-04-23 12:44:22 -07:00
Harshavardhana 5ae8d34f76 Donut mode will now take multiple paths as argument, updated docs as well 2015-04-23 11:36:14 -07:00
Harshavardhana a5729e8c02 Disable verifying ACL until we get webcli up and running 2015-04-23 03:21:08 -07:00
Harshavardhana bcc4449de7 Add proper help and several other cleanup 2015-04-23 02:42:50 -07:00
Harshavardhana e95604ff86 Finishing all the test case support for ACL and other fixes 2015-04-23 01:24:46 -07:00
Frederick F. Kautz IV 0f8626ab1a Refactoring cli to take commands 2015-04-23 01:05:44 -07:00
Harshavardhana 848c4ee31c Further fixes for ACL support, currently code is disabled in all the handlers 
Disabled because due to lack of testing support. Once we get that in we can
uncomment them back.
 2015-04-22 19:31:45 -07:00
Harshavardhana 1c0ff2c758 ACL driver/storage layer support 2015-04-22 19:31:45 -07:00
Harshavardhana c8713fd650 Bring in full fledged acl support 2015-04-22 18:26:35 -07:00
Harshavardhana 2c1455af1b Implement x-amz-acl tests 2015-04-22 18:26:35 -07:00
Harshavardhana 107e077ec0 Implement x-amz-acl handling 2015-04-22 18:26:35 -07:00
Frederick F. Kautz IV c56f7380d6 Memory driver now limits by size 2015-04-22 17:50:26 -07:00
Harshavardhana b121c8588f Remove bucketpolicy handlers and all its references 2015-04-22 14:46:51 -07:00
Harshavardhana 3f33643d39 File driver to take the high road 2015-04-16 02:16:39 -07:00
Frederick F. Kautz IV 59a81a73c6 Adding lru to memory driver, not wired to command line opts 2015-04-15 12:49:41 -07:00
Frederick F. Kautz IV 0cab3fd151 Better locking in memory driver 2015-04-15 11:48:56 -07:00
Harshavardhana 148935392d Fix golint complaint 2015-04-08 23:42:59 -07:00
Frederick F. Kautz IV 26d8deee40 Removing unneeded files 2015-04-08 20:58:42 -07:00
Frederick F. Kautz IV 7212b3406a Adding comments to iodine 2015-04-08 19:44:44 -07:00
Frederick F. Kautz IV c12d5ea8a9 Adding iodine error conversion 2015-04-08 19:32:37 -07:00
Harshavardhana 38155e86ac iodine is part of minio now 2015-04-08 18:05:48 -07:00
Harshavardhana 7fd1cc073c Add everything back into one project hood, still missing iodine 2015-04-08 17:47:56 -07:00
Frederick F. Kautz IV 6241c9880b Moving object storage drivers back to minio 2015-04-08 16:28:14 -07:00
Harshavardhana 2a21b7d639 Implement headBucketHandler() and its related tests 
Remove internal testify/mock and use upstream - update all godeps
 2015-04-07 01:16:32 -07:00
Frederick F. Kautz IV ea961ef16d Updating to use objectdriver 2015-04-06 18:00:18 -07:00
Frederick F. Kautz IV e71cc8d833 Adding Debug logger to log package 2015-04-06 12:15:19 -07:00
Harshavardhana 319feb52f1 erasure is now external pull it and update from minio-io/erasure 2015-04-05 23:51:24 -07:00
Frederick F. Kautz IV 052817ff91 Fix fd leak 2015-04-05 21:33:24 -07:00
Harshavardhana d324c4b061 Return proper return for all drivers when BucketNotFound 2015-04-05 15:55:10 -07:00
Harshavardhana 0475d7d056 Migrate to external Donut package 2015-04-05 14:59:24 -07:00
Frederick F. Kautz IV 2f44438843 Reformatting imports 2015-04-04 13:59:59 -07:00
Frederick F. Kautz IV be4bc44b4d Replacing gopkg.in/check.v1 with minio-io/check 2015-04-04 13:37:30 -07:00
Frederick F. Kautz IV e6abca1a63 Adding iodine to object HEAD verb 2015-04-04 12:46:34 -07:00
Frederick F. Kautz IV 46badca07e Removing Godeps from pkg/api 2015-04-03 20:40:21 -07:00
Frederick F. Kautz IV 4171ad8b69 Adding test for invalid range in api 2015-04-03 20:35:39 -07:00
Harshavardhana 4079dd1d12 Commit testify,objx, gorilla/mux, minio-io/erasure gopkg.in/check.v1 2015-04-03 18:57:47 -07:00
Harshavardhana 4b25d4e8db Use verifyError instead of parsResponse, also get rid of mxj dependency 2015-04-03 18:56:47 -07:00
Frederick F. Kautz IV 9954e5c280 Better error handling + tests for get object 2015-04-03 18:53:27 -07:00
Frederick F. Kautz IV db1e6e5988 Adding tests for put bucket errors, fixed mismatching error 2015-04-03 18:37:39 -07:00
Frederick F. Kautz IV ebf1e784b3 Adding iodine and tests for for list buckets errors 2015-04-03 18:29:31 -07:00
Frederick F. Kautz IV 5865295f5d Adding better error support to api 2015-04-03 18:06:30 -07:00
Frederick F. Kautz IV 8757ccf9a3 Fixing race condition in api_test.go 2015-04-01 20:30:57 -07:00
Harshavardhana 85e547d854 Parse response headers in tests 2015-04-01 20:07:51 -07:00
Harshavardhana 92f408af5d Add erasure to godep 2015-04-01 16:56:43 -07:00
Frederick F. Kautz IV 818b6d2e33 Minor cleanup of memory get partial object 2015-04-01 15:50:59 -07:00
Frederick F. Kautz IV 0c5ed6f3ba Add partial object support to donut driver 2015-04-01 15:48:12 -07:00
Frederick F. Kautz IV 563a914046 Fix whitespace 2015-04-01 12:30:58 -07:00
Frederick F. Kautz IV 83f0f0971f Adding comments on why we subtract a second from date1 in an api test 2015-04-01 12:29:08 -07:00
Frederick F. Kautz IV 1f699a094b Minor modification for creating temp dir in api test 2015-04-01 12:07:33 -07:00
Frederick F. Kautz IV 5b1350b4bf api test now states which driver is currently running 2015-04-01 12:06:22 -07:00
Harshavardhana a56098b8c9 Make sure in API we handle all error type exclusively, and return certain values exclusively 2015-03-31 20:26:16 -07:00
Frederick F. Kautz IV 6136960cfd Moving api_test to fixtures for drivers 2015-03-31 19:53:25 -07:00
Frederick F. Kautz IV 959d90fcf7 Adding test for partial objects 2015-03-31 19:07:45 -07:00
Frederick F. Kautz IV 62c4c997c2 Simplifying error handling in api bucket handlers 2015-03-30 14:27:32 -07:00
Frederick F. Kautz IV 318d9d5aa8 API Bucket Handlers errors simplified 2015-03-30 11:40:28 -07:00
Frederick F. Kautz IV 7d657eb55c Simplifying error response writing for object handlers 2015-03-29 22:03:13 -07:00
Frederick F. Kautz IV c3cda5d9a8 Adding comments to errors 2015-03-29 22:03:13 -07:00
Frederick F. Kautz IV ded090a8b0 Normalizing imports in api 2015-03-29 21:41:38 -07:00
Frederick F. Kautz IV 8cbc1c2346 More api cleanup 2015-03-29 21:38:23 -07:00
Frederick F. Kautz IV 452f0f8335 Cleaning up more api code 2015-03-29 21:06:51 -07:00
Frederick F. Kautz IV d644233054 Rename writeObjectHeaders to setObjectHeaders, fix writing multiple headers bug 2015-03-29 18:49:32 -07:00
Harshavardhana b748cf359a More simplification to erasure also add two more test functions 2015-03-29 14:52:13 -07:00
Frederick F. Kautz IV 86b32c2147 Minor change from trace to error log 2015-03-28 18:47:36 -07:00
Frederick F. Kautz IV d5db891094 Simplfying code in api, round 1. Decoupling functions which both write headers and data. 2015-03-28 18:01:00 -07:00
Frederick F. Kautz IV b165efdfcb Refactoring pkg/utils/log, copying from golang's log and modifying 2015-03-28 15:58:48 -07:00
Harshavardhana f3f0a3a5f7 Merge pull request #403 from harshavardhana/pr_out_let_donut_driver_use_log_errorln_wrapper 2015-03-28 12:43:42 -07:00
Harshavardhana 767f4d1d8f Let donut driver use log.Errorln wrapper 2015-03-28 12:43:26 -07:00
Frederick F. Kautz IV c901d0fb32 Fix for readers closing in erasure early 2015-03-27 23:49:14 -07:00
Frederick F. Kautz IV 96e26335e2 Adding tests and a fix for get object where bucket is not found 2015-03-27 23:23:22 -07:00
Frederick F. Kautz IV ad370f341b Fixing donut location 2015-03-27 12:44:13 -07:00
Frederick F. Kautz IV 53439d7768 Updating iodine to match new api 2015-03-26 15:55:06 -07:00
Frederick F. Kautz IV c5f10df957 Removing TODO 2015-03-26 11:33:50 -07:00
Frederick F. Kautz IV eaeeb4bef0 Refactoring erasure, each block is encoded by a function. Iodine included. 2015-03-26 11:32:39 -07:00
Frederick F. Kautz IV 31746fdeb2 Adding iodine to donutObjectWriter.Write 2015-03-26 10:49:03 -07:00
Frederick F. Kautz IV bfbb97c98f Merge pull request #393 from fkautz/pr_out_adding_iodine_to_object_writer 2015-03-26 10:42:57 -07:00
Frederick F. Kautz IV f7a300400e Merge pull request #392 from fkautz/pr_out_adding_iodine_to_node_local 2015-03-26 10:42:50 -07:00
Frederick F. Kautz IV caf7e41499 Adding iodine to object_writer 2015-03-26 10:41:40 -07:00
Frederick F. Kautz IV 5cdf2564a5 Adding iodine to node_local 2015-03-26 10:21:24 -07:00
Frederick F. Kautz IV 70a2217eb2 AddNode fails with proper error code when bucketid is malformed 2015-03-26 10:02:40 -07:00
Frederick F. Kautz IV cc64417481 Adding iodine to donut 2015-03-26 09:57:21 -07:00
Frederick F. Kautz IV 40e311a919 Updating iodine to support new iodine.Error(error, map[string]string) error method 2015-03-25 19:12:55 -07:00
Harshavardhana 68f36f5e08 Merge pull request #385 from abperiasamy/erasure_cache_tbls_matrix 2015-03-25 19:11:07 -07:00
Anand Babu (AB) Periasamy b44d2100db cache decode matrix tables and matrix 2015-03-25 19:09:34 -07:00
Harshavardhana 2d0be3e3b2 fix golint complaints 2015-03-25 19:01:36 -07:00
Anand Babu (AB) Periasamy bd39768de2 janitor duty on erasure-decode 2015-03-25 18:02:03 -07:00
Frederick F. Kautz IV eff48971ad Adding iodine to donut bucket 2015-03-25 17:05:26 -07:00
Frederick F. Kautz IV d52dd5ed5b Adding iodine to pkg/server/server.go 2015-03-25 17:04:26 -07:00
Harshavardhana 58082cd8dc Add gocyclo into source build, choosing cyclomatic complexity tolerance of 15 and below 2015-03-25 15:57:17 -07:00
Harshavardhana c366fdd2d8 loop only on readers once 2015-03-25 14:48:46 -07:00
Harshavardhana e1f4ed4650 Comment exported functions GetEncodedBlocksLen, GetEncodedBlockLen 2015-03-25 13:32:35 -07:00
Harshavardhana 2cbd15b690 Golint fixes 2015-03-25 13:25:25 -07:00
Harshavardhana d967babbaa Remove unnecessary if <block> 2015-03-25 12:58:15 -07:00
Frederick F. Kautz IV be229473b5 Updating iodine 2015-03-25 00:43:48 -07:00
Harshavardhana 0f8021abd1 Fix nil pointer dereference when err is nil 2015-03-25 00:37:25 -07:00
Frederick F. Kautz IV 1be3766ae6 Adding iodine annotation to donut.Start(path) 2015-03-25 00:27:45 -07:00
Frederick F. Kautz IV 12a17f1ed7 Merge pull request #364 from fkautz/pr_out_using_iodine_in_donut_start_ 2015-03-25 00:25:41 -07:00
Frederick F. Kautz IV d06f5afdfc Using iodine in donut.Start() 2015-03-25 00:25:27 -07:00
Harshavardhana b3c3a8d6f2 Avoid empty if else body.. 2015-03-25 00:24:41 -07:00
Harshavardhana ada98c56c0 Fix minio hang in donut driver. 
remove sending err to errorChannel until we figure out the right fix.
 2015-03-25 00:15:36 -07:00
Frederick F. Kautz IV 1ae74591f6 Adding iodine 2015-03-24 23:54:49 -07:00
Frederick F. Kautz IV dd5ba865ff Merge remote-tracking branch 'origin/master' into HEAD 
Conflicts:
	pkg/drivers/donut/donut.go
	pkg/storage/donut/bucket.go
	pkg/storage/donut/donut.go
	pkg/storage/donut/donut_test.go
	pkg/storage/donut/donutdriver.go
 2015-03-24 21:04:08 -07:00
Frederick F. Kautz IV cf2550b267 Populate buckets at start 2015-03-24 19:58:03 -07:00
Harshavardhana 653ea50a59 Erasure technique is now uint8, handle it appropriately 2015-03-24 19:12:52 -07:00
Anand Babu (AB) Periasamy fa9b1f341b erasure encode janitor duty 2015-03-24 18:47:53 -07:00
Harshavardhana 6a87ed001c Handle errors properly during erasure Decoding, also populate technique and verify 2015-03-24 18:14:13 -07:00
Harshavardhana 0474439b43 Code restructuring, assigning proper subsystems to each 
- Drivers contain
    * donut/*
    * file/*
    * memory/*

 - Storage format contains
    * donut/*
      - GetObject() --> renamed to GetObjectReader()
      - Deleted stale objectwriter.go, renamed donutwriter.go to object_writer.go

Simplifying, and documenting codebase further
 2015-03-24 06:47:10 -07:00
Frederick F. Kautz IV 3c4012f1e7 Adding donut backend and setting as default 2015-03-23 20:07:22 -07:00
Harshavardhana 7c1c4b8a02 Merge pull request #351 from harshavardhana/pr_out_implement_log_package_as_drop_in_replacement_for_handling_debug_log_level 2015-03-23 18:55:14 -07:00
Harshavardhana 914962bd93 Implement log package as drop-in replacement for handling Debug log-level 2015-03-23 18:18:11 -07:00
Frederick F. Kautz IV f91f5e59ee Running gofmt manually, auto gofmt didn't simplify code 2015-03-23 12:26:41 -07:00
Frederick F. Kautz IV 56a298fd6a Refactoring method variable names for erasureWriter struct 2015-03-23 12:25:20 -07:00
Frederick F. Kautz IV d57e9d6c18 Adding sys. as reserved in erasureWriter 2015-03-23 11:56:24 -07:00
Frederick F. Kautz IV b2f47ef446 Switching content hashing from sha512 to md5 2015-03-23 11:46:19 -07:00
Harshavardhana b8e9834faa Rename parms as --> params for brevity and misc cleanup 2015-03-22 20:26:01 -07:00
Frederick F. Kautz IV b2c3172095 Adding sha512 support for pre-encoded data and verification on decode 2015-03-22 20:11:25 -07:00
Frederick F. Kautz IV befac7d047 Exorcising donutbox 2015-03-22 19:55:15 -07:00
Frederick F. Kautz IV eb8005cc65 Reading correct block size in erasure coded data 2015-03-22 18:49:30 -07:00
Anand Babu (AB) Periasamy b9b0b35edf removed private k and m and some more cleanups 2015-03-22 18:22:09 -07:00
Frederick F. Kautz IV 293e438830 Merge pull request #341 from abperiasamy/janitor 2015-03-22 17:34:25 -07:00
Anand Babu (AB) Periasamy 3a76229551 encoded size function is now public + some janitor duty 2015-03-22 17:17:53 -07:00
Harshavardhana 3929135c34 Make sure we use O_EXCL with O_CREATE to make sure we don't trip over existing file 2015-03-22 15:55:40 -07:00
Harshavardhana 2fa9320df2 De-couple donut into smaller files, useful for ease in external integration 2015-03-22 15:43:37 -07:00
Frederick F. Kautz IV 55f3659bb3 Removing log tracers 2015-03-22 15:39:50 -07:00
Frederick F. Kautz IV 03f51eac29 Adding list objects to donut 2015-03-22 15:33:52 -07:00
Harshavardhana 4b9fbd3b3a DonutfileWriter should return value of file.Close() 2015-03-22 14:51:18 -07:00
Frederick F. Kautz IV ced6edc9fe Minor change, removing unnecessary commented method 2015-03-22 12:44:45 -07:00
Frederick F. Kautz IV 99cf484488 Adding GetObjectMetadata to donut 2015-03-22 12:36:51 -07:00
Frederick F. Kautz IV 703af6c3ce Removing gob from donut driver 2015-03-22 12:17:12 -07:00
Harshavardhana 95affcb119 Fix build issues for new DonutDriver() 2015-03-22 02:53:21 -07:00
Frederick F. Kautz IV 933d44c656 New donut driver with local file encoding 2015-03-22 00:40:21 -07:00
Harshavardhana a8056c4c12 Return error during JoinFiles back to the caller for inspection 2015-03-20 03:17:08 -07:00
Harshavardhana 2b0b5f014a Update license header Mini should be Minimalist really 2015-03-19 14:35:50 -07:00
Harshavardhana c3602967ab Fix bucketpolicy to handle s3 and minio policy names separately. 
Also fix an apparent bug in GetBucketPolicy{}
 2015-03-19 12:22:14 -07:00
Harshavardhana 6eb5315249 Resource list updates 2015-03-19 00:47:34 -07:00
Harshavardhana a1baed3bbc Refactor to move config handling into api/ 2015-03-18 22:49:02 -07:00
Harshavardhana 7ce3ab3193 Restructure minio api, move signature checks from utils to Api. 2015-03-18 18:16:19 -07:00
Frederick F. Kautz IV 2c69433588 Adding interfaces for donut driver 2015-03-18 14:27:10 -07:00
Frederick F. Kautz IV bf86e78714 Renaming files to match 2015-03-18 12:26:44 -07:00
Frederick F. Kautz IV 1c9e550ee6 Renaming encoded driver to donutstorage 2015-03-18 12:25:37 -07:00
Frederick F. Kautz IV d7f00194b5 Adding interfaces for implementing against 2015-03-17 22:33:59 -07:00
Frederick F. Kautz IV 982a52e0d4 Merge pull request #320 from fkautz/pr_out_removing_block_size_from_donut_writer_no_longer_necessary 2015-03-17 14:25:08 -07:00
Frederick F. Kautz IV d6be8d3517 Removing block size from donut writer, no longer necessary 2015-03-17 14:24:59 -07:00
Harshavardhana 74b3d092f2 Now MinioAPI handles Content-MD5 set during putObject() 
- This change also facilitates proper error response
    in case of md5 mismatch or corruption
  - TODO a test function needs to be implemented
 2015-03-17 13:32:13 -07:00
Harshavardhana 42006c2ab0 Merge pull request #318 from harshavardhana/pr_out_remove_contenttype_pseudo_func_ 2015-03-17 00:47:46 -07:00
Harshavardhana dccf32e5c4 Remove contenttype pseudo func() 2015-03-17 00:47:22 -07:00
Frederick F. Kautz IV 90b337a3e8 Fixing order of input and layout in time.Parse 2015-03-17 00:47:17 -07:00
Frederick F. Kautz IV 25725c442a Moving to more accurate RFC3339 2015-03-17 00:45:50 -07:00
Frederick F. Kautz IV e36fa95f1f Moving from iso8601 to RFC3339 2015-03-17 00:42:08 -07:00
Frederick F. Kautz IV 598eb5192e Encoded now passes tests, enabling. 2015-03-17 00:33:49 -07:00
Harshavardhana 6d71935ea5 Encoded storage type now feature complete 2015-03-16 23:42:45 -07:00
Frederick F. Kautz IV 34cf6682fe list objects in buckets 2015-03-16 20:43:05 -07:00
Frederick F. Kautz IV d99435963b Merge pull request #311 from fkautz/pr_out_switching_from_base64_to_hex_for_etag 2015-03-16 18:39:07 -07:00
Frederick F. Kautz IV 1a88b36c55 Switching from base64 to hex for etag 2015-03-16 18:38:42 -07:00
Harshavardhana 81f1d3c195 Merge pull request #310 from harshavardhana/pr_out_make_blocksize_as_constant 2015-03-16 18:20:22 -07:00
Harshavardhana b495f15178 Make blockSize as constant 2015-03-16 18:20:01 -07:00
Frederick F. Kautz IV 211b084e50 Performing etag off of bytes rather than md5 string 2015-03-16 18:15:37 -07:00
Frederick F. Kautz IV 583c32c774 Metadata now stores hex md5, not base64. Minio API now converts md5 hex to base64. 2015-03-16 18:03:22 -07:00
Frederick F. Kautz IV 10ae089c46 Better handling of metadata in encoded 2015-03-16 17:40:59 -07:00
Harshavardhana 15f68972a5 Some necessary cleanup 
- Rename 'fs' as 'file' for brevity
  - Rename 'inmemory' as 'memory' for brevity
  - Change everywhere else
 2015-03-16 11:46:16 -07:00
Harshavardhana 765fde13f1 Remove x509 package, janitor continuation monad 2015-03-15 19:44:55 -07:00
Harshavardhana bf04d125fd Merge pull request #304 from harshavardhana/pr_out_move_from_fmt_errorf_to_errors_new 2015-03-15 19:42:23 -07:00
Harshavardhana 596eeb097e Move from fmt.Errorf to errors.New 
In many cases fmt.Errorf is good enough, but since
error is an interface, you can use arbitrary data
structures as error values, to allow callers to
inspect the details of the error.
 2015-03-15 19:41:16 -07:00
Frederick F. Kautz IV a589ddaddf Decoding and returning objects now works 2015-03-15 19:02:00 -07:00
Frederick F. Kautz IV 682bf085c4 Metadata and writing the object are now both committed together. 2015-03-15 17:42:33 -07:00
Frederick F. Kautz IV 96d0a089b8 Adding create bucket to encoded 2015-03-15 12:28:24 -07:00
Harshavardhana 3f95c55067 Skip uinimplemented tests and some cleanup 2015-03-15 01:23:57 -07:00
Frederick F. Kautz IV 3e453a394a Working in memory donut driver 2015-03-15 00:41:06 -07:00
Frederick F. Kautz IV 15d5e74837 More testing 2015-03-14 19:29:54 -07:00
Frederick F. Kautz IV 7e61696901 Inmemory donutbox is now thread safe 2015-03-14 19:21:34 -07:00
Frederick F. Kautz IV 85da9fc627 Adding start of inmem donut 2015-03-14 18:02:27 -07:00
Frederick F. Kautz IV 54472c18fe Renaming singledisk to encoded 2015-03-14 11:37:43 -07:00
Frederick F. Kautz IV 348c4d9a88 Removing old donut format 2015-03-14 11:36:19 -07:00
Frederick F. Kautz IV 2f4b92eb8d Minor refactor to prepare for file version of donut 2015-03-14 11:34:45 -07:00
Harshavardhana f7b62caccc Update erasure README with supported architecture list 2015-03-12 16:37:08 -07:00
Harshavardhana f7fd526b28 Cosmetic changes 2015-03-12 02:11:04 -07:00
Harshavardhana 1e4a47ac5f Merge pull request #290 from harshavardhana/pr_out_add_crc32c_license_headers_golang_and_minio 2015-03-12 02:09:06 -07:00
Harshavardhana 4c86e594d6 Add crc32c license headers Golang and Minio 2015-03-12 02:05:05 -07:00
Harshavardhana e27ba44d94 Remove minio_calc_chunk_size() C code to write one in Golang 2015-03-12 02:03:04 -07:00
Frederick F. Kautz IV 94fe03cfc1 More refactoring to meet new donut spec 2015-03-12 01:52:14 -07:00
Harshavardhana 67888cbaa9 Restructure and use human readable names 2015-03-12 00:36:33 -07:00
Harshavardhana e21020b337 Merge pull request #280 from harshavardhana/pr_out_http_range_support_wip 
Get object range request added
 2015-03-12 00:11:38 -07:00
Harshavardhana a42a48a455 Get object range request added 2015-03-12 00:07:27 -07:00
Frederick F. Kautz IV 24d5018037 Rename StoreBucket to CreateBucket 2015-03-11 20:14:40 -07:00
Frederick F. Kautz IV 8c6aa45458 Adding copyright 2015-03-11 19:59:20 -07:00
Frederick F. Kautz IV 7971b66e60 Adding object donut type 2015-03-11 19:56:18 -07:00
Frederick F. Kautz IV ec9b20a94f Adding stubes for new single disk driver 2015-03-11 13:36:21 -07:00
Harshavardhana a24bfb007d Rename dateFormat with understandable name ``iso8601Format`` 2015-03-10 01:51:34 -07:00
Harshavardhana b952855779 Add strict checks with vet, golint and gofmt 2015-03-09 16:17:38 -07:00
Harshavardhana 41653774fc Simplify fs codebase, split them into separate files and more commenting 2015-03-08 18:01:48 -07:00
Harshavardhana 01e98eb326 Make BucketPolicy strictly typed 2015-03-08 15:57:20 -07:00
Harshavardhana 8866b2cc5f Add delimiter and prefix tests, also add inmemory to support delimiters 2015-03-08 14:51:31 -07:00
Harshavardhana 7a01db8005 Strict checks for delimiter and prefix if absent or present 2015-03-08 13:01:39 -07:00
Harshavardhana ba85ed7566 Revert "RFC1123 is wrong protocol format for AWS" 
This reverts commit c7b1f8ded8.
 2015-03-08 03:06:23 -07:00
Harshavardhana c7b1f8ded8 RFC1123 is wrong protocol format for AWS 2015-03-08 01:55:25 -08:00
Harshavardhana 9e89af0e3c Remove redundant uint32 2015-03-07 19:15:51 -08:00
Harshavardhana 9553eb6b4a Merge pull request #269 from harshavardhana/pr_out_get_object_metadata_needs_to_filter_based_of_prefix_when_delimiter_is_specified 2015-03-07 19:14:41 -08:00
Harshavardhana f8e2b75e23 Get object metadata needs to filter based of Prefix when delimiter is specified 2015-03-07 19:13:15 -08:00
Frederick F. Kautz IV b39eac36ea Merge pull request #268 from fkautz/pr_out_fragment_reader_implemented 2015-03-07 18:04:29 -08:00
Frederick F. Kautz IV 679c6ace3a Fragment reader implemented 2015-03-07 18:04:20 -08:00
Harshavardhana df69038b04 Merge pull request #267 from harshavardhana/pr_out_add_md5_support_for_fsstorage_also_re_purpose_getobjectmetadata_for_listobjects 2015-03-07 17:16:00 -08:00
Harshavardhana 4751ed0e30 Add MD5 support for fsStorage, also re-purpose GetObjectMetadata for ListObjects 2015-03-07 17:15:00 -08:00
Frederick F. Kautz IV 1ec0cd6683 Merge pull request #266 from fkautz/pr_out_adding_read_header_for_erasure 2015-03-07 15:54:17 -08:00
Frederick F. Kautz IV ca1a4b616c Adding read header for erasure 2015-03-07 15:53:41 -08:00
Frederick F. Kautz IV 00381c887a Adding erasure package to wrap versioned erasure. Ensures we always write latest version. 2015-03-07 13:04:02 -08:00
Harshavardhana bf5a314144 clang lacks proper gas support, implement stubs for sha256,sha512 for darwin. 
REF: http://llvm.org/bugs/show_bug.cgi?id=18918
 2015-03-07 02:55:48 -08:00
Harshavardhana 1637138f86 If incoming request IP bucketfromHostname needs to be empty string 2015-03-07 02:53:29 -08:00
Harshavardhana a18bfa2fda Rename test-data to testdata, go will ignore it automatically 2015-03-07 01:32:35 -08:00
Harshavardhana d992bccd9a Move pkg/storage/erasure to pkg/encoding/erasure - and other cleanups 2015-03-07 00:16:48 -08:00
Frederick F. Kautz IV 856e0100c0 Erasure layer now writes using new technique 2015-03-06 16:37:44 -08:00
Harshavardhana 856b7680df Now able to use crc32c and sha1 on Darwin, with OSX specific code 2015-03-06 02:38:29 -08:00
Harshavardhana e5af8a3f5d Further fixes - 
- All test files have been renamed to their respective <package>_test name,
    this is done in accordance with
      - https://github.com/golang/go/wiki/CodeReviewComments#import-dot
        imports are largely used in testing, but to avoid namespace collision
        and circular dependencies

  - Never use _* in package names other than "_test" change fragment_v1 to expose
    fragment just like 'gopkg.in/check.v1'
 2015-03-06 02:04:21 -08:00
Harshavardhana c00d1461b9 Golint cleanup from top level 2015-03-05 23:37:37 -08:00
Harshavardhana 540723d7ae Golint cleanup pkg/api 2015-03-05 21:09:19 -08:00
Harshavardhana 76e601b26b Golint cleanup pkg/storage 2015-03-05 21:09:19 -08:00
Harshavardhana 256faddab5 Golint cleanup utils/crypto/sha*,cpu,md5 2015-03-05 21:09:19 -08:00
Harshavardhana 66e31445ff Golint cleanup pkg/utils/crypto/keys 2015-03-05 21:09:19 -08:00
Harshavardhana b33e2d2f9b Golint cleanup pkg/utils/config 2015-03-05 21:09:19 -08:00
Harshavardhana 861f13d5bd Golint cleanup pkg/utils/cpu 2015-03-05 21:09:19 -08:00
Harshavardhana c36450a83a Golint cleanup pkg/utils/policy 2015-03-05 21:09:19 -08:00
Harshavardhana 3e321b6631 Golint cleanup pkg/utils/split 2015-03-05 21:09:19 -08:00
Frederick F. Kautz IV 1186c47603 Fragment now implements new version 2015-03-05 20:56:02 -08:00
Frederick F. Kautz IV 665af44c70 Renaming donut packages to match layout 2015-03-05 19:44:01 -08:00
Frederick F. Kautz IV 6ec7894312 Removing innerxml 2015-03-04 14:24:56 -08:00
Frederick F. Kautz IV 6f7840677a Fixing xml tags 2015-03-04 11:02:37 -08:00
Harshavardhana f2e238199b Handle individual objects too 2015-03-04 01:54:35 -08:00
Harshavardhana e400e85639 Add proper path prefix to be delimited 2015-03-04 00:54:39 -08:00
Harshavardhana df135d7307 Merge pull request #249 from harshavardhana/pr_out_pathprefix_is_filtered_out_properly_and_delimited_as_well 2015-03-03 23:08:30 -08:00
Harshavardhana cc46475c86 Pathprefix is filtered out properly and delimited as well 2015-03-03 19:45:54 -08:00
Frederick F. Kautz IV 34c1e58332 Renaming Package to goPackage to fix golint error 2015-03-03 12:56:19 -08:00
Harshavardhana 0ebaac3594 Janitor duty v2 2015-03-03 02:43:24 -08:00
Harshavardhana 3a3c8645fc Update comments across the codebase 2015-03-03 02:39:38 -08:00
Harshavardhana 137584d658 Add comments 2015-03-03 01:25:45 -08:00
Harshavardhana c3ad0906e0 Add deadcode code which recursivley goes into all directories and verifies dangling variables. 2015-03-02 18:44:20 -08:00
Frederick F. Kautz IV 32f8c17174 Renaming AppendUstr to AppendUniqStr 2015-03-01 17:06:00 -08:00
Anand Babu (AB) Periasamy 81ede6c3fe Merge pull request #241 from abperiasamy/janitor 
On janitor duty.
 2015-03-01 16:43:04 -08:00
Frederick F. Kautz IV de6131cc7c Renaming AppendUInt to AppendUniqInt 2015-03-01 16:39:45 -08:00
Anand Babu (AB) Periasamy d9c9634138 removed custom Assert 2015-03-01 15:28:42 -08:00
Frederick F. Kautz IV 6f300c9ef0 Adding test at byte level 2015-03-01 15:13:09 -08:00
Anand Babu (AB) Periasamy 177121868b removed HomeDir 2015-03-01 15:13:08 -08:00
Frederick F. Kautz IV 0b08312ddd Specifying more exact types for data 2015-03-01 14:43:11 -08:00
Harshavardhana 9f43659135 Add tests, add NewHeader() for populating Header and ValidateHeader() 2015-03-01 13:19:06 -08:00
Anand Babu (AB) Periasamy 6515678e87 Merge pull request #236 from fkautz/pr_out_adding_data_writer 
Adding data writer
 2015-03-01 11:36:35 -08:00
Frederick F. Kautz IV 7414dc9b75 Adding data writer 2015-03-01 11:32:30 -08:00
Harshavardhana eb9a4a83ac Move AppendUstr(), AppendUint() into helpers 2015-03-01 11:19:03 -08:00
Frederick F. Kautz IV be12a6b3d9 Changing K and M in data.go to match new datatype in erasure package 2015-03-01 01:28:33 -08:00
Harshavardhana 33601ff567 Merge pull request #233 from harshavardhana/pr_out_make_k_m_to_be_uint8_and_technique_becomes_its_own_type 2015-03-01 01:26:42 -08:00
Harshavardhana b77c62e25d Make K,M to be uint8 and Technique becomes its own type 2015-03-01 01:25:28 -08:00
Frederick F. Kautz IV bb0441917c Making changes to data based on discussion 2015-03-01 01:16:22 -08:00
Harshavardhana 2f04a1299e Avoid allocating 1gig memory, avoid such mistakes 2015-03-01 00:43:12 -08:00
Frederick F. Kautz IV 2916b1278e Adding data.go, does nothing for now. 2015-03-01 00:27:52 -08:00
Frederick F. Kautz IV b4ef522a9b Donut frame is now its own package 2015-03-01 00:02:14 -08:00
Frederick F. Kautz IV 6c2428833a Renaming Write to WriteFrame 2015-02-28 23:52:56 -08:00
Harshavardhana 6b397844b9 Add donut benchmark for 64MB, 128MB, 256MB, 512MB 
Current average values

~~~
> minio/pkg/storage/donut/v1 *> go test -bench .
OK: 2 passed
PASS
BenchmarkDonut64M              5         266249634 ns/op         252.05 MB/s
BenchmarkDonut128M             3         436507394 ns/op         307.48 MB/s
BenchmarkDonut256M             2         836116359 ns/op         321.05 MB/s
BenchmarkDonut512M             1        1676738951 ns/op         320.19 MB/s
ok      github.com/minio-io/minio/pkg/storage/donut/v1  8.430s
> minio/pkg/storage/donut/v1 *>
~~~
 2015-02-28 17:05:46 -08:00
Harshavardhana 65f22083ca donut_gen works with new Donut Frame 2015-02-28 16:30:18 -08:00
Frederick F. Kautz IV 4d98c355c9 Merge pull request #224 from abperiasamy/units 2015-02-28 16:12:10 -08:00
Frederick F. Kautz IV ddc7cf835e Donut now tests amount written should match expected amount to write 2015-02-28 16:09:52 -08:00
Frederick F. Kautz IV d3d1205eb1 Updating comment describing data type 2015-02-28 15:59:52 -08:00
Frederick F. Kautz IV 58a04ee831 New donut frame implemented 2015-02-28 15:54:13 -08:00
Anand Babu (AB) Periasamy 317a1141c0 removed unitconv 2015-02-28 15:53:44 -08:00
Frederick F. Kautz IV 90cd8b7b47 Adding streaming sha512 2015-02-28 15:51:33 -08:00
Harshavardhana 56bd413a16 Merge pull request #219 from harshavardhana/pr_out_add_1mb_block_benchmarks 2015-02-28 15:25:24 -08:00
Harshavardhana 7feca3f0d1 Add 1MB block benchmarks 2015-02-28 15:00:24 -08:00
Harshavardhana 0c2d58bc6d Implement proper delimiter and prefix handling 
With this change Minio server now responds with, delimited
'object names' in conjunction with prefix filtering

~~~
<ListBucketResult>
  <Name>example-bucket</Name>
  <Prefix></Prefix>
  <Marker></Marker>
  <MaxKeys>1000</MaxKeys>
  <Delimiter>/</Delimiter>
  <IsTruncated>false</IsTruncated>
  <Contents>
    <Key>sample.html</Key>
    <LastModified>2011-02-26T01:56:20.000Z</LastModified>
    <ETag>example-bucket#sample.html</ETag>
    <Size>142863</Size>
    <Owner>
      <ID>minio</ID>
      <DisplayName>minio</DisplayName>
    </Owner>
    <StorageClass>STANDARD</StorageClass>
  </Contents>
  <CommonPrefixes>
    <Prefix>photos/</Prefix>
  </CommonPrefixes>
</ListBucketResult>
~~~

~~~
<ListBucketResult>
  <Name>example-bucket</Name>
  <Prefix>photos/2006/</Prefix>
  <Marker></Marker>
  <MaxKeys>1000</MaxKeys>
  <Delimiter>/</Delimiter>
  <IsTruncated>false</IsTruncated>

  <CommonPrefixes>
    <Prefix>photos/2006/feb/</Prefix>
  </CommonPrefixes>
  <CommonPrefixes>
    <Prefix>photos/2006/jan/</Prefix>
  </CommonPrefixes>
</ListBucketResult>
~~~
 2015-02-28 14:47:19 -08:00
Harshavardhana d5e5a3b3e6 Use validString 2015-02-27 19:49:18 -08:00
Harshavardhana 6ebb48b4ea Add object name validation 2015-02-27 19:42:04 -08:00
Harshavardhana 9acb1f728c Sort list of objects 2015-02-27 17:52:55 -08:00
Harshavardhana 53669a0854 Implement delimiter, path prefix 2015-02-27 16:23:05 -08:00
Harshavardhana 3f8b3462e6 Remove donut_gen binary and update crc32c function 2015-02-27 14:41:53 -08:00
Harshavardhana 9aea7fd97b Merge pull request #213 from harshavardhana/pr_out_crc32c_re_implementation_with_convenience_functions 2015-02-27 14:35:11 -08:00
Harshavardhana 8ce9b84b69 Crc32c re-implementation with Convenience functions 
- Sum32([]byte) --> uint32
   - Sum(io.Reader) --> uint32, error
 2015-02-27 13:39:17 -08:00
Frederick F. Kautz IV 2e37791681 Merge pull request #212 from fkautz/pr_out_adding_comparison_for_footer_s_inim 2015-02-27 12:40:10 -08:00
Frederick F. Kautz IV 1f7a3c13a5 Adding comparison for footer's INIM 2015-02-27 12:39:52 -08:00
Frederick F. Kautz IV 2ca62f13a5 Moving donut_gen to its own dir and renaming it as donut_gen_v1 2015-02-27 12:39:14 -08:00
Frederick F. Kautz IV be768d01a2 Seek to beginning for tempbuffer 2015-02-27 12:18:27 -08:00
Frederick F. Kautz IV 5c52da6ebe Adding crc32c to write 2015-02-26 23:04:07 -08:00
Harshavardhana a9aaff0ee0 Remove HTTP server timeouts write and read. 
Implement it later with a proper timer thread. Large file
transfers from 'mc' fail with i/o timeout without this change.
 2015-02-24 13:55:24 -08:00
Frederick F. Kautz IV ef442289da Making donut use io.ReadWriteSeek instead of io.Writer 2015-02-23 20:27:25 -08:00
Harshavardhana 5e1e5ad786 More updates on documentation 2015-02-23 17:44:55 -08:00
Harshavardhana b7868d23ac Update minioapi documentation 2015-02-23 16:46:48 -08:00
Harshavardhana f4b5519451 Update erasure documentation 2015-02-23 13:39:16 -08:00
Anand Babu (AB) Periasamy de41e465c8 move httpserver under server 2015-02-23 11:03:01 -08:00
Harshavardhana 51e80eaa6d Add domain and subdomain support for MinioAPI 
This change brings in domain and subdomain support

   - ./minio --domain "yourminiodomain.com"

This change brings in a much needed feature by keeping
bucketnames as part of your 'DNS' name.

All your existing applications can be migrated off from s3 to
Minio without little to no modifications.

NOTE: Setting up DNS for your `buckets` is out of scope of this feature
 2015-02-23 02:25:01 -08:00
Harshavardhana a92c6cb2b2 Purge tiedot wrapper 2015-02-22 20:02:09 -08:00
Harshavardhana 49a714d1a4 Add license header for Minio modifications, improvements for sha1,sha256,sha512 implementations 
Also bring in SSE3 optimized public domain implementation of SHA1 from Intel ``FIPS PUB 180-1``
 2015-02-21 15:50:47 -08:00
Harshavardhana 12cff1be58 Implement sha1 as intel optimized set 2015-02-21 11:25:09 -08:00
Anis Elleuch 1d4d7ff274 Add missing asm instruction to correct sha256 calculation in avx mode 2015-02-21 19:34:36 +01:00
Anis Elleuch e89271b8f5 Use Intel optimized code (ssse3, avx, avx2) to calculate sha256 2015-02-21 13:11:15 +01:00
Harshavardhana 3151f9b6c3 Add proper entries for GobHeader, Data copy. 
This commit also adds 'donut_gen.go' for generating sample donut file
 2015-02-21 01:44:27 -08:00
Frederick F. Kautz IV 8129d8fd0b Writing data out with error handling 2015-02-20 23:24:01 -08:00
Frederick F. Kautz IV 98ddbd840e Moving donut code to v1 in donut package 2015-02-20 23:07:43 -08:00
Frederick F. Kautz IV 9833647fb7 Fixing magic number for INIM 2015-02-20 22:42:58 -08:00
Anand Babu (AB) Periasamy ec36a598c9 Magic as numbers are faster to encode and compare 2015-02-20 18:20:21 -08:00
Harshavardhana 741e14f749 Remove go cover and remove unncessary Makefiles 2015-02-20 17:44:26 -08:00
Harshavardhana 19a4998fcb SHA512 Implemention with Intel assembly code 2015-02-20 16:32:19 -08:00
Harshavardhana c4f0aad9b5 Add donut tests, just for verification 2015-02-18 19:46:14 -08:00
Frederick F. Kautz IV f7f34d08e8 Donut structure refined 2015-02-18 17:18:33 -08:00
Frederick F. Kautz IV ab80c0ce12 Merge pull request #180 from abperiasamy/donut 2015-02-18 15:57:16 -08:00
Anand Babu (AB) Periasamy 9e304b53de donut spec v1 prototype 2015-02-18 15:52:12 -08:00
Harshavardhana fab954f13f Run govet and fix 2015-02-18 15:33:55 -08:00
Harshavardhana f7a06a5d1e Some more cleanup 2015-02-18 12:15:33 -08:00
Frederick F. Kautz IV 05ac09bf69 Adding header version 2015-02-18 10:21:02 -08:00
Harshavardhana a3e5121f7b Make channels more aware of errors and shutdown on error - some cleanup 2015-02-18 00:24:19 -08:00
Harshavardhana cf6d03b907 Expose policy code, for api router usage 2015-02-17 19:06:01 -08:00
Harshavardhana 5235badec9 Merge pull request #169 from harshavardhana/pr_out_add_policy_validation_code_more_rigorous 2015-02-17 18:10:00 -08:00
Frederick F. Kautz IV fe9a5c14a0 Adding scaffolding for writing donut 2015-02-17 17:47:10 -08:00
Harshavardhana 980d229272 Add policy validation code more rigorous 2015-02-17 13:23:16 -08:00
Harshavardhana f04beab7c2 Skip policy files 2015-02-16 18:37:09 -08:00
Harshavardhana eeae64935e Implement bucket policy handler and with galore of cleanup 2015-02-16 18:20:40 -08:00
Harshavardhana ba3e2d41f7 Add license and fix development scripts 2015-02-15 01:59:48 -08:00
Harshavardhana 966cdd4f05 Reply in xml from validate and ignore handlers 2015-02-11 22:00:45 -08:00
Harshavardhana ac4f07906c Implement S3 Style ErrorCodes and Response 
This patchset also brings in lot of cleanup in terms of minioapi codebase
 2015-02-11 21:27:28 -08:00
Harshavardhana 680848bdcb Add license header 2015-02-08 03:00:32 -08:00
Harshavardhana 8087ca2450 Verify both 'x-amz-date' and 'date' header before discarding request 2015-02-08 02:37:19 -08:00
Harshavardhana 92f83982db Changes to tests pending from #142 2015-02-06 20:47:21 -08:00
Harshavardhana ecb70a4c19 Avoid trailing '/' for buckets and also re-directs - in accordance with aws clients 2015-02-06 20:42:09 -08:00
Harshavardhana 81fc11ee5d Implement authorization support 2015-02-06 02:08:52 -08:00
Harshavardhana 0e8b16a55d Ignore EOF in ReadConfig() 2015-02-05 16:10:49 -08:00
Harshavardhana 72d0999dcb Reply back AccessKey, Secretkey through json 2015-02-05 15:59:44 -08:00
Harshavardhana 95f214c99b Append to Users list 2015-02-05 15:40:51 -08:00
Harshavardhana b3b73ff4cb Merge pull request #136 from harshavardhana/pr_out_store_credentials_through_webui 2015-02-05 14:20:06 -08:00
Harshavardhana 4c90017b02 Store credentials through webui 2015-02-05 14:19:48 -08:00
Frederick F. Kautz IV 620435cea6 Content type is now passed from minioapi to storage 2015-02-05 14:05:43 -08:00
Frederick F. Kautz IV 288bc3fcc7 Adding content type to backend storage drivers 2015-02-04 17:32:40 -08:00
Frederick F. Kautz IV 816310825e Removing XMLName from json request 2015-02-04 14:21:47 -08:00
Frederick F. Kautz IV 7152712f34 Accessing dir as object should return ObjectNotFound 2015-02-03 16:45:00 -08:00
Harshavardhana 582ce3df36 Make sure config testing is done in tempdir 2015-02-03 00:25:33 -08:00
Harshavardhana b1db70c807 Add json config reader/writer 2015-02-03 00:12:46 -08:00
Harshavardhana aff4c5826b Use helper HomeDir() 2015-02-02 21:44:17 -08:00
Harshavardhana dc47ca1431 Fix build issue 2015-02-01 17:21:17 -08:00
Frederick F. Kautz IV a8eef13835 Minioapi TestPutObject properly creates bucket at start of test 2015-02-01 14:46:45 -08:00
Frederick F. Kautz IV a092b44cf1 Adding test to ensure list objects returns an error for non existant buckets to match previous bug fix and fix for inmemory 2015-02-01 10:48:31 -08:00
Harshavardhana 5d197dd063 Reply back errors properly 2015-02-01 03:10:28 -08:00
Harshavardhana b635501fb7 Add StrictSlash for path re-direction 2015-02-01 02:49:09 -08:00
Harshavardhana e1b3836c60 Service a new WebUI server at port 8081. 
It is provided to avoid the router clutter on the Object API
end. This is just a first cut sample we are going to build on
top of this the new management console UI.

General interest comes from subsequently writing a small UI to
generate `AccessKey` and `SecretKey` to build authorization
layer.
 2015-02-01 02:44:53 -08:00
Frederick F. Kautz IV 54026dbc78 Setting order when retrieving buckets 2015-02-01 00:16:25 -08:00
Harshavardhana 26aac9a439 Add bucket resources to ignore handler 2015-01-31 22:45:52 -08:00
Frederick F. Kautz IV 94e124470c Converting unimplemented methods list into an http handler 2015-01-31 19:50:32 -08:00
Harshavardhana 46751f0b28 Add database query function, and some re-factoring 2015-01-31 15:51:18 -08:00
Frederick F. Kautz IV 0cd54031a7 Adding list api test 2015-01-30 17:36:53 -08:00
Harshavardhana 8a4128d304 Force clients to have certs 2015-01-30 16:51:33 -08:00
Frederick F. Kautz IV 84c5df8441 Using generic make test ./... and make build ./..., disabled signers code for now to fix build. 2015-01-30 10:58:47 -08:00
Frederick F. Kautz IV 7ca87e06aa Adding list bucket tests and fixing inmemory behavior. 2015-01-29 11:23:14 -08:00
Harshavardhana 82baccb8bf Prefix if empty skip it before IsValidBucket 2015-01-29 01:07:37 -08:00
Frederick F. Kautz IV c6d537e66e Reducing code duplication relating to content type 2015-01-28 20:47:23 -08:00
Frederick F. Kautz IV 96a5fc0ee7 Updating godep 2015-01-28 20:26:52 -08:00