4e34e03dd4
xl/fs: Split object layer into interface. ( #1415 )
2016-04-29 14:24:10 -07:00
a98a7fb1ad
Implement XL layer - preliminary work.
2016-04-25 12:47:31 -07:00
ff4e04d942
atomic/fs: use safe package for atomic writes, even in multipart.
2016-04-06 16:05:30 -07:00
379e0abf03
cleanup: Remove old donut/xl code and erasure implementation.
...
This is a change to bring in 'klauspost/reedsolomon' library
in #1270 patch.
2016-04-02 17:30:35 -07:00
6037fe66e9
minio: Simplify for `gosimple` tool complaints.
2016-04-02 17:28:54 -07:00
efc80343e3
fs: Break fs package to top-level and introduce ObjectAPI interface.
...
ObjectAPI interface brings in changes needed for XL ObjectAPI layer.
The new interface for any ObjectAPI layer is as below
```
// ObjectAPI interface.
type ObjectAPI interface {
// Bucket resource API.
DeleteBucket(bucket string) *probe.Error
ListBuckets() ([]BucketInfo, *probe.Error)
MakeBucket(bucket string) *probe.Error
GetBucketInfo(bucket string) (BucketInfo, *probe.Error)
// Bucket query API.
ListObjects(bucket, prefix, marker, delimiter string, maxKeys int) (ListObjectsResult, *probe.Error)
ListMultipartUploads(bucket string, resources BucketMultipartResourcesMetadata) (BucketMultipartResourcesMetadata, *probe.Error)
// Object resource API.
GetObject(bucket, object string, startOffset int64) (io.ReadCloser, *probe.Error)
GetObjectInfo(bucket, object string) (ObjectInfo, *probe.Error)
PutObject(bucket string, object string, size int64, data io.Reader, metadata map[string]string) (ObjectInfo, *probe.Error)
DeleteObject(bucket, object string) *probe.Error
// Object query API.
NewMultipartUpload(bucket, object string) (string, *probe.Error)
PutObjectPart(bucket, object, uploadID string, partID int, size int64, data io.Reader, md5Hex string) (string, *probe.Error)
ListObjectParts(bucket, object string, resources ObjectResourcesMetadata) (ObjectResourcesMetadata, *probe.Error)
CompleteMultipartUpload(bucket string, object string, uploadID string, parts []CompletePart) (ObjectInfo, *probe.Error)
AbortMultipartUpload(bucket, object, uploadID string) *probe.Error
}
```
2016-04-01 15:58:39 -07:00
1502e2f29f
Merge pull request #1265 from vadmeste/add_fbsd_support
...
Add simple FreeBSD support, make the minio project compilable
2016-03-26 22:28:07 -07:00
9dca46e156
signature: Use a layered approach for signature verification.
...
Signature calculation has now moved out from being a package to
top-level as a layered mechanism.
In case of payload calculation with body, go-routines are initiated
to simultaneously write and calculate shasum. Errors are sent
over the writer so that the lower layer removes the temporary files
properly.
2016-03-26 15:21:05 -07:00
663f24064b
Add simple FreeBSD support, make the minio project compilable
2016-03-26 22:39:34 +01:00
aaf97ea02c
config/main: Re-write config files - add to new config v3
...
- New config format.
```
{
"version": "3",
"address": ":9000",
"backend": {
"type": "fs",
"disk": "/path"
},
"credential": {
"accessKey": "WLGDGYAQYIGI833EV05A",
"secretKey": "BYvgJM101sHngl2uzjXS/OBF/aMxAN06JrJ3qJlF"
},
"region": "us-east-1",
"logger": {
"file": {
"enable": false,
"fileName": "",
"level": "error"
},
"syslog": {
"enable": false,
"address": "",
"level": "debug"
},
"console": {
"enable": true,
"level": "fatal"
}
}
}
```
New command lines in lieu of supporting XL.
Minio initialize filesystem backend.
~~~
$ minio init fs <path>
~~~
Minio initialize XL backend.
~~~
$ minio init xl <url1>...<url16>
~~~
For 'fs' backend it starts the server.
~~~
$ minio server
~~~
For 'xl' backend it waits for servers to join.
~~~
$ minio server
... [PROGRESS BAR] of servers connecting
~~~
Now on other servers execute 'join' and they connect.
~~~
....
minio join <url1> -- from <url2> && minio server
minio join <url1> -- from <url3> && minio server
...
...
minio join <url1> -- from <url16> && minio server
~~~
2016-03-23 19:16:09 -07:00
c8570edaab
Issue of 'mc ls' when prefix is a directory fixed, tests added for GetObjectInfo
2016-03-23 19:42:04 +05:30
7be79b507b
Changing to empty response when prefixDir doesn't exist
2016-03-23 04:46:10 +05:30
ff41c050d5
Formatting issues fixed.
2016-03-22 15:55:29 +05:30
b55922effe
Fix for Istruncated set to true under certain conditions.
...
Optimizing List Objects by using binary sort to discard entries in cases
where prefix or marker is set.
Adding test coverage to ListObjects.
Adding benchmark to ListObjects.
2016-03-22 10:09:16 +05:30
99af0444b7
Handling maxKeys=0 case with a empty response
2016-03-18 15:16:30 +05:30
88714e7c8e
bucketpolicy: Improve bucket policy validation, avoid nested rules.
...
Bucket policy validation is more stricter now, to avoid nested
rules. The reason to do this is keep the rules simpler and more
meaningful avoiding conflicts.
This patch implements stricter checks.
Example policy to be generally avoided.
```
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"s3:GetObject",
"s3:DeleteObject"
],
"Effect": "Allow",
"Principal": {
"AWS": [
"*"
]
},
"Resource": [
"arn:aws:s3:::jarjarbing/*"
]
},
{
"Action": [
"s3:GetObject",
"s3:DeleteObject"
],
"Effect": "Deny",
"Principal": {
"AWS": [
"*"
]
},
"Resource": [
"arn:aws:s3:::jarjarbing/restic/key/*"
]
}
]
}
```
2016-03-15 17:50:23 -07:00
583e4ecff6
pkg/fs: optimize GetObject syscalls for common case
...
In the common case, GetObject is called on a bucket that exists and an
object that exists and is not a directory. It should be optimized for
this case, thus error-related syscalls are pushed back until they are
necessary.
This should not impact performance negatively in the uncommon case, and
instead drops two otherwise unnecessary os.Stat's in the common case.
The race conditions around a proper error being returned were present
beforehand.
It also renames 'err' to 'e'.
2016-03-13 13:56:33 -07:00
b2257682e4
pkg/fs: add benchmark for GetObject
...
This commit adds a benchmark for GetObject. It doesn't leverage the I/O
as much because it uses short text for data, just 58 chars.
2016-03-13 11:13:06 -07:00
53a76439a2
test for GetBucketInfo
2016-03-12 14:31:30 +05:30
5282a79eda
cleanup: Remove unecessary packages and tests. Simplify.
2016-03-11 19:53:55 -08:00
52751d81cb
cleanup: Rename ObjectMetadata as ObjectInfo.
...
Fixes #1215
2016-03-11 16:58:08 -08:00
ec8c1d4ef6
Better structuring of fs-utils test
2016-03-11 19:19:47 +05:30
fdf3d64793
error: Add proper prefixes for s3Error codes.
...
This patch adds 'Err' prefix for all standard API
error codes and also adds a proper type for them.
2016-03-10 18:38:46 -08:00
373d335d94
Merge pull request #1214 from brendanashworth/improve-listbuckets
...
ListBuckets test & improvement, IsValid{Bucket,Object}Name fix, test, docs
2016-03-10 18:20:18 -08:00
c70bc2209e
api: refactor list object handling in fs backend
...
When list object is invoked, it creates a goroutine if not available
for given parameters else uses existing goroutine. These goroutines
are alive for 15 seconds for further continuation list object request
else they exit.
Fixes #1076
2016-03-11 02:20:51 +05:30
d5057b3c51
accessPolicy: Implement Put, Get, Delete access policy.
...
This patch implements Get,Put,Delete bucket policies
Supporting - http://docs.aws.amazon.com/AmazonS3/latest/dev/access-policy-language-overview.html
Currently supports following actions.
"*": true,
"s3:*": true,
"s3:GetObject": true,
"s3:ListBucket": true,
"s3:PutObject": true,
"s3:CreateBucket": true,
"s3:GetBucketLocation": true,
"s3:DeleteBucket": true,
"s3:DeleteObject": true,
"s3:AbortMultipartUpload": true,
"s3:ListBucketMultipartUploads": true,
"s3:ListMultipartUploadParts": true,
following conditions for "StringEquals" and "StringNotEquals"
"s3:prefix", "s3:max-keys"
2016-03-08 17:44:50 -08:00
cd3eb63c4a
pkg/fs: test, document, and fix IsValid{Bucket,Object}Name
...
This commit improves the docs for both functions (more Go-like) and
drops an unnecessary condition in IsValidBucketName. This also drops a
condition in IsValidObjectName where "" (empty string) was a valid
object name. This has been fixed and will no longer return true.
This commit also adds tests for both functions, including a regression
test for the bug fix.
2016-03-07 19:59:24 -08:00
a5d0bef4e2
pkg/fs: test, bench, and drop unnecessary check in ListBuckets
...
There is now a simple test and a benchmark for ListBuckets. I also
dropped an unnecessary check that was simply repeated from above,
guaranteed to be true.
2016-03-07 19:58:33 -08:00
fab45aae40
pkg/fs: add bucket test and benchmarks
...
Lots of useful benchmarks and a simple test addition!
2016-03-07 00:07:11 -08:00
7399d8ceaa
pkg/fs: skip unnecessary os.Stat system call
2016-03-07 00:07:11 -08:00
0a0451a0fb
pkg/fs: DRY SetBucketMetadata
...
It had a lot of code that was the same as GetBucketMetadata, so instead
call GBM from SBM so as to reduce doing the same thing in two different
spots. Theoretically this will induce a small overhead as now at least
two calls of denormalizeBucket are made, although this shouldn't be
noticeable.
2016-03-07 00:07:11 -08:00
294ea814bf
pkg/fs: for locks, prefer defer and read-only ops
...
This commit prefers the use of 'defer' for fs.Unlock (and fs.RUnlock)
because it is more idiomatic Go and reduces repetition in the code,
lending to a cleaner code base.
It also switches a few uses of the lock to read-only locks, which should
improve performance of those functions dramatically in certain contexts.
2016-03-07 00:07:11 -08:00
aed62788d9
api: Implement multiple objects Delete api - fixes #956
...
This API takes input XML input in following form.
```
<?xml version="1.0" encoding="UTF-8"?>
<Delete>
<Quiet>true</Quiet>
<Object>
<Key>Key</Key>
</Object>
<Object>
<Key>Key</Key>
</Object>
...
</Delete>
```
and responds the list of successful deletes, list of errors
for all the deleted objects.
```
<?xml version="1.0" encoding="UTF-8"?>
<DeleteResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/ ">
<Deleted>
<Key>sample1.txt</Key>
</Deleted>
<Error>
<Key>sample2.txt</Key>
<Code>AccessDenied</Code>
<Message>Access Denied</Message>
</Error>
</DeleteResult>
```
2016-03-06 18:31:50 -08:00
f111997184
multipart: remove proper MD5, rather create MD5 based on parts to be s3 compatible.
...
This increases the performance phenominally.
2016-03-02 14:20:49 -08:00
17d145df3a
signature: Fix signature handling of parallel requests.
...
Signature struct should be immutable, this fixes an issue
with AWS cli not being able to do multipart put operations.
2016-03-02 11:49:50 -08:00
6651f5b368
go vet shadow error patch
2016-03-02 09:55:00 +05:30
c7021f6a95
list: Fix handling of maxKeys and prefixes.
...
This fixes a problem of requeuing the same request
and also fixes a major problem of sending truncated
for full key prefixes.
Fixes #1177
2016-03-01 17:34:31 -08:00
3ff8a1b719
api: Implement CopyObject s3 API, doing server side copy.
...
Fixes #1172
2016-02-27 19:51:59 -08:00
408aa72146
build/vet: Fix all the shadowing reports with go1.6
...
Golang 1.6 is default version for the build now.
Additionally set 'GODEBUG=cgocheck=0' for now, until
we fix the erasure coding package.
Readmore here https://tip.golang.org/doc/go1.6#cgo
2016-02-23 14:34:39 -08:00
07da31f8b8
Merge pull request #1150 from harshavardhana/signature
...
signV4: Move pkg/signature to pkg/s3/signature4
2016-02-23 12:39:28 +05:30
653ceee9ee
signV4: Move pkg/signature to pkg/s3/signature4
...
Cleanup and move this to relevant path.
2016-02-22 22:47:09 -08:00
800b19d8e5
cleanup: Remove definitions and move them to its relative places accordingly
...
- Move fs-definitions.go and break them into fs-datatypes.go, fs-bucket-acl.go
and fs-utils.go
- Move api-definitions.go to api-response.go, where they should be.
- Move web-definitions to its related handlers.
2016-02-22 10:41:27 -08:00
91a092792a
presigned: Fix a bug in presigned request verification.
...
Additionally add Docker proxy configuration.
2016-02-18 02:23:12 -08:00
dd9aaa855c
web/rpc: Merge ports with API server.
...
Fixes #1081 and #1130
2016-02-17 20:28:15 -08:00
5a9333a67b
signature: Rewrite signature handling and move it into a library.
2016-02-16 17:28:16 -08:00
9e10ee7e47
cpu: Remove pkg/cpu in favor of better klauspost/cpuid.
...
Fixes #1128
2016-02-15 13:50:33 -08:00
bbca70e13b
rewrite minio runtime checks
2016-02-15 17:56:56 +05:30
fbab7128d5
fs/multipart: Handle un-ordered creation of multiparts.
...
Fixes #1125
2016-02-14 00:39:15 -08:00
62f6ffb6db
xl: Moved to minio/minio - fixes #1112
2016-02-11 15:43:36 -08:00
6e9d73426b
pkg/ioutils: True should be true
2016-02-10 13:33:36 -08:00
Harshavardhana
Harshavardhana
Anis Elleuch
Karthic Rao
Brendan Ashworth
Anand Babu (AB) Periasamy
Bala.FA
Anand Babu (AB) Periasamy