Optimizing List Objects by using binary sort to discard entries in cases
where prefix or marker is set.
Adding test coverage to ListObjects.
Adding benchmark to ListObjects.
In the common case, GetObject is called on a bucket that exists and an
object that exists and is not a directory. It should be optimized for
this case, thus error-related syscalls are pushed back until they are
necessary.
This should not impact performance negatively in the uncommon case, and
instead drops two otherwise unnecessary os.Stat's in the common case.
The race conditions around a proper error being returned were present
beforehand.
It also renames 'err' to 'e'.
When list object is invoked, it creates a goroutine if not available
for given parameters else uses existing goroutine. These goroutines
are alive for 15 seconds for further continuation list object request
else they exit.
Fixes#1076
This commit improves the docs for both functions (more Go-like) and
drops an unnecessary condition in IsValidBucketName. This also drops a
condition in IsValidObjectName where "" (empty string) was a valid
object name. This has been fixed and will no longer return true.
This commit also adds tests for both functions, including a regression
test for the bug fix.
There is now a simple test and a benchmark for ListBuckets. I also
dropped an unnecessary check that was simply repeated from above,
guaranteed to be true.
It had a lot of code that was the same as GetBucketMetadata, so instead
call GBM from SBM so as to reduce doing the same thing in two different
spots. Theoretically this will induce a small overhead as now at least
two calls of denormalizeBucket are made, although this shouldn't be
noticeable.
This commit prefers the use of 'defer' for fs.Unlock (and fs.RUnlock)
because it is more idiomatic Go and reduces repetition in the code,
lending to a cleaner code base.
It also switches a few uses of the lock to read-only locks, which should
improve performance of those functions dramatically in certain contexts.
This API takes input XML input in following form.
```
<?xml version="1.0" encoding="UTF-8"?>
<Delete>
<Quiet>true</Quiet>
<Object>
<Key>Key</Key>
</Object>
<Object>
<Key>Key</Key>
</Object>
...
</Delete>
```
and responds the list of successful deletes, list of errors
for all the deleted objects.
```
<?xml version="1.0" encoding="UTF-8"?>
<DeleteResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Deleted>
<Key>sample1.txt</Key>
</Deleted>
<Error>
<Key>sample2.txt</Key>
<Code>AccessDenied</Code>
<Message>Access Denied</Message>
</Error>
</DeleteResult>
```
Golang 1.6 is default version for the build now.
Additionally set 'GODEBUG=cgocheck=0' for now, until
we fix the erasure coding package.
Readmore here https://tip.golang.org/doc/go1.6#cgo
- Move fs-definitions.go and break them into fs-datatypes.go, fs-bucket-acl.go
and fs-utils.go
- Move api-definitions.go to api-response.go, where they should be.
- Move web-definitions to its related handlers.
Minor optimization.
- Add 1000 entries buffered channel for walkerCh.
- Reset marker after the lexical order has reached and
compare only if the marker is set.
- Fixes initiating parallel uploads, and configs being quickly
re-written by another incoming request.
- Parallel uploads work smoothly now and return expected behavior.
Fail createObject() if a file already exists and one attempts
to create a prefix/directory by same name.
Send an approriate error back to the client as 409 Conflict.
Currently the server would set 'application/octet-stream' for all
objects, set this value based on the file extension transparently.
This is useful in case of minio browser to facilitate displaying
proper icons for the different mime data types.
Incoming request params in presigned can come in different order
for different implementations. Rather than verifying a full string
we should verify individual params instead.
This patch fixes an incompatibility issue with AWS SDK Java.
Fixes#1059 - Thanks to @notnoopci for reporting this problem.
listObjects was returning inconsistent results, i.e missing
entries during recursive and non-recursive listing. This led
to 'mc mirror' copying contents repeatedly consisdering
these files to be missing on the destination.
This patch addresses this problem - fixes#1056
HMAC is a much simpler implementation, providing the same
benefits as RSA, avoids additional steps and keeps the code
simpler.
This patch also additionally
- Implements PutObjectURL API.
- GetObjectURL, PutObjectURL take TargetHost as another
argument for generating URL's for proper target destination.
- Adds experimental TLS support for JSON RPC calls.
Without this change listObjects() goes into an infinite loop for
files which have special characters i.e "++" encoded with "%2B%2B".
We have to unescape and convert them to their native representation
before being used internally.
Fixes#1052
Existing code
```
{
if os.IsNotExist(e) {
e = os.MkdirAll(objectDir, 0700)
if e != nil {
return "", probe.NewError(e)
}
}
return "", probe.NewError(e) ---> Error was here.
}
```
For a successful 'MkdirAll' it would still return an empty uploadID,
but the 'error' would be nil. This would succeed the request but
client would fail.
Fix is to re-arrange the logic. Thanks to Alexander Neumann @fd0, for
reporting this problem.
- limit list buckets to limit only 100 buckets, all uppercase buckets
are now lowercase and work transparently with all calls.
- Change disk.Stat to disk.GetInfo and return back disk.Info{} struct.
- Introduce new ioutils package which implements ReadDirN(path, n),
ReadDirNamesN(path, n)
Golang http server strips off 'Expect' header, if the
client sent this as part of signed headers we need to
handle otherwise we would see a signature mismatch.
`aws-cli` sets this as part of signed headers which is
a bad idea since servers trying to implement AWS
Signature version '4' will all encounter this issue.
According to
http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.20
Expect header is always of form:
Expect = "Expect" ":" 1#expectation
expectation = "100-continue" | expectation-extension
So it safe to assume that '100-continue' is what would
be sent, for the time being keep this work around.
Remove option of providing Technique and handling errors based on that
choose a matrix type automatically based on number of data blocks.
INTEL recommends on using cauchy for consistent invertible matrices,
while vandermonde is faster we should default to cauchy for large
data blocks.
Stream reading needs to check for length parameter being non zero,
after Reading() a predefined set of buffer length an EOF might be returned
with length == 0.
Erasure taking this zeroed data in might wrongly encode it as part of existing
data blocks which leads to errors while decoding even when the other contents
are intact.
- Upon first time invocation ``minio controller`` would create access keys and secret id
- Upon request passing 'keys' arg ``minio controller`` would provide the keys
- Add colorized notification
This change brings a new SignatureHandler where Presigned.
Requests without Payload are handled very early before even
going through the call.
This change simplifies Donut codebase to not have signature related
logic for all API's.
Simplification is still needed for Payload based signature eg. PUT/POST calls
, which are still part of the donut codebase, which will be done subsequently
after donut re-write.
Karthic Rao
Harshavardhana
Brendan Ashworth
Anand Babu (AB) Periasamy
Bala.FA
Harshavardhana
Anand Babu (AB) Periasamy
Krishna Srinivas
Anis ELLEUCH