This change brings in changes at multiple places
- Reuse buffers at almost all locations ranging
from rpc, fs, xl, checksum etc.
- Change caching behavior to disable itself
under low memory conditions i.e < 8GB of RAM.
- Only objects cached are of size 1/10th the size
of the cache for example if 4GB is the cache size
the maximum object size which will be cached
is going to be 400MB. This change is an
optimization to cache more objects rather
than few larger objects.
- If object cache is enabled default GC
percent has been reduced to 20% in lieu
with newly found behavior of GC. If the cache
utilization reaches 75% of the maximum value
GC percent is reduced to 10% to make GC
more aggressive.
- Do not use *bytes.Buffer* due to its growth
requirements. For every allocation *bytes.Buffer*
allocates an additional buffer for its internal
purposes. This is undesirable for us, so
implemented a new cappedWriter which is capped to a
desired size, beyond this all writes rejected.
Possible fix for #3403.
Do not attempt to fetch volume/drive information for
each i/o situation. In our case we do this in all calls
`posix.go` this in-turn created a terrible situation for
windows. This issue does not affect the i/o path on Unix
platforms since statvfs calls are in the range of micro
seconds on these platforms.
This verification is only needed during startup and we
let things fail at a later stage on windows.
This API is precursor before implementing `minio lambda` and `mc` continous replication.
This new api is an extention to BucketNofication APIs.
// Request
```
GET /bucket?notificationARN=arn:minio:lambda:us-east-1:10:minio HTTP/1.1
...
...
```
// Response
```
{"Records": ...}
...
...
...
{"Records": ...}
```
- Support for '?' wildcard for resource matching.
- Wildcard package is added with Match functions.
- Wildcard.Match supports '*' and wild.MatchExtended supports both '*'
and '?' wildcards in the pattern string.
- Tests for the same for the wide range of cases.
* XL/erasure-read: optimize memory allocation during erasure-read by using temporary buffer pool.
With the change the buffer needed during GetObject by erasureReadFile is allocated only once.
Change brings in a new signVerifyReader which provides a io.Reader
compatible reader, additionally implements Verify() function.
Verify() function validates the signature present in the incoming
request. This approach is choosen to avoid complexities involved
in using io.Pipe().
Thanks to Krishna for his inputs on this.
Fixes#2058Fixes#2054Fixes#2087
Previously checkDiskFree() checks for free available space. This
patch enables checkDiskFree() also checks for free inodes in linux and
free clusters in windows.
Fixes#2075
The object cache implementation is XL cache, which defaults
to 8GB worth of read cache. Currently GetObject() transparently
writes to this cache upon first client read and then subsequently
serves reads from the same cache.
Currently expiration is not implemented.
Some environments might disable access to `/dev/tty`, fall
back to '80' in such scenarios.
Move to 'cheggaaa/pb' package for better cross platform
support on fetching terminal width.
Fixes#1891
The functionality provided by minhttp will be implemented
cleanly through our own APIs. Since we are not going to
send SIGUSR2 and manage configuration in that manner, it
doesn't make sense to use minhttp.
Fixes#1586
Signature calculation has now moved out from being a package to
top-level as a layered mechanism.
In case of payload calculation with body, go-routines are initiated
to simultaneously write and calculate shasum. Errors are sent
over the writer so that the lower layer removes the temporary files
properly.
Optimizing List Objects by using binary sort to discard entries in cases
where prefix or marker is set.
Adding test coverage to ListObjects.
Adding benchmark to ListObjects.
In the common case, GetObject is called on a bucket that exists and an
object that exists and is not a directory. It should be optimized for
this case, thus error-related syscalls are pushed back until they are
necessary.
This should not impact performance negatively in the uncommon case, and
instead drops two otherwise unnecessary os.Stat's in the common case.
The race conditions around a proper error being returned were present
beforehand.
It also renames 'err' to 'e'.
When list object is invoked, it creates a goroutine if not available
for given parameters else uses existing goroutine. These goroutines
are alive for 15 seconds for further continuation list object request
else they exit.
Fixes#1076
This commit improves the docs for both functions (more Go-like) and
drops an unnecessary condition in IsValidBucketName. This also drops a
condition in IsValidObjectName where "" (empty string) was a valid
object name. This has been fixed and will no longer return true.
This commit also adds tests for both functions, including a regression
test for the bug fix.
There is now a simple test and a benchmark for ListBuckets. I also
dropped an unnecessary check that was simply repeated from above,
guaranteed to be true.
It had a lot of code that was the same as GetBucketMetadata, so instead
call GBM from SBM so as to reduce doing the same thing in two different
spots. Theoretically this will induce a small overhead as now at least
two calls of denormalizeBucket are made, although this shouldn't be
noticeable.
This commit prefers the use of 'defer' for fs.Unlock (and fs.RUnlock)
because it is more idiomatic Go and reduces repetition in the code,
lending to a cleaner code base.
It also switches a few uses of the lock to read-only locks, which should
improve performance of those functions dramatically in certain contexts.
This API takes input XML input in following form.
```
<?xml version="1.0" encoding="UTF-8"?>
<Delete>
<Quiet>true</Quiet>
<Object>
<Key>Key</Key>
</Object>
<Object>
<Key>Key</Key>
</Object>
...
</Delete>
```
and responds the list of successful deletes, list of errors
for all the deleted objects.
```
<?xml version="1.0" encoding="UTF-8"?>
<DeleteResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Deleted>
<Key>sample1.txt</Key>
</Deleted>
<Error>
<Key>sample2.txt</Key>
<Code>AccessDenied</Code>
<Message>Access Denied</Message>
</Error>
</DeleteResult>
```
Golang 1.6 is default version for the build now.
Additionally set 'GODEBUG=cgocheck=0' for now, until
we fix the erasure coding package.
Readmore here https://tip.golang.org/doc/go1.6#cgo
- Move fs-definitions.go and break them into fs-datatypes.go, fs-bucket-acl.go
and fs-utils.go
- Move api-definitions.go to api-response.go, where they should be.
- Move web-definitions to its related handlers.
Minor optimization.
- Add 1000 entries buffered channel for walkerCh.
- Reset marker after the lexical order has reached and
compare only if the marker is set.
- Fixes initiating parallel uploads, and configs being quickly
re-written by another incoming request.
- Parallel uploads work smoothly now and return expected behavior.
Anis Elleuch
Harshavardhana
Krishnan Parthasarathi
karthic rao
Krishna Srinivas
Bala FA
Harshavardhana
karthic rao
Karthic Rao
Brendan Ashworth
Anand Babu (AB) Periasamy
Anand Babu (AB) Periasamy