f3c6c55719
posix: Fix windows performance issues. ( #3132 )
...
Do not attempt to fetch volume/drive information for
each i/o situation. In our case we do this in all calls
`posix.go` this in-turn created a terrible situation for
windows. This issue does not affect the i/o path on Unix
platforms since statvfs calls are in the range of micro
seconds on these platforms.
This verification is only needed during startup and we
let things fail at a later stage on windows.
2016-10-31 09:34:44 -07:00
9417614a8e
Recalculate free minimum disk space ( #2788 )
...
* Fix calculating free space disk by using blocks available for unprivileged user
* Use fixed minimal free disk space instead of percentage
2016-09-27 12:46:38 -07:00
669783f875
Purge stale object cache entry ( #2770 )
2016-09-23 19:55:28 -07:00
7270ca4157
pkg/wildcard: Simplify the wildcard logic further. ( #2555 )
2016-08-27 00:27:17 -07:00
ec4260d260
api: BucketNotification should disallow duplicate notification. ( #2539 )
...
Added checks to look for duplicated notification configs.
Fixes #2472
2016-08-23 18:42:30 -07:00
4dec50ba51
build: Add platform specific fixes.
2016-08-16 14:40:41 -07:00
76d56c6ff2
typo: Fix typos across the codebase. ( #2442 )
2016-08-15 02:44:48 -07:00
7e46055a15
api/handlers: Implement streaming signature v4 support. ( #2370 )
...
* api/handlers: Implement streaming signature v4 support.
Fixes #2326
* tests: Add tests for quick/safe
2016-08-08 20:56:29 -07:00
b23605a2b5
pkg/objcache: Add more tests. ( #2371 )
2016-08-06 10:22:14 -07:00
064c51162d
api: Add new ListenBucketNotificationHandler. ( #2336 )
...
This API is precursor before implementing `minio lambda` and `mc` continous replication.
This new api is an extention to BucketNofication APIs.
// Request
```
GET /bucket?notificationARN=arn:minio:lambda:us-east-1:10:minio HTTP/1.1
...
...
```
// Response
```
{"Records": ...}
...
...
...
{"Records": ...}
```
2016-08-04 22:01:58 -07:00
de5d5ff241
pkg/crypto: Deprecate cgo sha256 version. ( #2354 )
2016-08-04 03:19:36 -07:00
2e0742e309
bucket policy: Support for '?' wildcard. ( #2353 )
...
- Support for '?' wildcard for resource matching.
- Wildcard package is added with Match functions.
- Wildcard.Match supports '*' and wild.MatchExtended supports both '*'
and '?' wildcards in the pattern string.
- Tests for the same for the wide range of cases.
2016-08-04 00:41:32 -07:00
69fd196471
Object-cache: enforce cache size to be less than RAM. ( #2338 )
2016-08-02 10:04:35 -07:00
043ddbd834
optimize memory allocation during erasure-read by using temporary buffer pool. ( #2259 )
...
* XL/erasure-read: optimize memory allocation during erasure-read by using temporary buffer pool.
With the change the buffer needed during GetObject by erasureReadFile is allocated only once.
2016-07-25 14:17:01 -07:00
c0c8a8430e
XL/PutObject: Add single putObject and multipart caching. ( #2115 )
...
- Additionally adds test cases as well for object cache.
- Adds auto-expiry with expiration and cleanup time interval.
Fixes #2080
Fixes #2091
2016-07-08 20:34:27 -07:00
01cbacd803
object-cache: use golang bytes.Buffer and bytes.NewReader instead of custom implementation. ( #2108 )
2016-07-06 01:29:49 -07:00
8a028a9efb
handler/PUT: Handle signature verification through a custom reader. ( #2066 )
...
Change brings in a new signVerifyReader which provides a io.Reader
compatible reader, additionally implements Verify() function.
Verify() function validates the signature present in the incoming
request. This approach is choosen to avoid complexities involved
in using io.Pipe().
Thanks to Krishna for his inputs on this.
Fixes #2058
Fixes #2054
Fixes #2087
2016-07-05 01:04:50 -07:00
1ad5fb8f76
posix: checkDiskFree() also checks free inodes. ( #2086 )
...
Previously checkDiskFree() checks for free available space. This
patch enables checkDiskFree() also checks for free inodes in linux and
free clusters in windows.
Fixes #2075
2016-07-03 22:34:45 -07:00
d2557bb538
XL: GetObject caching implemented for XL. ( #2017 )
...
The object cache implementation is XL cache, which defaults
to 8GB worth of read cache. Currently GetObject() transparently
writes to this cache upon first client read and then subsequently
serves reads from the same cache.
Currently expiration is not implemented.
2016-07-03 17:15:38 -07:00
e10934a88e
bitrot: Start using blake2b algorithm and remove sha512 usage. ( #1957 )
...
Fixes #1952
2016-06-22 17:13:26 -07:00
4ab57f7d60
server: terminal width should fallback to 80x25. ( #1895 )
...
Some environments might disable access to `/dev/tty`, fall
back to '80' in such scenarios.
Move to 'cheggaaa/pb' package for better cross platform
support on fetching terminal width.
Fixes #1891
2016-06-12 19:35:28 -07:00
51f3d4e0ca
XL/multipart: statPart should ignore errDiskNotFound. ( #1862 )
...
startPart should also take uploadId and partName as arguments.
2016-06-07 18:15:04 -07:00
51bb613fdf
pkg/safe: remove temporary file on failure ( #1774 )
2016-05-28 15:15:53 -07:00
64b0976e1b
Remove probe and tasker. ( #1733 )
...
Fixes #1717
2016-05-24 18:43:33 -07:00
49141eb3e4
http: Remove minhttp package and use standard Golang http. ( #1587 )
...
The functionality provided by minhttp will be implemented
cleanly through our own APIs. Since we are not going to
send SIGUSR2 and manage configuration in that manner, it
doesn't make sense to use minhttp.
Fixes #1586
2016-05-10 18:03:00 -07:00
d1fa1d9352
Remove binary files from previous commit.
2016-05-10 15:49:17 -07:00
26e2c4bf4d
Replacing fastsha256 with crypto/sha256 package from golang standard package ( #1584 )
2016-05-10 14:20:11 -07:00
5133ea50bd
xl/fs: Make i/o operations atomic. ( #1496 )
2016-05-05 20:28:22 -07:00
7ae40eb1bb
minhttp: Remove probe usage, move to golang error. ( #1459 )
...
Fixes #1454
2016-05-03 01:07:34 -07:00
4e34e03dd4
xl/fs: Split object layer into interface. ( #1415 )
2016-04-29 14:24:10 -07:00
a98a7fb1ad
Implement XL layer - preliminary work.
2016-04-25 12:47:31 -07:00
ff4e04d942
atomic/fs: use safe package for atomic writes, even in multipart.
2016-04-06 16:05:30 -07:00
379e0abf03
cleanup: Remove old donut/xl code and erasure implementation.
...
This is a change to bring in 'klauspost/reedsolomon' library
in #1270 patch.
2016-04-02 17:30:35 -07:00
6037fe66e9
minio: Simplify for `gosimple` tool complaints.
2016-04-02 17:28:54 -07:00
efc80343e3
fs: Break fs package to top-level and introduce ObjectAPI interface.
...
ObjectAPI interface brings in changes needed for XL ObjectAPI layer.
The new interface for any ObjectAPI layer is as below
```
// ObjectAPI interface.
type ObjectAPI interface {
// Bucket resource API.
DeleteBucket(bucket string) *probe.Error
ListBuckets() ([]BucketInfo, *probe.Error)
MakeBucket(bucket string) *probe.Error
GetBucketInfo(bucket string) (BucketInfo, *probe.Error)
// Bucket query API.
ListObjects(bucket, prefix, marker, delimiter string, maxKeys int) (ListObjectsResult, *probe.Error)
ListMultipartUploads(bucket string, resources BucketMultipartResourcesMetadata) (BucketMultipartResourcesMetadata, *probe.Error)
// Object resource API.
GetObject(bucket, object string, startOffset int64) (io.ReadCloser, *probe.Error)
GetObjectInfo(bucket, object string) (ObjectInfo, *probe.Error)
PutObject(bucket string, object string, size int64, data io.Reader, metadata map[string]string) (ObjectInfo, *probe.Error)
DeleteObject(bucket, object string) *probe.Error
// Object query API.
NewMultipartUpload(bucket, object string) (string, *probe.Error)
PutObjectPart(bucket, object, uploadID string, partID int, size int64, data io.Reader, md5Hex string) (string, *probe.Error)
ListObjectParts(bucket, object string, resources ObjectResourcesMetadata) (ObjectResourcesMetadata, *probe.Error)
CompleteMultipartUpload(bucket string, object string, uploadID string, parts []CompletePart) (ObjectInfo, *probe.Error)
AbortMultipartUpload(bucket, object, uploadID string) *probe.Error
}
```
2016-04-01 15:58:39 -07:00
1502e2f29f
Merge pull request #1265 from vadmeste/add_fbsd_support
...
Add simple FreeBSD support, make the minio project compilable
2016-03-26 22:28:07 -07:00
9dca46e156
signature: Use a layered approach for signature verification.
...
Signature calculation has now moved out from being a package to
top-level as a layered mechanism.
In case of payload calculation with body, go-routines are initiated
to simultaneously write and calculate shasum. Errors are sent
over the writer so that the lower layer removes the temporary files
properly.
2016-03-26 15:21:05 -07:00
663f24064b
Add simple FreeBSD support, make the minio project compilable
2016-03-26 22:39:34 +01:00
aaf97ea02c
config/main: Re-write config files - add to new config v3
...
- New config format.
```
{
"version": "3",
"address": ":9000",
"backend": {
"type": "fs",
"disk": "/path"
},
"credential": {
"accessKey": "WLGDGYAQYIGI833EV05A",
"secretKey": "BYvgJM101sHngl2uzjXS/OBF/aMxAN06JrJ3qJlF"
},
"region": "us-east-1",
"logger": {
"file": {
"enable": false,
"fileName": "",
"level": "error"
},
"syslog": {
"enable": false,
"address": "",
"level": "debug"
},
"console": {
"enable": true,
"level": "fatal"
}
}
}
```
New command lines in lieu of supporting XL.
Minio initialize filesystem backend.
~~~
$ minio init fs <path>
~~~
Minio initialize XL backend.
~~~
$ minio init xl <url1>...<url16>
~~~
For 'fs' backend it starts the server.
~~~
$ minio server
~~~
For 'xl' backend it waits for servers to join.
~~~
$ minio server
... [PROGRESS BAR] of servers connecting
~~~
Now on other servers execute 'join' and they connect.
~~~
....
minio join <url1> -- from <url2> && minio server
minio join <url1> -- from <url3> && minio server
...
...
minio join <url1> -- from <url16> && minio server
~~~
2016-03-23 19:16:09 -07:00
c8570edaab
Issue of 'mc ls' when prefix is a directory fixed, tests added for GetObjectInfo
2016-03-23 19:42:04 +05:30
7be79b507b
Changing to empty response when prefixDir doesn't exist
2016-03-23 04:46:10 +05:30
ff41c050d5
Formatting issues fixed.
2016-03-22 15:55:29 +05:30
b55922effe
Fix for Istruncated set to true under certain conditions.
...
Optimizing List Objects by using binary sort to discard entries in cases
where prefix or marker is set.
Adding test coverage to ListObjects.
Adding benchmark to ListObjects.
2016-03-22 10:09:16 +05:30
99af0444b7
Handling maxKeys=0 case with a empty response
2016-03-18 15:16:30 +05:30
88714e7c8e
bucketpolicy: Improve bucket policy validation, avoid nested rules.
...
Bucket policy validation is more stricter now, to avoid nested
rules. The reason to do this is keep the rules simpler and more
meaningful avoiding conflicts.
This patch implements stricter checks.
Example policy to be generally avoided.
```
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"s3:GetObject",
"s3:DeleteObject"
],
"Effect": "Allow",
"Principal": {
"AWS": [
"*"
]
},
"Resource": [
"arn:aws:s3:::jarjarbing/*"
]
},
{
"Action": [
"s3:GetObject",
"s3:DeleteObject"
],
"Effect": "Deny",
"Principal": {
"AWS": [
"*"
]
},
"Resource": [
"arn:aws:s3:::jarjarbing/restic/key/*"
]
}
]
}
```
2016-03-15 17:50:23 -07:00
583e4ecff6
pkg/fs: optimize GetObject syscalls for common case
...
In the common case, GetObject is called on a bucket that exists and an
object that exists and is not a directory. It should be optimized for
this case, thus error-related syscalls are pushed back until they are
necessary.
This should not impact performance negatively in the uncommon case, and
instead drops two otherwise unnecessary os.Stat's in the common case.
The race conditions around a proper error being returned were present
beforehand.
It also renames 'err' to 'e'.
2016-03-13 13:56:33 -07:00
b2257682e4
pkg/fs: add benchmark for GetObject
...
This commit adds a benchmark for GetObject. It doesn't leverage the I/O
as much because it uses short text for data, just 58 chars.
2016-03-13 11:13:06 -07:00
53a76439a2
test for GetBucketInfo
2016-03-12 14:31:30 +05:30
5282a79eda
cleanup: Remove unecessary packages and tests. Simplify.
2016-03-11 19:53:55 -08:00
52751d81cb
cleanup: Rename ObjectMetadata as ObjectInfo.
...
Fixes #1215
2016-03-11 16:58:08 -08:00
ec8c1d4ef6
Better structuring of fs-utils test
2016-03-11 19:19:47 +05:30
fdf3d64793
error: Add proper prefixes for s3Error codes.
...
This patch adds 'Err' prefix for all standard API
error codes and also adds a proper type for them.
2016-03-10 18:38:46 -08:00
373d335d94
Merge pull request #1214 from brendanashworth/improve-listbuckets
...
ListBuckets test & improvement, IsValid{Bucket,Object}Name fix, test, docs
2016-03-10 18:20:18 -08:00
c70bc2209e
api: refactor list object handling in fs backend
...
When list object is invoked, it creates a goroutine if not available
for given parameters else uses existing goroutine. These goroutines
are alive for 15 seconds for further continuation list object request
else they exit.
Fixes #1076
2016-03-11 02:20:51 +05:30
d5057b3c51
accessPolicy: Implement Put, Get, Delete access policy.
...
This patch implements Get,Put,Delete bucket policies
Supporting - http://docs.aws.amazon.com/AmazonS3/latest/dev/access-policy-language-overview.html
Currently supports following actions.
"*": true,
"s3:*": true,
"s3:GetObject": true,
"s3:ListBucket": true,
"s3:PutObject": true,
"s3:CreateBucket": true,
"s3:GetBucketLocation": true,
"s3:DeleteBucket": true,
"s3:DeleteObject": true,
"s3:AbortMultipartUpload": true,
"s3:ListBucketMultipartUploads": true,
"s3:ListMultipartUploadParts": true,
following conditions for "StringEquals" and "StringNotEquals"
"s3:prefix", "s3:max-keys"
2016-03-08 17:44:50 -08:00
cd3eb63c4a
pkg/fs: test, document, and fix IsValid{Bucket,Object}Name
...
This commit improves the docs for both functions (more Go-like) and
drops an unnecessary condition in IsValidBucketName. This also drops a
condition in IsValidObjectName where "" (empty string) was a valid
object name. This has been fixed and will no longer return true.
This commit also adds tests for both functions, including a regression
test for the bug fix.
2016-03-07 19:59:24 -08:00
a5d0bef4e2
pkg/fs: test, bench, and drop unnecessary check in ListBuckets
...
There is now a simple test and a benchmark for ListBuckets. I also
dropped an unnecessary check that was simply repeated from above,
guaranteed to be true.
2016-03-07 19:58:33 -08:00
fab45aae40
pkg/fs: add bucket test and benchmarks
...
Lots of useful benchmarks and a simple test addition!
2016-03-07 00:07:11 -08:00
7399d8ceaa
pkg/fs: skip unnecessary os.Stat system call
2016-03-07 00:07:11 -08:00
0a0451a0fb
pkg/fs: DRY SetBucketMetadata
...
It had a lot of code that was the same as GetBucketMetadata, so instead
call GBM from SBM so as to reduce doing the same thing in two different
spots. Theoretically this will induce a small overhead as now at least
two calls of denormalizeBucket are made, although this shouldn't be
noticeable.
2016-03-07 00:07:11 -08:00
294ea814bf
pkg/fs: for locks, prefer defer and read-only ops
...
This commit prefers the use of 'defer' for fs.Unlock (and fs.RUnlock)
because it is more idiomatic Go and reduces repetition in the code,
lending to a cleaner code base.
It also switches a few uses of the lock to read-only locks, which should
improve performance of those functions dramatically in certain contexts.
2016-03-07 00:07:11 -08:00
aed62788d9
api: Implement multiple objects Delete api - fixes #956
...
This API takes input XML input in following form.
```
<?xml version="1.0" encoding="UTF-8"?>
<Delete>
<Quiet>true</Quiet>
<Object>
<Key>Key</Key>
</Object>
<Object>
<Key>Key</Key>
</Object>
...
</Delete>
```
and responds the list of successful deletes, list of errors
for all the deleted objects.
```
<?xml version="1.0" encoding="UTF-8"?>
<DeleteResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/ ">
<Deleted>
<Key>sample1.txt</Key>
</Deleted>
<Error>
<Key>sample2.txt</Key>
<Code>AccessDenied</Code>
<Message>Access Denied</Message>
</Error>
</DeleteResult>
```
2016-03-06 18:31:50 -08:00
f111997184
multipart: remove proper MD5, rather create MD5 based on parts to be s3 compatible.
...
This increases the performance phenominally.
2016-03-02 14:20:49 -08:00
17d145df3a
signature: Fix signature handling of parallel requests.
...
Signature struct should be immutable, this fixes an issue
with AWS cli not being able to do multipart put operations.
2016-03-02 11:49:50 -08:00
6651f5b368
go vet shadow error patch
2016-03-02 09:55:00 +05:30
c7021f6a95
list: Fix handling of maxKeys and prefixes.
...
This fixes a problem of requeuing the same request
and also fixes a major problem of sending truncated
for full key prefixes.
Fixes #1177
2016-03-01 17:34:31 -08:00
3ff8a1b719
api: Implement CopyObject s3 API, doing server side copy.
...
Fixes #1172
2016-02-27 19:51:59 -08:00
408aa72146
build/vet: Fix all the shadowing reports with go1.6
...
Golang 1.6 is default version for the build now.
Additionally set 'GODEBUG=cgocheck=0' for now, until
we fix the erasure coding package.
Readmore here https://tip.golang.org/doc/go1.6#cgo
2016-02-23 14:34:39 -08:00
07da31f8b8
Merge pull request #1150 from harshavardhana/signature
...
signV4: Move pkg/signature to pkg/s3/signature4
2016-02-23 12:39:28 +05:30
653ceee9ee
signV4: Move pkg/signature to pkg/s3/signature4
...
Cleanup and move this to relevant path.
2016-02-22 22:47:09 -08:00
800b19d8e5
cleanup: Remove definitions and move them to its relative places accordingly
...
- Move fs-definitions.go and break them into fs-datatypes.go, fs-bucket-acl.go
and fs-utils.go
- Move api-definitions.go to api-response.go, where they should be.
- Move web-definitions to its related handlers.
2016-02-22 10:41:27 -08:00
91a092792a
presigned: Fix a bug in presigned request verification.
...
Additionally add Docker proxy configuration.
2016-02-18 02:23:12 -08:00
dd9aaa855c
web/rpc: Merge ports with API server.
...
Fixes #1081 and #1130
2016-02-17 20:28:15 -08:00
5a9333a67b
signature: Rewrite signature handling and move it into a library.
2016-02-16 17:28:16 -08:00
9e10ee7e47
cpu: Remove pkg/cpu in favor of better klauspost/cpuid.
...
Fixes #1128
2016-02-15 13:50:33 -08:00
bbca70e13b
rewrite minio runtime checks
2016-02-15 17:56:56 +05:30
fbab7128d5
fs/multipart: Handle un-ordered creation of multiparts.
...
Fixes #1125
2016-02-14 00:39:15 -08:00
62f6ffb6db
xl: Moved to minio/minio - fixes #1112
2016-02-11 15:43:36 -08:00
6e9d73426b
pkg/ioutils: True should be true
2016-02-10 13:33:36 -08:00
5e4b13f4bd
remove unused functions
2016-02-10 13:32:53 -08:00
255505a83b
pkg/ioutils: remove usage of os.Lstat() in FTW()
...
As os.Readdir() is used get file entries where statinfo is already
present. This patch fixes to use statinfo provided by os.Readdir().
2016-02-10 13:32:53 -08:00
9b29af8bbe
listObjects: list objects minor optimization.
...
Minor optimization.
- Add 1000 entries buffered channel for walkerCh.
- Reset marker after the lexical order has reached and
compare only if the marker is set.
2016-02-09 21:45:19 -08:00
42fcb27308
pkg/user: Support 32bit darwin in user package.
2016-02-08 01:34:25 -08:00
99fbc0fcb3
getObject: Add support for special response headers.
...
Supports now response-content-type, response-content-disposition,
response-cache-control, response-expires.
2016-02-07 03:55:16 -08:00
f4c8120cf9
server: Remove max-buckets option and now max buckets is unlimited.
...
minio server max-buckets option removed. min-free-disk option is
now a flag.
2016-02-06 18:25:47 -08:00
4e6e78598f
multipart: Increase locked critical for CompleteMultipart.
2016-02-06 01:46:05 -08:00
8df201ef30
multipart: Multipart session map now is based on uploadID.
...
- Fixes initiating parallel uploads, and configs being quickly
re-written by another incoming request.
- Parallel uploads work smoothly now and return expected behavior.
2016-02-05 23:32:30 -08:00
69bd001c8b
multipart: Multipart resume simplify further.
2016-02-05 17:40:08 -08:00
8bf1045645
setBucketMetadata: Fix a deadlock.
2016-02-05 15:48:08 -08:00
6f80380497
fs: Use mimedb now.
2016-02-05 15:09:23 -08:00
35dcccb4cd
Merge remote-tracking branch 'abperiasamy/mimedb' into mimedb
2016-02-05 15:03:47 -08:00
a4c005ce30
multipart: Code cleanup
...
- More locking cleanup. Fix naming convention.
- Simplify concatenation and blocking calls.
2016-02-05 14:42:09 -08:00
d8abb36653
contentdb replaced by new mimedb
2016-02-05 03:49:24 -08:00
8557cbc9b7
fs: Add granular locking.
2016-02-04 20:40:58 -08:00
7a3409c309
fs: Cleanup Golang errors to be called 'e' and probe to be called as 'err'
...
- Replace the ACL checks back, remove them when bucket
policy is implemented.
- Move FTW (File Tree Walk) into ioutils package.
2016-02-04 13:43:52 -08:00
835b297ba7
fs: Fail createObject with appropriate message.
...
Fail createObject() if a file already exists and one attempts
to create a prefix/directory by same name.
Send an approriate error back to the client as 409 Conflict.
2016-02-03 21:49:36 -08:00
454d71cafa
expiry: Remove auto-expiry.
...
Move the logic outside and use scripting, cronjob to delete files.
Fixes #1019
2016-02-02 19:35:51 -08:00
81fcbd2a54
statObject: Make sure to lowercase file extensions.
2016-02-02 17:54:59 -08:00
0aedb67de0
contentType: Reply back proper contentTypes based on the file extension.
...
Currently the server would set 'application/octet-stream' for all
objects, set this value based on the file extension transparently.
This is useful in case of minio browser to facilitate displaying
proper icons for the different mime data types.
2016-02-01 12:19:58 -08:00
2469c9c591
presign: Verify query params for presign individually
...
Incoming request params in presigned can come in different order
for different implementations. Rather than verifying a full string
we should verify individual params instead.
This patch fixes an incompatibility issue with AWS SDK Java.
Fixes #1059 - Thanks to @notnoopci for reporting this problem.
2016-01-28 12:16:56 -08:00
5934a00058
listObjects: ListObjects should have idempotent behavior.
...
listObjects was returning inconsistent results, i.e missing
entries during recursive and non-recursive listing. This led
to 'mc mirror' copying contents repeatedly consisdering
these files to be missing on the destination.
This patch addresses this problem - fixes #1056
2016-01-28 03:17:40 -08:00
db387912f2
jwt: Deprecate RSA usage, use HMAC instead.
...
HMAC is a much simpler implementation, providing the same
benefits as RSA, avoids additional steps and keeps the code
simpler.
This patch also additionally
- Implements PutObjectURL API.
- GetObjectURL, PutObjectURL take TargetHost as another
argument for generating URL's for proper target destination.
- Adds experimental TLS support for JSON RPC calls.
2016-01-27 03:38:33 -08:00
9ca3372870
listObjects: Marker should be unescaped before being used internally.
...
Without this change listObjects() goes into an infinite loop for
files which have special characters i.e "++" encoded with "%2B%2B".
We have to unescape and convert them to their native representation
before being used internally.
Fixes #1052
2016-01-26 23:32:59 -08:00
5d87fdb35c
Merge pull request #1051 from harshavardhana/fix-multipart
...
multipart: NewMultipartUpload shouldn't return empty UploadID
2016-01-26 15:15:44 -08:00
2e311168ee
multipart: NewMultipartUpload shouldn't return empty UploadID
...
Existing code
```
{
if os.IsNotExist(e) {
e = os.MkdirAll(objectDir, 0700)
if e != nil {
return "", probe.NewError(e)
}
}
return "", probe.NewError(e) ---> Error was here.
}
```
For a successful 'MkdirAll' it would still return an empty uploadID,
but the 'error' would be nil. This would succeed the request but
client would fail.
Fix is to re-arrange the logic. Thanks to Alexander Neumann @fd0, for
reporting this problem.
2016-01-26 15:00:34 -08:00
18375b7794
ioutils: Add tests
2016-01-26 12:34:04 -08:00
1341fb79c3
listBuckets: Bump up the limit of max buckets to 1000.
2016-01-26 11:49:17 -08:00
f5d6be158e
listObjects: Simplify channel based changes.
2016-01-26 02:19:55 -08:00
682020ef2f
listObjects: Channel based changes.
...
Supports:
- prefixes
- marker
2016-01-25 20:39:38 -08:00
9e18bfa60e
listObjects: Channel based ftw - initial implementation.
2016-01-25 18:58:07 -08:00
2ec9b16667
deleteBucket: Directory not empty error on windows is "directory is not empty"
2016-01-25 17:58:43 -08:00
ae2f15c6d0
api: More cleanups at WebAPI.
...
- Fixes a bug where bucketName was not denormalized.
- Remove unneeded functions from jwt.go
2016-01-25 17:30:08 -08:00
497f13d733
api: Various fixes.
...
- limit list buckets to limit only 100 buckets, all uppercase buckets
are now lowercase and work transparently with all calls.
- Change disk.Stat to disk.GetInfo and return back disk.Info{} struct.
- Introduce new ioutils package which implements ReadDirN(path, n),
ReadDirNamesN(path, n)
2016-01-25 16:08:27 -08:00
8cdaf87c8f
signature: Add aws-cli work-around for now.
...
Golang http server strips off 'Expect' header, if the
client sent this as part of signed headers we need to
handle otherwise we would see a signature mismatch.
`aws-cli` sets this as part of signed headers which is
a bad idea since servers trying to implement AWS
Signature version '4' will all encounter this issue.
According to
http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.20
Expect header is always of form:
Expect = "Expect" ":" 1#expectation
expectation = "100-continue" | expectation-extension
So it safe to assume that '100-continue' is what would
be sent, for the time being keep this work around.
2016-01-09 10:47:59 -08:00
d955ce4123
s3cmd: Fix signature issues related to s3cmd.
...
Support regions both 'us-east-1' and 'US' (short hand for US Standard)
honored by S3.
2015-12-28 18:05:28 -08:00
1bfb490f90
pkg/user: Add pending tests.
2015-12-07 14:13:54 -08:00
4fc161ddb1
bucketName: relax bucket names, now allow numbers as starting characters.
2015-12-07 13:58:55 -08:00
836f5204af
minio: Add config-folder option.
...
Fixes #997
2015-12-07 12:34:09 -08:00
8e8538175b
fixes race in Init
2015-12-03 01:08:05 -08:00
25df427383
contentdb file.ext to content-type lookups
2015-12-02 16:55:26 -08:00
7c91a8495f
signature/region: Remove 'milkyway' and use 'us-east-1' as default.
...
Fixes #980
2015-11-23 13:40:23 -08:00
a328120e4d
fs: Filter out $multiparts properly.
...
Relax md5 requirement during complete multipart upload - ref #977
2015-11-22 01:49:57 -08:00
35b9f965f1
fs: use new atomic package - use FileCreateWithPrefix() API
2015-11-17 16:32:20 -08:00
f8e59e8399
386: Support minio server on 32bit linux.
2015-11-14 00:29:18 -08:00
d668117a99
fs: Improve upon proper lexical ordering for ListObjects()
...
Handle sorting properly making sure that we treat fs like a
flat namespace.
2015-11-11 14:52:03 -08:00
440bec28d9
docker: the docker image will now contain just the static binary
2015-11-06 20:44:58 -08:00
ab15f56a61
fs/bucket: Move bucket metadata into buckets.json
2015-11-01 21:25:01 -08:00
baf66988e9
Simplify bucket delete - remove only bucket directory, no need to recursively traverse
2015-10-30 16:03:18 -07:00
0010f0ee10
Remove empty directories while removing an oobject
2015-10-29 23:30:16 -07:00
53adfb38f4
Leverage sort Interface to provide sortUnique function
2015-10-23 15:55:41 -07:00
dbaa4d8643
Move ListObjects into its own file
2015-10-22 15:39:04 -07:00
1f66f4869b
Fix all remaining windows path issues.
2015-10-22 00:05:10 -07:00
afa27b9847
On windows translate Prefix, Marker and Delimiter for paths
2015-10-21 22:00:03 -07:00
b74852116a
Fix portability issues for arm on raspberry pi
2015-10-20 11:22:00 -07:00
dddb1650de
Add 5% cumulative reduction in total size of the disk
...
This is done due to filesystem holding additional metadata and inode space
which is unaccounted for during min-free-disk calculation.
2015-10-19 10:33:34 -07:00
179d2d7dac
Add initial cut of auto expiry of objects
2015-10-19 01:34:31 -07:00
c065be656c
Implement min-free-disk as a subcommand, deprecate flag
2015-10-19 00:59:20 -07:00
5b2fa33bdb
Implementing min-free-disk
2015-10-18 00:23:14 -07:00
a8a935f5fd
Improve disk code to return back disk StatFS{} structure
...
```
StatFS {
Total int64
Free int64
FSType string
}
```
Provides more information in a cross platform way.
2015-10-17 20:19:26 -07:00
aee0845b2e
Add disk package
...
Implements
- Stat returns total and free disk space supported across platforms
- Type returns type of the filesystem underneath
2015-10-17 16:48:24 -07:00
1256ca86d0
Add fs separator
2015-10-17 12:05:12 -07:00
2ec679a089
If directory already removed, return nil and move on
2015-10-17 00:13:46 -07:00
d534fc5a4f
Implement delete bucket properly with proper error handlings
2015-10-17 00:01:12 -07:00
c9af01d807
Enhance listing further, this time handle cases related to common prefixes
2015-10-16 23:11:41 -07:00
0eb7f078f9
Implement Bucket ACL support
2015-10-16 19:47:30 -07:00
762b798767
Migrate this project to minio micro services code
2015-10-16 11:26:08 -07:00
3318cba132
Add windows support for minhttp library
2015-10-11 01:08:16 -07:00
7fea9cb550
Add rpc signature handler
2015-10-08 22:28:11 -07:00
11048708bb
Implement GetBucketACL - fixes #893
2015-10-08 11:12:44 -07:00
bf901d3b9a
Merge pull request #895 from abperiasamy/tasker
...
new task model minio server
2015-10-08 02:23:29 -07:00
b52697e6ad
new task model minio server
2015-10-08 02:20:24 -07:00
a060b158c8
Add quick.CheckVersion() to verify config version quickly before unmarshalling the full struct
...
This is needed during migration where we would need to verify the underlying version number
in a quick way.
2015-10-07 17:44:33 -07:00
ab5ea997ab
Make erasure Encode and Decode atomic to avoid races
2015-10-06 23:05:01 -07:00
1b42398e8b
Canonicalize all the incoming input values, now PresignedPostPolicy works with minio-go
2015-10-06 10:21:28 -07:00
d5ce2f6944
Make erasure matrix type not optional choose automatically
...
Remove option of providing Technique and handling errors based on that
choose a matrix type automatically based on number of data blocks.
INTEL recommends on using cauchy for consistent invertible matrices,
while vandermonde is faster we should default to cauchy for large
data blocks.
2015-10-05 22:38:02 -07:00
4ed50a8004
Fix encoding bug in donut during encoding phase
...
Stream reading needs to check for length parameter being non zero,
after Reading() a predefined set of buffer length an EOF might be returned
with length == 0.
Erasure taking this zeroed data in might wrongly encode it as part of existing
data blocks which leads to errors while decoding even when the other contents
are intact.
2015-10-05 22:12:53 -07:00
f0a8dbecae
Add --json output formatter for server
2015-10-05 00:20:49 -07:00
c4faf47e64
First time mode for controller
...
- Upon first time invocation ``minio controller`` would create access keys and secret id
- Upon request passing 'keys' arg ``minio controller`` would provide the keys
- Add colorized notification
2015-10-04 16:42:16 -07:00
cfdb29cac0
Simplify signature handling
...
This change brings a new SignatureHandler where Presigned.
Requests without Payload are handled very early before even
going through the call.
This change simplifies Donut codebase to not have signature related
logic for all API's.
Simplification is still needed for Payload based signature eg. PUT/POST calls
, which are still part of the donut codebase, which will be done subsequently
after donut re-write.
2015-10-04 13:15:33 -07:00
c8de5bad2f
Implement presigned policy
2015-10-04 00:01:34 -07:00
b5ea05d839
A better way to print prettified json syntax error msg
2015-10-03 12:25:44 -07:00
3b070dee16
Fix an important metadata getObject bug in donut
2015-10-01 10:18:03 -07:00
81cc017f91
Implement presigned signature v4 support
2015-10-01 10:17:47 -07:00
50750efb52
Reduce memory usage for memory multipart write by doing io.Pipe() streaming copy
2015-09-30 20:53:30 -07:00
8c7c5df770
Prober stringer should avoid frivolous newlines
2015-09-29 10:13:11 -07:00
cd52d7a11c
Fix Linux/Mac OS X erasure decoding failure with new Golang version 1.5.1
...
Fixes #871
2015-09-24 21:42:59 -07:00
3785489153
Disable multipart for donut backend from being used
...
Will enable it later after cleanup
2015-09-24 18:54:30 -07:00
b938e40fb5
Move atomic package to the top and simplify its tests
2015-09-20 13:51:38 -07:00
d54488f144
Move all server and controller packages into top-level
2015-09-19 01:07:42 -07:00
89a86948b5
new version format and some cleanup
2015-09-18 23:33:28 -07:00
d1f1b7ac31
new version format and some cleanup
2015-09-18 23:27:04 -07:00
778f8cd222
Add new rpc tests for Server.Add and Server.List, improve Version.Get RPC to provide more details
2015-09-18 17:44:46 -07:00
2a15dd5eab
Enhance signature handler - throw back valid error messages
2015-09-18 15:14:55 -07:00
ac93bbb41d
Merge pull request #846 from harshavardhana/new-changes
...
With new auth config changes, restructure the API code to use the new style
2015-09-18 03:45:03 -07:00
6a5e5c1826
With new auth config changes, restructure the API code to use the new style
2015-09-18 03:41:05 -07:00
b4ce1e8c1d
Add missing reply.Name and add possible failure tests
2015-09-18 03:15:19 -07:00
f8bb85aeb7
Enhance auth JSONRPC, now provides persistent output
...
Implements
- Auth.Generate("user")
- Auth.Fetch("user")
- Auth.Reset("user")
This patch also adds testing for each of these cases
2015-09-18 03:02:39 -07:00
4bcd86408b
Fix minio header in accordance with rfc2616.txt
2015-09-17 23:46:10 -07:00
7093a05ab1
Version is a package now, will be re-used across codebase.
2015-09-17 20:17:33 -07:00
1887114444
Fix all the golint complaints about newly added changes
...
Do not use func(this *server), such generic names should not be used
for writing struct methods.
2015-09-17 18:53:42 -07:00
03ef6533c8
Add more documentation for probe
2015-09-17 18:10:42 -07:00
c49407ced4
rename rpc/server.go -> rpc/rpc.go. rpc/server.go will accomodate ServerService
2015-09-17 15:46:51 -07:00
3f4b98ca4c
Consolidate controller, move rpc package into controller - remove dangling code in pkg/server
2015-09-15 19:38:36 -07:00
8d5f6e0b96
Merge pull request #826 from krishnasrinivas/middleware-cleanup
...
Remove unneeded functions in middleware init
2015-09-15 18:11:45 -07:00
b1b387b157
Remove unneeded functions in middleware init
2015-09-15 18:09:09 -07:00
45d8898019
Merge branch 'parallel-read2' of https://github.com/krishnasrinivas/minio into krishnasrinivas-parallel-read2
...
Make few more changes and rebased with current master
2015-09-15 17:33:33 -07:00
6930e4d668
logrus logger
2015-09-09 20:28:09 -07:00
1e2c010174
Avoid shadowing variables and enable checks to avoid them during build
2015-09-09 15:14:55 -07:00
b649eff3fb
Disable logging for now
2015-09-05 21:28:35 -07:00
d0f945f8e7
Simplify erasure package for OSX
2015-09-05 20:19:43 -07:00
afff3f8885
Revert "Enable controller to have CORS"
...
This reverts commit f39ac24e99
2015-08-31 17:15:49 -07:00
1e82ee1192
Add CORS support to minio s3 server
2015-08-31 16:59:52 -07:00
f39ac24e99
Enable controller to have CORS
2015-08-31 01:47:05 -07:00
dcf0c71ca3
Merge pull request #810 from harshavardhana/restructure
...
Restructure server code, controller now runs in silo
2015-08-27 17:10:54 -07:00
025f95b1d6
Restructure server code, controller now runs in silo
2015-08-27 17:07:32 -07:00
c11aa1c892
return *probe.Error for Untrace() as well.
2015-08-24 03:35:24 -07:00
988d39a5b6
Migrate to golang1.5 release with GO15VENDOREXPERIMENT=1 enabled
2015-08-22 18:35:37 -07:00
0e416ea699
Tests were running 4 times due to multiple times the TestingT{} was being called
...
Calling multiple times TestingT{} will hook up runner for Suites for that many times
which would lead to repeated running tests.
Fix it by only initializing it once for all the Suites
2015-08-20 22:32:50 -07:00
068d1d1ba9
Add a new quick.Load() function to load directly any config file provided a quick compatible struct{} is also provided
2015-08-20 20:33:49 -07:00
b49b8cdbe8
return call stack in reverse
2015-08-19 22:40:27 -07:00
826202716f
skip Env in json printinf if empty
2015-08-19 01:21:28 -07:00
76c40e075a
make tracePoint public as well
2015-08-19 01:02:39 -07:00
ac928b5092
Fix typo in probe
2015-08-18 23:42:41 -07:00
cdf93e534c
simplify probe APIs
2015-08-18 19:30:17 -07:00
e9c5a51bc6
Migrate pkg/quick from mc
2015-08-13 16:29:55 -07:00
61175ef091
Migrate to govendor to avoid limitations of godep
...
- over the course of a project history every maintainer needs to update
its dependency packages, the problem essentially with godep is manipulating
GOPATH - this manipulation leads to static objects created at different locations
which end up conflicting with the overall functionality of golang.
This also leads to broken builds. There is no easier way out of this other than
asking developers to do 'godep restore' all the time. Which perhaps as a practice
doesn't sound like a clean solution. On the other hand 'godep restore' has its own
set of problems.
- govendor is a right tool but a stop gap tool until we wait for golangs official
1.5 version which fixes this vendoring issue once and for all.
- govendor provides consistency in terms of how import paths should be handled unlike
manipulation GOPATH.
This has advantages
- no more compiled objects being referenced in GOPATH and build time GOPATH
manging which leads to conflicts.
- proper import paths referencing the exact package a project is dependent on.
govendor is simple and provides the minimal necessary tooling to achieve this.
For now this is the right solution.
2015-08-12 19:24:57 -07:00
45b59b8456
Probe revamped to provide for a new WrappedError struct to wrap probes as error interface
...
This convenience was necessary to be used for golang library functions like io.Copy and io.Pipe
where we shouldn't be writing proxies and alternatives returning *probe.Error
This change also brings more changes across code base for clear separation regarding where an error
interface should be passed encapsulating *probe.Error and where it should be used as is.
2015-08-08 00:16:38 -07:00
f8141493bd
use command not found helper
2015-08-03 18:06:19 -07:00
d09fd8b0a1
Migrate from iodine to probe
2015-08-03 16:33:44 -07:00
884e9771b2
More changes to probe to avoid nil dereferences
2015-08-03 01:47:37 -07:00
65e4aede82
Minor changes to probe
2015-08-02 20:33:49 -07:00
697009c0a1
trace on New and add read locks
2015-08-02 11:58:28 -07:00
d9493909d8
Merge pull request #788 from krishnasrinivas/open-and-openfile
...
rename disk.OpenFile to Open which will do os.Open (which will be rea…
2015-08-02 11:05:04 -07:00
ee4432bc40
rename disk.OpenFile to Open which will do os.Open (which will be read-only). disk.OpenFile will do os.OpenFile (which can be rw, append)
2015-08-02 17:34:29 +05:30
a728ddc027
probe package to trace & return errors
2015-08-02 02:38:08 -07:00
aabfd541e1
Merge cmd/donut into minio cmd, deprecate controller RPC request
2015-07-31 12:57:15 -07:00
2671b2dbf4
crypto/cleanup: remove unused functions
2015-07-29 13:09:55 -07:00
fdd2c22fa5
Do go fmt on bucket.go
2015-07-29 23:49:25 +05:30
ae8089c9b6
when signature is not available there is no need to compute sha256
2015-07-29 21:09:02 +05:30
bdc00624fd
get erros from buffered channel. Return error during object read only if we have readers < k
2015-07-29 19:36:41 +05:30
d346250f1c
Collapse GetPartialObject() into GetObject()
2015-07-28 19:33:56 -07:00
e1280779ed
Read from the disks in parallel during object read
2015-07-28 18:04:17 +05:30
1ea91d2fa2
Use missingEncodedBlocksCount directly instead of "-1" workaround in missingEncodedBlocks[]
...
Makes Code more readable
2015-07-28 16:56:06 +05:30
b0ea64a04f
Strip off quotes from ETag for verifying complete multipart upload
2015-07-25 16:10:05 -07:00
e082f26e10
Improving EncoderStream to return error only upon non io.EOF.
...
io.EOF is okay since io.ReadFull will not have read any bytes at all.
Also making error channel receive only for go routine.
2015-07-25 15:57:30 -07:00
bcfaa12a4d
Encoder now directly reads from the object stream. Using split.Stream() was causing lot of redundant memory operations.
2015-07-26 03:54:39 +05:30
d6a0e0cc55
Rename more
2015-07-24 18:09:53 -07:00
63c9cf0c4b
Move from Minimalist Object Storage to Minio Cloud Storage
2015-07-24 17:51:40 -07:00
43c908d5b9
ListObjects now considers multipart objects, also move to upstream check.v1
2015-07-18 15:49:41 -07:00
966786c78e
remove unnecessary updateAt()
2015-07-17 19:48:09 -07:00
c1da10a207
Donut multipart support no get, listobjects() support yet
2015-07-17 16:23:51 -07:00
86a887f9d4
Add corresponding tests
2015-07-16 17:40:19 -07:00
5eae32f2b0
Return proper InvalidArgument messages like s3 for invalid data for ListObjects(), ListObjectParts(), ListMultipartUploads()
2015-07-16 17:22:45 -07:00
e605787e65
Method not allowed is right response for DELETE() operations and add tests
2015-07-16 14:15:24 -07:00
1f2e6a40a0
Fix an issue with reusing closed network connetion, changing the way rateLimitedListener is initialized
2015-07-16 10:56:14 -07:00
5507a39840
Heal buckets upon init if needed, adding new disks and HUP works
2015-07-16 09:59:05 -07:00
e4574c7d6f
Add File() method to Ratelimitlistener for extracting underlying fd()
2015-07-16 00:11:17 -07:00
2e8f154f34
Iodine should indent with EmitJSON()
2015-07-16 00:10:55 -07:00
4498662c16
Remove scsi non portable code, instead "donut make" implements functionality to instantiate a donut
2015-07-15 11:55:57 -07:00
6baf45e360
Fix a crash during listObjects() populating NextMarker
2015-07-15 10:04:15 -07:00
2553654e24
Add initial version of heal, remove rebalance
2015-07-14 22:56:41 -07:00
e37c5315ec
Live multiple disk removal works properly
2015-07-14 20:46:14 -07:00
e1e4908515
Wire up sha512 matching inside donut along with md5sum
2015-07-14 19:47:50 -07:00
e885259584
Handle removal of disks - getObject() now reads if disks are missing underneath, add initial stub healing code
2015-07-14 18:53:00 -07:00
efbf3eabb7
Return x-amz-request-id for all replies
2015-07-14 14:44:03 -07:00
da8b9fd112
Remove global custom config path variables, use get/set methods instead
2015-07-14 11:56:15 -07:00
c4cf7635bf
Read req.Body for PutBucket() if any
2015-07-14 09:30:10 -07:00
45ddec925c
Fix an ugly multipart bug
2015-07-13 21:38:01 -07:00
7ae60a6d10
For missing parts reply back as InvalidPart{}
2015-07-13 19:31:01 -07:00
634f70f3b0
WriteObject() encoded data in go routine, fix another multipart issue
2015-07-13 15:56:54 -07:00
1bad92356d
Add a ratelimited listener than a ratelimited handler - more precise
2015-07-13 11:12:17 -07:00
8af5933b07
Merge pull request #743 from flandr/fix-osx
...
Fix OS X build
2015-07-13 10:15:53 -07:00
ec347f96fd
Fix OS X build
...
- Explicitly cast statfs_t members to int64 (this structure is
platform-specific)
- Add pass-through New methods to Darwin SHA package
- Move scsi pkg types to common translation unit (package was empty)
- Add stub implementations mount/disk ops for OS X
2015-07-13 10:06:55 -07:00
55e4d0c6a5
mkdonut now creates a donut processing cli args
2015-07-12 21:21:31 -07:00
58a1d865a9
Renaming nimble to minhttp
2015-07-12 18:01:52 -07:00
847440196e
Add signature v4 tests
2015-07-12 13:36:02 -07:00
7615a6bfe5
HEAD shouldn't have any body, handle it in writeErrorResponse()
2015-07-11 10:34:55 -07:00
8977f9a524
cached api test should also have a custom config path, would conflict with your .minio local path
2015-07-10 20:39:30 -07:00
97d4a27c7e
Do not reply on ignoredHeaders for server, rely on SignedHeaders sent as part of Authorization header
2015-07-10 19:37:22 -07:00
53f5d2c32b
nodejs http library sends Connection header during HTTP request, this clobbers up the signature handling ignore it
2015-07-10 15:44:47 -07:00
29838bb851
Cleanup temporary writers upon errors during putObject(), all metadata() write operations
2015-07-10 14:11:04 -07:00
15dd0df187
Support signature v4 at rest
2015-07-10 11:49:27 -07:00
7fa514351c
Rename definitions to log.go, add valid prefixes
2015-07-10 11:11:20 -07:00
d5ffc16f25
Add abbreviated close response, to avoid any leaks
2015-07-10 10:20:00 -07:00
e5006c738d
Handle both space and non-space characters, in signature v4 - add errors for all API's
2015-07-09 21:49:58 -07:00
84f427f14a
All other API's now support signature v4
2015-07-09 19:45:56 -07:00
00890c254e
CompleteMultipartUpload and CreateObjectPart now fully support signature v4
2015-07-09 19:01:15 -07:00
89c1215194
PutObject handler gets initial support for signature v4, working
2015-07-09 16:44:38 -07:00
d461fa5ab1
Add mountinfo functions for detecting mount disks, and other rpc changes
2015-07-09 12:25:29 -07:00
8654ddb566
Generate auth now saves in ${HOME}/.minio/users.json, also authHandler verifies request validity
2015-07-08 21:53:13 -07:00
375860077d
HTTP header Content-Length signifies body length of the request, if its smaller reply appropriately
...
This patch also handles large individual part sizes > 5MB by using less memory copies.
2015-07-08 20:56:41 -07:00
ec33d79d57
Add server side signaturev4 check, not wired up to the readers yet.
2015-07-08 16:57:03 -07:00
396b728031
Add auth rpc service to generate access keys, add corresponding test
2015-07-08 14:40:39 -07:00
770fd23afa
Renaming keys as auth, working towards signature v4 support for all put objects
2015-07-08 14:17:16 -07:00
2413a110e6
Fix SSL support, pointer indirection caused nil buffers
2015-07-08 11:02:15 -07:00
d1deda3a96
Add API tests for both donut on disk and donut cache
2015-07-07 19:39:46 -07:00
ece797c16e
Add rpc tests
2015-07-07 17:27:34 -07:00
676b9058de
Separate out memory statistics and system information into two different services
2015-07-07 16:59:20 -07:00
8abb96c030
If NodeDisks are not empty do not impose cache maxSize restriction
2015-07-07 16:41:40 -07:00
a50a44b0ca
Add nimbleNet tests
2015-07-07 16:15:47 -07:00
317096a0c4
Add net.Addr wrapper with IsEqual() and use it.
2015-07-07 15:33:08 -07:00
11b893804c
Moving os.MkdirAll() inside atomic for auto parent directory creates
2015-07-07 12:35:57 -07:00
52cd23ad9f
Move atomic file writes into its own package, use them inside quick and disk packages
2015-07-07 12:29:14 -07:00
3622fbc87d
Across donut, split, nimble some code cleanup
2015-07-06 21:55:21 -07:00
bbb89b5776
Add multi-thread protection and also allow atomic file creates, rename upon Close()
2015-07-06 18:54:32 -07:00
c2c7bdf0cd
Cleanup nimble http
2015-07-06 18:51:20 -07:00
b029d0a5f0
Avoid config reload all the time, reload is manually triggerred from outside
2015-07-06 17:26:35 -07:00
8b94c53345
Fix issues with multipart upload
2015-07-06 16:22:27 -07:00
474954022e
Add modified grace library from facebookgo, rename it as nimble
2015-07-06 15:40:12 -07:00
1d64e4b6c1
Add Donut rpc service for sending changes to configuration files
2015-07-06 11:10:06 -07:00
10b082144e
Add updateConfig code to load config changes if possible for every function
2015-07-05 22:46:42 -07:00
ba0a5ed416
Add sighup, sigusr2 into trapping code, to trap signals for reloading configuration.
...
Need to still figure out a way of graceful restarts - gave facebookgo/httpdown a shot,
but it is not suitable.
2015-07-05 21:40:53 -07:00
4a27ab0e58
Fix another deadlock inside CreateObjectPart() code, premature return without Unlocking()
...
Also this patch changes the cache key element to be interface{} type not string.
2015-07-05 20:26:32 -07:00
75788c7a1d
Fix go installation check on amazon instance
2015-07-05 18:12:58 -07:00
7f0c14f2b7
Add basic controller code, initiating json rpc connection getting list of disks and memstats for now.
2015-07-05 17:17:41 -07:00
a3ccb9d405
Rename stuttered service names and make them appropriate
2015-07-05 10:19:23 -07:00
e66a84242a
Add disk detection for Linux, add new RPC service GetDiskInfoService(), remove dummy HelloService()
2015-07-05 02:08:33 -07:00
bab4a47525
Move to container/list datastructure from map[string][]byte
2015-07-04 17:08:23 -07:00
aa67a19e99
Implement new CPU detection using cpuid, cpuidex plan9 instructions from klauspost/cpuid project, remove C code
2015-07-04 14:28:16 -07:00
0a827305ad
Implement metadata cache, metadata cache is used by top level donut right now. Rename trove as data cache
...
We should use it internally everywhere.
2015-07-03 21:09:57 -07:00
14844f48dd
Make donut do everything as an atomic operation, this avoids all the deadlocks and races
2015-07-03 17:16:58 -07:00
30fc14e703
Restructure codebase move crypto, checksum to top-level, move ``split`` into donut, move crypto/keys into api/auth
2015-07-03 15:24:51 -07:00
8a4e7bcdcf
Add full API tests, move storage/donut to donut, add disk tests as well
2015-07-03 14:36:29 -07:00
7c37e9d06a
Make donut fully integrated back into API handlers
2015-07-02 21:04:04 -07:00
12bde7df30
Add simple Ticket Master which pro-actively sends messages on proceedChannel
...
Handlers are going to wait on proceedChannel, this the initial step towards
providing priority for different set of API operations
2015-07-02 21:04:04 -07:00
5cfb05465e
Add cache, donut tests separately - fix behavior differences
...
Remove priority queue, implement it using a simpler channels
2015-07-02 21:04:04 -07:00
ebe61d99d9
Use cache Append() for saving objects in memory, GetObject() caches un-cached entries while reading
2015-07-02 21:04:04 -07:00
bce93c1b3a
Integrate cache with donut, add tests
2015-07-02 21:04:04 -07:00
0533abf6a8
Make priority queue lambda function return error over a channel
2015-07-02 21:04:04 -07:00
38a6ce36e5
Remove slow AppendUniq code, rolling through over a slice is in-efficient
...
Remove it and use map instead
2015-07-02 21:04:04 -07:00
84810162f5
Add simple Version and GetSysInfo services
2015-07-02 21:04:04 -07:00
14ec42d646
Add initial implementation of priority queue, uses container/heap
2015-07-02 21:04:04 -07:00
eb5aa19dfa
Remove custom Config, will use quick Config instead for user access keys
2015-07-02 21:04:04 -07:00
701c3e5242
Add new RPC helpers wrapping over regular rpc packages, add middleware chaining ability
2015-07-02 21:04:04 -07:00
4addf7a996
Restructure API handlers, add JSON RPC simple HelloService right now.
2015-07-02 21:04:04 -07:00
335c7827eb
More donut, cache, api cleanup
2015-07-02 21:04:04 -07:00
dc0df3dc0e
Breakaway from driver model, move cache into donut
2015-07-02 21:04:03 -07:00
72572d6c71
Remove some api server code bringing in new cleanup
2015-07-02 21:04:03 -07:00
c2031ca066
Add server and control command
2015-07-02 21:04:03 -07:00
cfbc169034
Fixing API definitions
2015-07-02 13:14:21 -07:00
8f61d6b6be
Move memory code out, add it as layer on top of existing donut code
...
Just like how http.Handlers can be overlayed on top of each other
with each implementing ServeHTTP().
drivers.Driver can be overlayed on top of each other in similar manner
which would implement the drivers.Driver interface.
API <----> cache <----> donut <----> donut(format)
2015-06-30 10:09:12 -07:00
ab6e16bb41
Add dummy driver for community to submit new drivers
2015-06-29 16:43:50 -07:00
12de98fb62
Rename memory driver as cache
2015-06-29 16:43:50 -07:00
2571342451
Filesystem goes the high road *again*
2015-06-29 16:43:42 -07:00
f74d6138da
IsValidBucket() is sufficient we don't need to verify for "."
2015-06-29 15:15:54 -07:00
3109909355
Handle couple of cases of OOM conditions, move caching to GetObject() rather than PutObject()
2015-06-29 12:28:50 -07:00
d07d0c670a
Return back proper errors in writeObjectData(), rename few functions
2015-06-29 11:46:35 -07:00
10c807f233
Put object on successful write returns full metadata, to avoid subsequent GetObjectMetadata() calls in driver
2015-06-29 11:15:46 -07:00
6921328b93
Avoid frivolous GetObjectMetadata() calls at driver level, return back all the information in donut ListObjects()
2015-06-29 11:14:58 -07:00
d8f7896a43
Expand http server struct to store more values
2015-06-29 00:12:28 -07:00
42c0287943
Use errorChannels only for services not for drivers, reduce them to use simple functions
2015-06-28 23:59:47 -07:00
91e5f648cb
Move to Set() not Append() due to large memory reference copy
2015-06-28 10:13:12 -07:00
ac4d8fe478
Add free() method for proxyReader to aggressively de-allocate Read data, to handle certain out of memory conditions
...
There are still some more out there
2015-06-27 20:43:25 -07:00
05f8654e3d
Add Append() method to trove cache for appending data to an existing key
...
This largely avoids a large buffer copy which would accumulate inside proxyReader{}
This patch also implements "initialize()" function to init and populate data
on all the existing buckets, avoiding the redundant ListBuckets() invoked by
every API call.
2015-06-27 20:25:24 -07:00
367772b988
Make sure to populate on disk data into memory upon first API requests
2015-06-27 18:25:21 -07:00
45a7eab804
An attempt to bring in memory layer into donut driver
2015-06-27 17:23:34 -07:00
f3c25bcfc4
Keeping the lexical order same add optimizations, provide a comprehensive response from ListObjects()
2015-06-27 13:12:44 -07:00
ae66ae42c4
Rename functions for their purpose
2015-06-27 12:39:11 -07:00
9a4680475f
Object metadata was wrongly misconstrued to be mutable, handle it
2015-06-26 23:22:53 -07:00
3aa6d90c5e
Avoid racy maps, read from disk on success return quickly. Many more optimizations
2015-06-26 19:49:37 -07:00
68974918ac
Donut moves to typed metadata, removing the necessity for strconv, parsing and string converstions
2015-06-26 16:23:12 -07:00
767d3743ee
Add new metadata definitions and use them
2015-06-26 13:34:09 -07:00
9958e34772
Import quick key value store from Minio Client for persistent state files, primarily for donut
2015-06-25 19:57:31 -07:00
fb9adb5524
Donut cleanup another set
...
- Make sure to close all readers
- Fix errors in api_testsuite c.Assert(err, IsNil) should be done right after each function call
2015-06-25 18:54:34 -07:00
eec66f195a
Take all the ListObjects into bucket handlers
...
Earlier the listing would wait for all the objects to be processed
this is essentially very time consuming considering even for 100,000
files.
2015-06-25 18:04:29 -07:00
45e9d25931
Remove more bloated code - simplify
2015-06-25 13:02:08 -07:00
82dcbf262d
Add simple locking for donut API for now - fixes #671
2015-06-25 11:29:11 -07:00
03b4d3b308
Go vet fixes for donut
2015-06-24 21:07:03 -07:00
57a2b53178
Removing further bloated code simplifying
2015-06-24 21:03:39 -07:00
a2c205ff2e
Use external package disk for donut.
2015-06-24 21:03:02 -07:00
Harshavardhana
Anis Elleuch
Krishnan Parthasarathi
karthic rao
Krishna Srinivas
Bala FA
Harshavardhana
karthic rao
Karthic Rao
Brendan Ashworth
Anand Babu (AB) Periasamy
Anand Babu (AB) Periasamy
Krishna Srinivas
Anis ELLEUCH
Anand Babu (AB) Periasamy
Nate Rosenblum
Frederick F. Kautz IV