Commit Graph

3789 Commits

Author SHA1 Message Date
Harshavardhana
b363709c11 caching: Optimize memory allocations. (#3405)
This change brings in changes at multiple places

 - Reuse buffers at almost all locations ranging
   from rpc, fs, xl, checksum etc.
 - Change caching behavior to disable itself
   under low memory conditions i.e < 8GB of RAM.
 - Only objects cached are of size 1/10th the size
   of the cache for example if 4GB is the cache size
   the maximum object size which will be cached
   is going to be 400MB. This change is an
   optimization to cache more objects rather
   than few larger objects.
 - If object cache is enabled default GC
   percent has been reduced to 20% in lieu
   with newly found behavior of GC. If the cache
   utilization reaches 75% of the maximum value
   GC percent is reduced to 10% to make GC
   more aggressive.
 - Do not use *bytes.Buffer* due to its growth
   requirements. For every allocation *bytes.Buffer*
   allocates an additional buffer for its internal
   purposes. This is undesirable for us, so
   implemented a new cappedWriter which is capped to a
   desired size, beyond this all writes rejected.

Possible fix for #3403.
2016-12-08 20:35:07 -08:00
Anis Elleuch
410b579e87 startup: Show elapsed time in disks format process (#3413) 2016-12-07 10:22:00 -08:00
Karthic Rao
7b7c0bba58 Use a non member mutex lock for serverConfig access. (#3411)
- This is to ensure that the any new config references made to the
  serverConfig is also backed by a mutex lock.
- Otherwise any new config assigment will also replace the member mutex
  which is currently used for safe access.
2016-12-07 03:41:54 -08:00
Anis Elleuch
0cef971832 Fix max cache size calculation when system RAM is inferior to the default cache size (#3410) 2016-12-06 16:09:26 -08:00
Anis Elleuch
5c9a95df32 srv-mux: do not print peek protocol EOF err msg (#3402)
EOF err message in Peek Protocol is shown when a client closes the
connection in the middle of peek protocol, this commit hides it since it
doesn't make sense to show it
2016-12-05 14:49:32 -08:00
Anis Elleuch
3b455d6137 tests: Add tests for xl-v1-list-objects-heal (#3399) 2016-12-05 09:40:33 -08:00
Anis Elleuch
b2a0e5754b bucket-handlers: More tests for post form handler (#3392) 2016-12-04 12:23:19 -08:00
Anis Elleuch
63d9bb626a postform: fix check when ${filename} is provided (#3391)
Checking key condition when ${filename} is provided wasn't working well,
this patch fixes the wrong behavior
2016-12-04 10:30:52 -08:00
Anis Elleuch
372da5eaf5 tests: Enhance checkPostPolicy() coverage (#3389) 2016-12-03 12:41:07 -08:00
Harshavardhana
cf17fc7774 fs: PutObject create 0byte objects properly. (#3387)
Current code always appends to a file only if 1byte or
more was sent on the wire was affecting both PutObject
and PutObjectPart uploads.

This patch fixes such a situation and resolves #3385
2016-12-03 11:53:12 -08:00
Krishnan Parthasarathi
67509453d3 FS: sync abortMultipart cleanup and bg append (#3388)
backgroundAppend type's abort method should wait for appendParts to finish
writing ongoing appending of parts in the background before cleaning up
the part files.
2016-12-02 23:33:06 -08:00
Harshavardhana
d31f256020 Fail on lint errors during CI build. 2016-12-02 18:08:12 -08:00
Harshavardhana
d67f47927c api: Fix the formatting issues in last patch. 2016-12-02 17:39:21 -08:00
Anis Elleuch
85bb5870a9 Post Policy Form: exhaustive post policy check (#3386)
Add support of all conditions check described in
http://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-HTTPPOSTConstructPolicy.html
and simplify a little bit the existing code
2016-12-02 17:00:33 -08:00
Harshavardhana
4a9b205a15 docs: Add missing comments for exported functions. 2016-12-02 14:39:23 -08:00
koolhead17
f3b346cbb3 docs: removed the Edge tag reference. (#3366) 2016-12-02 10:47:18 -08:00
Harshavardhana
ff4ce0ee14 fs/xl: Combine input checks into re-usable functions. (#3383)
Repeated code around both object layers are moved
and combined into simple re-usable functions.
2016-12-01 23:15:17 -08:00
Anis Elleuch
918924796f fs: Enable shutdown test with faulty disks (#3380) 2016-12-01 13:59:06 -08:00
Krishnan Parthasarathi
feb6685359 posix: Use preparePath only for paths used with syscall or os functions (#3377) 2016-11-30 20:56:15 -08:00
Bala FA
0d59ea1e94 postpolicy: handle Amazon S3 compatible content-length-range condition (#3376)
Previously minio server expects content-length-range values as integer
in JSON.  However Amazon S3 handles content-length-range values as
integer and strings.

This patch adds support for string values.
2016-11-30 18:30:59 -08:00
Krishna Srinivas
38edd94282 ListBuckets: Allow listBuckets request to be signed with region configured in config.json (#3374)
Fixes #3373
2016-11-30 13:55:56 -08:00
Krishna Srinivas
8021061bd8 Implement BucketUpdater interface to call BucketMetaState methods. (#3375) 2016-11-30 13:37:38 -08:00
Anand Babu (AB) Periasamy
bc9509bc8a edits on Limits 2016-11-30 01:58:31 -08:00
Krishna Srinivas
e3b4910b66 FS/CompleteMultipart: lock the namespace before renaming the appended tmp file. (#3371) 2016-11-29 23:26:36 -08:00
Harshavardhana
d056f19d07 api: Allow reconnection of policy/notification rpc clients. (#3368)
Since we moved out reconnection logic from net-rpc-client.go
we should do it from the top-layer properly and bring back
the code to reconnect properly in-case the connection is lost.
2016-11-29 22:39:32 -08:00
Harshavardhana
834007728c fs: Do not print redundant md5Sum response header. (#3369)
For both GET and HEAD requests.
2016-11-29 16:47:01 -08:00
Krishnan Parthasarathi
a609a4126c Vendorize recent fixes in dsync (#3365)
* Update 'i' only if lock grant read from buffered channel
2016-11-28 23:03:46 -08:00
Krishna Srinivas
bcd1a2308b FS/Shutdown: cleanup and delete .minio.sys during Shutdown() (#3360) 2016-11-28 22:54:48 -08:00
koolhead17
694bad434c docs: Modified the Docker doc to reflect distributed release image & (#3362)
modification.
2016-11-28 12:45:35 -08:00
Anis Elleuch
01f625824a args: Honor config-dir & quiet wherever they are (#3356)
setGlobalsFromContext() is added to set global variables after parsing
command line arguments. Thus, global flags will be honored wherever
they are placed in minio command.
2016-11-28 12:15:36 -08:00
Bala FA
9ccfb70104 Minor cleanup. (#3361) 2016-11-28 12:14:24 -08:00
Harshavardhana
201a20ac02 handlers: Handle re-direction properly for S3 requests. (#3355)
Make sure all S3 signature requests are not re-directed
to `/minio`. This should be only done for JWT and some
Anonymous requests.

This also fixes a bug found from https://github.com/bji/libs3

```
$ s3 -u list

ERROR: XmlParseFailure
```

Now after this fix shows proper output
```
$ s3 -u list
                         Bucket                                 Created
--------------------------------------------------------  --------------------
andoria                                                   2016-11-27T08:19:06Z
```
2016-11-27 16:30:46 -08:00
Krishna Srinivas
f3322e94c8 FS: Skip creating fs.json for objects created by minio (ex. policy.json) (#3353) 2016-11-27 11:33:08 -08:00
Harshavardhana
46a6fde813 xl/fs: Fix initializing meta volume bug. 2016-11-25 18:17:53 -08:00
Anis Elleuch
fd1f09a66c log: Enable loggers just after configuration load (#3348)
It would make sense to enable logger just after config initialisation.
That way, errorIf() and fatalIf() will be usable and can catch error
like invalid access and key errors.
2016-11-25 10:39:00 -08:00
Remco Verhoef
752ed7915b Add windows services docs with nssm instead of sc. 2016-11-24 16:53:10 -08:00
Bala FA
d3064e40b3 isDocker() logs error than fatal error. (#3347) 2016-11-24 16:06:49 -08:00
Bala FA
39f9324616 Remove uncessary err != nil check. (#3346) 2016-11-24 15:22:33 -08:00
Aditya Manthramurthy
a822b8e782 docs: Fix command sample to be copy-pastable (#3345) 2016-11-24 11:38:19 -08:00
Harshavardhana
a6922f94ee docs: Fix more formatting issues. 2016-11-23 20:25:05 -08:00
Harshavardhana
b4d57bf4c4 docs: Fix some formatting issues for rendering 2016-11-23 20:20:12 -08:00
Bala FA
0f2e493c9a Use isErrIgnored() function wherever applicable. (#3343) 2016-11-23 20:05:04 -08:00
Nitish Tiwari
4ef2d8940c Rename quick-start-guide.md to README.md (#3344) 2016-11-23 20:04:29 -08:00
Nitish Tiwari
96099a1e97 Added distributed Minio architecture images and quick start guide (#3342) 2016-11-23 19:30:23 -08:00
Harshavardhana
0e87f29de9 Disable heal message printing, comment it out as todo. 2016-11-23 17:54:29 -08:00
Harshavardhana
dd74e5a809 Revert "init: Honor config-dir flag when it is passed as global or local flag (#3337)"
This reverts commit e2ef95af7d.

This is reverted since the previous patch caused crashes.
2016-11-23 17:31:36 -08:00
Harshavardhana
12c1abed98 Vendorize with bug fixes from minio browser. (#3341)
This patch brings in changes from miniobrowser repo.

- Bucket policy UI and functionality fixes by @krishnasrinivas
- Bucket policy implementation by @balamurugana
- UI changes and new functionality changing password etc. @rushenn
- UI and new functionality for sharing URLs, deleting files
  @rushenn and @krishnasrinivas.
- Other misc fixes by @vadmeste @brendanashworth
2016-11-23 17:31:11 -08:00
Anis Elleuch
e2ef95af7d init: Honor config-dir flag when it is passed as global or local flag (#3337)
setGlobalsFromContext() is added to sets global variable after parsing
command line arguments.
2016-11-23 17:13:40 -08:00
Harshavardhana
d711ff454e logs: Do not log common successful errors. (#3340)
Errors like `BucketNotFound`, `BucketExists` shouldn't be logged.

Fixes #3229
2016-11-23 16:36:26 -08:00
Anis Elleuch
c667d20dfc config-migrate: Fix buggy continuous re-migration of v9 to v10 config (#3338) 2016-11-23 15:53:55 -08:00