Commit Graph

412 Commits

Author SHA1 Message Date
Anis Elleuch
5ff30777e1 Rewrite connection muxer peek process to avoid server blocking by silent clients (#3187) 2016-11-06 11:41:01 -08:00
Anis Elleuch
754c0770d6 Merge ListenAndServe and ListenAndServeTLS for simplification purpose (#3186) 2016-11-05 20:32:13 -07:00
Harshavardhana
1105508453 connection muxer should use bufio.Reader to be simpler. (#3177) 2016-11-05 12:57:31 -07:00
Harshavardhana
1ba497950c Fix net.Listener to fully close the underlying socket. (#3171)
Leads to races and accepting connections. This patch implements
a way to reject accepting new connections.
2016-11-05 09:43:28 -07:00
Aditya Manthramurthy
eb1bc67db1 Refactor: simplify signV4TrimAll() (#3179) 2016-11-04 13:52:22 -07:00
Krishna Srinivas
8408dfaa6c bootup-validation: Allow port configuration only using --address option. (#3166) 2016-11-04 12:14:19 -07:00
Harshavardhana
d192044915 router: PathPrefix router was wrong. (#3172) 2016-11-04 12:13:22 -07:00
Anis Elleuch
d9bab6b3bd sigv4: Trim and shrink spaces in headers values (#3162) 2016-11-03 16:41:25 -07:00
Anis Elleuch
e6965ca066 Quit initializing disks process when term signal is invoked (#3163) 2016-11-02 15:27:36 -07:00
Harshavardhana
d9674f7524 Improve coverage of web-handlers.go (#3157)
This patch additionally relaxes the requirement for
accesskeys to be in a regexy set of values.

Fixes #3063
2016-11-02 14:45:11 -07:00
Anis Elleuch
79601d27f2 Use endpoint url when printing disks status in distributed mode (#3151) 2016-11-02 08:51:06 -07:00
Aditya Manthramurthy
226a69fe15 Add test for updateUploadJSON (Fixes #3060) (#3155) 2016-11-01 20:04:32 -07:00
Harshavardhana
9bb799462e Start all listeners when a given host resolves to multiple IPs. (#3145)
Default golang net.Listen only listens on the first IP when
host resolves to multiple IPs.

This change addresses a problem for example your ``/etc/hosts``
has entries as following

```
127.0.1.1  minio1

192.168.1.10 minio1
```

Trying to start minio as

```
minio server --address "minio1:9001" ~/Photos
```

Causes the minio server to be bound only to "127.0.1.1" which
is an incorrect behavior since we are generally interested in
`192.168.1.10` as well.

This patch addresses this issue if the hostname is resolvable
and gives back list of addresses associated with that hostname
we just bind on all of them as it is the expected behavior.
2016-11-01 15:38:28 -07:00
Karthic Rao
8bffa78f7f Fix Instance type during benchmarks. (#3147)
- The benchmark initialization function was not taking into account the
  instance type (FS/XL), was using XL ObjectLayer even for FS
  benchmarks.
- This was leading to incorrect benchmark results for FS related
  benchmarks.
- The fix takes into account the instance type (FS/XL) and correctly
  returns FS backend for FS benchmarks.
2016-11-01 10:21:16 -07:00
Anis Elleuch
4b302173ae fs: Remove object append file if found since it is not always used (#3139) 2016-11-01 00:56:03 -07:00
Karthic Rao
9db2205db3 Fix List-Object Benchmark failure. (#3142) 2016-10-31 13:57:30 -07:00
Aditya Manthramurthy
6914fe436c Remove unused function. (#3143) 2016-10-31 12:18:15 -07:00
Anis Elleuch
807cc3c28d Tracing locking errors for better debugging (#3140) 2016-10-31 10:17:14 -07:00
Harshavardhana
f3c6c55719 posix: Fix windows performance issues. (#3132)
Do not attempt to fetch volume/drive information for
each i/o situation. In our case we do this in all calls
`posix.go` this in-turn created a terrible situation for
windows. This issue does not affect the i/o path on Unix
platforms since statvfs calls are in the range of micro
seconds on these platforms.

This verification is only needed during startup and we
let things fail at a later stage on windows.
2016-10-31 09:34:44 -07:00
Karthic Rao
3e8cb8c937 build: ineffassign fixes. (#3134) 2016-10-30 23:32:46 -07:00
Aditya Manthramurthy
dd6ecf1193 Read/write quorum algo with uploads.json in xl (Fixes #3123) (#3124)
- Reads and writes of uploads.json in XL now uses quorum for
  newMultipart, completeMultipart and abortMultipart operations.

- Each disk's `uploads.json` file is read and updated independently for
  adding or removing an upload id from the file. Quorum is used to
  decide if the high-level operation actually succeeded.

- Refactor FS code to simplify the flow, and fix a bug while reading
  uploads.json.
2016-10-30 09:27:29 -07:00
Anis Elleuch
a47ce7ab22 Add support of fallocate for FS and XL backends (#3032) 2016-10-29 12:44:44 -07:00
Krishna Srinivas
0b3282ac9f Logging: errorIf fatalIf print in the format [file.go:82:funcName()] (#3127) 2016-10-29 02:34:16 -07:00
Krishna Srinivas
79b98b5c25 boot: getPath() should take care of simple directory exports. (#3122)
fixes #3120
2016-10-28 13:08:56 -07:00
Krishnan Parthasarathi
6a57f2c1f0 XL: Add more information to panic msg (#3119) 2016-10-28 08:46:03 -07:00
Krishna Srinivas
0fa2477cb0 boot: check parameter syntax before initializing the system. (#3114)
fixes #3112
2016-10-28 08:45:32 -07:00
Anis Elleuch
7ffb337cd7 Update control cmd USAGE to indicate that it is possible to put access secret keys in URL (#3115) 2016-10-27 13:26:11 -07:00
Harshavardhana
9e2d0ac50b Move to URL based syntax formatting. (#3092)
For command line arguments we are currently following

- <node-1>:/path ... <node-n>:/path

This patch changes this to

- http://<node-1>/path ... http://<node-n>/path
2016-10-27 03:30:52 -07:00
Aditya Manthramurthy
30dc11a931 No listener.json for single-node mode (Fixes #3052) (#3108)
In FS or single-node XL mode, there is no need to save listener
configuration to persistent storage. As there is only one server, if it
is restarted, any connected listenBucketAPI clients were disconnected
and will have to reconnect - so there is nothing to actually store.

This incidentally solves #3052 by avoiding the problem.
2016-10-26 20:13:00 -07:00
Anis Elleuch
a15dc5fed5 Print message when creating the config file (#3089) 2016-10-26 18:44:22 -07:00
Anis Elleuch
f7c20b97a1 control cmds: Extract access and secret keys from URL if specified (#3109) 2016-10-26 18:42:12 -07:00
Harshavardhana
e9c45102b0 posix: Use sync.Pool buffers to copy in large buffers. (#3106)
These fixes are borrowed from the fixes required for GlusterFS i/o throughput.
2016-10-26 17:14:05 -07:00
Anis Elleuch
8871eb8e1e Show offline nodes after a fixed number of init retry (#3107) 2016-10-26 16:09:06 -07:00
Krishna Srinivas
0f32efb825 PostPolicy - rename of files/functions + add testcases (#3104) 2016-10-26 10:15:57 -07:00
Karthic Rao
63f1b4fdf4 mispell fixes. (#3100) 2016-10-26 08:46:14 -07:00
Krishnan Parthasarathi
31f2db6880 Remove leftover debug statement from PutObject StreamingSignature unit-test (#3099) 2016-10-26 03:17:47 -07:00
Krishnan Parthasarathi
2c9b406f6c Add TLS based tests to functional test suite (#3083) 2016-10-26 02:30:31 -07:00
Krishnan Parthasarathi
12cd2da265 Add PutObjectHandler unit tests covering failure cases (#3096) 2016-10-26 02:06:22 -07:00
Harshavardhana
485c0ea8bf tests: Combine v2 tests with the Suite itself. (#3088) 2016-10-25 13:34:14 -07:00
Krishnan Parthasarathi
49ba07d1d6 Use net.ParseCIDR instead of custom-built parsers (#3055)
Removes avoidable conversion to and from net.IP to string.
2016-10-25 11:14:47 -07:00
Krishna Srinivas
35e541e0b1 content-length-range policy should be honored for the uploaded object sizes. (#3076) 2016-10-24 23:47:03 -07:00
Aditya Manthramurthy
3977d6b7bd Lock bucket while modifying its metadata (Fixes #2979) (#3019)
- When modifying notification configuration
- When modifying listener configuration
- When modifying policy configuration

With this change we also stop early checking if the bucket exists, since
that uses a Read-lock and causes a deadlock due to the outer Write-lock.
2016-10-24 19:52:24 -07:00
Harshavardhana
0905398459 Fix benchmark tests. (#3082)
Fixes #3081
2016-10-24 18:45:06 -07:00
Aditya Manthramurthy
f41faf96b7 Fix newMultipartUpload to not leave stale uploads.json (Fixes #3014) (#3079) 2016-10-24 17:37:18 -07:00
Bala FA
36639b65a9 rename completeMultipartMD5() into getCompleteMultipartMD5(). (#3051) 2016-10-24 13:56:13 -07:00
Harshavardhana
7fc598b73f Fix user-agent prefix to have docker instead of suffix. (#3074) 2016-10-24 13:44:15 -07:00
Krishna Srinivas
21d41ad7fd init[windows]: Fix to handle the case when export path is a relative path. (#3054)
ex. to handle "minio server export"
2016-10-24 08:26:28 -07:00
Harshavardhana
fe56220d1a Do not print nil when hostname is provided as --address (#3053)
Fixes #3018
2016-10-23 23:55:12 -07:00
Harshavardhana
5782ec3ada Fix peers and web UIVersion validation. (#3048) 2016-10-23 12:32:35 -07:00
Krishnan Parthasarathi
8839c5105a Pass values to closures esp. when passed to defer statement. (#3050)
opsID, a variable on the stack, changes over the course of
Completemultipartupload function in xl-v1-multipart.go.  This was
being used in a function closure which was passed to defer
statement. The variables used in the closure depend on their values at
the time of evaluation which is indeterminate behaviour. It is
incorrect to depend on values of variables on stack at the end of
function, when deferred functions are executed.
2016-10-23 09:57:52 -07:00
Anis Elleuch
6c2d5e3d05 Correct the number of failed disks that we can withstand in startup message (#3045) 2016-10-22 10:36:50 -07:00
Krishna Srinivas
5999a23d3e When object whose size is greater than 5G is uploaded using presigned POST we should return error. (#3033)
fixes #2961
2016-10-22 09:05:01 -07:00
Krishna Srinivas
e51be73ac7 PresignedPost: Support for Signature V2 presigned POST Policy. (#3043)
fixes #2993
2016-10-22 08:57:12 -07:00
Harshavardhana
4b5b363c6c tests: Fix race between SetBucketListenerConfig and eventNotifyForBucketNotifications (#3041) 2016-10-22 02:35:33 -07:00
Krishna Srinivas
f2b0c08e34 logging: print file:line:funcName of the caller of errorIf and fatalIf (#3035) 2016-10-22 02:24:34 -07:00
Harshavardhana
83b364891d tests: Fix a potential race in ListenBucketNotificationHandler. (#3040) 2016-10-21 22:56:27 -07:00
Harshavardhana
e3ab478c70 tests: Fix a potential race in RemoveBucketNotification. (#3037)
Fixes #3036
2016-10-21 17:12:56 -07:00
Harshavardhana
ecaccefd2e tests: Implement GetBucketNotification handler tests. (#3029) 2016-10-21 02:39:37 -07:00
Harshavardhana
ece559afe2 api: Do not use sqs for ListenBucketNotification. (#3023)
Add more tests. Fixes #3024
2016-10-21 01:25:17 -07:00
Krishna Srinivas
d3aaf50a40 posix: Split on ":" in path d:\export makes minio use wrong disk. (#3027)
As the host/path split happens at a higher layer now, split at posix is not needed.
fixes part of #2987
2016-10-20 23:39:33 -07:00
Krishna Srinivas
32c3a558e9 distributed-XL: Support to run one minio process per export even on the same machine. (#2999)
fixes #2983
2016-10-20 18:31:02 -07:00
Anis Elleuch
41f9ab1c69 Translate storage access denied error to S3 Access Denied response (#3015) 2016-10-20 16:09:55 -07:00
Aditya Manthramurthy
8876e0a80a Delete bucket listener config file from disk (#3016) 2016-10-20 16:09:19 -07:00
Anis Elleuch
c21ac80268 Validate access/secret keys found in the config file and enhance invalid keys messages (#3017) 2016-10-20 16:07:24 -07:00
Frank
0e2cd1a64d Added clear subcommand for control lock (#3013)
Added clear subcommand for control lock with following options:

```
  3. Clear lock named 'bucket/object' (exact match).
    $ minio control lock clear http://localhost:9000/bucket/object

  4. Clear all locks with names that start with 'bucket/prefix' (wildcard match).
    $ minio control lock --recursive clear http://localhost:9000/bucket/prefix

  5. Clear all locks older than 10minutes.
    $ minio control lock --older-than=10m clear http://localhost:9000/

  6. Clear all locks with names that start with 'bucket/a' and that are older than 1hour.
    $ minio control lock --recursive --older-than=1h clear http://localhost:9000/bucket/a
```
2016-10-20 13:15:28 -07:00
Aditya Manthramurthy
6274727b71 Pick up server address from --address option (#3002) (#3008)
This makes sure that when SSL is enabled (for FS/single node mode),
the server address is picked up from the --address option (that needs
to include the hostname for SSL verification, and has to be input
appropriately by user), instead of just using ":<port>".
2016-10-20 11:39:10 -07:00
Harshavardhana
95567c68bf posix: Do not print errors in expected errors. (#3012)
Fixes #3011
2016-10-20 09:26:18 -07:00
Anis Elleuch
c189337b6e rpc: Support SNI in TLS certificates (#3009) 2016-10-20 07:43:31 -07:00
Krishnan Parthasarathi
6fc81dc162 Delete temp object/part when PutObject{,Part} fails (#3004) 2016-10-19 22:52:03 -07:00
Krishnan Parthasarathi
7d50361ca9 Move housekeeping before object layer initialization (#3001)
In a distributed setup that the server should not perform any operation
on the storage layer after it is exported via RPC. e.g, cleaning up of
temporary directories under .minio.sys/tmp may interfere with ongoing
PUT objects being served by the distributed setup.
2016-10-19 19:59:48 -07:00
Frank
19c51f3f3c Added ForceUnlock to namespace-lock (#2990) 2016-10-19 09:27:36 -07:00
Aditya Manthramurthy
c3bbadacbf Improve Peer RPC error handling (Fixes #2992) (#2995)
* Check for RPC connection shutdown and try again just once.

* Refactor SendRPC to use sync.WaitGroup
2016-10-18 21:26:58 -07:00
Anis Elleuch
2208992e6a More informative message when erasure fails to read a part of an object (#2989) 2016-10-18 13:09:26 -07:00
Anis Elleuch
bbba8e432a Add ssl support to s3/web peers connections (#2988) 2016-10-18 11:46:33 -07:00
Harshavardhana
39331b6b4e xl: GetCheckSumInfo() shouldn't fail if hash not available. (#2984)
In a multipart upload scenario disks going down and coming backup
can lead to certain parts missing on the disk/server which was
going down. This is a valid case since these blocks can be
missing and should be healed through heal operation. But we are
not supposed to fail prematurely since we have enough data on
the other disks as well within read-quorum.

This fix relaxes previous assumption, fixes a major corruption
issue reproduced by @vadmeste.

Fixes #2976
2016-10-18 11:13:25 -07:00
Mike Ralphson
6e748cb1cf Report when invalid bucket names are skipped in FS backend. (#2947) 2016-10-18 01:42:46 -07:00
Anis Elleuch
2005d656e6 Properly load creds from env and save them when server cmd is executed (#2970) 2016-10-17 23:14:41 -07:00
Aditya Manthramurthy
0f26ec8095 Propagate creds change to cluster (Fixes #2855) (#2929) 2016-10-17 20:18:08 -07:00
Harshavardhana
8d2347bc7b storage: DeleteFile should return errFileNotFound for ENOENT. (#2978) 2016-10-17 16:38:46 -07:00
Aditya Manthramurthy
0ff359ca0e Fix early init. problem for notifications (Fixes #2972) (#2977) 2016-10-17 16:38:29 -07:00
Harshavardhana
f8e13fb00e server: Startup sequence should be more idempotent. (#2974)
Fixes #2971 - honors ignore-disks option properly.
Fixes #2969 - change the net.Dial to have a timeout of 3secs.
2016-10-17 14:31:33 -07:00
Harshavardhana
686a610fc3 api: Nanosecond precision for API responses is valid with S3. (#2957)
Wqe need to be compatible as well fixes #2955
2016-10-17 08:44:55 -07:00
Krishnan Parthasarathi
b89609dc2e XL: Filter out md5Sum from user defined headers (#2962) 2016-10-17 08:41:33 -07:00
Anis Elleuch
fa50312220 Avoid returning disk corrupted by servers in the middle of init all disks formats (#2964) 2016-10-17 08:39:55 -07:00
Harshavardhana
fee3f99a6e xl: heal bucket should validate if bucket exists first. (#2953)
Fixes #2944
2016-10-17 02:10:23 -07:00
Frank
ea406754a6 New dsync and added ForceUnlock to lock rpc server (#2956)
* Update dsync and added ForceUnlock function
* Added test cases for ForceUnlock
2016-10-17 01:53:29 -07:00
Aditya Manthramurthy
d02cb963d5 Fix listen-bucket (Fixes #2942) (#2949)
Don't close socket while re-initializing notify-listeners, as the rpc
client object is shared between notify-listeners and peer clients.

Also, improves SendRPC() readability by using GetPeerClient().
2016-10-16 20:52:10 -07:00
Anis Elleuch
334cdb5d64 XL total/free space calculation is done inside xl module (#2945) 2016-10-16 14:24:15 -07:00
Anis Elleuch
5c3639c1b7 Redirect /minio to /minio/ when requests come from browsers (#2937) 2016-10-15 06:21:51 -07:00
Krishna Srinivas
903574db90 copy-object: Do not use ETag of source as MD5 as it will not be MD5 if source was uploaded as multipart. (#2938)
fixes #2934
2016-10-15 06:20:55 -07:00
Anis Elleuch
f463d3ce42 Fix a crash when service shutdown is signaled and object API is not ready yet (#2939) 2016-10-15 06:20:16 -07:00
Aditya Manthramurthy
17eeec6895 Bucket policy propagation (Fixes #2930) (#2932)
Fixes a serialisation bug - encoding/gob does not directly support
serializing `map[string]interface{}`, so we serialise to JSON and send a
byte array in the RPC call, and deserialize and update on the receiver.
2016-10-14 22:49:51 -07:00
Karthic Rao
070d3610ff tests: V2 Signature tests for object-handlers. (#2931) 2016-10-14 20:52:46 -07:00
Harshavardhana
f22862aa28 heal: Refactor heal command. (#2901)
- return errors for heal operation through rpc replies.
  - implement rotating wheel for healing status.

Fixes #2491
2016-10-14 19:57:40 -07:00
Harshavardhana
2f520ed92f Remove errors package, add comments and simplify. (#2925) 2016-10-14 12:31:00 -07:00
Mateusz Gajewski
c03ce0f74a Display SSL expiry warnings (#2925) 2016-10-14 12:30:36 -07:00
Krishna Srinivas
0320a77dc0 HealBucket: create the bucket if it is missing in one of the disks. (#2924) 2016-10-14 11:12:17 -07:00
Aditya Manthramurthy
31be826f51 Fix missing error check for jsonrpc.Server.RegisterService() (#2921) 2016-10-13 17:34:10 -07:00
Karthic Rao
17e49a9ed2 signature-v2 fix. (#2918)
- Return errors similar to V4 Sign processsing.
- Return ErrMissing fields when Auth Header fields are missing.
- Return InvalidAccessID when accessID doesn't match.

* tests: Adding V2 signature tests for bucket handler API's.
2016-10-13 09:25:56 -07:00
Aditya Manthramurthy
0aabc1d8d9 Use Peer RPC to propagate bucket policy changes (#2891) 2016-10-13 09:19:04 -07:00