Commit Graph

3895 Commits

Author SHA1 Message Date
Harshavardhana
cae62ce543 browser: Handle proper login errors. (#3518)
Also additionally log the remote address.

Fixes #3514
2017-01-03 01:33:00 -08:00
Harshavardhana
7bbb532b4b Add a isErr function to check for errs.
DisksInfo() should handle collection of some
base errors as offlineDisks.
2017-01-02 10:52:43 -08:00
Bala.FA
6d10f4c19a Adopt dsync interface changes and major cleanup on RPC server/client.
* Rename GenericArgs to AuthRPCArgs
* Rename GenericReply to AuthRPCReply
* Remove authConfig.loginMethod and add authConfig.ServiceName
* Rename loginServer to AuthRPCServer
* Rename RPCLoginArgs to LoginRPCArgs
* Rename RPCLoginReply to LoginRPCReply
* Version and RequestTime are added to LoginRPCArgs and verified by
  server side, not client side.
* Fix data race in lockMaintainence loop.
2017-01-02 20:57:42 +05:30
Matthew Hall
cde6496172 checkdeps.sh: support unusual git version strings (for OS X et. al.) (#3512) 2016-12-31 19:57:33 -08:00
Harshavardhana
8562b22823 Fix delays and iterim fix for the partial fix in #3502 (#3511)
This patch uses a technique where in a retryable storage
before object layer initialization has a higher delay
and waits for longer period upto 4 times with time unit
of seconds.

And uses another set of configuration after the disks
have been formatted, i.e use a lower retry backoff rate
and retrying only once per 5 millisecond.

Network IO error count is reduced to a lower value i.e 256
before we reject the disk completely. This is done so that
combination of retry logic and total error count roughly
come to around 2.5secs which is when we basically take the
disk offline completely.

NOTE: This patch doesn't fix the issue of what if the disk
is completely dead and comes back again after the initialization.
Such a mutating state requires a change in our startup sequence
which will be done subsequently. This is an interim fix to alleviate
users from these issues.
2016-12-30 17:08:02 -08:00
Harshavardhana
dd68cdd802 Auto-reconnect for regular authRPC client. (#3506)
Implement a storage rpc specific rpc client,
which does not reconnect unnecessarily.

Instead reconnect is handled at a different
layer for storage alone.

Rest of the calls using AuthRPC automatically
reconnect, i.e upon an error equal to `rpc.ErrShutdown`
they dial again and call the requested method again.
2016-12-29 19:42:02 -08:00
Harshavardhana
41cf580bb1 Improve reconnection logic, allow jitters. (#3502)
Attempt a reconnect also if disk not found.

This is needed since any network operation error
is converted to disk not found but we also need
to make sure if disk is really not available. 

Additionally we also need to retry more than
once because the server might be in startup
sequence which would render other servers to
wrongly think that the server is offline.
2016-12-29 03:13:51 -08:00
Dee Koder
00baec1746 syntax highlighting : Remove sh from code block which incorrectly highlights shell commands. (#3504) 2016-12-28 16:07:05 -08:00
Harshavardhana
f9025652df Support building specific architectures. 2016-12-27 17:52:24 -08:00
Bala FA
ee0172dfe4 Have simpler JWT authentication. (#3501) 2016-12-27 08:28:10 -08:00
Harshavardhana
69559aa101 objAPI: Implement CopyObject API. (#3487)
This is written so that to simplify our handler code
and provide a way to only update metadata instead of
the data when source and destination in CopyObject
request are same.

Fixes #3316
2016-12-26 16:29:26 -08:00
Krishnan Parthasarathi
36fd317eb2 Clean up lock-instrumentation and improve comments (#3499)
- Add a lockStat type to group counters
- Remove unnecessary helper functions
- Fix stats computation on force unlock
- Removed unnecessary checks and cleaned up comments
2016-12-26 10:29:55 -08:00
Bala FA
e8ce3b64ed Generate and use access/secret keys properly (#3498) 2016-12-26 10:21:23 -08:00
Andreas Auernhammer
6ee27daac1 fix blake2b tests on non-amd64 machines (#3496)
Fix the TestHashes Test for non-amd64 machines
2016-12-25 17:03:08 -08:00
koolhead17
cdc6c2d578 docs: Removed unmaintained docs. (#3493)
* docs: Removed unmaintained docs.

* docs: removed associated screenshots with the older docs.
2016-12-23 22:06:01 -08:00
Harshavardhana
855ef4f1aa Fix typo in erasure-utils.go 2016-12-22 10:47:10 -08:00
Harshavardhana
5878fcc086 bit-rot: Default to sha256 on ARM64. (#3488)
This is to utilize an optimized version of
sha256 checksum which @fwessels implemented.

blake2b lacks such optimizations on ARM platform,
this can provide us significant boost in performance.

blake2b on ARM64 as expected would be slower.
```
BenchmarkSize1K-4           	   30000	     44015 ns/op	  23.26 MB/s
BenchmarkSize8K-4           	    5000	    335448 ns/op	  24.42 MB/s
BenchmarkSize32K-4          	    1000	   1333960 ns/op	  24.56 MB/s
BenchmarkSize128K-4         	     300	   5328286 ns/op	  24.60 MB/s
```

sha256 on ARM64 is faster by orders of magnitude giving close to
AVX performance of blake2b.
```
BenchmarkHash8Bytes-4	 1000000	      1446 ns/op	   5.53 MB/s
BenchmarkHash1K-4    	  500000	      3229 ns/op	 317.12 MB/s
BenchmarkHash8K-4    	  100000	     14430 ns/op	 567.69 MB/s
BenchmarkHash1M-4    	    1000	   1640126 ns/op	 639.33 MB/s
```
2016-12-22 08:25:03 -08:00
Andreas Auernhammer
1ac36a95aa replace blake2b implementation (#3481)
* replace blake2b implementation
replace the blake2b-simd with the golang/x/crypto implementation

```
name        old time/op    new time/op     delta
Size64-8       715ns ±13%      614ns ± 3%    ~     (p=0.084 n=6+6)
Size128-8      612ns ± 5%      634ns ± 8%    ~     (p=0.084 n=6+6)
Size1K-8      2.18µs ± 5%     2.09µs ± 7%    ~     (p=0.084 n=6+6)
Size8K-8      13.1µs ± 2%     13.4µs ± 3%    ~     (p=0.084 n=6+6)
Size32K-8     48.5µs ± 1%     49.5µs ± 3%    ~     (p=0.775 n=6+6)
Size128K-8     199µs ± 0%      198µs ± 3%    ~     (p=0.468 n=6+6)

name        old speed      new speed       delta
Size64-8    92.6MB/s ±11%  104.2MB/s ± 3%    ~     (p=0.139 n=6+6)
Size128-8    208MB/s ± 6%    202MB/s ± 8%    ~     (p=0.102 n=6+6)
Size1K-8     466MB/s ± 7%    492MB/s ± 7%    ~     (p=0.139 n=6+6)
Size8K-8     621MB/s ± 2%    610MB/s ± 3%    ~     (p=0.102 n=6+6)
Size32K-8    672MB/s ± 2%    669MB/s ± 1%    ~     (p=0.818 n=6+6)
Size128K-8   657MB/s ± 1%    672MB/s ± 0%  +2.28%  (p=0.002 n=6+6)

name        old time/op   new time/op   delta
Size64-4      334ns ± 1%    243ns ± 0%  -27.14%  (p=0.029 n=4+4)
Size128-4     296ns ± 1%    242ns ± 0%  -18.21%  (p=0.029 n=4+4)
Size1K-4     1.44µs ± 0%   1.28µs ± 0%  -10.83%  (p=0.029 n=4+4)
Size8K-4     10.0µs ± 0%    9.4µs ± 0%   -6.23%  (p=0.029 n=4+4)
Size32K-4    39.8µs ± 1%   37.3µs ± 0%   -6.31%  (p=0.029 n=4+4)
Size128K-4    162µs ± 3%    149µs ± 0%   -7.72%  (p=0.029 n=4+4)

name        old speed     new speed     delta
Size64-4    192MB/s ± 1%  263MB/s ± 0%  +37.24%  (p=0.029 n=4+4)
Size128-4   431MB/s ± 0%  526MB/s ± 0%  +22.04%  (p=0.029 n=4+4)
Size1K-4    713MB/s ± 0%  800MB/s ± 0%  +12.17%  (p=0.029 n=4+4)
Size8K-4    815MB/s ± 0%  869MB/s ± 0%   +6.64%  (p=0.029 n=4+4)
Size32K-4   823MB/s ± 1%  878MB/s ± 0%   +6.72%  (p=0.029 n=4+4)
Size128K-4  810MB/s ± 3%  877MB/s ± 0%   +8.23%  (p=0.029 n=4+4)
```
See: https://go-review.googlesource.com/#/c/34319/
2016-12-21 14:20:01 -08:00
Harshavardhana
15b4c49621 fs/xl: Simplify bucket metadata reading. (#3486)
ObjectLayer GetObject() now returns the entire object
if starting offset is 0 and length is negative. This
also allows to simplify handler layer code where
we always had to use GetObjectInfo() before proceeding
to read bucket metadata files examples `policy.json`.

This also reduces one additional call overhead.
2016-12-21 11:29:32 -08:00
Harshavardhana
f57f773189 admin: Add missing madmin examples and API docs. (#3483) 2016-12-20 18:49:48 -08:00
Harshavardhana
e7b4e4e105 admin: ServiceStatus() shouldn't have to write double http headers. (#3484)
Fixes #3482
2016-12-20 18:05:25 -08:00
Harshavardhana
1d134c1a94 admin: Fix wrong vendorized location for s3signer and s3utils. 2016-12-20 15:18:37 -08:00
Anis Elleuch
329a910b86 Admin Lib: Implement Service API (#3426)
Three APIs were added to control a minio server

* NewAdminClient()
* ServiceStop()
* ServiceRestart()
* ServiceStatus()
2016-12-20 14:45:17 -08:00
Harshavardhana
4309727354 docs: Add docs for minio limitations. (#3477) 2016-12-20 14:42:13 -08:00
Anis Elleuch
ef3319a49d Fix: Typo in non canonicalized header extraction (#3480)
Extracting metadata from headers was doing wrong when Headers are not well canonicalized, fixing typo.
2016-12-20 11:59:08 -08:00
Anis Elleuch
d8e4d3c9c8 POSTForm: Return http 303 if redirect is specified (#3479)
success_action_redirect in the sent Form means that the server needs to return 303 in addition to a well specific redirection url, this commit adds this feature
2016-12-20 09:32:17 -08:00
Harshavardhana
faa6b1e925 vendorize deps for snappy, blake2b and sha256 (#3476)
Bring in new optimization and portability changes.

Fixes https://github.com/minio/minio-go/issues/578
2016-12-19 19:32:55 -08:00
Krishnan Parthasarathi
85c6bb9809 server: Sort disk arguments for consistent ordering (#3469)
This is important in a distributed setup, where the server hosting the
first disk formats a fresh setup. Sorting ensures that all servers
arrive at the same 'first' server.

Note: This change doesn't protect against different disk arguments
with some disks being same across servers.
2016-12-19 17:04:31 -08:00
Anis Elleuch
5404dddcea PostForm: Save supported headers in obj metadata (#3474)
Supported Headers like Content-Type, Cache-Control, Content-Encoding, X-Amz-* , etc.. are now saved in object metadata
2016-12-19 16:14:04 -08:00
Anis Elleuch
4692fdbb8f PostForm: Follow success_action_status requirement (#3467)
S3 spec requires that Post Object response depends on the passed success_action_status. This commit implements that requirement.
2016-12-18 13:39:56 -08:00
Bala FA
1875a47495 rpcclient: fix leaky connection (#3471)
Previously, more than one goroutine calls RPCClient.dial(), each
goroutine gets a new rpc.Client but only one such client is stored
into RPCClient object.  This leads to leaky connection at the server
side.  This is fixed by taking lock at top of dial() and release on
return.
2016-12-17 18:17:40 -08:00
Harshavardhana
9c9f390350 fs: validate filesystem path argument properly. (#3470)
FS should fail for invalid paths like

 - file:///
 - ftp://
 - http://
2016-12-17 13:43:26 -08:00
Bala FA
1b2b16998f Remove regexp usage (#3456)
This patch removes regexp usage in cacheControlHandler.ServeHTTP() and
server-main.checkEndpointsSyntax()
2016-12-17 11:00:16 -08:00
koolhead17
7a17b2a585 Browser: Added character limit for Bucket Name. (#3454) 2016-12-16 09:59:37 -08:00
Aditya Manthramurthy
f7766b49aa Amend ZFS line in README - to make it about Minio on FreeNAS (#3461) 2016-12-16 09:59:02 -08:00
Krishnan Parthasarathi
b2f920a868 Add service API handler stubs for status, stop and restart (#3417) 2016-12-15 22:26:15 -08:00
Anis Elleuch
8ceb969445 tests: Use testTmpDir var to specify tmp directory (#3459)
To be able to specify the directory where tests will be done.  This way,
it will be easier to run Minio tests on a mounted directory like NFS, ..
2016-12-15 22:25:05 -08:00
Harshavardhana
0db484c8f6 signv2: Do not use path encoding for query values. (#3458)
Use query unescape before comparing signature.
2016-12-15 14:56:18 -08:00
dcharbonnier
5c481fbf6e Fix Minio Quickstart Guide link (#3457) 2016-12-15 11:59:16 -08:00
Aditya Manthramurthy
8e6e9301ce Add support for Kafka as a notifications target (#2869) (#3439) 2016-12-15 08:23:48 -08:00
Harshavardhana
664ff063a1 server: checkEndpoints syntax properly. (#3451) 2016-12-14 20:42:19 -08:00
koolhead17
d9fd6f9a96 docs: Removed $/# from code block to make code copy easier. (#3448) 2016-12-14 17:45:47 -08:00
Harshavardhana
b28ff50126 lock/server: Check if the lock server itself is skewed back. (#3447) 2016-12-14 08:02:32 -08:00
Karthic Rao
3fe2d77b70 Adding functions for resetting globals. (#3421) 2016-12-13 11:51:48 -08:00
Krishnan Parthasarathi
ab49498fc3 server: Exit gracefully if no endpoint is local to it. (#3442) 2016-12-13 11:18:31 -08:00
Minio Trusted
46fdd70114 Fix docker file and avoid creating README.md 2016-12-13 09:31:19 -08:00
Krishnan Parthasarathi
29d72b84c0 loginServer should be a member of lockServer (#3441)
Add a unit test to catch a missing LoginHandler method in lockServer.
2016-12-13 08:01:47 -08:00
Anand Babu (AB) Periasamy
f53fcdf10e keep make messages consistent (#3437) 2016-12-13 00:26:45 -08:00
Krishna Srinivas
8e665105b1 Presign-v4: Allow requests that were signed slightly ahead of the current time. (#3435) 2016-12-12 16:32:00 -08:00
Harshavardhana
2062add05f fs/posix: On windows use helpers and init format.json properly. (#3434)
Fixes #3433
2016-12-12 15:43:41 -08:00