1b85302161
Fix spelling and golint errors. ( #3266 )
...
Fixes #3263
2016-11-15 18:14:23 -08:00
6512d9978e
Add support of user self signed certificates
...
Additionally add documentation about how to configure TLS with Minio
2016-11-15 16:15:23 -08:00
e216201901
Remove control command from minio binary ( Fixes #3264 ) ( #3265 )
2016-11-15 13:39:02 -08:00
7abcededf2
Add tests for storage rpc handlers ( #3262 )
2016-11-15 12:12:06 -08:00
398421b9f5
xl/bootup: Server bootup shouldn't return for missing buckets. ( #3255 )
...
Ref #3196
2016-11-14 15:45:00 -08:00
b8f0d9352f
signature-v2: encode path and query strings when calculating signature ( #3253 )
2016-11-14 10:23:21 -08:00
f234c35020
lock: slice length of lock clients should be precisely urls. ( #3254 )
...
This patch fixes a possible bug, reproduced rarely only seen
once.
```
panic: runtime error: index out of range
goroutine 136 [running]:
panic(0xac1a40, 0xc4200120b0)
/usr/local/go/src/runtime/panic.go:500 +0x1a1
github.com/minio/minio/vendor/github.com/minio/dsync.lock.func1(0xc4203d2240, 0x4, 0xc420474080, 0x4, 0x4, 0xc4202abb60, 0x0, 0xa86d01, 0xefcfc0, 0xc420417a80)
/go/src/github.com/minio/minio/vendor/github.com/minio/dsync/drwmutex.go:170 +0x69b
created by github.com/minio/minio/vendor/github.com/minio/dsync.lock
/go/src/github.com/minio/minio/vendor/github.com/minio/dsync/drwmutex.go:191 +0xf4
```
2016-11-14 10:18:56 -08:00
0c042a622a
Return objects content types in Web List Objects handler ( #3249 )
2016-11-13 12:26:40 -08:00
c57a358c9d
Validate date header only for Signed{,V2} and StreamingSigned. ( #3248 )
...
Fixes #2941
2016-11-13 12:08:24 -08:00
380d6c6435
Use getObjectInfo() in both FS and XL ListObjects() to simplify and to return complete object information ( #3247 )
2016-11-13 11:48:02 -08:00
716316f711
Reduce number of envs and options from command line. ( #3230 )
...
Ref #3229
After review with @abperiasamy we decided to remove all the unnecessary options
- MINIO_BROWSER (Implemented as a security feature but now deemed obsolete
since even if blocking access to MINIO_BROWSER, s3 API port is open)
- MINIO_CACHE_EXPIRY (Defaults to 72h)
- MINIO_MAXCONN (No one used this option and we don't test this)
- MINIO_ENABLE_FSMETA (Enable FSMETA all the time)
Remove --ignore-disks option - this option was implemented when XL layer
would initialize the backend disks and heal them automatically to disallow
XL accidentally using the root partition itself this option was introduced.
This behavior has been changed XL no longer automatically initializes
`format.json` a HEAL is controlled activity, so ignore-disks is not
useful anymore. This change also addresses the problems of our documentation
going forward and keeps things simple. This patch brings in reduction of
options and defaulting them to a valid known inputs. This patch also
serves as a guideline of limiting many ways to do the same thing.
2016-11-11 16:40:55 -08:00
98e79b4b50
Use 307 StatusTemporaryRedirect to redirect clients from http to https with forcing them to keep HTTP Verb ( #3246 )
2016-11-11 15:04:51 -08:00
e2216a0936
exponentialBackoffWait returns zero after some retries, limit attempt number inside ( #3245 )
2016-11-11 09:49:30 -08:00
e8d9d710d0
rpc: Protect racy access of internal auth states. ( #3238 )
...
Fixes #3232
2016-11-11 00:14:32 -08:00
a8ab02a73a
v4/presign: Fix presign requests when there are more signed headers. ( #3222 )
...
This fix removes a wrong logic which fails for requests which
have more signed headers in a presign request.
Fixes #3217
2016-11-10 21:57:15 -08:00
cf022de4d5
Add tests for s3PeerAPIHandlers ( Fixes #3067 ) ( #3242 )
2016-11-10 16:43:04 -08:00
3995e21c5b
fix: Ignore object not found error in RemoveObject() in web-handler. ( #3228 )
...
Fixes #3181
2016-11-10 15:02:03 -08:00
2f7fb78692
rpc: Our rpcClient should make an attempt to reconnect. ( #3221 )
...
rpcClient should attempt a reconnect if the call fails
with 'rpc.ErrShutdown' this is needed since at times when
the servers are taken down and brought back up.
The hijacked connection from net.Dial is usually closed.
So upon first attempt rpcClient might falsely indicate that
disk to be down, to avoid this state make another dial attempt
to really fail.
Fixes #3206
Fixes #3205
2016-11-10 07:44:41 -08:00
cf2fb30ac7
event: Add event notification for object deletion in web browser. ( #3226 )
...
Fixes #3201
2016-11-10 07:42:55 -08:00
51d1e6f75b
Fix missing CompleteMultipartUpload Etag. ( #3227 )
...
Fixes #3224
2016-11-10 07:41:02 -08:00
2f373684f5
Fix the server startup messages and help text. ( #3211 )
2016-11-09 23:37:12 -08:00
dd0698d14c
Improve namespace lock API: ( #3203 )
...
- abstract out instrumentation information.
- use separate lockInstance type that encapsulates the nsMutex, volume,
path and opsID as the frontend or top-level lock object.
2016-11-09 10:58:41 -08:00
3e67bfcc88
heal: Print heal command appropriately without export path. ( #3208 )
...
Fixes #3204
2016-11-09 10:50:14 -08:00
ea579f5b69
Avoid shutdown fs multiple times and create a new fs in each loop ( #3213 )
2016-11-09 10:10:14 -08:00
d44e9d6da9
Prevent weird messages from rpc lib on start ( Fixes #3209 ): ( #3212 )
...
This is done by not making the methods of the BucketMetaState interface
as methods (via type nesting) on the type implementing
RPCs (s3PeerAPIHandlers).
2016-11-08 23:47:44 -08:00
9c2cfb5cb6
tests: Add missing unit test cases for AbortMultipartUploadHandler(). ( #3200 )
...
Fixes #3070
2016-11-08 16:25:00 -08:00
91a0ade908
tests: add unit test for HeadObjectHandler ( #3197 )
...
Fixes #3068
2016-11-07 16:02:27 -08:00
85a5c358d8
Add bucket metadata state client/handler ( Fixes #3022 ) ( #3152 )
...
- Adds an interface to update in-memory bucket metadata state called
BucketMetaState - this interface has functions to:
- update bucket notification configuration,
- bucket listener configuration,
- bucket policy configuration, and
- send bucket event
- This interface is implemented by `localBMS` a type for manipulating
local node in-memory bucket metadata, and by `remoteBMS` a type for
manipulating remote node in-memory bucket metadata.
- The remote node interface, makes an RPC call, but the local node
interface does not - it updates in-memory bucket state directly.
- Rename mkPeersFromEndpoints to makeS3Peers and refactored it.
- Use arrayslice instead of map in s3Peers struct
- `s3Peers.SendUpdate` now receives an arrayslice of peer indexes to
send the request to, with a special nil value slice indicating that
all peers should be sent the update.
- `s3Peers.SendUpdate` now returns an arrayslice of errors, representing
errors from peers when sending an update. The array positions
correspond to peer array s3Peers.peers
Improve globalS3Peers:
- Make isDistXL a global `globalIsDistXL` and remove from s3Peers
- Make globalS3Peers an array of (address, bucket-meta-state) pairs.
- Fix code and tests.
2016-11-07 12:09:24 -08:00
33c771bb3e
tests: Add tests for browser peer rpc. Fixes #3062 ( #3189 )
2016-11-07 11:43:35 -08:00
286a8924fd
Add leak detection to object-handler tests. ( #3195 )
2016-11-06 21:53:50 -08:00
efca29b00e
Fix typos and comments in leak_detect_test.go. ( #3193 )
2016-11-06 21:02:29 -08:00
9161016962
tests: Improve coverage on signature v4 tests. ( #3188 )
...
Fixes #3065
2016-11-06 11:47:16 -08:00
5ff30777e1
Rewrite connection muxer peek process to avoid server blocking by silent clients ( #3187 )
2016-11-06 11:41:01 -08:00
754c0770d6
Merge ListenAndServe and ListenAndServeTLS for simplification purpose ( #3186 )
2016-11-05 20:32:13 -07:00
1105508453
connection muxer should use bufio.Reader to be simpler. ( #3177 )
2016-11-05 12:57:31 -07:00
1ba497950c
Fix net.Listener to fully close the underlying socket. ( #3171 )
...
Leads to races and accepting connections. This patch implements
a way to reject accepting new connections.
2016-11-05 09:43:28 -07:00
eb1bc67db1
Refactor: simplify signV4TrimAll() ( #3179 )
2016-11-04 13:52:22 -07:00
8408dfaa6c
bootup-validation: Allow port configuration only using --address option. ( #3166 )
2016-11-04 12:14:19 -07:00
d192044915
router: PathPrefix router was wrong. ( #3172 )
2016-11-04 12:13:22 -07:00
d9bab6b3bd
sigv4: Trim and shrink spaces in headers values ( #3162 )
2016-11-03 16:41:25 -07:00
e6965ca066
Quit initializing disks process when term signal is invoked ( #3163 )
2016-11-02 15:27:36 -07:00
d9674f7524
Improve coverage of web-handlers.go ( #3157 )
...
This patch additionally relaxes the requirement for
accesskeys to be in a regexy set of values.
Fixes #3063
2016-11-02 14:45:11 -07:00
79601d27f2
Use endpoint url when printing disks status in distributed mode ( #3151 )
2016-11-02 08:51:06 -07:00
226a69fe15
Add test for updateUploadJSON ( Fixes #3060 ) ( #3155 )
2016-11-01 20:04:32 -07:00
9bb799462e
Start all listeners when a given host resolves to multiple IPs. ( #3145 )
...
Default golang net.Listen only listens on the first IP when
host resolves to multiple IPs.
This change addresses a problem for example your ``/etc/hosts``
has entries as following
```
127.0.1.1 minio1
192.168.1.10 minio1
```
Trying to start minio as
```
minio server --address "minio1:9001" ~/Photos
```
Causes the minio server to be bound only to "127.0.1.1" which
is an incorrect behavior since we are generally interested in
`192.168.1.10` as well.
This patch addresses this issue if the hostname is resolvable
and gives back list of addresses associated with that hostname
we just bind on all of them as it is the expected behavior.
2016-11-01 15:38:28 -07:00
8bffa78f7f
Fix Instance type during benchmarks. ( #3147 )
...
- The benchmark initialization function was not taking into account the
instance type (FS/XL), was using XL ObjectLayer even for FS
benchmarks.
- This was leading to incorrect benchmark results for FS related
benchmarks.
- The fix takes into account the instance type (FS/XL) and correctly
returns FS backend for FS benchmarks.
2016-11-01 10:21:16 -07:00
4b302173ae
fs: Remove object append file if found since it is not always used ( #3139 )
2016-11-01 00:56:03 -07:00
9db2205db3
Fix List-Object Benchmark failure. ( #3142 )
2016-10-31 13:57:30 -07:00
6914fe436c
Remove unused function. ( #3143 )
2016-10-31 12:18:15 -07:00
807cc3c28d
Tracing locking errors for better debugging ( #3140 )
2016-10-31 10:17:14 -07:00
f3c6c55719
posix: Fix windows performance issues. ( #3132 )
...
Do not attempt to fetch volume/drive information for
each i/o situation. In our case we do this in all calls
`posix.go` this in-turn created a terrible situation for
windows. This issue does not affect the i/o path on Unix
platforms since statvfs calls are in the range of micro
seconds on these platforms.
This verification is only needed during startup and we
let things fail at a later stage on windows.
2016-10-31 09:34:44 -07:00
3e8cb8c937
build: ineffassign fixes. ( #3134 )
2016-10-30 23:32:46 -07:00
dd6ecf1193
Read/write quorum algo with uploads.json in xl ( Fixes #3123 ) ( #3124 )
...
- Reads and writes of uploads.json in XL now uses quorum for
newMultipart, completeMultipart and abortMultipart operations.
- Each disk's `uploads.json` file is read and updated independently for
adding or removing an upload id from the file. Quorum is used to
decide if the high-level operation actually succeeded.
- Refactor FS code to simplify the flow, and fix a bug while reading
uploads.json.
2016-10-30 09:27:29 -07:00
a47ce7ab22
Add support of fallocate for FS and XL backends ( #3032 )
2016-10-29 12:44:44 -07:00
0b3282ac9f
Logging: errorIf fatalIf print in the format [file.go:82:funcName()] ( #3127 )
2016-10-29 02:34:16 -07:00
79b98b5c25
boot: getPath() should take care of simple directory exports. ( #3122 )
...
fixes #3120
2016-10-28 13:08:56 -07:00
6a57f2c1f0
XL: Add more information to panic msg ( #3119 )
2016-10-28 08:46:03 -07:00
0fa2477cb0
boot: check parameter syntax before initializing the system. ( #3114 )
...
fixes #3112
2016-10-28 08:45:32 -07:00
7ffb337cd7
Update control cmd USAGE to indicate that it is possible to put access secret keys in URL ( #3115 )
2016-10-27 13:26:11 -07:00
9e2d0ac50b
Move to URL based syntax formatting. ( #3092 )
...
For command line arguments we are currently following
- <node-1>:/path ... <node-n>:/path
This patch changes this to
- http://<node-1>/path ... http://<node-n>/path
2016-10-27 03:30:52 -07:00
30dc11a931
No `listener.json` for single-node mode ( Fixes #3052 ) ( #3108 )
...
In FS or single-node XL mode, there is no need to save listener
configuration to persistent storage. As there is only one server, if it
is restarted, any connected listenBucketAPI clients were disconnected
and will have to reconnect - so there is nothing to actually store.
This incidentally solves #3052 by avoiding the problem.
2016-10-26 20:13:00 -07:00
a15dc5fed5
Print message when creating the config file ( #3089 )
2016-10-26 18:44:22 -07:00
f7c20b97a1
control cmds: Extract access and secret keys from URL if specified ( #3109 )
2016-10-26 18:42:12 -07:00
e9c45102b0
posix: Use sync.Pool buffers to copy in large buffers. ( #3106 )
...
These fixes are borrowed from the fixes required for GlusterFS i/o throughput.
2016-10-26 17:14:05 -07:00
8871eb8e1e
Show offline nodes after a fixed number of init retry ( #3107 )
2016-10-26 16:09:06 -07:00
0f32efb825
PostPolicy - rename of files/functions + add testcases ( #3104 )
2016-10-26 10:15:57 -07:00
63f1b4fdf4
mispell fixes. ( #3100 )
2016-10-26 08:46:14 -07:00
31f2db6880
Remove leftover debug statement from PutObject StreamingSignature unit-test ( #3099 )
2016-10-26 03:17:47 -07:00
2c9b406f6c
Add TLS based tests to functional test suite ( #3083 )
2016-10-26 02:30:31 -07:00
12cd2da265
Add PutObjectHandler unit tests covering failure cases ( #3096 )
2016-10-26 02:06:22 -07:00
485c0ea8bf
tests: Combine v2 tests with the Suite itself. ( #3088 )
2016-10-25 13:34:14 -07:00
49ba07d1d6
Use net.ParseCIDR instead of custom-built parsers ( #3055 )
...
Removes avoidable conversion to and from net.IP to string.
2016-10-25 11:14:47 -07:00
35e541e0b1
content-length-range policy should be honored for the uploaded object sizes. ( #3076 )
2016-10-24 23:47:03 -07:00
3977d6b7bd
Lock bucket while modifying its metadata ( Fixes #2979 ) ( #3019 )
...
- When modifying notification configuration
- When modifying listener configuration
- When modifying policy configuration
With this change we also stop early checking if the bucket exists, since
that uses a Read-lock and causes a deadlock due to the outer Write-lock.
2016-10-24 19:52:24 -07:00
0905398459
Fix benchmark tests. ( #3082 )
...
Fixes #3081
2016-10-24 18:45:06 -07:00
f41faf96b7
Fix newMultipartUpload to not leave stale uploads.json ( Fixes #3014 ) ( #3079 )
2016-10-24 17:37:18 -07:00
36639b65a9
rename completeMultipartMD5() into getCompleteMultipartMD5(). ( #3051 )
2016-10-24 13:56:13 -07:00
7fc598b73f
Fix user-agent prefix to have docker instead of suffix. ( #3074 )
2016-10-24 13:44:15 -07:00
21d41ad7fd
init[windows]: Fix to handle the case when export path is a relative path. ( #3054 )
...
ex. to handle "minio server export"
2016-10-24 08:26:28 -07:00
fe56220d1a
Do not print nil when hostname is provided as --address ( #3053 )
...
Fixes #3018
2016-10-23 23:55:12 -07:00
5782ec3ada
Fix peers and web UIVersion validation. ( #3048 )
2016-10-23 12:32:35 -07:00
8839c5105a
Pass values to closures esp. when passed to defer statement. ( #3050 )
...
opsID, a variable on the stack, changes over the course of
Completemultipartupload function in xl-v1-multipart.go. This was
being used in a function closure which was passed to defer
statement. The variables used in the closure depend on their values at
the time of evaluation which is indeterminate behaviour. It is
incorrect to depend on values of variables on stack at the end of
function, when deferred functions are executed.
2016-10-23 09:57:52 -07:00
6c2d5e3d05
Correct the number of failed disks that we can withstand in startup message ( #3045 )
2016-10-22 10:36:50 -07:00
5999a23d3e
When object whose size is greater than 5G is uploaded using presigned POST we should return error. ( #3033 )
...
fixes #2961
2016-10-22 09:05:01 -07:00
e51be73ac7
PresignedPost: Support for Signature V2 presigned POST Policy. ( #3043 )
...
fixes #2993
2016-10-22 08:57:12 -07:00
4b5b363c6c
tests: Fix race between SetBucketListenerConfig and eventNotifyForBucketNotifications ( #3041 )
2016-10-22 02:35:33 -07:00
f2b0c08e34
logging: print file:line:funcName of the caller of errorIf and fatalIf ( #3035 )
2016-10-22 02:24:34 -07:00
83b364891d
tests: Fix a potential race in ListenBucketNotificationHandler. ( #3040 )
2016-10-21 22:56:27 -07:00
e3ab478c70
tests: Fix a potential race in RemoveBucketNotification. ( #3037 )
...
Fixes #3036
2016-10-21 17:12:56 -07:00
ecaccefd2e
tests: Implement GetBucketNotification handler tests. ( #3029 )
2016-10-21 02:39:37 -07:00
ece559afe2
api: Do not use sqs for ListenBucketNotification. ( #3023 )
...
Add more tests. Fixes #3024
2016-10-21 01:25:17 -07:00
d3aaf50a40
posix: Split on ":" in path d:\export makes minio use wrong disk. ( #3027 )
...
As the host/path split happens at a higher layer now, split at posix is not needed.
fixes part of #2987
2016-10-20 23:39:33 -07:00
32c3a558e9
distributed-XL: Support to run one minio process per export even on the same machine. ( #2999 )
...
fixes #2983
2016-10-20 18:31:02 -07:00
41f9ab1c69
Translate storage access denied error to S3 Access Denied response ( #3015 )
2016-10-20 16:09:55 -07:00
8876e0a80a
Delete bucket listener config file from disk ( #3016 )
2016-10-20 16:09:19 -07:00
c21ac80268
Validate access/secret keys found in the config file and enhance invalid keys messages ( #3017 )
2016-10-20 16:07:24 -07:00
0e2cd1a64d
Added clear subcommand for control lock ( #3013 )
...
Added clear subcommand for control lock with following options:
```
3. Clear lock named 'bucket/object' (exact match).
$ minio control lock clear http://localhost:9000/bucket/object
4. Clear all locks with names that start with 'bucket/prefix' (wildcard match).
$ minio control lock --recursive clear http://localhost:9000/bucket/prefix
5. Clear all locks older than 10minutes.
$ minio control lock --older-than=10m clear http://localhost:9000/
6. Clear all locks with names that start with 'bucket/a' and that are older than 1hour.
$ minio control lock --recursive --older-than=1h clear http://localhost:9000/bucket/a
```
2016-10-20 13:15:28 -07:00
6274727b71
Pick up server address from --address option ( #3002 ) ( #3008 )
...
This makes sure that when SSL is enabled (for FS/single node mode),
the server address is picked up from the --address option (that needs
to include the hostname for SSL verification, and has to be input
appropriately by user), instead of just using ":<port>".
2016-10-20 11:39:10 -07:00
95567c68bf
posix: Do not print errors in expected errors. ( #3012 )
...
Fixes #3011
2016-10-20 09:26:18 -07:00
c189337b6e
rpc: Support SNI in TLS certificates ( #3009 )
2016-10-20 07:43:31 -07:00
6fc81dc162
Delete temp object/part when PutObject{,Part} fails ( #3004 )
2016-10-19 22:52:03 -07:00
7d50361ca9
Move housekeeping before object layer initialization ( #3001 )
...
In a distributed setup that the server should not perform any operation
on the storage layer after it is exported via RPC. e.g, cleaning up of
temporary directories under .minio.sys/tmp may interfere with ongoing
PUT objects being served by the distributed setup.
2016-10-19 19:59:48 -07:00
19c51f3f3c
Added ForceUnlock to namespace-lock ( #2990 )
2016-10-19 09:27:36 -07:00
c3bbadacbf
Improve Peer RPC error handling ( Fixes #2992 ) ( #2995 )
...
* Check for RPC connection shutdown and try again just once.
* Refactor SendRPC to use sync.WaitGroup
2016-10-18 21:26:58 -07:00
2208992e6a
More informative message when erasure fails to read a part of an object ( #2989 )
2016-10-18 13:09:26 -07:00
bbba8e432a
Add ssl support to s3/web peers connections ( #2988 )
2016-10-18 11:46:33 -07:00
39331b6b4e
xl: GetCheckSumInfo() shouldn't fail if hash not available. ( #2984 )
...
In a multipart upload scenario disks going down and coming backup
can lead to certain parts missing on the disk/server which was
going down. This is a valid case since these blocks can be
missing and should be healed through heal operation. But we are
not supposed to fail prematurely since we have enough data on
the other disks as well within read-quorum.
This fix relaxes previous assumption, fixes a major corruption
issue reproduced by @vadmeste.
Fixes #2976
2016-10-18 11:13:25 -07:00
6e748cb1cf
Report when invalid bucket names are skipped in FS backend. ( #2947 )
2016-10-18 01:42:46 -07:00
2005d656e6
Properly load creds from env and save them when server cmd is executed ( #2970 )
2016-10-17 23:14:41 -07:00
0f26ec8095
Propagate creds change to cluster ( Fixes #2855 ) ( #2929 )
2016-10-17 20:18:08 -07:00
8d2347bc7b
storage: DeleteFile should return errFileNotFound for ENOENT. ( #2978 )
2016-10-17 16:38:46 -07:00
0ff359ca0e
Fix early init. problem for notifications ( Fixes #2972 ) ( #2977 )
2016-10-17 16:38:29 -07:00
f8e13fb00e
server: Startup sequence should be more idempotent. ( #2974 )
...
Fixes #2971 - honors ignore-disks option properly.
Fixes #2969 - change the net.Dial to have a timeout of 3secs.
2016-10-17 14:31:33 -07:00
686a610fc3
api: Nanosecond precision for API responses is valid with S3. ( #2957 )
...
Wqe need to be compatible as well fixes #2955
2016-10-17 08:44:55 -07:00
b89609dc2e
XL: Filter out md5Sum from user defined headers ( #2962 )
2016-10-17 08:41:33 -07:00
fa50312220
Avoid returning disk corrupted by servers in the middle of init all disks formats ( #2964 )
2016-10-17 08:39:55 -07:00
fee3f99a6e
xl: heal bucket should validate if bucket exists first. ( #2953 )
...
Fixes #2944
2016-10-17 02:10:23 -07:00
ea406754a6
New dsync and added ForceUnlock to lock rpc server ( #2956 )
...
* Update dsync and added ForceUnlock function
* Added test cases for ForceUnlock
2016-10-17 01:53:29 -07:00
d02cb963d5
Fix listen-bucket ( Fixes #2942 ) ( #2949 )
...
Don't close socket while re-initializing notify-listeners, as the rpc
client object is shared between notify-listeners and peer clients.
Also, improves SendRPC() readability by using GetPeerClient().
2016-10-16 20:52:10 -07:00
334cdb5d64
XL total/free space calculation is done inside xl module ( #2945 )
2016-10-16 14:24:15 -07:00
5c3639c1b7
Redirect /minio to /minio/ when requests come from browsers ( #2937 )
2016-10-15 06:21:51 -07:00
903574db90
copy-object: Do not use ETag of source as MD5 as it will not be MD5 if source was uploaded as multipart. ( #2938 )
...
fixes #2934
2016-10-15 06:20:55 -07:00
f463d3ce42
Fix a crash when service shutdown is signaled and object API is not ready yet ( #2939 )
2016-10-15 06:20:16 -07:00
17eeec6895
Bucket policy propagation ( Fixes #2930 ) ( #2932 )
...
Fixes a serialisation bug - encoding/gob does not directly support
serializing `map[string]interface{}`, so we serialise to JSON and send a
byte array in the RPC call, and deserialize and update on the receiver.
2016-10-14 22:49:51 -07:00
070d3610ff
tests: V2 Signature tests for object-handlers. ( #2931 )
2016-10-14 20:52:46 -07:00
f22862aa28
heal: Refactor heal command. ( #2901 )
...
- return errors for heal operation through rpc replies.
- implement rotating wheel for healing status.
Fixes #2491
2016-10-14 19:57:40 -07:00
2f520ed92f
Remove errors package, add comments and simplify. ( #2925 )
2016-10-14 12:31:00 -07:00
c03ce0f74a
Display SSL expiry warnings ( #2925 )
2016-10-14 12:30:36 -07:00
0320a77dc0
HealBucket: create the bucket if it is missing in one of the disks. ( #2924 )
2016-10-14 11:12:17 -07:00
31be826f51
Fix missing error check for jsonrpc.Server.RegisterService() ( #2921 )
2016-10-13 17:34:10 -07:00
17e49a9ed2
signature-v2 fix. ( #2918 )
...
- Return errors similar to V4 Sign processsing.
- Return ErrMissing fields when Auth Header fields are missing.
- Return InvalidAccessID when accessID doesn't match.
* tests: Adding V2 signature tests for bucket handler API's.
2016-10-13 09:25:56 -07:00
0aabc1d8d9
Use Peer RPC to propagate bucket policy changes ( #2891 )
2016-10-13 09:19:04 -07:00
55f6828750
Do not print update message unless there is an update. ( #2919 )
2016-10-13 09:17:08 -07:00
6303f26330
Protect map from concurrent access ( Fixes #2915 ) ( #2916 )
...
Protects the Peers RPC clients map from concurrent access to fix a data race condition.
2016-10-13 01:33:50 -07:00
b59bac670a
Handle err returned by rpc.Server.RegisterName ( #2910 )
2016-10-12 23:13:24 -07:00
84acc820c7
Fix free drive space calculation in XL mode ( #2917 )
2016-10-12 20:22:15 -07:00
fdaa129a5b
Fix dockerfile container image. ( #2892 )
2016-10-12 18:09:08 -07:00
df59967f59
Avoid checking date header of web requests by properly applying generic handlers ( #2914 )
2016-10-12 12:58:36 -07:00
73982c8cb6
Listen bucket notification for multiple prefixes/suffixes ( #2911 )
...
* Listen bucket notification for multiple prefixes/suffixes
* After review fixes
2016-10-12 11:02:15 -07:00
6199aa0707
Peer RPCs for bucket notifications ( #2877 )
...
* Implements a Peer RPC router that sends info to all Minio servers in the cluster.
* Bucket notifications are propagated to all nodes via this RPC router.
* Bucket listener configuration is persisted to separate object layer
file (`listener.json`) and peer RPCs are used to communicate changes
throughout the cluster.
* When events are generated, RPC calls to send them to other servers
where bucket listeners may be connected is implemented.
* Some bucket notification tests are now disabled as they cannot work in
the new design.
* Minor fix in `funcFromPC` to use `path.Join`
2016-10-12 01:03:50 -07:00
a5921b5743
Use same timestamp for all chunks in chunked signature ( #2908 )
2016-10-11 23:46:51 -07:00
ff91ecb177
tests: Adding unknown signature type test for API handlers. ( #2905 )
2016-10-11 20:38:10 -07:00
a6357502c1
Correct typo in error string ( #2902 )
2016-10-11 08:56:02 -07:00
fa8ea41cd9
lock/instrumentation: Cleanup and print in user friendly form. ( #2807 )
2016-10-11 00:50:27 -07:00
3ac6790ca2
tests: Add Object Layer nil test for bucket-handler API's ( #2899 )
2016-10-11 00:00:02 -07:00
268b96058f
ns-lock: lock namespace during FS object operations. ( #2896 )
2016-10-10 10:20:04 -07:00
0d031c432b
Fix typo in serverVersion ( #2894 )
2016-10-10 10:11:56 -07:00
6e8f3224c5
Test coverage for lock rpc server ( #2893 )
...
* Add test coverage for removeEntry and removeEntryIfExists
* Initial test framework for Lock/Unlock functionality
* Add clarification comments
* Add test coverage code for RLock() and RUnlock()
* Add test coverage for Expired() function
* Have all lock-rpc-server test functions start with the same prefix
* Properly initialize JWT security token
2016-10-10 10:11:29 -07:00
9c53e9f4c3
tests: Enhance coverage for bucket policy handlers. ( #2895 )
2016-10-10 09:29:56 -07:00
2d5e988a6d
Refactor streaming signatureV4 w/ state machine ( #2862 )
...
* Refactor streaming signatureV4 w/ state machine
- Used state machine to make transitions between reading chunk header,
chunk data and trailer explicit.
* debug: add print/panic statements to gather more info on CI failure
* Persist lastChunk status between Read() on ChunkReader
... remove panic() which was added as interim aid for debugging.
* Add unit-tests to cover v4 streaming signature
2016-10-10 01:42:32 -07:00
3cfb23750a
control: Implement service command 'stop,restart,status'. ( #2883 )
...
- stop - stops all the servers.
- restart - restart all the servers.
- status - prints status of storage info about the cluster.
2016-10-09 23:03:10 -07:00
57f75b1d9b
Ignore copy conditions when ETag is not available ( #2888 )
2016-10-09 16:21:42 -07:00
e213172431
tests: Missing anonymous tests for bucket-handlers. ( #2885 )
2016-10-09 09:21:37 -07:00
f5f007e183
Test: Add test case for xl.HealObject() ( #2884 )
...
fixes #2842
2016-10-08 17:08:17 -07:00
09463265ce
tests: Adding anonymous requests tests for bucket policy handlers. ( #2882 )
2016-10-08 01:04:26 -07:00
8f4cf2a7d0
tests: anonymous/unsigned tests for object handler API's . ( #2881 )
2016-10-07 23:28:50 -07:00
30183c4a9a
tests: cleanup and unsigned request test. ( #2880 )
...
- Cleaning up of ListMultipartUpload API test for improving readability,
code maintainance and extensibility.
- Moving ListMultipartUploads to Go 1.7 sub tests.
- Using the new Anonymous request helper function for
ListMultipartUploads.
2016-10-07 20:16:57 -07:00
d1df5e0ae1
tests: Add helper function for API handler anonymous request tests. ( #2876 )
...
- Add helper function for API handler anonymous request tests.
- Add PutObject Part Anonymous request case using the new helper
function to validate its functionality.
2016-10-07 11:16:11 -07:00
f1bc9343a1
prep: Initialization should wait instead of exit the servers. ( #2872 )
...
- Servers do not exit for invalid credentials instead they print and wait.
- Servers do not exit for version mismatch instead they print and wait.
- Servers do not exit for time differences between nodes they print and wait.
2016-10-07 11:15:55 -07:00
97f4989945
tests: cleaning up. ( #2875 )
...
- Clean up PutObjectPart and ListObjectPart API handler tests.
- Add more comments, make the tests more readable.
- Add verification for HTTP response status code.
- Initialize the test using object Layer.
- Move to Go 1.7 sub tests.
2016-10-07 08:02:37 -07:00
1e5e213d24
auth: Make sure we initialize or change config before RPC requests. ( #2867 )
2016-10-06 13:35:56 -07:00
a8105ec068
- Test utility function for easy asserting of cases wherein objectLayer ( #2865 )
...
is `nil` in API handlers.
- Remove the existing tests for the `nil` check and use the new method
to test for object layer being `nil`.
2016-10-06 13:34:33 -07:00
bb9be02228
minio-browser: do not redirect to /minio if MINIO_BROWSER=off ( #2863 )
...
fixes #2837
2016-10-06 08:30:32 -07:00
64f37bbf5b
rpc: Add RPC client tests. ( #2858 )
2016-10-06 02:30:54 -07:00
0fc96fa25c
Refactor bucket policy handler test to use API test initializer. ( #2859 )
2016-10-06 02:02:42 -07:00
2d8c6f8288
unit test for bucketPolicyConditionMatch function. ( #2857 )
2016-10-06 00:23:46 -07:00
b94211bd66
api: ListObjectsV1 compliance with AWS S3. ( #2856 )
...
XSD - xml schema definition for SOAP operations
on S3 provides positional restrictions on XML
output.
Fix the response by re-arranging the positions in
accordance with S3 behavior.
Fixes #2849
2016-10-05 20:12:47 -07:00
6494b77d41
server: Add more elaborate startup messages. ( #2731 )
...
These messages based on our prep stage during XL
and prints more informative message regarding
drive information.
This change also does a much needed refactoring.
2016-10-05 12:48:07 -07:00
63a7ca1af0
web: fix jwt token expiry set to one day by default. ( #2819 )
...
Fixes #2818
2016-10-05 10:18:55 -07:00
95f544657a
Signature-V2: use raw resource/query from the request for signature calculation. ( #2833 )
2016-10-05 09:18:53 -07:00
740a919e25
config: Use migrateV8 to v9 function properly. ( #2852 )
2016-10-05 02:28:04 -07:00
402c92beda
Add listObjectParts test w/ unknown request signature type ( #2847 )
2016-10-04 07:57:35 -07:00
6a9013b97c
misspell fixes. ( #2835 )
2016-10-04 00:09:21 -07:00
73b50aea2d
Add preSign auth type tests for ListObjectPartsHandler and PutObjectPartHandler ( #2834 )
2016-10-03 22:05:33 -07:00
315e66858c
Add PostgreSQL notifier ( #2739 ) ( #2824 )
...
* The user is required to specify a table name and database connection
information in the configuration file.
* INSERTs and DELETEs are done via prepared statements for speed.
* Assumes a table structure, and requires PostgreSQL 9.5 or above due to
the use of UPSERT.
* Creates the table if it does not exist with the given table name using
a query like:
CREATE TABLE myminio (
key varchar PRIMARY KEY,
value JSONB
);
* Vendors some required libraries.
2016-10-03 17:29:55 -07:00
4f902d42b2
Add unit-tests for ListObjectParts API handler ( #2826 )
...
* Add missing uploadID test
... make variables in test code unexported.
* Add ServerNotInitialized test for ListObjectPartsHandler
* Add tests for ListObjectParts with signatureV2 and Anonymous requests
* Add failure test cases for ListObjectParts
2016-10-03 08:54:57 -07:00
61a18ed48f
sha256: Verify sha256 along with md5sum, signature is verified on the request early. ( #2813 )
2016-10-02 15:51:49 -07:00
b5a6dd1395
Avoid path-cleaning policy resources for a better compliance with S3 ( #2823 )
2016-10-01 21:30:25 -07:00
83e6e1060e
Layer LimitReader responsibly allowing sign verification to work ( #2821 )
2016-10-01 09:37:40 -07:00
ddeb8242d8
PutObjectPartHandler unit-tests ( #2810 )
2016-10-01 08:23:26 -07:00
5ecba587f7
api: Relax object name validation. ( #2814 )
...
Fixes #2812
2016-09-30 16:56:36 -07:00
db3da97a50
signature/v2: Fix presigned requests.
2016-09-30 15:22:00 -07:00
5885ffc8ae
signature: Add legacy signature v2 support transparently. ( #2811 )
...
Add new tests as well.
2016-09-30 14:32:13 -07:00
9fb1c89f81
Add TLS encryption capability to RPC clients ( #2789 )
2016-09-29 23:42:37 -07:00
1e6afac3bd
Add NATS notifier ( #2795 )
2016-09-29 23:42:10 -07:00
64083b9227
signature: Region changes should be handled just like AWS. ( #2805 )
...
- PutBucket happens with 'us-east-1'.
- ListBuckets happens with any region.
- GetBucketLocation happens with 'us-east-1' and location is returned.
2016-09-29 15:51:00 -07:00
5fdd768903
Make addition of TopicConfig to globalEventNotifier go-routine safe ( #2806 )
2016-09-28 22:46:19 -07:00
428629f577
Add unit tests for server-main.go ( #2802 )
2016-09-28 11:19:07 -07:00
1edd74dda2
update: Deprecate the usage of update=yes query param. ( #2801 )
...
Fixes #2799
2016-09-28 02:41:21 -07:00
740ecf530c
Add PutBucketNotification, ListenBucketNotification handler unit-tests. ( #2787 )
2016-09-28 01:08:03 -07:00
10d2ef5449
Remove comments relating to deprecated MINIO_DEBUG envvar ( #2797 )
2016-09-27 18:28:46 -07:00
8ea571c7f7
Remove MINIO_DEBUG environment variable ( #2794 )
...
Removes the unimplemented settings of MINIO_DEBUG=mem and makes
MINIO_DEBUG=lock the default behaviour.
2016-09-27 14:35:43 -07:00
ca3022d545
api: Change ListenBucketNotification with new API format. ( #2791 )
...
Take prefix, suffix and events as query params.
2016-09-27 13:17:43 -07:00
9417614a8e
Recalculate free minimum disk space ( #2788 )
...
* Fix calculating free space disk by using blocks available for unprivileged user
* Use fixed minimal free disk space instead of percentage
2016-09-27 12:46:38 -07:00
70d52bbc4c
Add unit test for rate-limit-handler ( #2661 ) ( #2784 )
2016-09-26 21:31:12 -07:00
6aa2fc95c0
Revert "bucket: refactor policies and fix bugs related to enforcing policies. ( #2766 )"
...
This reverts commit ca5ca8332b
2016-09-26 19:32:33 -07:00
cfbab22237
web: Remove bucket policy when we have no more statements. ( #2779 )
2016-09-26 03:11:22 -07:00
be0e06c0aa
web: Simplify and rename GetAllBucketPolicy --> ListAllBucketPolicies. ( #2778 )
2016-09-25 21:53:19 -07:00
1c941fd787
rpc: Should validate server versions. ( #2775 )
...
Fixes #2764
2016-09-24 03:34:45 -07:00
669783f875
Purge stale object cache entry ( #2770 )
2016-09-23 19:55:28 -07:00
27e474b3d2
Improve code coverage in bucket-notification-handlers.go ( #2759 )
...
* Fix incorrect test cases for bucket-notification handler
* Add tests covering failure cases for bucket notification
2016-09-23 13:32:51 -07:00
1e53316241
Add tests for presigned-get ( #2767 )
...
* web-handlers: support for presigned-get json-rpc call for MinioBrowser's "share" feature.
* Add tests for presigned-get
2016-09-23 01:25:49 -07:00
ca5ca8332b
bucket: refactor policies and fix bugs related to enforcing policies. ( #2766 )
...
This patch also addresses the problem of double caching at
object layer once at XL and another at handler layer.
2016-09-22 23:47:48 -07:00
aa579bbc20
web: add method to get all policies for given bucket name. ( #2756 )
...
Refer #1858
2016-09-22 23:06:45 -07:00
e375d822da
bucket: SetBucketPolicy should save a valid Version and validate. ( #2762 )
2016-09-22 22:27:21 -07:00
fc783f8407
More tests for web handlers ( #2755 )
...
* Return negative values of Total and Free in StorageInfo() when we fail to get disk info
* Return consistent messages in web handlers when the server is not initialized
2016-09-22 16:35:12 -07:00
ef22330563
Require go 1.7.1 to build Minio server ( #2727 )
2016-09-22 10:33:52 -07:00
1148f95292
ineffassign fixes ( #2758 )
2016-09-21 23:03:54 -07:00
f7430ec09c
use runtime/debug.Stack() in leak detect test ( #2757 )
2016-09-21 22:04:35 -07:00
b8903d842c
api/complete-multipart: fixes and tests. ( #2719 )
...
* api/complete-multipart: tests and simplification.
- Removing the logic of sending white space characters.
- Fix for incorrect HTTP response status for certain cases.
- Tests for New Multipart Upload and Complete Multipart Upload.
* tests: test for Delelete Object API handler
2016-09-21 20:08:08 -07:00
32f097b4d6
Controller rpc tests ( #2709 )
...
* Test code for controller-handler operations:
* Heal operations
* List operation
* Switch to "testing" lib, moving away from gocheck
* Minor refactors
* Remove extra call to initGracefulShutdown
* Remove dead code in mainControl:
Dead code found by the TestControlMain() test function that always
passes.
* Add tests for control-*-main.go
2016-09-21 19:58:50 -07:00
559ad38b8c
Add bucket-notification-handler tests ( #2750 )
2016-09-21 17:41:34 -07:00
90417d2dd6
Check for bucket existence in Set/Get/Remove bucket policy workflow + tests ( #2745 )
2016-09-21 16:38:50 -07:00
e66fb4bd7b
configMigrate() returns errors + tests ( #2735 )
2016-09-21 09:44:57 -07:00
018c90dae7
events: ElasticSearch doesnt support objects with '/' in them. ( #2747 )
...
Fix this by using a unique sha256 generated for each unique key.
2016-09-20 16:36:18 -07:00
a5066e8f76
Better code coverage of handler-utils.go ( #2746 )
2016-09-20 10:40:46 -07:00
0a3448c8b6
events: Change event notifiers to delete and update keys. ( #2742 )
...
ElasticSearch and Redis are both treated like a database.
Each indexs are based on the object names uniquely indentifying
the event. Upon each delete event of the named object deletes
the index on elasticsearch and redis respectively.
2016-09-20 02:11:17 -07:00
c4964232eb
config: Fail to start for config mistakes. ( #2740 )
2016-09-19 15:23:49 -07:00
ef3c807b4a
policies: Parser should handle Principals with various forms. ( #2733 )
...
Handles cases for these three combinations
- "Principal": "*",
- "Principal": { "AWS" : "*" }
- "Principal": { "AWS" : [ "*" ]}
Fixes #2732
2016-09-19 13:52:28 -07:00
113b93346b
lock: Make some cleanup and moving the code around. ( #2718 )
...
This patch just avoids lot of ifs and inverts some logic.
2016-09-19 13:14:55 -07:00
a955676986
Signature-V4: Dump the request with error message on signature mismatch. ( #2734 )
...
fixes #2691
2016-09-19 10:17:46 -07:00
725df557b5
tests: Add tests for bucket-notification-utils ( #2726 )
...
Part - 2 final fix #2711
2016-09-17 03:19:39 -07:00
490056eee3
tests: Add tests for bucket-notification-utils ( #2723 )
...
Part fix - 1 for #2711
2016-09-16 17:26:27 -07:00
797d749322
tests: Add tests for filterRuleMatch ( #2722 )
...
Part-1 fix for #2418
2016-09-16 16:44:44 -07:00
79888bfff7
tests: Add auth-handler. ( #2721 )
...
Fixes #2658
2016-09-16 15:17:49 -07:00
010f61e91f
Add more tests for event-notifier code ( #2716 )
2016-09-16 14:26:05 -07:00
9216981262
tests: Add test for diskCount. ( #2717 )
...
Fixes #2312
2016-09-16 13:44:52 -07:00
b89a1cd482
tests: Implemented more tests for fs-v1*.go ( #2686 )
2016-09-16 13:06:49 -07:00
7d37dea449
tests: Add more streaming signature tests. ( #2713 )
...
Part fix for #2621
2016-09-16 02:45:42 -07:00
df2ef64d20
Upgrade to new dsync version incl. stale lock detection ( #2708 )
2016-09-16 00:30:55 -07:00
7a549096de
XL and FS use different tree walk ignored errors ( #2707 )
2016-09-15 13:43:40 -07:00
a1ff351f21
tests: Fix ListMultipartUploadsHandler tests. ( #2705 )
2016-09-15 01:44:19 -07:00
03430d0db8
tests: Add ListBucketHandler tests. ( #2701 )
...
part-3 final fix for #2412
2016-09-14 23:53:42 -07:00
6f73d597e0
Fix tracing twice an error in fs Complete Multipart Upload ( #2703 )
2016-09-14 21:24:54 -07:00
a84548d7ea
Fix FS remove bucket regression bug ( #2693 )
2016-09-14 16:41:39 -07:00
1e6d67b16d
server: Remove deadcode. ( #2699 )
2016-09-14 13:43:08 -07:00
6533927237
Lock-free rate-limit algorithm + bug-fix ( #2694 )
2016-09-14 11:27:37 -07:00
da9ae574df
server: We should fail properly during server startup. ( #2689 )
...
Fixes #2688
2016-09-14 01:11:03 -07:00
ee7e70c992
tests: Add tests for ListMultipartUploads, DeleteMultipleObjects. ( #2649 )
...
Additionally adds PostPolicyHandler tests.
2016-09-13 21:22:31 -07:00
54a9f59a13
Init: Print SQS ARNs after globalEventNotifier is inited. ( #2682 )
...
fixes #2681
2016-09-13 21:18:30 -07:00
e6fd664331
tests: Fix format-config tests.
2016-09-13 21:18:30 -07:00
b247ec9352
tests: refactor object-handler tests. ( #2656 )
...
- Move the initialization to a common executor for object Layer API
tests.d
2016-09-13 21:18:30 -07:00
43befab8ef
Change distributed server wording.
2016-09-13 21:18:30 -07:00
eae0281c64
tests: Add GetBucketLocation, HeadBucket tests. ( #2644 )
2016-09-13 21:18:30 -07:00
8bd78fbdfb
performance: gjson parsing for readXLMeta, listParts, getObjectInfo. ( #2631 )
...
- Using gjson for constructing xlMetaV1{} in realXLMeta.
- Test for parsing constructing xlMetaV1{} using gjson.
- Changes made since benchmarks showed 30-40% improvement in speed.
- Follow up comments in issue https://github.com/minio/minio/issues/2208
for more details.
- gjson parsing of parts from xl.json for listParts.
- gjson parsing of statInfo from xl.json for getObjectInfo.
- Vendorizing gjson dependency.
2016-09-13 21:18:30 -07:00
66459a4ce0
Add unit-tests for formatting disks during initialization ( #2635 )
...
* Add unit-tests for formatting disks during initialization
- Fixed corresponding code at places where it was deviating from the
tabular spec.
* Added more test cases and simplified algo
... based on feedback from ``go test -coverprofile``.
2016-09-13 21:18:30 -07:00
182109f0de
xl: Heal `format.json` properly on fresh disks.
2016-09-13 21:18:30 -07:00
9998e9ea19
api: Response timeFormat do not need to have nano-second precision.
...
Fixes an error reported by s3verify.
2016-09-13 21:18:30 -07:00
ba2ba328da
server: Fixes for various conditions
...
- Fix distributed branch to be able to run FS version.
- Fix distributed branch to be able to run XL local disks.
- Ignore initialization failures of notification and bucket
policies, the codepath should load whatever is possible.
2016-09-13 21:18:30 -07:00
67b8080144
Fix control lock rpc name in control lock cmd ( #2627 )
2016-09-13 21:18:30 -07:00
239a34ca97
Add tests for regular and streaming v4 PutObject Handler ( #2618 )
2016-09-13 21:18:30 -07:00
81d8263ae2
binary-update: Do not fetch update info for minio binary compiled from source.
...
fixes #2494
2016-09-13 21:18:30 -07:00
b4e4846e9f
PutObject: object layer now returns ObjectInfo instead of md5sum to avoid extra GetObjectInfo call. ( #2599 )
...
From the S3 layer after PutObject we were calling GetObjectInfo for bucket notification. This can
be avoided if PutObjectInfo returns ObjectInfo.
fixes #2567
2016-09-13 21:18:30 -07:00
92e49eab5a
FS/Multipart: Do not rename append files to another tmp file as the append files are already in tmp location. ( #2612 )
2016-09-13 21:18:30 -07:00
c4a7b950a0
fs: Fix asynchronous multipart bug.
...
Construct part path properly.
2016-09-13 21:18:30 -07:00
1ce339abeb
Fixing ineffssaign errors ( #2608 )
2016-09-13 21:18:30 -07:00
3e284162d7
Add global flags to all commands and subcommands ( #2605 )
2016-09-13 21:18:30 -07:00
ff99392102
Enhance minio server help template ( #2603 )
2016-09-13 21:18:30 -07:00
9358ee011b
logging: Print stack trace in case of errors.
...
fixes #1827
2016-09-13 21:18:30 -07:00
37cbcae6ba
xl: Remove an unecessary lock with isBucketExist() ( #2593 )
...
Fixes #2566
2016-09-13 21:18:30 -07:00
ae64b7fac8
XL: Handle object layer initialization properly.
...
Initialization when disk was down the network disk
reported an incorrect error rather than errDiskNotFound.
This resulted in incorrect error handling during
prepInitStorage() stage.
Fixes #2577
2016-09-13 21:18:30 -07:00
d936ed90ae
Avoid testing on system errors strings in posix ( #2583 )
2016-09-13 21:18:30 -07:00
7cc77eba45
XL/Healing: errDiskNotFound is the only pardonable error in xlShouldHeal. ( #2586 )
...
This is so that we try to heal a file for all the "bad" cases except when the disk is down.
2016-09-13 21:18:30 -07:00
07d232c7b4
instrumentation: instrumentation for locks. ( #2584 )
...
- Instrumentation for locks.
- Detailed test coverage.
- Adding RPC control handler to fetch lock instrumentation.
- RPC control handlers suite tests with a test RPC server.
2016-09-13 21:18:30 -07:00
de67bca211
Move formatting of disks out of object layer initialization ( #2572 )
2016-09-13 21:18:30 -07:00
bca1385683
lock: Fix support single node XL locking as well.
2016-09-13 21:18:30 -07:00
cbe87cb2ed
Fix fd-leak in rpcClient close it pro-actively.
2016-09-13 21:18:30 -07:00
0513b3ed07
Add Heal Disk Metadata RPC API + tests ( #2556 )
2016-09-13 21:18:30 -07:00
7f92165c79
Single use DRWMutex, RPCClient refactor and added missing cases to lock-rpc-server ( #2560 )
...
This PR contains various fixes for the distributed release:
- Use DRWMutex in namespace-lock only for a single Lock()/RLock() call in conformance to server-side rw-locking as implemented in minio/dsync
- Implement missing cases in lock-rpc-server to catch Unlock() for active read locks and RUnlock() for an active write lock
- Refactor RPCClient to release local mutex while making actual RPC.Call()
2016-09-13 21:18:30 -07:00
780ccc26f7
server: Validate server arguments for duplicates. ( #2554 )
...
- Validates invalid format inputs.
- Validates duplicate entries.
- Validates sufficient amount of disks.
Partially fixes #2502
2016-09-13 21:18:30 -07:00
339425fd52
server: Fetch StorageInfo() from underlying disks transparently. ( #2549 )
...
Fixes #2511
2016-09-13 21:18:30 -07:00
fa6e9540a8
server: We shouldn't exit the server in lazy init. ( #2548 )
...
Avoid fatalIf instead these are non-critical errors,
continue running the server.
- initializing bucket notifications
- initializing bucket policies.
- migrating bucket policies failure.
Fixes #2547
2016-09-13 21:18:30 -07:00
9605fde04d
controller/auth: Implement JWT based authorization for controller. ( #2544 )
...
Fixes #2474
2016-09-13 21:18:30 -07:00
200d327737
List only objects that need healing ( #2546 )
2016-09-13 21:18:30 -07:00
e1b0985b5b
rpc: Refactor authentication and login changes. ( #2543 )
...
- Cache login requests.
- Converge validating distributed setup.
2016-09-13 21:18:30 -07:00
c8dfc4cda4
auth-rpc: Reset token on disconnect ( #2542 )
2016-09-13 21:18:30 -07:00
7922a54c9a
rpc-client: remove unwanted nil check of rpcClient. ( #2538 )
2016-09-13 21:18:30 -07:00
bda6bcd5be
Layered rpc-client implementation ( #2512 )
2016-09-13 21:18:30 -07:00
7e3e24b394
rpc: client login should ignore server versions.
2016-09-13 21:18:30 -07:00
bb0466f4ce
control: Fix controller CLI handling with distributed server object layer.
...
Object layer initialization is done lazily fix it.
2016-09-13 21:18:30 -07:00
8797952409
server: Add server command line for running in distributed mode
2016-09-13 21:18:30 -07:00
a4691611a7
Move initGracefulShutdown before objectLayer initialization ( #2468 )
2016-09-13 21:18:30 -07:00
7c7eb1475d
splitNetPath: Add support for windows paths including volumeNames e.g ip:C:\network\path
2016-09-13 21:18:30 -07:00
0bce3d6d63
Fix web handler initialize with distributed lazy init.
2016-09-13 21:18:30 -07:00
804d91ef61
storage/rpc-client: Reconnect on network disconnect ( #2436 )
2016-09-13 21:18:30 -07:00
43098df9d8
rpc: Re-factor ReadFile behavior client <--> server.
...
Current code did not marshal/unmarshal buffers properly from
the server, The reason being buffers have to be allocated and
sent properly back to client to be consumable.
2016-09-13 21:18:30 -07:00
6908a0dcd4
Extract rpc server wrapped errors and translate to storage error.
2016-09-13 21:18:30 -07:00
cae5761f16
rpc/client: Add missing rpcTokens for each rpc calls.
2016-09-13 21:18:30 -07:00
61af764f8a
Add rpc layer authentication.
2016-09-13 21:18:30 -07:00
b4172ad3c8
Rename rpc-{client,server} storage-rpc-{client,server}
2016-09-13 21:18:30 -07:00
4917038f55
Move the ObjectAPI() resource to be beginning of each handlers.
...
We should return back proper errors so that the clients can
retry until server has been initialized.
2016-09-13 21:18:30 -07:00
b0f3f94163
unify single-node and distributed namespace locking ( #2401 )
2016-09-13 21:18:30 -07:00
b7c169d71d
object layer initialization using distributed locking ( #2397 )
...
* vendorized latest minio/dsync
* wip - object layer initialization using distributed locking
2016-09-13 21:18:30 -07:00
3939c75345
Added distributed RWMutex ( #2369 )
2016-09-13 21:18:30 -07:00
e55926e8cf
distribute: Make server work with multiple remote disks
...
This change initializes rpc servers associated with disks that are
local. It makes object layer initialization on demand, namely on the
first request to the object layer.
Also adds lock RPC service vendorized minio/dsync
2016-09-13 21:18:30 -07:00
f82f535509
Fix possible race in shutdown callbacks process and simplify shuttting down profiler ( #2684 )
2016-09-13 18:43:45 -07:00
51e337228e
Avoid hiding disk errors in some cases in FS Shutdown ( #2668 )
2016-09-13 11:01:10 -07:00
241c56e6d7
Use Set instead of Add in the event that the request already contains the content-length ( #2683 )
2016-09-13 10:59:40 -07:00
895471afa1
Fixes #2678 ( #2679 )
...
Refactor `ratelimit.acquire()` to properly enforce the *globalMaxConn*
limit.
2016-09-12 15:53:54 -07:00
85e2d886bc
bucketPolicy: Trim incoming paths properly for policy verification.
2016-09-11 10:36:54 -07:00
8acf4d112a
Return Owner information in ListObjectV2 only when fetch-owner is specified ( #2654 )
2016-09-10 10:44:38 -07:00
e6abfb3b67
More tests for format-config for a better coverage ( #2655 )
2016-09-10 09:54:22 -07:00
040af08473
server: Startup message now prints configured ARNs. ( #2653 )
...
Fixes #2628
2016-09-10 02:23:28 -07:00
e3de237eb8
Move config to v7 to fix previous v6 bug on null Notification entries. ( #2650 )
...
Thanks to @koolhead17 for reporting this.
2016-09-10 00:51:25 -07:00
11785529fc
Add Request Not Ready error when presigned date request is not valid ( #2646 )
2016-09-10 00:38:07 -07:00
32201a18ab
Forbid requests generated at least 15 min in the past or in the future ( #2648 )
2016-09-09 15:33:35 -07:00
421cccb1d7
Add naughty disk to simulate disk errors ( #2645 )
2016-09-09 12:53:09 -07:00
400e9309f1
Signature-V4: Include content-length for signature calculation. ( #2643 )
...
This is to be compatible with clients which includes content-length for signature calculation
(and hence deviating from AWS Signature-v4 spec)
2016-09-09 09:38:07 -07:00
a4afb312d4
S3-layer: Multiple-delete should return success for non-existent objects. ( #2633 )
...
Fixes #2630
2016-09-07 11:49:12 -07:00
afe874f15a
api: CopyObject pipe should be closed pro-actively upon error. ( #2625 )
...
Fixes a deadlock reproduced while running s3verify during
RemoveObject(). Previously held lock by GetObject() inside
the go-routine was never relenquished.
2016-09-06 02:23:32 -07:00
b291dbe9c5
s3: Fix compliance bugs reported by s3verify. ( #2624 )
...
- Fixes couple of error strings reported are mismatching.
- Fixes a error HTTP status which was wrong fixed.
- Remove usage of an deprecated PostResponse, au contraire
to their documentation there is no response body in
PostPolicy.
2016-09-06 02:18:02 -07:00
bc8f34bfe7
server/mux: Remove unused waitgroup from listenerMux. ( #2614 )
...
Change struct names to be more meaningful.
2016-09-02 19:07:42 -07:00
2dc7ecc59b
notifications: Fix bucket notifications for DeleteMultipleObjects. ( #2609 )
...
Now reports bucket notifications for DeleteMultipleObjects API.
Also deletes multiple objects in parallel.
2016-09-02 01:59:08 -07:00
7398d737b5
profiler: Fix it properly and generate/save profiles even failure situations. ( #2607 )
...
Fixes #2594
2016-09-01 20:13:11 -07:00
511a8cbe04
profiler: Fix how we generate profiles. ( #2596 )
...
Fixes #2594
2016-09-01 03:50:31 -07:00
e79d2381fc
Fix rare 'go test -race' failure in ListenServe{Plain,TLS} ( #2588 )
2016-08-31 16:07:44 -07:00
a3b4199e9b
FS/Multipart: Allow for parallel uploads of parts with same part number. ( #2587 )
...
CBL client does not close connection when the backup process is stopped, this causes
read() on the stream on the server side to block and hence the lock held on the part
is not released. When the backup process is restarted, we again try to lock on the
part and this will block. Using a unique tmp name and not locking it fixes the problem.
2016-08-31 16:01:32 -07:00
3aa0574c6b
FS/multipart: Append the parts to a file as and when the parts arrive. ( #2513 )
2016-08-31 13:42:57 -07:00
5716f1c199
Fix incorrect error strings. ( #2580 )
2016-08-30 13:47:09 -07:00
2020afd200
server: http.Server do not add deadlines causes issues. ( #2581 )
...
Adding deadlines is a no go since Golang doesn't back off
the timers if there is an active i/o in progress.
It is meant to be for applications to handle this themselves
and manually progress the deadlines.
Fixes #2561
2016-08-30 12:52:19 -07:00
7431acb2c4
web: add handlers for set/get bucket policy. ( #2486 )
...
Refer #1858
2016-08-30 10:04:50 -07:00
2f9975c76c
notifications: GetObjectInfo() later if bucket notifications are set. ( #2575 )
2016-08-30 03:42:37 -07:00
7270ca4157
pkg/wildcard: Simplify the wildcard logic further. ( #2555 )
2016-08-27 00:27:17 -07:00
b67c8970c9
bucket-handlers: fix missing return statement ( #2553 )
...
If the location was invalid, it would write an error response but then
continue to attempt to make the bucket. Whether or not it would succeed,
it would attempt to call response.WriteHeaders twice in a row, which
would cause a message to be logged to the server console (bad).
Here is the relevant Go code:
c80e0d374b/src/net/http/server.go (L878-L881)
2016-08-25 20:00:47 -07:00
418921de89
minor cleanup
...
- Reused contains() from utils.go at a couple of places
- Cleanup in return statements and boolean checks
2016-08-24 22:54:34 -07:00
ec4260d260
api: BucketNotification should disallow duplicate notification. ( #2539 )
...
Added checks to look for duplicated notification configs.
Fixes #2472
2016-08-23 18:42:30 -07:00
c39d3db7a0
server/tls: allocate tls.Config{} properly. ( #2537 )
...
There is a golang bug which crashes the server, allocate tls.Config properly to avoid
this problem.
```
2016/08/22 20:21:03 http: panic serving 127.0.0.1:40744: runtime error: invalid memory address or nil pointer dereference
goroutine 38 [running]:
net/http.(*conn).serve.func1(0xc820526680)
/home/harsha/.gimme/versions/go1.6.2.linux.amd64/src/net/http/server.go:1389 +0xc1
panic(0xbefa80, 0xc820010140)
/home/harsha/.gimme/versions/go1.6.2.linux.amd64/src/runtime/panic.go:443 +0x4e9
crypto/tls.(*Conn).serverHandshake(0xc820368300, 0x0, 0x0)
/home/harsha/.gimme/versions/go1.6.2.linux.amd64/src/crypto/tls/handshake_server.go:43 +0x4d6
crypto/tls.(*Conn).Handshake(0xc820368300, 0x0, 0x0)
/home/harsha/.gimme/versions/go1.6.2.linux.amd64/src/crypto/tls/conn.go:1035 +0x169
net/http.(*conn).serve(0xc820526680)
/home/harsha/.gimme/versions/go1.6.2.linux.amd64/src/net/http/server.go:1405 +0x382
created by net/http.(*Server).Serve
/home/harsha/.gimme/versions/go1.6.2.linux.amd64/src/net/http/server.go:2137 +0x44e
```
Fixes #2536
2016-08-23 00:20:15 -07:00
2da0cfc904
signature: Use the client sent region if region is properly validated. ( #2534 )
...
A properly validated region string from the client should be once
validated properly.
Fixes #2521
2016-08-22 19:43:36 -07:00
9dd0e3dc44
config: Initialize notify for fresh configs. ( #2531 )
2016-08-22 16:04:04 -07:00
45c928e2f5
boot: checkPortAvailability() should fail only for EADDRINUSE error and ignore other errors. ( #2527 )
...
fixes #2510
2016-08-22 10:20:01 -07:00
07506358ff
Refactor Heal RPC and add Shutdown RPC ( #2488 )
2016-08-21 12:06:53 -07:00
975eb31973
api/bucketPolicy: Use minio-go/pkg/set and fix bucket policy regression. ( #2506 )
...
Current master has a regression 'mc policy <policy-type> alias/bucket/prefix'
does not work anymore, due to the way new minio-go changes do json marshalling.
This led to a regression on server side when a ``prefix`` is provided
policy is rejected as malformed from th server which is not the case with
AWS S3.
This patch uses the new ``minio-go/pkg/set`` package to address the
unmarshalling problems.
Fixes #2503
2016-08-20 03:16:38 -07:00
f2fd8b0265
Adding test coverage for server-mux.go by creating TestListenAndServeTLS(). ( #2493 )
...
Editing config.go to add lock on customConfigPath to avoid race condition
when setGlobalConfigPath() and getConfigPath() are called concurrently.
2016-08-19 01:29:50 -07:00
a8052889fe
server-mux_test.go updating test to fix race condition with TestListenAndServe ( #2485 )
2016-08-18 19:45:03 -07:00
bccf549463
server: Move all the top level files into cmd folder. ( #2490 )
...
This change brings a change which was done for the 'mc'
package to allow for clean repo and have a cleaner
github drop in experience.
2016-08-18 16:23:42 -07:00
aabfd541e1
Merge cmd/donut into minio cmd, deprecate controller RPC request
2015-07-31 12:57:15 -07:00
f15375426a
minor: Add commands into donut template
2015-07-28 19:40:02 -07:00
0eefbdef0c
use new app.ExtraInfo inside minio and donut commands properly
2015-07-24 23:55:18 -07:00
63c9cf0c4b
Move from Minimalist Object Storage to Minio Cloud Storage
2015-07-24 17:51:40 -07:00
4498662c16
Remove scsi non portable code, instead "donut make" implements functionality to instantiate a donut
2015-07-15 11:55:57 -07:00
1e80925ca7
Remove dependency on minio/cli for make.go fixes #738
2015-07-13 11:39:28 -07:00
f360ee0ab5
Add mkdonut examples
2015-07-12 21:37:57 -07:00
55e4d0c6a5
mkdonut now creates a donut processing cli args
2015-07-12 21:21:31 -07:00
535bcc3eac
Add mkdonut command
2015-07-12 19:16:36 -07:00
432275e966
Full restructure in accordance with
...
- pkg/{subsystem}/{package} style
- modify Makefile to reflect the new style,
consolidate various entries
- add a dummy ``main.go`` at top level
2015-01-14 11:29:04 -08:00
462808b87a
Disable sha1,sha256,sha512 avx,avx2,sse3 crypto implementations.
...
Re-implement them later, once stable
2015-01-06 17:02:25 -08:00
992dea1bf3
Adding index command
2015-01-06 22:26:09 +13:00
fff6b589cf
Rename split-file --> split
2015-01-03 18:03:21 -08:00
f5010e1ee2
Adding initial split-file command
2015-01-04 14:47:37 +13:00
0212079cd5
Remove template files, make them const strings instead
2015-01-01 17:31:51 -08:00
0e4a26e3b4
Add Diskattrmap, Scsiattrmap for probed scsi devices
...
Additional changes
- Use ``iota`` for constants
- Remove unncessary C header files
- ``new-cmd`` now depends on codegangsta cli
2014-12-30 02:37:47 -08:00
6b36b5c551
A full restructure
2014-12-29 21:22:26 -08:00
13650e088c
Implement md5c function, slower than Golang's implementation
...
keeping it in repo to make further improvements and also rename
minio-hash ---> crypto
2014-12-21 03:50:33 -08:00
2278df9910
Adding minio-hash with streaming crypto hashes
2014-12-21 13:04:41 +13:00
d4c05f11c7
Update documentation
2014-12-20 09:09:35 -08:00
b0e986c82c
Rename appname to commandname for consistency
2014-12-19 00:16:41 -08:00
5268a62166
Add minio-cli stub generator
2014-12-19 00:00:14 -08:00
65e2031cfc
Implement add, commit - instead of put, encoded
2014-12-15 21:55:54 -08:00
98119707fb
Move wrappers at pkgs/minio back to cmd/minio - as their usage is limited
2014-12-15 20:09:50 -08:00
3ec5c2fb8f
Implement commands
...
Commands implemented
$ minio put <objectname> <file>
$ minio encode <objectname>
$ minio get <objectname>
$ minio list
TODO :
$ minio init
$ minio verify
...
...
2014-12-15 14:59:15 -08:00
5efc0d54f8
storage.ObjectStorage List() is now List(objectPath string)
2014-12-15 19:30:08 +13:00
05baa6d9a2
Removing fsstorage
2014-12-15 19:15:22 +13:00
fc50291cac
fsStorage should use appendStorage with offsets
...
- also takes in 'blockSize' cli option.
- robustness fixes
2014-12-12 17:18:16 -08:00
0a0e1111cd
Add list() object support for erasure and append storage drivers
...
- Reply back objects with their protectionlevel and md5sum
- // TODO hash value
- Calculate md5sum after "storeBlocks()", to make sure data is committed
2014-12-12 02:52:37 -08:00
f71b323e32
Provide new cli options for 'put' command
...
- --protection-level
- --block-size
2014-12-11 22:56:12 -08:00
0bd2b8516e
Build fix, fixing without a PR
2014-12-11 20:44:06 -08:00
dddc555491
Use objectName with basename for the filepath
2014-12-11 15:35:25 -08:00
568ca4e1bc
erasure encoded demo
2014-12-11 00:32:32 -08:00
a1040ae657
erasure-demo put now takes 1 or 2 arguments
2014-12-10 20:59:49 -08:00
19da9760b3
ObjectStorage refactor to take io.Reader instead of []byte
2014-12-10 20:57:59 -08:00
cae9b288b0
Adding List(path) to object storage definition and accompanying definition to fs
2014-12-10 19:15:03 -08:00
9dd3dd8b53
Merge pull request #100 from harshavardhana/pr_out_add_crc32c_checksum_support_for_erasure_chunks
...
Add crc32c checksum support for erasure chunks
2014-12-10 11:53:23 -08:00
01c0e45b5c
Add crc32c checksum support for erasure chunks
...
Now upon conditions
- if checksum '.cksum' file is missing
- or corrupted chunk (mismatching chunk)
- or missing chunk entirely
Decoding works consistently
2014-12-10 00:58:15 -08:00
f6a5b61b00
Make get/put more robust
...
- init folder if not present already in 'get()' call
- put.go was referencing wrong args - fix it
- if no OBJECTNAME is specified during 'get' - grab a
list of files under "~/.minio"
- Provide new API GetList() for ObjectStorage struct
2014-12-09 03:32:31 -08:00
d1ebf935da
Adding extremely simple fs based get and put in erasure-demo
2014-12-08 20:20:25 -08:00
7314b5e37d
Fixes #71 - some crazy races inside erasure and simplify the code
...
- This change also brings in changing 'unsigned char' to 'uint8_t'
for brevity
2014-12-07 01:33:49 -08:00
9a5003dd25
Provide JoinFilesWithPrefix case with Gochannel
2014-12-05 16:02:46 -08:00
46b08681a4
Adding chunking by block to erasure-demo via --block-size parameter
2014-12-01 21:06:36 -08:00
323a7b8492
Refactoring input parsing for erasure-demo
2014-11-30 22:18:23 -08:00
6382ee161e
Renaming minio-demo to erasure-demo
2014-11-30 13:54:31 -08:00
30687a8c66
Increasing parity from 5 to 6
2014-11-29 22:59:44 -08:00
3c09029049
Merging encode and decode to single minio-demo binary
2014-11-29 22:48:37 -08:00
5074542636
Fix function name change
2014-11-29 17:29:38 -08:00
becb165770
Simplifying code
2014-11-26 23:33:36 -08:00
3cefaf5ad8
Restructure directory, add 'pkgs' folder now
2014-11-24 15:10:25 -08:00
6592ef3bd2
Make use of LRU and higher order erasure functions
2014-11-24 07:16:54 -08:00
ee7ff7c436
Decode on chunks missing should build and continue
2014-11-24 06:58:27 -08:00
5a3eb47583
Adding minio-decode
2014-11-23 19:32:01 -08:00
cec0a0ef68
Adding k and m as optional parameters for encoding in minio-encode
2014-11-23 15:49:26 -08:00
6faa55e25a
Adding input and output file parameters to minio-encode.
2014-11-22 19:25:46 -08:00
3b9495d91a
Vandermonde --> Cauchy
2014-11-22 18:52:07 -08:00
a15e935a33
Adding minio-encode binary
2014-11-22 18:39:59 -08:00
9c18624e58
Removing binaries
2014-11-22 15:16:38 -08:00
f8f9f65652
Moving minio command to cmd/minio
2014-11-14 18:20:40 -07:00
Harshavardhana
Anis Elleuch
Aditya Manthramurthy
Krishnan Parthasarathi
Anis Elleuch
Bala FA
Karthic Rao
Krishna Srinivas
Krishnan Parthasarathi
Krishna Srinivas
Frank
Mike Ralphson
Mateusz Gajewski
Mateusz Gajewski
awwalker
Kevin Qiu
Brendan Ashworth
Mohit Agarwal
Jesse Lucas
Harshavardhana
Frederick F. Kautz IV
Frederick F. Kautz IV