MyFavMirrors/minio
1
0
Fork 0
mirror of https://github.com/minio/minio.git synced 2024-12-26 23:25:54 -05:00
Code Issues Packages Projects Releases Wiki Activity
4,350 Commits 4 Branches 506 Tags 189 MiB
f7889e1af7
Commit Graph

898 Commits

Author SHA1 Message Date
Bala FA
05dc52a206 fix: use constants for access/secret key min/max length (#3271) 2016-11-16 17:33:55 -08:00
Harshavardhana
c91d3791f9 heal: Add healing support for bucket, bucket metadata files. (#3252) 
This patch implements healing in general but it is only used
as part of quickHeal().

Fixes #3237
 2016-11-16 16:42:23 -08:00
Aditya Manthramurthy
2f43709f85 Prevent gorilla mux from normalizing path (Fixes #3256) (#3268) 
Also fix test to not use a bucket name with a leading slash - this
causes the bucket name to become empty and go to an unintended API
call (listbuckets).
 2016-11-16 16:23:22 -08:00
Bala FA
61d67a061c tests: add unit test for DeleteMultipleObjectsHandler. (#3267) 
Fixes #3058
 2016-11-16 09:46:09 -08:00
Harshavardhana
1b85302161 Fix spelling and golint errors. (#3266) 
Fixes #3263
 2016-11-15 18:14:23 -08:00
Anis Elleuch
6512d9978e Add support of user self signed certificates 
Additionally add documentation about how to configure TLS with Minio
 2016-11-15 16:15:23 -08:00
Aditya Manthramurthy
e216201901 Remove control command from minio binary (Fixes #3264) (#3265) 2016-11-15 13:39:02 -08:00
Krishnan Parthasarathi
7abcededf2 Add tests for storage rpc handlers (#3262) 2016-11-15 12:12:06 -08:00
Harshavardhana
398421b9f5 xl/bootup: Server bootup shouldn't return for missing buckets. (#3255) 
Ref #3196
 2016-11-14 15:45:00 -08:00
Anis Elleuch
b8f0d9352f signature-v2: encode path and query strings when calculating signature (#3253) 2016-11-14 10:23:21 -08:00
Harshavardhana
f234c35020 lock: slice length of lock clients should be precisely urls. (#3254) 
This patch fixes a possible bug, reproduced rarely only seen
once.

```
panic: runtime error: index out of range

goroutine 136 [running]:
panic(0xac1a40, 0xc4200120b0)
    /usr/local/go/src/runtime/panic.go:500 +0x1a1
github.com/minio/minio/vendor/github.com/minio/dsync.lock.func1(0xc4203d2240, 0x4, 0xc420474080, 0x4, 0x4, 0xc4202abb60, 0x0, 0xa86d01, 0xefcfc0, 0xc420417a80)
    /go/src/github.com/minio/minio/vendor/github.com/minio/dsync/drwmutex.go:170 +0x69b
created by github.com/minio/minio/vendor/github.com/minio/dsync.lock
    /go/src/github.com/minio/minio/vendor/github.com/minio/dsync/drwmutex.go:191 +0xf4
```
 2016-11-14 10:18:56 -08:00
Anis Elleuch
0c042a622a Return objects content types in Web List Objects handler (#3249) 2016-11-13 12:26:40 -08:00
Harshavardhana
c57a358c9d Validate date header only for Signed{,V2} and StreamingSigned. (#3248) 
Fixes #2941
 2016-11-13 12:08:24 -08:00
Anis Elleuch
380d6c6435 Use getObjectInfo() in both FS and XL ListObjects() to simplify and to return complete object information (#3247) 2016-11-13 11:48:02 -08:00
Harshavardhana
716316f711 Reduce number of envs and options from command line. (#3230) 
Ref #3229

After review with @abperiasamy we decided to remove all the unnecessary options

- MINIO_BROWSER (Implemented as a security feature but now deemed obsolete
  since even if blocking access to MINIO_BROWSER, s3 API port is open)
- MINIO_CACHE_EXPIRY (Defaults to 72h)
- MINIO_MAXCONN (No one used this option and we don't test this)
- MINIO_ENABLE_FSMETA (Enable FSMETA all the time)

Remove --ignore-disks option - this option was implemented when XL layer
 would initialize the backend disks and heal them automatically to disallow
 XL accidentally using the root partition itself this option was introduced.

This behavior has been changed XL no longer automatically initializes
`format.json`  a HEAL is controlled activity, so ignore-disks is not
useful anymore. This change also addresses the problems of our documentation
going forward and keeps things simple. This patch brings in reduction of
options and defaulting them to a valid known inputs.  This patch also
serves as a guideline of limiting many ways to do the same thing.
 2016-11-11 16:40:55 -08:00
Anis Elleuch
98e79b4b50 Use 307 StatusTemporaryRedirect to redirect clients from http to https with forcing them to keep HTTP Verb (#3246) 2016-11-11 15:04:51 -08:00
Anis Elleuch
e2216a0936 exponentialBackoffWait returns zero after some retries, limit attempt number inside (#3245) 2016-11-11 09:49:30 -08:00
Harshavardhana
e8d9d710d0 rpc: Protect racy access of internal auth states. (#3238) 
Fixes #3232
 2016-11-11 00:14:32 -08:00
Harshavardhana
a8ab02a73a v4/presign: Fix presign requests when there are more signed headers. (#3222) 
This fix removes a wrong logic which fails for requests which
have more signed headers in a presign request.

Fixes #3217
 2016-11-10 21:57:15 -08:00
Aditya Manthramurthy
cf022de4d5 Add tests for s3PeerAPIHandlers (Fixes #3067) (#3242) 2016-11-10 16:43:04 -08:00
Bala FA
3995e21c5b fix: Ignore object not found error in RemoveObject() in web-handler. (#3228) 
Fixes #3181
 2016-11-10 15:02:03 -08:00
Harshavardhana
2f7fb78692 rpc: Our rpcClient should make an attempt to reconnect. (#3221) 
rpcClient should attempt a reconnect if the call fails
with 'rpc.ErrShutdown' this is needed since at times when
the servers are taken down and brought back up.

The hijacked connection from net.Dial is usually closed.

So upon first attempt rpcClient might falsely indicate that
disk to be down, to avoid this state make another dial attempt
to really fail.

Fixes #3206
Fixes #3205
 2016-11-10 07:44:41 -08:00
Bala FA
cf2fb30ac7 event: Add event notification for object deletion in web browser. (#3226) 
Fixes #3201
 2016-11-10 07:42:55 -08:00
Harshavardhana
51d1e6f75b Fix missing CompleteMultipartUpload Etag. (#3227) 
Fixes #3224
 2016-11-10 07:41:02 -08:00
Harshavardhana
2f373684f5 Fix the server startup messages and help text. (#3211) 2016-11-09 23:37:12 -08:00
Aditya Manthramurthy
dd0698d14c Improve namespace lock API: (#3203) 
- abstract out instrumentation information.
- use separate lockInstance type that encapsulates the nsMutex, volume,
  path and opsID as the frontend or top-level lock object.
 2016-11-09 10:58:41 -08:00
Harshavardhana
3e67bfcc88 heal: Print heal command appropriately without export path. (#3208) 
Fixes #3204
 2016-11-09 10:50:14 -08:00
Anis Elleuch
ea579f5b69 Avoid shutdown fs multiple times and create a new fs in each loop (#3213) 2016-11-09 10:10:14 -08:00
Aditya Manthramurthy
d44e9d6da9 Prevent weird messages from rpc lib on start (Fixes #3209): (#3212) 
This is done by not making the methods of the BucketMetaState interface
as methods (via type nesting) on the type implementing
RPCs (s3PeerAPIHandlers).
 2016-11-08 23:47:44 -08:00
Bala FA
9c2cfb5cb6 tests: Add missing unit test cases for AbortMultipartUploadHandler(). (#3200) 
Fixes #3070
 2016-11-08 16:25:00 -08:00
Bala FA
91a0ade908 tests: add unit test for HeadObjectHandler (#3197) 
Fixes #3068
 2016-11-07 16:02:27 -08:00
Aditya Manthramurthy
85a5c358d8 Add bucket metadata state client/handler (Fixes #3022) (#3152) 
- Adds an interface to update in-memory bucket metadata state called
  BucketMetaState - this interface has functions to:
     - update bucket notification configuration,
     - bucket listener configuration,
     - bucket policy configuration, and
     - send bucket event

- This interface is implemented by `localBMS` a type for manipulating
  local node in-memory bucket metadata, and by `remoteBMS` a type for
  manipulating remote node in-memory bucket metadata.

- The remote node interface, makes an RPC call, but the local node
  interface does not - it updates in-memory bucket state directly.

- Rename mkPeersFromEndpoints to makeS3Peers and refactored it.

- Use arrayslice instead of map in s3Peers struct

- `s3Peers.SendUpdate` now receives an arrayslice of peer indexes to
  send the request to, with a special nil value slice indicating that
  all peers should be sent the update.

- `s3Peers.SendUpdate` now returns an arrayslice of errors, representing
  errors from peers when sending an update. The array positions
  correspond to peer array s3Peers.peers

Improve globalS3Peers:

- Make isDistXL a global `globalIsDistXL` and remove from s3Peers

- Make globalS3Peers an array of (address, bucket-meta-state) pairs.

- Fix code and tests.
 2016-11-07 12:09:24 -08:00
Harshavardhana
33c771bb3e tests: Add tests for browser peer rpc. Fixes #3062 (#3189) 2016-11-07 11:43:35 -08:00
Karthic Rao
286a8924fd Add leak detection to object-handler tests. (#3195) 2016-11-06 21:53:50 -08:00
Karthic Rao
efca29b00e Fix typos and comments in leak_detect_test.go. (#3193) 2016-11-06 21:02:29 -08:00
Harshavardhana
9161016962 tests: Improve coverage on signature v4 tests. (#3188) 
Fixes #3065
 2016-11-06 11:47:16 -08:00
Anis Elleuch
5ff30777e1 Rewrite connection muxer peek process to avoid server blocking by silent clients (#3187) 2016-11-06 11:41:01 -08:00
Anis Elleuch
754c0770d6 Merge ListenAndServe and ListenAndServeTLS for simplification purpose (#3186) 2016-11-05 20:32:13 -07:00
Harshavardhana
1105508453 connection muxer should use bufio.Reader to be simpler. (#3177) 2016-11-05 12:57:31 -07:00
Harshavardhana
1ba497950c Fix net.Listener to fully close the underlying socket. (#3171) 
Leads to races and accepting connections. This patch implements
a way to reject accepting new connections.
 2016-11-05 09:43:28 -07:00
Aditya Manthramurthy
eb1bc67db1 Refactor: simplify signV4TrimAll() (#3179) 2016-11-04 13:52:22 -07:00
Krishna Srinivas
8408dfaa6c bootup-validation: Allow port configuration only using --address option. (#3166) 2016-11-04 12:14:19 -07:00
Harshavardhana
d192044915 router: PathPrefix router was wrong. (#3172) 2016-11-04 12:13:22 -07:00
Anis Elleuch
d9bab6b3bd sigv4: Trim and shrink spaces in headers values (#3162) 2016-11-03 16:41:25 -07:00
Anis Elleuch
e6965ca066 Quit initializing disks process when term signal is invoked (#3163) 2016-11-02 15:27:36 -07:00
Harshavardhana
d9674f7524 Improve coverage of web-handlers.go (#3157) 
This patch additionally relaxes the requirement for
accesskeys to be in a regexy set of values.

Fixes #3063
 2016-11-02 14:45:11 -07:00
Anis Elleuch
79601d27f2 Use endpoint url when printing disks status in distributed mode (#3151) 2016-11-02 08:51:06 -07:00
Aditya Manthramurthy
226a69fe15 Add test for updateUploadJSON (Fixes #3060) (#3155) 2016-11-01 20:04:32 -07:00
Harshavardhana
9bb799462e Start all listeners when a given host resolves to multiple IPs. (#3145) 
Default golang net.Listen only listens on the first IP when
host resolves to multiple IPs.

This change addresses a problem for example your ``/etc/hosts``
has entries as following

```
127.0.1.1  minio1

192.168.1.10 minio1
```

Trying to start minio as

```
minio server --address "minio1:9001" ~/Photos
```

Causes the minio server to be bound only to "127.0.1.1" which
is an incorrect behavior since we are generally interested in
`192.168.1.10` as well.

This patch addresses this issue if the hostname is resolvable
and gives back list of addresses associated with that hostname
we just bind on all of them as it is the expected behavior.
 2016-11-01 15:38:28 -07:00
Karthic Rao
8bffa78f7f Fix Instance type during benchmarks. (#3147) 
- The benchmark initialization function was not taking into account the
  instance type (FS/XL), was using XL ObjectLayer even for FS
  benchmarks.
- This was leading to incorrect benchmark results for FS related
  benchmarks.
- The fix takes into account the instance type (FS/XL) and correctly
  returns FS backend for FS benchmarks.
 2016-11-01 10:21:16 -07:00
Anis Elleuch
4b302173ae fs: Remove object append file if found since it is not always used (#3139) 2016-11-01 00:56:03 -07:00
Karthic Rao
9db2205db3 Fix List-Object Benchmark failure. (#3142) 2016-10-31 13:57:30 -07:00
Aditya Manthramurthy
6914fe436c Remove unused function. (#3143) 2016-10-31 12:18:15 -07:00
Anis Elleuch
807cc3c28d Tracing locking errors for better debugging (#3140) 2016-10-31 10:17:14 -07:00
Harshavardhana
f3c6c55719 posix: Fix windows performance issues. (#3132) 
Do not attempt to fetch volume/drive information for
each i/o situation. In our case we do this in all calls
`posix.go` this in-turn created a terrible situation for
windows. This issue does not affect the i/o path on Unix
platforms since statvfs calls are in the range of micro
seconds on these platforms.

This verification is only needed during startup and we
let things fail at a later stage on windows.
 2016-10-31 09:34:44 -07:00
Karthic Rao
3e8cb8c937 build: ineffassign fixes. (#3134) 2016-10-30 23:32:46 -07:00
Aditya Manthramurthy
dd6ecf1193 Read/write quorum algo with uploads.json in xl (Fixes #3123) (#3124) 
- Reads and writes of uploads.json in XL now uses quorum for
  newMultipart, completeMultipart and abortMultipart operations.

- Each disk's `uploads.json` file is read and updated independently for
  adding or removing an upload id from the file. Quorum is used to
  decide if the high-level operation actually succeeded.

- Refactor FS code to simplify the flow, and fix a bug while reading
  uploads.json.
 2016-10-30 09:27:29 -07:00
Anis Elleuch
a47ce7ab22 Add support of fallocate for FS and XL backends (#3032) 2016-10-29 12:44:44 -07:00
Krishna Srinivas
0b3282ac9f Logging: errorIf fatalIf print in the format [file.go:82:funcName()] (#3127) 2016-10-29 02:34:16 -07:00
Krishna Srinivas
79b98b5c25 boot: getPath() should take care of simple directory exports. (#3122) 
fixes #3120
 2016-10-28 13:08:56 -07:00
Krishnan Parthasarathi
6a57f2c1f0 XL: Add more information to panic msg (#3119) 2016-10-28 08:46:03 -07:00
Krishna Srinivas
0fa2477cb0 boot: check parameter syntax before initializing the system. (#3114) 
fixes #3112
 2016-10-28 08:45:32 -07:00
Anis Elleuch
7ffb337cd7 Update control cmd USAGE to indicate that it is possible to put access secret keys in URL (#3115) 2016-10-27 13:26:11 -07:00
Harshavardhana
9e2d0ac50b Move to URL based syntax formatting. (#3092) 
For command line arguments we are currently following

- <node-1>:/path ... <node-n>:/path

This patch changes this to

- http://<node-1>/path ... http://<node-n>/path
 2016-10-27 03:30:52 -07:00
Aditya Manthramurthy
30dc11a931 No listener.json for single-node mode (Fixes #3052) (#3108) 
In FS or single-node XL mode, there is no need to save listener
configuration to persistent storage. As there is only one server, if it
is restarted, any connected listenBucketAPI clients were disconnected
and will have to reconnect - so there is nothing to actually store.

This incidentally solves #3052 by avoiding the problem.
 2016-10-26 20:13:00 -07:00
Anis Elleuch
a15dc5fed5 Print message when creating the config file (#3089) 2016-10-26 18:44:22 -07:00
Anis Elleuch
f7c20b97a1 control cmds: Extract access and secret keys from URL if specified (#3109) 2016-10-26 18:42:12 -07:00
Harshavardhana
e9c45102b0 posix: Use sync.Pool buffers to copy in large buffers. (#3106) 
These fixes are borrowed from the fixes required for GlusterFS i/o throughput.
 2016-10-26 17:14:05 -07:00
Anis Elleuch
8871eb8e1e Show offline nodes after a fixed number of init retry (#3107) 2016-10-26 16:09:06 -07:00
Krishna Srinivas
0f32efb825 PostPolicy - rename of files/functions + add testcases (#3104) 2016-10-26 10:15:57 -07:00
Karthic Rao
63f1b4fdf4 mispell fixes. (#3100) 2016-10-26 08:46:14 -07:00
Krishnan Parthasarathi
31f2db6880 Remove leftover debug statement from PutObject StreamingSignature unit-test (#3099) 2016-10-26 03:17:47 -07:00
Krishnan Parthasarathi
2c9b406f6c Add TLS based tests to functional test suite (#3083) 2016-10-26 02:30:31 -07:00
Krishnan Parthasarathi
12cd2da265 Add PutObjectHandler unit tests covering failure cases (#3096) 2016-10-26 02:06:22 -07:00
Harshavardhana
485c0ea8bf tests: Combine v2 tests with the Suite itself. (#3088) 2016-10-25 13:34:14 -07:00
Krishnan Parthasarathi
49ba07d1d6 Use net.ParseCIDR instead of custom-built parsers (#3055) 
Removes avoidable conversion to and from net.IP to string.
 2016-10-25 11:14:47 -07:00
Krishna Srinivas
35e541e0b1 content-length-range policy should be honored for the uploaded object sizes. (#3076) 2016-10-24 23:47:03 -07:00
Aditya Manthramurthy
3977d6b7bd Lock bucket while modifying its metadata (Fixes #2979) (#3019) 
- When modifying notification configuration
- When modifying listener configuration
- When modifying policy configuration

With this change we also stop early checking if the bucket exists, since
that uses a Read-lock and causes a deadlock due to the outer Write-lock.
 2016-10-24 19:52:24 -07:00
Harshavardhana
0905398459 Fix benchmark tests. (#3082) 
Fixes #3081
 2016-10-24 18:45:06 -07:00
Aditya Manthramurthy
f41faf96b7 Fix newMultipartUpload to not leave stale uploads.json (Fixes #3014) (#3079) 2016-10-24 17:37:18 -07:00
Bala FA
36639b65a9 rename completeMultipartMD5() into getCompleteMultipartMD5(). (#3051) 2016-10-24 13:56:13 -07:00
Harshavardhana
7fc598b73f Fix user-agent prefix to have docker instead of suffix. (#3074) 2016-10-24 13:44:15 -07:00
Krishna Srinivas
21d41ad7fd init[windows]: Fix to handle the case when export path is a relative path. (#3054) 
ex. to handle "minio server export"
 2016-10-24 08:26:28 -07:00
Harshavardhana
fe56220d1a Do not print nil when hostname is provided as --address (#3053) 
Fixes #3018
 2016-10-23 23:55:12 -07:00
Harshavardhana
5782ec3ada Fix peers and web UIVersion validation. (#3048) 2016-10-23 12:32:35 -07:00
Krishnan Parthasarathi
8839c5105a Pass values to closures esp. when passed to defer statement. (#3050) 
opsID, a variable on the stack, changes over the course of
Completemultipartupload function in xl-v1-multipart.go.  This was
being used in a function closure which was passed to defer
statement. The variables used in the closure depend on their values at
the time of evaluation which is indeterminate behaviour. It is
incorrect to depend on values of variables on stack at the end of
function, when deferred functions are executed.
 2016-10-23 09:57:52 -07:00
Anis Elleuch
6c2d5e3d05 Correct the number of failed disks that we can withstand in startup message (#3045) 2016-10-22 10:36:50 -07:00
Krishna Srinivas
5999a23d3e When object whose size is greater than 5G is uploaded using presigned POST we should return error. (#3033) 
fixes #2961
 2016-10-22 09:05:01 -07:00
Krishna Srinivas
e51be73ac7 PresignedPost: Support for Signature V2 presigned POST Policy. (#3043) 
fixes #2993
 2016-10-22 08:57:12 -07:00
Harshavardhana
4b5b363c6c tests: Fix race between SetBucketListenerConfig and eventNotifyForBucketNotifications (#3041) 2016-10-22 02:35:33 -07:00
Krishna Srinivas
f2b0c08e34 logging: print file:line:funcName of the caller of errorIf and fatalIf (#3035) 2016-10-22 02:24:34 -07:00
Harshavardhana
83b364891d tests: Fix a potential race in ListenBucketNotificationHandler. (#3040) 2016-10-21 22:56:27 -07:00
Harshavardhana
e3ab478c70 tests: Fix a potential race in RemoveBucketNotification. (#3037) 
Fixes #3036
 2016-10-21 17:12:56 -07:00
Harshavardhana
ecaccefd2e tests: Implement GetBucketNotification handler tests. (#3029) 2016-10-21 02:39:37 -07:00
Harshavardhana
ece559afe2 api: Do not use sqs for ListenBucketNotification. (#3023) 
Add more tests. Fixes #3024
 2016-10-21 01:25:17 -07:00
Krishna Srinivas
d3aaf50a40 posix: Split on ":" in path d:\export makes minio use wrong disk. (#3027) 
As the host/path split happens at a higher layer now, split at posix is not needed.
fixes part of #2987
 2016-10-20 23:39:33 -07:00
Krishna Srinivas
32c3a558e9 distributed-XL: Support to run one minio process per export even on the same machine. (#2999) 
fixes #2983
 2016-10-20 18:31:02 -07:00
Anis Elleuch
41f9ab1c69 Translate storage access denied error to S3 Access Denied response (#3015) 2016-10-20 16:09:55 -07:00
Aditya Manthramurthy
8876e0a80a Delete bucket listener config file from disk (#3016) 2016-10-20 16:09:19 -07:00
Anis Elleuch
c21ac80268 Validate access/secret keys found in the config file and enhance invalid keys messages (#3017) 2016-10-20 16:07:24 -07:00
Frank
0e2cd1a64d Added clear subcommand for control lock (#3013) 
Added clear subcommand for control lock with following options:

```
  3. Clear lock named 'bucket/object' (exact match).
    $ minio control lock clear http://localhost:9000/bucket/object

  4. Clear all locks with names that start with 'bucket/prefix' (wildcard match).
    $ minio control lock --recursive clear http://localhost:9000/bucket/prefix

  5. Clear all locks older than 10minutes.
    $ minio control lock --older-than=10m clear http://localhost:9000/

  6. Clear all locks with names that start with 'bucket/a' and that are older than 1hour.
    $ minio control lock --recursive --older-than=1h clear http://localhost:9000/bucket/a
```
 2016-10-20 13:15:28 -07:00
Aditya Manthramurthy
6274727b71 Pick up server address from --address option (#3002) (#3008) 
This makes sure that when SSL is enabled (for FS/single node mode),
the server address is picked up from the --address option (that needs
to include the hostname for SSL verification, and has to be input
appropriately by user), instead of just using ":<port>".
 2016-10-20 11:39:10 -07:00
Harshavardhana
95567c68bf posix: Do not print errors in expected errors. (#3012) 
Fixes #3011
 2016-10-20 09:26:18 -07:00
Anis Elleuch
c189337b6e rpc: Support SNI in TLS certificates (#3009) 2016-10-20 07:43:31 -07:00
Krishnan Parthasarathi
6fc81dc162 Delete temp object/part when PutObject{,Part} fails (#3004) 2016-10-19 22:52:03 -07:00
Krishnan Parthasarathi
7d50361ca9 Move housekeeping before object layer initialization (#3001) 
In a distributed setup that the server should not perform any operation
on the storage layer after it is exported via RPC. e.g, cleaning up of
temporary directories under .minio.sys/tmp may interfere with ongoing
PUT objects being served by the distributed setup.
 2016-10-19 19:59:48 -07:00
Frank
19c51f3f3c Added ForceUnlock to namespace-lock (#2990) 2016-10-19 09:27:36 -07:00
Aditya Manthramurthy
c3bbadacbf Improve Peer RPC error handling (Fixes #2992) (#2995) 
* Check for RPC connection shutdown and try again just once.

* Refactor SendRPC to use sync.WaitGroup
 2016-10-18 21:26:58 -07:00
Anis Elleuch
2208992e6a More informative message when erasure fails to read a part of an object (#2989) 2016-10-18 13:09:26 -07:00
Anis Elleuch
bbba8e432a Add ssl support to s3/web peers connections (#2988) 2016-10-18 11:46:33 -07:00
Harshavardhana
39331b6b4e xl: GetCheckSumInfo() shouldn't fail if hash not available. (#2984) 
In a multipart upload scenario disks going down and coming backup
can lead to certain parts missing on the disk/server which was
going down. This is a valid case since these blocks can be
missing and should be healed through heal operation. But we are
not supposed to fail prematurely since we have enough data on
the other disks as well within read-quorum.

This fix relaxes previous assumption, fixes a major corruption
issue reproduced by @vadmeste.

Fixes #2976
 2016-10-18 11:13:25 -07:00
Mike Ralphson
6e748cb1cf Report when invalid bucket names are skipped in FS backend. (#2947) 2016-10-18 01:42:46 -07:00
Anis Elleuch
2005d656e6 Properly load creds from env and save them when server cmd is executed (#2970) 2016-10-17 23:14:41 -07:00
Aditya Manthramurthy
0f26ec8095 Propagate creds change to cluster (Fixes #2855) (#2929) 2016-10-17 20:18:08 -07:00
Harshavardhana
8d2347bc7b storage: DeleteFile should return errFileNotFound for ENOENT. (#2978) 2016-10-17 16:38:46 -07:00
Aditya Manthramurthy
0ff359ca0e Fix early init. problem for notifications (Fixes #2972) (#2977) 2016-10-17 16:38:29 -07:00
Harshavardhana
f8e13fb00e server: Startup sequence should be more idempotent. (#2974) 
Fixes #2971 - honors ignore-disks option properly.
Fixes #2969 - change the net.Dial to have a timeout of 3secs.
 2016-10-17 14:31:33 -07:00
Harshavardhana
686a610fc3 api: Nanosecond precision for API responses is valid with S3. (#2957) 
Wqe need to be compatible as well fixes #2955
 2016-10-17 08:44:55 -07:00
Krishnan Parthasarathi
b89609dc2e XL: Filter out md5Sum from user defined headers (#2962) 2016-10-17 08:41:33 -07:00
Anis Elleuch
fa50312220 Avoid returning disk corrupted by servers in the middle of init all disks formats (#2964) 2016-10-17 08:39:55 -07:00
Harshavardhana
fee3f99a6e xl: heal bucket should validate if bucket exists first. (#2953) 
Fixes #2944
 2016-10-17 02:10:23 -07:00
Frank
ea406754a6 New dsync and added ForceUnlock to lock rpc server (#2956) 
* Update dsync and added ForceUnlock function
* Added test cases for ForceUnlock
 2016-10-17 01:53:29 -07:00
Aditya Manthramurthy
d02cb963d5 Fix listen-bucket (Fixes #2942) (#2949) 
Don't close socket while re-initializing notify-listeners, as the rpc
client object is shared between notify-listeners and peer clients.

Also, improves SendRPC() readability by using GetPeerClient().
 2016-10-16 20:52:10 -07:00
Anis Elleuch
334cdb5d64 XL total/free space calculation is done inside xl module (#2945) 2016-10-16 14:24:15 -07:00
Anis Elleuch
5c3639c1b7 Redirect /minio to /minio/ when requests come from browsers (#2937) 2016-10-15 06:21:51 -07:00
Krishna Srinivas
903574db90 copy-object: Do not use ETag of source as MD5 as it will not be MD5 if source was uploaded as multipart. (#2938) 
fixes #2934
 2016-10-15 06:20:55 -07:00
Anis Elleuch
f463d3ce42 Fix a crash when service shutdown is signaled and object API is not ready yet (#2939) 2016-10-15 06:20:16 -07:00
Aditya Manthramurthy
17eeec6895 Bucket policy propagation (Fixes #2930) (#2932) 
Fixes a serialisation bug - encoding/gob does not directly support
serializing `map[string]interface{}`, so we serialise to JSON and send a
byte array in the RPC call, and deserialize and update on the receiver.
 2016-10-14 22:49:51 -07:00
Karthic Rao
070d3610ff tests: V2 Signature tests for object-handlers. (#2931) 2016-10-14 20:52:46 -07:00
Harshavardhana
f22862aa28 heal: Refactor heal command. (#2901) 
- return errors for heal operation through rpc replies.
  - implement rotating wheel for healing status.

Fixes #2491
 2016-10-14 19:57:40 -07:00
Harshavardhana
2f520ed92f Remove errors package, add comments and simplify. (#2925) 2016-10-14 12:31:00 -07:00
Mateusz Gajewski
c03ce0f74a Display SSL expiry warnings (#2925) 2016-10-14 12:30:36 -07:00
Krishna Srinivas
0320a77dc0 HealBucket: create the bucket if it is missing in one of the disks. (#2924) 2016-10-14 11:12:17 -07:00
Aditya Manthramurthy
31be826f51 Fix missing error check for jsonrpc.Server.RegisterService() (#2921) 2016-10-13 17:34:10 -07:00
Karthic Rao
17e49a9ed2 signature-v2 fix. (#2918) 
- Return errors similar to V4 Sign processsing.
- Return ErrMissing fields when Auth Header fields are missing.
- Return InvalidAccessID when accessID doesn't match.

* tests: Adding V2 signature tests for bucket handler API's.
 2016-10-13 09:25:56 -07:00
Aditya Manthramurthy
0aabc1d8d9 Use Peer RPC to propagate bucket policy changes (#2891) 2016-10-13 09:19:04 -07:00
Harshavardhana
55f6828750 Do not print update message unless there is an update. (#2919) 2016-10-13 09:17:08 -07:00
Aditya Manthramurthy
6303f26330 Protect map from concurrent access (Fixes #2915) (#2916) 
Protects the Peers RPC clients map from concurrent access to fix a data race condition.
 2016-10-13 01:33:50 -07:00
Krishnan Parthasarathi
b59bac670a Handle err returned by rpc.Server.RegisterName (#2910) 2016-10-12 23:13:24 -07:00
Anis Elleuch
84acc820c7 Fix free drive space calculation in XL mode (#2917) 2016-10-12 20:22:15 -07:00
Harshavardhana
fdaa129a5b Fix dockerfile container image. (#2892) 2016-10-12 18:09:08 -07:00
Anis Elleuch
df59967f59 Avoid checking date header of web requests by properly applying generic handlers (#2914) 2016-10-12 12:58:36 -07:00
Mateusz Gajewski
73982c8cb6 Listen bucket notification for multiple prefixes/suffixes (#2911) 
* Listen bucket notification for multiple prefixes/suffixes

* After review fixes
 2016-10-12 11:02:15 -07:00
Aditya Manthramurthy
6199aa0707 Peer RPCs for bucket notifications (#2877) 
* Implements a Peer RPC router that sends info to all Minio servers in the cluster.
* Bucket notifications are propagated to all nodes via this RPC router.
* Bucket listener configuration is persisted to separate object layer
  file (`listener.json`) and peer RPCs are used to communicate changes
  throughout the cluster.
* When events are generated, RPC calls to send them to other servers
  where bucket listeners may be connected is implemented.
* Some bucket notification tests are now disabled as they cannot work in
  the new design.
* Minor fix in `funcFromPC` to use `path.Join`
 2016-10-12 01:03:50 -07:00
Krishnan Parthasarathi
a5921b5743 Use same timestamp for all chunks in chunked signature (#2908) 2016-10-11 23:46:51 -07:00
Karthic Rao
ff91ecb177 tests: Adding unknown signature type test for API handlers. (#2905) 2016-10-11 20:38:10 -07:00
Frank
a6357502c1 Correct typo in error string (#2902) 2016-10-11 08:56:02 -07:00
Harshavardhana
fa8ea41cd9 lock/instrumentation: Cleanup and print in user friendly form. (#2807) 2016-10-11 00:50:27 -07:00
Karthic Rao
3ac6790ca2 tests: Add Object Layer nil test for bucket-handler API's (#2899) 2016-10-11 00:00:02 -07:00
Krishna Srinivas
268b96058f ns-lock: lock namespace during FS object operations. (#2896) 2016-10-10 10:20:04 -07:00
Frank
0d031c432b Fix typo in serverVersion (#2894) 2016-10-10 10:11:56 -07:00
Frank
6e8f3224c5 Test coverage for lock rpc server (#2893) 
* Add test coverage for removeEntry and removeEntryIfExists
* Initial test framework for Lock/Unlock functionality
* Add clarification comments
* Add test coverage code for RLock() and RUnlock()
* Add test coverage for Expired() function
* Have all lock-rpc-server test functions start with the same prefix
* Properly initialize JWT security token
 2016-10-10 10:11:29 -07:00
Karthic Rao
9c53e9f4c3 tests: Enhance coverage for bucket policy handlers. (#2895) 2016-10-10 09:29:56 -07:00
Krishnan Parthasarathi
2d5e988a6d Refactor streaming signatureV4 w/ state machine (#2862) 
* Refactor streaming signatureV4 w/ state machine

- Used state machine to make transitions between reading chunk header,
  chunk data and trailer explicit.

* debug: add print/panic statements to gather more info on CI failure

* Persist lastChunk status between Read() on ChunkReader

... remove panic() which was added as interim aid for debugging.

* Add unit-tests to cover v4 streaming signature
 2016-10-10 01:42:32 -07:00
Harshavardhana
3cfb23750a control: Implement service command 'stop,restart,status'. (#2883) 
- stop - stops all the servers.
- restart - restart all the servers.
- status - prints status of storage info about the cluster.
 2016-10-09 23:03:10 -07:00
Anis Elleuch
57f75b1d9b Ignore copy conditions when ETag is not available (#2888) 2016-10-09 16:21:42 -07:00
Karthic Rao
e213172431 tests: Missing anonymous tests for bucket-handlers. (#2885) 2016-10-09 09:21:37 -07:00
Krishna Srinivas
f5f007e183 Test: Add test case for xl.HealObject() (#2884) 
fixes #2842
 2016-10-08 17:08:17 -07:00
Karthic Rao
09463265ce tests: Adding anonymous requests tests for bucket policy handlers. (#2882) 2016-10-08 01:04:26 -07:00
Karthic Rao
8f4cf2a7d0 tests: anonymous/unsigned tests for object handler API's . (#2881) 2016-10-07 23:28:50 -07:00
Karthic Rao
30183c4a9a tests: cleanup and unsigned request test. (#2880) 
- Cleaning up of ListMultipartUpload API test for improving readability,
  code maintainance and extensibility.
- Moving ListMultipartUploads to Go 1.7 sub tests.
- Using the new Anonymous request helper function for
  ListMultipartUploads.
 2016-10-07 20:16:57 -07:00
Karthic Rao
d1df5e0ae1 tests: Add helper function for API handler anonymous request tests. (#2876) 
- Add helper function for API handler anonymous request tests.
- Add PutObject Part Anonymous request case using the new helper
  function to validate its functionality.
 2016-10-07 11:16:11 -07:00
Harshavardhana
f1bc9343a1 prep: Initialization should wait instead of exit the servers. (#2872) 
- Servers do not exit for invalid credentials instead they print and wait.
- Servers do not exit for version mismatch instead they print and wait.
- Servers do not exit for time differences between nodes they print and wait.
 2016-10-07 11:15:55 -07:00
Karthic Rao
97f4989945 tests: cleaning up. (#2875) 
- Clean up PutObjectPart and ListObjectPart API handler tests.
- Add more comments, make the tests more readable.
- Add verification for HTTP response status code.
- Initialize the test using object Layer.
- Move to Go 1.7 sub tests.
 2016-10-07 08:02:37 -07:00
Harshavardhana
1e5e213d24 auth: Make sure we initialize or change config before RPC requests. (#2867) 2016-10-06 13:35:56 -07:00
Karthic Rao
a8105ec068 - Test utility function for easy asserting of cases wherein objectLayer (#2865) 
is `nil` in API handlers.
- Remove the existing tests for the `nil` check and use the new method
  to test for object layer being `nil`.
 2016-10-06 13:34:33 -07:00
Krishna Srinivas
bb9be02228 minio-browser: do not redirect to /minio if MINIO_BROWSER=off (#2863) 
fixes #2837
 2016-10-06 08:30:32 -07:00
Harshavardhana
64f37bbf5b rpc: Add RPC client tests. (#2858) 2016-10-06 02:30:54 -07:00
Karthic Rao
0fc96fa25c Refactor bucket policy handler test to use API test initializer. (#2859) 2016-10-06 02:02:42 -07:00
Karthic Rao
2d8c6f8288 unit test for bucketPolicyConditionMatch function. (#2857) 2016-10-06 00:23:46 -07:00
Harshavardhana
b94211bd66 api: ListObjectsV1 compliance with AWS S3. (#2856) 
XSD - xml schema definition for SOAP operations
on S3 provides positional restrictions on XML
output.

Fix the response by re-arranging the positions in
accordance with S3 behavior.

Fixes #2849
 2016-10-05 20:12:47 -07:00
Harshavardhana
6494b77d41 server: Add more elaborate startup messages. (#2731) 
These messages based on our prep stage during XL
and prints more informative message regarding
drive information.

This change also does a much needed refactoring.
 2016-10-05 12:48:07 -07:00
Bala FA
63a7ca1af0 web: fix jwt token expiry set to one day by default. (#2819) 
Fixes #2818
 2016-10-05 10:18:55 -07:00
Krishna Srinivas
95f544657a Signature-V2: use raw resource/query from the request for signature calculation. (#2833) 2016-10-05 09:18:53 -07:00
Harshavardhana
740a919e25 config: Use migrateV8 to v9 function properly. (#2852) 2016-10-05 02:28:04 -07:00
Krishnan Parthasarathi
402c92beda Add listObjectParts test w/ unknown request signature type (#2847) 2016-10-04 07:57:35 -07:00
Karthic Rao
6a9013b97c misspell fixes. (#2835) 2016-10-04 00:09:21 -07:00
Krishnan Parthasarathi
73b50aea2d Add preSign auth type tests for ListObjectPartsHandler and PutObjectPartHandler (#2834) 2016-10-03 22:05:33 -07:00
Aditya Manthramurthy
315e66858c Add PostgreSQL notifier (#2739) (#2824) 
* The user is required to specify a table name and database connection
  information in the configuration file.

* INSERTs and DELETEs are done via prepared statements for speed.

* Assumes a table structure, and requires PostgreSQL 9.5 or above due to
  the use of UPSERT.

* Creates the table if it does not exist with the given table name using
  a query like:

    CREATE TABLE myminio (
        key varchar PRIMARY KEY,
        value JSONB
    );

* Vendors some required libraries.
 2016-10-03 17:29:55 -07:00
Krishnan Parthasarathi
4f902d42b2 Add unit-tests for ListObjectParts API handler (#2826) 
* Add missing uploadID test
... make variables in test code unexported.
* Add ServerNotInitialized test for ListObjectPartsHandler
* Add tests for ListObjectParts with signatureV2 and Anonymous requests
* Add failure test cases for ListObjectParts
 2016-10-03 08:54:57 -07:00
Krishna Srinivas
61a18ed48f sha256: Verify sha256 along with md5sum, signature is verified on the request early. (#2813) 2016-10-02 15:51:49 -07:00
Anis Elleuch
b5a6dd1395 Avoid path-cleaning policy resources for a better compliance with S3 (#2823) 2016-10-01 21:30:25 -07:00
Krishnan Parthasarathi
83e6e1060e Layer LimitReader responsibly allowing sign verification to work (#2821) 2016-10-01 09:37:40 -07:00
Krishnan Parthasarathi
ddeb8242d8 PutObjectPartHandler unit-tests (#2810) 2016-10-01 08:23:26 -07:00
Harshavardhana
5ecba587f7 api: Relax object name validation. (#2814) 
Fixes #2812
 2016-09-30 16:56:36 -07:00
Harshavardhana
db3da97a50 signature/v2: Fix presigned requests. 2016-09-30 15:22:00 -07:00
Harshavardhana
5885ffc8ae signature: Add legacy signature v2 support transparently. (#2811) 
Add new tests as well.
 2016-09-30 14:32:13 -07:00
Anis Elleuch
9fb1c89f81 Add TLS encryption capability to RPC clients (#2789) 2016-09-29 23:42:37 -07:00
Anis Elleuch
1e6afac3bd Add NATS notifier (#2795) 2016-09-29 23:42:10 -07:00
Harshavardhana
64083b9227 signature: Region changes should be handled just like AWS. (#2805) 
- PutBucket happens with 'us-east-1'.
- ListBuckets happens with any region.
- GetBucketLocation happens with 'us-east-1' and location is returned.
 2016-09-29 15:51:00 -07:00
Krishnan Parthasarathi
5fdd768903 Make addition of TopicConfig to globalEventNotifier go-routine safe (#2806) 2016-09-28 22:46:19 -07:00
Krishnan Parthasarathi
428629f577 Add unit tests for server-main.go (#2802) 2016-09-28 11:19:07 -07:00
Harshavardhana
1edd74dda2 update: Deprecate the usage of update=yes query param. (#2801) 
Fixes #2799
 2016-09-28 02:41:21 -07:00
Krishnan Parthasarathi
740ecf530c Add PutBucketNotification, ListenBucketNotification handler unit-tests. (#2787) 2016-09-28 01:08:03 -07:00
Aditya Manthramurthy
10d2ef5449 Remove comments relating to deprecated MINIO_DEBUG envvar (#2797) 2016-09-27 18:28:46 -07:00
Aditya Manthramurthy
8ea571c7f7 Remove MINIO_DEBUG environment variable (#2794) 
Removes the unimplemented settings of MINIO_DEBUG=mem and makes
MINIO_DEBUG=lock the default behaviour.
 2016-09-27 14:35:43 -07:00
Harshavardhana
ca3022d545 api: Change ListenBucketNotification with new API format. (#2791) 
Take prefix, suffix and events as query params.
 2016-09-27 13:17:43 -07:00
Anis Elleuch
9417614a8e Recalculate free minimum disk space (#2788) 
* Fix calculating free space disk by using blocks available for unprivileged user

* Use fixed minimal free disk space instead of percentage
 2016-09-27 12:46:38 -07:00
Aditya Manthramurthy
70d52bbc4c Add unit test for rate-limit-handler (#2661) (#2784) 2016-09-26 21:31:12 -07:00
Harshavardhana
6aa2fc95c0 Revert "bucket: refactor policies and fix bugs related to enforcing policies. (#2766)" 
This reverts commit ca5ca8332b.
 2016-09-26 19:32:33 -07:00
Harshavardhana
cfbab22237 web: Remove bucket policy when we have no more statements. (#2779) 2016-09-26 03:11:22 -07:00
Harshavardhana
be0e06c0aa web: Simplify and rename GetAllBucketPolicy --> ListAllBucketPolicies. (#2778) 2016-09-25 21:53:19 -07:00
Harshavardhana
1c941fd787 rpc: Should validate server versions. (#2775) 
Fixes #2764
 2016-09-24 03:34:45 -07:00
Krishnan Parthasarathi
669783f875 Purge stale object cache entry (#2770) 2016-09-23 19:55:28 -07:00
Krishnan Parthasarathi
27e474b3d2 Improve code coverage in bucket-notification-handlers.go (#2759) 
* Fix incorrect test cases for bucket-notification handler

* Add tests covering failure cases for bucket notification
 2016-09-23 13:32:51 -07:00
Krishna Srinivas
1e53316241 Add tests for presigned-get (#2767) 
* web-handlers: support for presigned-get json-rpc call for MinioBrowser's "share" feature.

* Add tests for presigned-get
 2016-09-23 01:25:49 -07:00
Harshavardhana
ca5ca8332b bucket: refactor policies and fix bugs related to enforcing policies. (#2766) 
This patch also addresses the problem of double caching at
object layer once at XL and another at handler layer.
 2016-09-22 23:47:48 -07:00
Bala FA
aa579bbc20 web: add method to get all policies for given bucket name. (#2756) 
Refer #1858
 2016-09-22 23:06:45 -07:00
Harshavardhana
e375d822da bucket: SetBucketPolicy should save a valid Version and validate. (#2762) 2016-09-22 22:27:21 -07:00
Anis Elleuch
fc783f8407 More tests for web handlers (#2755) 
* Return negative values of Total and Free in StorageInfo() when we fail to get disk info

* Return consistent messages in web handlers when the server is not initialized
 2016-09-22 16:35:12 -07:00
Anis Elleuch
ef22330563 Require go 1.7.1 to build Minio server (#2727) 2016-09-22 10:33:52 -07:00
Karthic Rao
1148f95292 ineffassign fixes (#2758) 2016-09-21 23:03:54 -07:00
Karthic Rao
f7430ec09c use runtime/debug.Stack() in leak detect test (#2757) 2016-09-21 22:04:35 -07:00
Karthic Rao
b8903d842c api/complete-multipart: fixes and tests. (#2719) 
* api/complete-multipart: tests and simplification.

- Removing the logic of sending white space characters.
- Fix for incorrect HTTP response status for certain cases.
- Tests for New Multipart Upload and Complete Multipart Upload.

* tests: test for Delelete Object API handler
 2016-09-21 20:08:08 -07:00
Aditya Manthramurthy
32f097b4d6 Controller rpc tests (#2709) 
* Test code for controller-handler operations:

* Heal operations
* List operation
* Switch to "testing" lib, moving away from gocheck
* Minor refactors

* Remove extra call to initGracefulShutdown

* Remove dead code in mainControl:

Dead code found by the TestControlMain() test function that always
passes.

* Add tests for control-*-main.go
 2016-09-21 19:58:50 -07:00
Krishnan Parthasarathi
559ad38b8c Add bucket-notification-handler tests (#2750) 2016-09-21 17:41:34 -07:00
Anis Elleuch
90417d2dd6 Check for bucket existence in Set/Get/Remove bucket policy workflow + tests (#2745) 2016-09-21 16:38:50 -07:00
Anis Elleuch
e66fb4bd7b configMigrate() returns errors + tests (#2735) 2016-09-21 09:44:57 -07:00
Harshavardhana
018c90dae7 events: ElasticSearch doesnt support objects with '/' in them. (#2747) 
Fix this by using a unique sha256 generated for each unique key.
 2016-09-20 16:36:18 -07:00
Anis Elleuch
a5066e8f76 Better code coverage of handler-utils.go (#2746) 2016-09-20 10:40:46 -07:00
Harshavardhana
0a3448c8b6 events: Change event notifiers to delete and update keys. (#2742) 
ElasticSearch and Redis are both treated like a database.
Each indexs are based on the object names uniquely indentifying
the event. Upon each delete event of the named object deletes
the index on elasticsearch and redis respectively.
 2016-09-20 02:11:17 -07:00
Harshavardhana
c4964232eb config: Fail to start for config mistakes. (#2740) 2016-09-19 15:23:49 -07:00
Harshavardhana
ef3c807b4a policies: Parser should handle Principals with various forms. (#2733) 
Handles cases for these three combinations

  - "Principal": "*",
  - "Principal": { "AWS" : "*" }
  - "Principal": { "AWS" : [ "*" ]}

Fixes #2732
 2016-09-19 13:52:28 -07:00
Harshavardhana
113b93346b lock: Make some cleanup and moving the code around. (#2718) 
This patch just avoids lot of ifs and inverts some logic.
 2016-09-19 13:14:55 -07:00
Krishna Srinivas
a955676986 Signature-V4: Dump the request with error message on signature mismatch. (#2734) 
fixes #2691
 2016-09-19 10:17:46 -07:00
Harshavardhana
725df557b5 tests: Add tests for bucket-notification-utils (#2726) 
Part - 2 final fix #2711
 2016-09-17 03:19:39 -07:00
Harshavardhana
490056eee3 tests: Add tests for bucket-notification-utils (#2723) 
Part fix - 1 for #2711
 2016-09-16 17:26:27 -07:00
Harshavardhana
797d749322 tests: Add tests for filterRuleMatch (#2722) 
Part-1 fix for #2418
 2016-09-16 16:44:44 -07:00
Harshavardhana
79888bfff7 tests: Add auth-handler. (#2721) 
Fixes #2658
 2016-09-16 15:17:49 -07:00
Anis Elleuch
010f61e91f Add more tests for event-notifier code (#2716) 2016-09-16 14:26:05 -07:00
Harshavardhana
9216981262 tests: Add test for diskCount. (#2717) 
Fixes #2312
 2016-09-16 13:44:52 -07:00
Anis Elleuch
b89a1cd482 tests: Implemented more tests for fs-v1*.go (#2686) 2016-09-16 13:06:49 -07:00
Harshavardhana
7d37dea449 tests: Add more streaming signature tests. (#2713) 
Part fix for #2621
 2016-09-16 02:45:42 -07:00
Frank
df2ef64d20 Upgrade to new dsync version incl. stale lock detection (#2708) 2016-09-16 00:30:55 -07:00
Anis Elleuch
7a549096de XL and FS use different tree walk ignored errors (#2707) 2016-09-15 13:43:40 -07:00
Harshavardhana
a1ff351f21 tests: Fix ListMultipartUploadsHandler tests. (#2705) 2016-09-15 01:44:19 -07:00
Harshavardhana
03430d0db8 tests: Add ListBucketHandler tests. (#2701) 
part-3 final fix for #2412
 2016-09-14 23:53:42 -07:00
Anis Elleuch
6f73d597e0 Fix tracing twice an error in fs Complete Multipart Upload (#2703) 2016-09-14 21:24:54 -07:00
Anis Elleuch
a84548d7ea Fix FS remove bucket regression bug (#2693) 2016-09-14 16:41:39 -07:00
Harshavardhana
1e6d67b16d server: Remove deadcode. (#2699) 2016-09-14 13:43:08 -07:00
Aditya Manthramurthy
6533927237 Lock-free rate-limit algorithm + bug-fix (#2694) 2016-09-14 11:27:37 -07:00
Harshavardhana
da9ae574df server: We should fail properly during server startup. (#2689) 
Fixes #2688
 2016-09-14 01:11:03 -07:00
Harshavardhana
ee7e70c992 tests: Add tests for ListMultipartUploads, DeleteMultipleObjects. (#2649) 
Additionally adds PostPolicyHandler tests.
 2016-09-13 21:22:31 -07:00
Krishna Srinivas
54a9f59a13 Init: Print SQS ARNs after globalEventNotifier is inited. (#2682) 
fixes #2681
 2016-09-13 21:18:30 -07:00
Harshavardhana
e6fd664331 tests: Fix format-config tests. 2016-09-13 21:18:30 -07:00
Karthic Rao
b247ec9352 tests: refactor object-handler tests. (#2656) 
- Move the initialization to a common executor for object Layer API
  tests.d
 2016-09-13 21:18:30 -07:00
Harshavardhana
43befab8ef Change distributed server wording. 2016-09-13 21:18:30 -07:00
Harshavardhana
eae0281c64 tests: Add GetBucketLocation, HeadBucket tests. (#2644) 2016-09-13 21:18:30 -07:00
Karthic Rao
8bd78fbdfb performance: gjson parsing for readXLMeta, listParts, getObjectInfo. (#2631) 
- Using gjson for constructing xlMetaV1{} in realXLMeta.
- Test for parsing constructing xlMetaV1{} using gjson.
- Changes made since benchmarks showed 30-40% improvement in speed.
- Follow up comments in issue https://github.com/minio/minio/issues/2208
  for more details.
- gjson parsing of parts from xl.json for listParts.
- gjson parsing of statInfo from xl.json for getObjectInfo.
- Vendorizing gjson dependency.
 2016-09-13 21:18:30 -07:00
Krishnan Parthasarathi
66459a4ce0 Add unit-tests for formatting disks during initialization (#2635) 
* Add unit-tests for formatting disks during initialization

- Fixed corresponding code at places where it was deviating from the
  tabular spec.

* Added more test cases and simplified algo

... based on feedback from ``go test -coverprofile``.
 2016-09-13 21:18:30 -07:00
Harshavardhana
182109f0de xl: Heal format.json properly on fresh disks. 2016-09-13 21:18:30 -07:00
Harshavardhana
9998e9ea19 api: Response timeFormat do not need to have nano-second precision. 
Fixes an error reported by s3verify.
 2016-09-13 21:18:30 -07:00
Harshavardhana
ba2ba328da server: Fixes for various conditions 
- Fix distributed branch to be able to run FS version.
- Fix distributed branch to be able to run XL local disks.
- Ignore initialization failures of notification and bucket
  policies, the codepath should load whatever is possible.
 2016-09-13 21:18:30 -07:00
Anis Elleuch
67b8080144 Fix control lock rpc name in control lock cmd (#2627) 2016-09-13 21:18:30 -07:00
Anis Elleuch
239a34ca97 Add tests for regular and streaming v4 PutObject Handler (#2618) 2016-09-13 21:18:30 -07:00
Krishna Srinivas
81d8263ae2 binary-update: Do not fetch update info for minio binary compiled from source. 
fixes #2494
 2016-09-13 21:18:30 -07:00
Krishna Srinivas
b4e4846e9f PutObject: object layer now returns ObjectInfo instead of md5sum to avoid extra GetObjectInfo call. (#2599) 
From the S3 layer after PutObject we were calling GetObjectInfo for bucket notification. This can
be avoided if PutObjectInfo returns ObjectInfo.

fixes #2567
 2016-09-13 21:18:30 -07:00
Krishna Srinivas
92e49eab5a FS/Multipart: Do not rename append files to another tmp file as the append files are already in tmp location. (#2612) 2016-09-13 21:18:30 -07:00
Harshavardhana
c4a7b950a0 fs: Fix asynchronous multipart bug. 
Construct part path properly.
 2016-09-13 21:18:30 -07:00
Karthic Rao
1ce339abeb Fixing ineffssaign errors (#2608) 2016-09-13 21:18:30 -07:00
Anis Elleuch
3e284162d7 Add global flags to all commands and subcommands (#2605) 2016-09-13 21:18:30 -07:00
Anis Elleuch
ff99392102 Enhance minio server help template (#2603) 2016-09-13 21:18:30 -07:00
Krishna Srinivas
9358ee011b logging: Print stack trace in case of errors. 
fixes #1827
 2016-09-13 21:18:30 -07:00
Harshavardhana
37cbcae6ba xl: Remove an unecessary lock with isBucketExist() (#2593) 
Fixes #2566
 2016-09-13 21:18:30 -07:00
Harshavardhana
ae64b7fac8 XL: Handle object layer initialization properly. 
Initialization when disk was down the network disk
reported an incorrect error rather than errDiskNotFound.

This resulted in incorrect error handling during
prepInitStorage() stage.

Fixes #2577
 2016-09-13 21:18:30 -07:00
Anis Elleuch
d936ed90ae Avoid testing on system errors strings in posix (#2583) 2016-09-13 21:18:30 -07:00
Krishna Srinivas
7cc77eba45 XL/Healing: errDiskNotFound is the only pardonable error in xlShouldHeal. (#2586) 
This is so that we try to heal a file for all the "bad" cases except when the disk is down.
 2016-09-13 21:18:30 -07:00
Karthic Rao
07d232c7b4 instrumentation: instrumentation for locks. (#2584) 
- Instrumentation for locks.
- Detailed test coverage.
- Adding RPC control handler to fetch lock instrumentation.
- RPC control handlers suite tests with a test RPC server.
 2016-09-13 21:18:30 -07:00
Krishnan Parthasarathi
de67bca211 Move formatting of disks out of object layer initialization (#2572) 2016-09-13 21:18:30 -07:00
Harshavardhana
bca1385683 lock: Fix support single node XL locking as well. 2016-09-13 21:18:30 -07:00
Harshavardhana
cbe87cb2ed Fix fd-leak in rpcClient close it pro-actively. 2016-09-13 21:18:30 -07:00
Anis Elleuch
0513b3ed07 Add Heal Disk Metadata RPC API + tests (#2556) 2016-09-13 21:18:30 -07:00
Frank
7f92165c79 Single use DRWMutex, RPCClient refactor and added missing cases to lock-rpc-server (#2560) 
This PR contains various fixes for the distributed release:
- Use DRWMutex in namespace-lock only for a single Lock()/RLock() call in conformance to server-side rw-locking as implemented in minio/dsync
- Implement missing cases in lock-rpc-server to catch Unlock() for active read locks and RUnlock() for an active write lock
- Refactor RPCClient to release local mutex while making actual RPC.Call()
 2016-09-13 21:18:30 -07:00
Harshavardhana
780ccc26f7 server: Validate server arguments for duplicates. (#2554) 
- Validates invalid format inputs.
- Validates duplicate entries.
- Validates sufficient amount of disks.

Partially fixes #2502
 2016-09-13 21:18:30 -07:00
Harshavardhana
339425fd52 server: Fetch StorageInfo() from underlying disks transparently. (#2549) 
Fixes #2511
 2016-09-13 21:18:30 -07:00
Harshavardhana
fa6e9540a8 server: We shouldn't exit the server in lazy init. (#2548) 
Avoid fatalIf instead these are non-critical errors,
continue running the server.

 - initializing bucket notifications
 - initializing bucket policies.
 - migrating bucket policies failure.

Fixes #2547
 2016-09-13 21:18:30 -07:00
Harshavardhana
9605fde04d controller/auth: Implement JWT based authorization for controller. (#2544) 
Fixes #2474
 2016-09-13 21:18:30 -07:00
Anis Elleuch
200d327737 List only objects that need healing (#2546) 2016-09-13 21:18:30 -07:00
Harshavardhana
e1b0985b5b rpc: Refactor authentication and login changes. (#2543) 
- Cache login requests.
- Converge validating distributed setup.
 2016-09-13 21:18:30 -07:00
Krishnan Parthasarathi
c8dfc4cda4 auth-rpc: Reset token on disconnect (#2542) 2016-09-13 21:18:30 -07:00
Bala FA
7922a54c9a rpc-client: remove unwanted nil check of rpcClient. (#2538) 2016-09-13 21:18:30 -07:00
Krishnan Parthasarathi
bda6bcd5be Layered rpc-client implementation (#2512) 2016-09-13 21:18:30 -07:00
Harshavardhana
7e3e24b394 rpc: client login should ignore server versions. 2016-09-13 21:18:30 -07:00
Harshavardhana
bb0466f4ce control: Fix controller CLI handling with distributed server object layer. 
Object layer initialization is done lazily fix it.
 2016-09-13 21:18:30 -07:00
Harshavardhana
8797952409 server: Add server command line for running in distributed mode 2016-09-13 21:18:30 -07:00
Krishnan Parthasarathi
a4691611a7 Move initGracefulShutdown before objectLayer initialization (#2468) 2016-09-13 21:18:30 -07:00
awwalker
7c7eb1475d splitNetPath: Add support for windows paths including volumeNames e.g ip:C:\network\path 2016-09-13 21:18:30 -07:00
Harshavardhana
0bce3d6d63 Fix web handler initialize with distributed lazy init. 2016-09-13 21:18:30 -07:00
Krishnan Parthasarathi
804d91ef61 storage/rpc-client: Reconnect on network disconnect (#2436) 2016-09-13 21:18:30 -07:00
Harshavardhana
43098df9d8 rpc: Re-factor ReadFile behavior client <--> server. 
Current code did not marshal/unmarshal buffers properly from
the server, The reason being buffers have to be allocated and
sent properly back to client to be consumable.
 2016-09-13 21:18:30 -07:00
Harshavardhana
6908a0dcd4 Extract rpc server wrapped errors and translate to storage error. 2016-09-13 21:18:30 -07:00
Harshavardhana
cae5761f16 rpc/client: Add missing rpcTokens for each rpc calls. 2016-09-13 21:18:30 -07:00
Harshavardhana
61af764f8a Add rpc layer authentication. 2016-09-13 21:18:30 -07:00
Harshavardhana
b4172ad3c8 Rename rpc-{client,server} storage-rpc-{client,server} 2016-09-13 21:18:30 -07:00
Harshavardhana
4917038f55 Move the ObjectAPI() resource to be beginning of each handlers. 
We should return back proper errors so that the clients can
retry until server has been initialized.
 2016-09-13 21:18:30 -07:00
Krishnan Parthasarathi
b0f3f94163 unify single-node and distributed namespace locking (#2401) 2016-09-13 21:18:30 -07:00
Krishnan Parthasarathi
b7c169d71d object layer initialization using distributed locking (#2397) 
* vendorized latest minio/dsync

* wip - object layer initialization using distributed locking
 2016-09-13 21:18:30 -07:00
Frank
3939c75345 Added distributed RWMutex (#2369) 2016-09-13 21:18:30 -07:00
Krishnan Parthasarathi
e55926e8cf distribute: Make server work with multiple remote disks 
This change initializes rpc servers associated with disks that are
local. It makes object layer initialization on demand, namely on the
first request to the object layer.

Also adds lock RPC service vendorized minio/dsync
 2016-09-13 21:18:30 -07:00
Anis Elleuch
f82f535509 Fix possible race in shutdown callbacks process and simplify shuttting down profiler (#2684) 2016-09-13 18:43:45 -07:00
Anis Elleuch
51e337228e Avoid hiding disk errors in some cases in FS Shutdown (#2668) 2016-09-13 11:01:10 -07:00
Kevin Qiu
241c56e6d7 Use Set instead of Add in the event that the request already contains the content-length (#2683) 2016-09-13 10:59:40 -07:00
Aditya Manthramurthy
895471afa1 Fixes #2678 (#2679) 
Refactor `ratelimit.acquire()` to properly enforce the *globalMaxConn*
limit.
 2016-09-12 15:53:54 -07:00
Harshavardhana
85e2d886bc bucketPolicy: Trim incoming paths properly for policy verification. 2016-09-11 10:36:54 -07:00
Anis Elleuch
8acf4d112a Return Owner information in ListObjectV2 only when fetch-owner is specified (#2654) 2016-09-10 10:44:38 -07:00
Anis Elleuch
e6abfb3b67 More tests for format-config for a better coverage (#2655) 2016-09-10 09:54:22 -07:00
Harshavardhana
040af08473 server: Startup message now prints configured ARNs. (#2653) 
Fixes #2628
 2016-09-10 02:23:28 -07:00
Harshavardhana
e3de237eb8 Move config to v7 to fix previous v6 bug on null Notification entries. (#2650) 
Thanks to @koolhead17 for reporting this.
 2016-09-10 00:51:25 -07:00
Anis Elleuch
11785529fc Add Request Not Ready error when presigned date request is not valid (#2646) 2016-09-10 00:38:07 -07:00
Anis Elleuch
32201a18ab Forbid requests generated at least 15 min in the past or in the future (#2648) 2016-09-09 15:33:35 -07:00
Anis Elleuch
421cccb1d7 Add naughty disk to simulate disk errors (#2645) 2016-09-09 12:53:09 -07:00
Krishna Srinivas
400e9309f1 Signature-V4: Include content-length for signature calculation. (#2643) 
This is to be compatible with clients which includes content-length for signature calculation
(and hence deviating from AWS Signature-v4 spec)
 2016-09-09 09:38:07 -07:00
Krishna Srinivas
a4afb312d4 S3-layer: Multiple-delete should return success for non-existent objects. (#2633) 
Fixes #2630
 2016-09-07 11:49:12 -07:00
Harshavardhana
afe874f15a api: CopyObject pipe should be closed pro-actively upon error. (#2625) 
Fixes a deadlock reproduced while running s3verify during
RemoveObject().  Previously held lock by GetObject() inside
the go-routine was never relenquished.
 2016-09-06 02:23:32 -07:00
Harshavardhana
b291dbe9c5 s3: Fix compliance bugs reported by s3verify. (#2624) 
- Fixes couple of error strings reported are mismatching.
- Fixes a error HTTP status which was wrong fixed.
- Remove usage of an deprecated PostResponse, au contraire
  to their documentation there is no response body in
  PostPolicy.
 2016-09-06 02:18:02 -07:00
Harshavardhana
bc8f34bfe7 server/mux: Remove unused waitgroup from listenerMux. (#2614) 
Change struct names to be more meaningful.
 2016-09-02 19:07:42 -07:00
Harshavardhana
2dc7ecc59b notifications: Fix bucket notifications for DeleteMultipleObjects. (#2609) 
Now reports bucket notifications for DeleteMultipleObjects API.
Also deletes multiple objects in parallel.
 2016-09-02 01:59:08 -07:00
Harshavardhana
7398d737b5 profiler: Fix it properly and generate/save profiles even failure situations. (#2607) 
Fixes #2594
 2016-09-01 20:13:11 -07:00
Harshavardhana
511a8cbe04 profiler: Fix how we generate profiles. (#2596) 
Fixes #2594
 2016-09-01 03:50:31 -07:00
Anis Elleuch
e79d2381fc Fix rare 'go test -race' failure in ListenServe{Plain,TLS} (#2588) 2016-08-31 16:07:44 -07:00
Krishna Srinivas
a3b4199e9b FS/Multipart: Allow for parallel uploads of parts with same part number. (#2587) 
CBL client does not close connection when the backup process is stopped, this causes
read() on the stream on the server side to block and hence the lock held on the part
is not released. When the backup process is restarted, we again try to lock on the
part and this will block. Using a unique tmp name and not locking it fixes the problem.
 2016-08-31 16:01:32 -07:00
Krishna Srinivas
3aa0574c6b FS/multipart: Append the parts to a file as and when the parts arrive. (#2513) 2016-08-31 13:42:57 -07:00
Harshavardhana
5716f1c199 Fix incorrect error strings. (#2580) 2016-08-30 13:47:09 -07:00
Harshavardhana
2020afd200 server: http.Server do not add deadlines causes issues. (#2581) 
Adding deadlines is a no go since Golang doesn't back off
the timers if there is an active i/o in progress.

It is meant to be for applications to handle this themselves
and manually progress the deadlines.

Fixes #2561
 2016-08-30 12:52:19 -07:00
Bala FA
7431acb2c4 web: add handlers for set/get bucket policy. (#2486) 
Refer #1858
 2016-08-30 10:04:50 -07:00
Harshavardhana
2f9975c76c notifications: GetObjectInfo() later if bucket notifications are set. (#2575) 2016-08-30 03:42:37 -07:00
Harshavardhana
7270ca4157 pkg/wildcard: Simplify the wildcard logic further. (#2555) 2016-08-27 00:27:17 -07:00
Brendan Ashworth
b67c8970c9 bucket-handlers: fix missing return statement (#2553) 
If the location was invalid, it would write an error response but then
continue to attempt to make the bucket. Whether or not it would succeed,
it would attempt to call response.WriteHeaders twice in a row, which
would cause a message to be logged to the server console (bad).

Here is the relevant Go code:
c80e0d374b/src/net/http/server.go (L878-L881)
 2016-08-25 20:00:47 -07:00
Mohit Agarwal
418921de89 minor cleanup 
- Reused contains() from utils.go at a couple of places
- Cleanup in return statements and boolean checks
 2016-08-24 22:54:34 -07:00
Harshavardhana
ec4260d260 api: BucketNotification should disallow duplicate notification. (#2539) 
Added checks to look for duplicated notification configs.

Fixes #2472
 2016-08-23 18:42:30 -07:00
Harshavardhana
c39d3db7a0 server/tls: allocate tls.Config{} properly. (#2537) 
There is a golang bug which crashes the server, allocate tls.Config properly to avoid
this problem.

```
2016/08/22 20:21:03 http: panic serving 127.0.0.1:40744: runtime error: invalid memory address or nil pointer dereference
goroutine 38 [running]:
net/http.(*conn).serve.func1(0xc820526680)
    /home/harsha/.gimme/versions/go1.6.2.linux.amd64/src/net/http/server.go:1389 +0xc1
panic(0xbefa80, 0xc820010140)
    /home/harsha/.gimme/versions/go1.6.2.linux.amd64/src/runtime/panic.go:443 +0x4e9
crypto/tls.(*Conn).serverHandshake(0xc820368300, 0x0, 0x0)
    /home/harsha/.gimme/versions/go1.6.2.linux.amd64/src/crypto/tls/handshake_server.go:43 +0x4d6
crypto/tls.(*Conn).Handshake(0xc820368300, 0x0, 0x0)
    /home/harsha/.gimme/versions/go1.6.2.linux.amd64/src/crypto/tls/conn.go:1035 +0x169
net/http.(*conn).serve(0xc820526680)
    /home/harsha/.gimme/versions/go1.6.2.linux.amd64/src/net/http/server.go:1405 +0x382
created by net/http.(*Server).Serve
    /home/harsha/.gimme/versions/go1.6.2.linux.amd64/src/net/http/server.go:2137 +0x44e
```

Fixes #2536
 2016-08-23 00:20:15 -07:00
Harshavardhana
2da0cfc904 signature: Use the client sent region if region is properly validated. (#2534) 
A properly validated region string from the client should be once
validated properly.

Fixes #2521
 2016-08-22 19:43:36 -07:00
Harshavardhana
9dd0e3dc44 config: Initialize notify for fresh configs. (#2531) 2016-08-22 16:04:04 -07:00
Krishna Srinivas
45c928e2f5 boot: checkPortAvailability() should fail only for EADDRINUSE error and ignore other errors. (#2527) 
fixes #2510
 2016-08-22 10:20:01 -07:00
Anis Elleuch
07506358ff Refactor Heal RPC and add Shutdown RPC (#2488) 2016-08-21 12:06:53 -07:00
Harshavardhana
975eb31973 api/bucketPolicy: Use minio-go/pkg/set and fix bucket policy regression. (#2506) 
Current master has a regression 'mc policy <policy-type> alias/bucket/prefix'
does not work anymore, due to the way new minio-go changes do json marshalling.
This led to a regression on server side when a ``prefix`` is provided
policy is rejected as malformed from th server which is not the case with
AWS S3.

This patch uses the new ``minio-go/pkg/set`` package to address the
unmarshalling problems.

Fixes #2503
 2016-08-20 03:16:38 -07:00
Jesse Lucas
f2fd8b0265 Adding test coverage for server-mux.go by creating TestListenAndServeTLS(). (#2493) 
Editing config.go to add lock on customConfigPath to avoid race condition
when setGlobalConfigPath() and getConfigPath() are called concurrently.
 2016-08-19 01:29:50 -07:00
Jesse Lucas
a8052889fe server-mux_test.go updating test to fix race condition with TestListenAndServe (#2485) 2016-08-18 19:45:03 -07:00
Harshavardhana
bccf549463 server: Move all the top level files into cmd folder. (#2490) 
This change brings a change which was done for the 'mc'
package to allow for clean repo and have a cleaner
github drop in experience.
 2016-08-18 16:23:42 -07:00
Harshavardhana
aabfd541e1 Merge cmd/donut into minio cmd, deprecate controller RPC request 2015-07-31 12:57:15 -07:00
Harshavardhana
f15375426a minor: Add commands into donut template 2015-07-28 19:40:02 -07:00
Harshavardhana
0eefbdef0c use new app.ExtraInfo inside minio and donut commands properly 2015-07-24 23:55:18 -07:00
Harshavardhana
63c9cf0c4b Move from Minimalist Object Storage to Minio Cloud Storage 2015-07-24 17:51:40 -07:00
Harshavardhana
4498662c16 Remove scsi non portable code, instead "donut make" implements functionality to instantiate a donut 2015-07-15 11:55:57 -07:00
Harshavardhana
1e80925ca7 Remove dependency on minio/cli for make.go fixes #738 2015-07-13 11:39:28 -07:00
Harshavardhana
f360ee0ab5 Add mkdonut examples 2015-07-12 21:37:57 -07:00
Harshavardhana
55e4d0c6a5 mkdonut now creates a donut processing cli args 2015-07-12 21:21:31 -07:00
Harshavardhana
535bcc3eac Add mkdonut command 2015-07-12 19:16:36 -07:00
Harshavardhana
432275e966 Full restructure in accordance with 
- pkg/{subsystem}/{package} style
  - modify Makefile to reflect the new style,
    consolidate various entries
  - add a dummy ``main.go`` at top level
 2015-01-14 11:29:04 -08:00
Harshavardhana
462808b87a Disable sha1,sha256,sha512 avx,avx2,sse3 crypto implementations. 
Re-implement them later, once stable
 2015-01-06 17:02:25 -08:00
Frederick F. Kautz IV
992dea1bf3 Adding index command 2015-01-06 22:26:09 +13:00
Harshavardhana
fff6b589cf Rename split-file --> split 2015-01-03 18:03:21 -08:00
Frederick F. Kautz IV
f5010e1ee2 Adding initial split-file command 2015-01-04 14:47:37 +13:00
Harshavardhana
0212079cd5 Remove template files, make them const strings instead 2015-01-01 17:31:51 -08:00
Harshavardhana
0e4a26e3b4 Add Diskattrmap, Scsiattrmap for probed scsi devices 
Additional changes

- Use ``iota`` for constants
- Remove unncessary C header files
- ``new-cmd`` now depends on codegangsta cli
 2014-12-30 02:37:47 -08:00
Harshavardhana
6b36b5c551 A full restructure 2014-12-29 21:22:26 -08:00
Harshavardhana
13650e088c Implement md5c function, slower than Golang's implementation 
keeping it in repo to make further improvements and also rename

        minio-hash ---> crypto
 2014-12-21 03:50:33 -08:00
Frederick F. Kautz IV
2278df9910 Adding minio-hash with streaming crypto hashes 2014-12-21 13:04:41 +13:00
Harshavardhana
d4c05f11c7 Update documentation 2014-12-20 09:09:35 -08:00
Harshavardhana
b0e986c82c Rename appname to commandname for consistency 2014-12-19 00:16:41 -08:00
Harshavardhana
5268a62166 Add minio-cli stub generator 2014-12-19 00:00:14 -08:00
Harshavardhana
65e2031cfc Implement add, commit - instead of put, encoded 2014-12-15 21:55:54 -08:00
Harshavardhana
98119707fb Move wrappers at pkgs/minio back to cmd/minio - as their usage is limited 2014-12-15 20:09:50 -08:00
Harshavardhana
3ec5c2fb8f Implement commands 
Commands implemented

   $ minio put <objectname> <file>
   $ minio encode <objectname>
   $ minio get <objectname>
   $ minio list

TODO :

   $ minio init
   $ minio verify
    ...
    ...
 2014-12-15 14:59:15 -08:00
Frederick F. Kautz IV
5efc0d54f8 storage.ObjectStorage List() is now List(objectPath string) 2014-12-15 19:30:08 +13:00
Frederick F. Kautz IV
05baa6d9a2 Removing fsstorage 2014-12-15 19:15:22 +13:00
Harshavardhana
fc50291cac fsStorage should use appendStorage with offsets 
- also takes in 'blockSize' cli option.
 - robustness fixes
 2014-12-12 17:18:16 -08:00
Harshavardhana
0a0e1111cd Add list() object support for erasure and append storage drivers 
- Reply back objects with their protectionlevel and md5sum
  - // TODO hash value
  - Calculate md5sum after "storeBlocks()", to make sure data is committed
 2014-12-12 02:52:37 -08:00
Harshavardhana
f71b323e32 Provide new cli options for 'put' command 
- --protection-level
  - --block-size
 2014-12-11 22:56:12 -08:00
Harshavardhana
0bd2b8516e Build fix, fixing without a PR 2014-12-11 20:44:06 -08:00
Harshavardhana
dddc555491 Use objectName with basename for the filepath 2014-12-11 15:35:25 -08:00
Frederick F. Kautz IV
568ca4e1bc erasure encoded demo 2014-12-11 00:32:32 -08:00
Frederick F. Kautz IV
a1040ae657 erasure-demo put now takes 1 or 2 arguments 2014-12-10 20:59:49 -08:00
Frederick F. Kautz IV
19da9760b3 ObjectStorage refactor to take io.Reader instead of []byte 2014-12-10 20:57:59 -08:00
Frederick F. Kautz IV
cae9b288b0 Adding List(path) to object storage definition and accompanying definition to fs 2014-12-10 19:15:03 -08:00
Harshavardhana
9dd3dd8b53 Merge pull request #100 from harshavardhana/pr_out_add_crc32c_checksum_support_for_erasure_chunks 
Add crc32c checksum support for erasure chunks
 2014-12-10 11:53:23 -08:00
Harshavardhana
01c0e45b5c Add crc32c checksum support for erasure chunks 
Now upon conditions

 - if checksum '.cksum' file is missing
 - or corrupted chunk (mismatching chunk)
 - or missing chunk entirely

Decoding works consistently
 2014-12-10 00:58:15 -08:00
Harshavardhana
f6a5b61b00 Make get/put more robust 
- init folder if not present already in 'get()' call
  - put.go was referencing wrong args - fix it
  - if no OBJECTNAME is specified during 'get' - grab a
    list of files under "~/.minio"
  - Provide new API GetList() for ObjectStorage struct
 2014-12-09 03:32:31 -08:00
Frederick F. Kautz IV
d1ebf935da Adding extremely simple fs based get and put in erasure-demo 2014-12-08 20:20:25 -08:00
Harshavardhana
7314b5e37d Fixes #71 - some crazy races inside erasure and simplify the code 
- This change also brings in changing 'unsigned char' to 'uint8_t'
for brevity
 2014-12-07 01:33:49 -08:00
Harshavardhana
9a5003dd25 Provide JoinFilesWithPrefix case with Gochannel 2014-12-05 16:02:46 -08:00
Frederick F. Kautz IV
46b08681a4 Adding chunking by block to erasure-demo via --block-size parameter 2014-12-01 21:06:36 -08:00
Frederick F. Kautz IV
323a7b8492 Refactoring input parsing for erasure-demo 2014-11-30 22:18:23 -08:00
Frederick F. Kautz IV
6382ee161e Renaming minio-demo to erasure-demo 2014-11-30 13:54:31 -08:00
Frederick F. Kautz IV
30687a8c66 Increasing parity from 5 to 6 2014-11-29 22:59:44 -08:00
Frederick F. Kautz IV
3c09029049 Merging encode and decode to single minio-demo binary 2014-11-29 22:48:37 -08:00
Harshavardhana
5074542636 Fix function name change 2014-11-29 17:29:38 -08:00
Frederick F. Kautz IV
becb165770 Simplifying code 2014-11-26 23:33:36 -08:00
Harshavardhana
3cefaf5ad8 Restructure directory, add 'pkgs' folder now 2014-11-24 15:10:25 -08:00
Harshavardhana
6592ef3bd2 Make use of LRU and higher order erasure functions 2014-11-24 07:16:54 -08:00
Harshavardhana
ee7ff7c436 Decode on chunks missing should build and continue 2014-11-24 06:58:27 -08:00
Frederick F. Kautz IV
5a3eb47583 Adding minio-decode 2014-11-23 19:32:01 -08:00
Frederick F. Kautz IV
cec0a0ef68 Adding k and m as optional parameters for encoding in minio-encode 2014-11-23 15:49:26 -08:00
Frederick F. Kautz IV
6faa55e25a Adding input and output file parameters to minio-encode. 2014-11-22 19:25:46 -08:00
Harshavardhana
3b9495d91a Vandermonde --> Cauchy 2014-11-22 18:52:07 -08:00
Frederick F. Kautz IV
a15e935a33 Adding minio-encode binary 2014-11-22 18:39:59 -08:00
Frederick F. Kautz IV
9c18624e58 Removing binaries 2014-11-22 15:16:38 -08:00
Frederick F. Kautz IV
f8f9f65652 Moving minio command to cmd/minio 2014-11-14 18:20:40 -07:00