MyFavMirrors/minio
1
0
Fork 0
mirror of https://github.com/minio/minio.git synced 2024-12-26 07:05:55 -05:00
Code Issues Packages Projects Releases Wiki Activity
3,727 Commits 4 Branches 506 Tags 189 MiB
d67f47927c
Commit Graph

453 Commits

Author SHA1 Message Date
Krishna Srinivas
d3aaf50a40 posix: Split on ":" in path d:\export makes minio use wrong disk. (#3027) 
As the host/path split happens at a higher layer now, split at posix is not needed.
fixes part of #2987
 2016-10-20 23:39:33 -07:00
Krishna Srinivas
32c3a558e9 distributed-XL: Support to run one minio process per export even on the same machine. (#2999) 
fixes #2983
 2016-10-20 18:31:02 -07:00
Anis Elleuch
41f9ab1c69 Translate storage access denied error to S3 Access Denied response (#3015) 2016-10-20 16:09:55 -07:00
Aditya Manthramurthy
8876e0a80a Delete bucket listener config file from disk (#3016) 2016-10-20 16:09:19 -07:00
Anis Elleuch
c21ac80268 Validate access/secret keys found in the config file and enhance invalid keys messages (#3017) 2016-10-20 16:07:24 -07:00
Frank
0e2cd1a64d Added clear subcommand for control lock (#3013) 
Added clear subcommand for control lock with following options:

```
  3. Clear lock named 'bucket/object' (exact match).
    $ minio control lock clear http://localhost:9000/bucket/object

  4. Clear all locks with names that start with 'bucket/prefix' (wildcard match).
    $ minio control lock --recursive clear http://localhost:9000/bucket/prefix

  5. Clear all locks older than 10minutes.
    $ minio control lock --older-than=10m clear http://localhost:9000/

  6. Clear all locks with names that start with 'bucket/a' and that are older than 1hour.
    $ minio control lock --recursive --older-than=1h clear http://localhost:9000/bucket/a
```
 2016-10-20 13:15:28 -07:00
Aditya Manthramurthy
6274727b71 Pick up server address from --address option (#3002) (#3008) 
This makes sure that when SSL is enabled (for FS/single node mode),
the server address is picked up from the --address option (that needs
to include the hostname for SSL verification, and has to be input
appropriately by user), instead of just using ":<port>".
 2016-10-20 11:39:10 -07:00
Harshavardhana
95567c68bf posix: Do not print errors in expected errors. (#3012) 
Fixes #3011
 2016-10-20 09:26:18 -07:00
Anis Elleuch
c189337b6e rpc: Support SNI in TLS certificates (#3009) 2016-10-20 07:43:31 -07:00
Krishnan Parthasarathi
6fc81dc162 Delete temp object/part when PutObject{,Part} fails (#3004) 2016-10-19 22:52:03 -07:00
Krishnan Parthasarathi
7d50361ca9 Move housekeeping before object layer initialization (#3001) 
In a distributed setup that the server should not perform any operation
on the storage layer after it is exported via RPC. e.g, cleaning up of
temporary directories under .minio.sys/tmp may interfere with ongoing
PUT objects being served by the distributed setup.
 2016-10-19 19:59:48 -07:00
Frank
19c51f3f3c Added ForceUnlock to namespace-lock (#2990) 2016-10-19 09:27:36 -07:00
Aditya Manthramurthy
c3bbadacbf Improve Peer RPC error handling (Fixes #2992) (#2995) 
* Check for RPC connection shutdown and try again just once.

* Refactor SendRPC to use sync.WaitGroup
 2016-10-18 21:26:58 -07:00
Anis Elleuch
2208992e6a More informative message when erasure fails to read a part of an object (#2989) 2016-10-18 13:09:26 -07:00
Anis Elleuch
bbba8e432a Add ssl support to s3/web peers connections (#2988) 2016-10-18 11:46:33 -07:00
Harshavardhana
39331b6b4e xl: GetCheckSumInfo() shouldn't fail if hash not available. (#2984) 
In a multipart upload scenario disks going down and coming backup
can lead to certain parts missing on the disk/server which was
going down. This is a valid case since these blocks can be
missing and should be healed through heal operation. But we are
not supposed to fail prematurely since we have enough data on
the other disks as well within read-quorum.

This fix relaxes previous assumption, fixes a major corruption
issue reproduced by @vadmeste.

Fixes #2976
 2016-10-18 11:13:25 -07:00
Mike Ralphson
6e748cb1cf Report when invalid bucket names are skipped in FS backend. (#2947) 2016-10-18 01:42:46 -07:00
Anis Elleuch
2005d656e6 Properly load creds from env and save them when server cmd is executed (#2970) 2016-10-17 23:14:41 -07:00
Aditya Manthramurthy
0f26ec8095 Propagate creds change to cluster (Fixes #2855) (#2929) 2016-10-17 20:18:08 -07:00
Harshavardhana
8d2347bc7b storage: DeleteFile should return errFileNotFound for ENOENT. (#2978) 2016-10-17 16:38:46 -07:00
Aditya Manthramurthy
0ff359ca0e Fix early init. problem for notifications (Fixes #2972) (#2977) 2016-10-17 16:38:29 -07:00
Harshavardhana
f8e13fb00e server: Startup sequence should be more idempotent. (#2974) 
Fixes #2971 - honors ignore-disks option properly.
Fixes #2969 - change the net.Dial to have a timeout of 3secs.
 2016-10-17 14:31:33 -07:00
Harshavardhana
686a610fc3 api: Nanosecond precision for API responses is valid with S3. (#2957) 
Wqe need to be compatible as well fixes #2955
 2016-10-17 08:44:55 -07:00
Krishnan Parthasarathi
b89609dc2e XL: Filter out md5Sum from user defined headers (#2962) 2016-10-17 08:41:33 -07:00
Anis Elleuch
fa50312220 Avoid returning disk corrupted by servers in the middle of init all disks formats (#2964) 2016-10-17 08:39:55 -07:00
Harshavardhana
fee3f99a6e xl: heal bucket should validate if bucket exists first. (#2953) 
Fixes #2944
 2016-10-17 02:10:23 -07:00
Frank
ea406754a6 New dsync and added ForceUnlock to lock rpc server (#2956) 
* Update dsync and added ForceUnlock function
* Added test cases for ForceUnlock
 2016-10-17 01:53:29 -07:00
Aditya Manthramurthy
d02cb963d5 Fix listen-bucket (Fixes #2942) (#2949) 
Don't close socket while re-initializing notify-listeners, as the rpc
client object is shared between notify-listeners and peer clients.

Also, improves SendRPC() readability by using GetPeerClient().
 2016-10-16 20:52:10 -07:00
Anis Elleuch
334cdb5d64 XL total/free space calculation is done inside xl module (#2945) 2016-10-16 14:24:15 -07:00
Anis Elleuch
5c3639c1b7 Redirect /minio to /minio/ when requests come from browsers (#2937) 2016-10-15 06:21:51 -07:00
Krishna Srinivas
903574db90 copy-object: Do not use ETag of source as MD5 as it will not be MD5 if source was uploaded as multipart. (#2938) 
fixes #2934
 2016-10-15 06:20:55 -07:00
Anis Elleuch
f463d3ce42 Fix a crash when service shutdown is signaled and object API is not ready yet (#2939) 2016-10-15 06:20:16 -07:00
Aditya Manthramurthy
17eeec6895 Bucket policy propagation (Fixes #2930) (#2932) 
Fixes a serialisation bug - encoding/gob does not directly support
serializing `map[string]interface{}`, so we serialise to JSON and send a
byte array in the RPC call, and deserialize and update on the receiver.
 2016-10-14 22:49:51 -07:00
Karthic Rao
070d3610ff tests: V2 Signature tests for object-handlers. (#2931) 2016-10-14 20:52:46 -07:00
Harshavardhana
f22862aa28 heal: Refactor heal command. (#2901) 
- return errors for heal operation through rpc replies.
  - implement rotating wheel for healing status.

Fixes #2491
 2016-10-14 19:57:40 -07:00
Harshavardhana
2f520ed92f Remove errors package, add comments and simplify. (#2925) 2016-10-14 12:31:00 -07:00
Mateusz Gajewski
c03ce0f74a Display SSL expiry warnings (#2925) 2016-10-14 12:30:36 -07:00
Krishna Srinivas
0320a77dc0 HealBucket: create the bucket if it is missing in one of the disks. (#2924) 2016-10-14 11:12:17 -07:00
Aditya Manthramurthy
31be826f51 Fix missing error check for jsonrpc.Server.RegisterService() (#2921) 2016-10-13 17:34:10 -07:00
Karthic Rao
17e49a9ed2 signature-v2 fix. (#2918) 
- Return errors similar to V4 Sign processsing.
- Return ErrMissing fields when Auth Header fields are missing.
- Return InvalidAccessID when accessID doesn't match.

* tests: Adding V2 signature tests for bucket handler API's.
 2016-10-13 09:25:56 -07:00
Aditya Manthramurthy
0aabc1d8d9 Use Peer RPC to propagate bucket policy changes (#2891) 2016-10-13 09:19:04 -07:00
Harshavardhana
55f6828750 Do not print update message unless there is an update. (#2919) 2016-10-13 09:17:08 -07:00
Aditya Manthramurthy
6303f26330 Protect map from concurrent access (Fixes #2915) (#2916) 
Protects the Peers RPC clients map from concurrent access to fix a data race condition.
 2016-10-13 01:33:50 -07:00
Krishnan Parthasarathi
b59bac670a Handle err returned by rpc.Server.RegisterName (#2910) 2016-10-12 23:13:24 -07:00
Anis Elleuch
84acc820c7 Fix free drive space calculation in XL mode (#2917) 2016-10-12 20:22:15 -07:00
Harshavardhana
fdaa129a5b Fix dockerfile container image. (#2892) 2016-10-12 18:09:08 -07:00
Anis Elleuch
df59967f59 Avoid checking date header of web requests by properly applying generic handlers (#2914) 2016-10-12 12:58:36 -07:00
Mateusz Gajewski
73982c8cb6 Listen bucket notification for multiple prefixes/suffixes (#2911) 
* Listen bucket notification for multiple prefixes/suffixes

* After review fixes
 2016-10-12 11:02:15 -07:00
Aditya Manthramurthy
6199aa0707 Peer RPCs for bucket notifications (#2877) 
* Implements a Peer RPC router that sends info to all Minio servers in the cluster.
* Bucket notifications are propagated to all nodes via this RPC router.
* Bucket listener configuration is persisted to separate object layer
  file (`listener.json`) and peer RPCs are used to communicate changes
  throughout the cluster.
* When events are generated, RPC calls to send them to other servers
  where bucket listeners may be connected is implemented.
* Some bucket notification tests are now disabled as they cannot work in
  the new design.
* Minor fix in `funcFromPC` to use `path.Join`
 2016-10-12 01:03:50 -07:00
Krishnan Parthasarathi
a5921b5743 Use same timestamp for all chunks in chunked signature (#2908) 2016-10-11 23:46:51 -07:00
Karthic Rao
ff91ecb177 tests: Adding unknown signature type test for API handlers. (#2905) 2016-10-11 20:38:10 -07:00
Frank
a6357502c1 Correct typo in error string (#2902) 2016-10-11 08:56:02 -07:00
Harshavardhana
fa8ea41cd9 lock/instrumentation: Cleanup and print in user friendly form. (#2807) 2016-10-11 00:50:27 -07:00
Karthic Rao
3ac6790ca2 tests: Add Object Layer nil test for bucket-handler API's (#2899) 2016-10-11 00:00:02 -07:00
Krishna Srinivas
268b96058f ns-lock: lock namespace during FS object operations. (#2896) 2016-10-10 10:20:04 -07:00
Frank
0d031c432b Fix typo in serverVersion (#2894) 2016-10-10 10:11:56 -07:00
Frank
6e8f3224c5 Test coverage for lock rpc server (#2893) 
* Add test coverage for removeEntry and removeEntryIfExists
* Initial test framework for Lock/Unlock functionality
* Add clarification comments
* Add test coverage code for RLock() and RUnlock()
* Add test coverage for Expired() function
* Have all lock-rpc-server test functions start with the same prefix
* Properly initialize JWT security token
 2016-10-10 10:11:29 -07:00
Karthic Rao
9c53e9f4c3 tests: Enhance coverage for bucket policy handlers. (#2895) 2016-10-10 09:29:56 -07:00
Krishnan Parthasarathi
2d5e988a6d Refactor streaming signatureV4 w/ state machine (#2862) 
* Refactor streaming signatureV4 w/ state machine

- Used state machine to make transitions between reading chunk header,
  chunk data and trailer explicit.

* debug: add print/panic statements to gather more info on CI failure

* Persist lastChunk status between Read() on ChunkReader

... remove panic() which was added as interim aid for debugging.

* Add unit-tests to cover v4 streaming signature
 2016-10-10 01:42:32 -07:00
Harshavardhana
3cfb23750a control: Implement service command 'stop,restart,status'. (#2883) 
- stop - stops all the servers.
- restart - restart all the servers.
- status - prints status of storage info about the cluster.
 2016-10-09 23:03:10 -07:00
Anis Elleuch
57f75b1d9b Ignore copy conditions when ETag is not available (#2888) 2016-10-09 16:21:42 -07:00
Karthic Rao
e213172431 tests: Missing anonymous tests for bucket-handlers. (#2885) 2016-10-09 09:21:37 -07:00
Krishna Srinivas
f5f007e183 Test: Add test case for xl.HealObject() (#2884) 
fixes #2842
 2016-10-08 17:08:17 -07:00
Karthic Rao
09463265ce tests: Adding anonymous requests tests for bucket policy handlers. (#2882) 2016-10-08 01:04:26 -07:00
Karthic Rao
8f4cf2a7d0 tests: anonymous/unsigned tests for object handler API's . (#2881) 2016-10-07 23:28:50 -07:00
Karthic Rao
30183c4a9a tests: cleanup and unsigned request test. (#2880) 
- Cleaning up of ListMultipartUpload API test for improving readability,
  code maintainance and extensibility.
- Moving ListMultipartUploads to Go 1.7 sub tests.
- Using the new Anonymous request helper function for
  ListMultipartUploads.
 2016-10-07 20:16:57 -07:00
Karthic Rao
d1df5e0ae1 tests: Add helper function for API handler anonymous request tests. (#2876) 
- Add helper function for API handler anonymous request tests.
- Add PutObject Part Anonymous request case using the new helper
  function to validate its functionality.
 2016-10-07 11:16:11 -07:00
Harshavardhana
f1bc9343a1 prep: Initialization should wait instead of exit the servers. (#2872) 
- Servers do not exit for invalid credentials instead they print and wait.
- Servers do not exit for version mismatch instead they print and wait.
- Servers do not exit for time differences between nodes they print and wait.
 2016-10-07 11:15:55 -07:00
Karthic Rao
97f4989945 tests: cleaning up. (#2875) 
- Clean up PutObjectPart and ListObjectPart API handler tests.
- Add more comments, make the tests more readable.
- Add verification for HTTP response status code.
- Initialize the test using object Layer.
- Move to Go 1.7 sub tests.
 2016-10-07 08:02:37 -07:00
Harshavardhana
1e5e213d24 auth: Make sure we initialize or change config before RPC requests. (#2867) 2016-10-06 13:35:56 -07:00
Karthic Rao
a8105ec068 - Test utility function for easy asserting of cases wherein objectLayer (#2865) 
is `nil` in API handlers.
- Remove the existing tests for the `nil` check and use the new method
  to test for object layer being `nil`.
 2016-10-06 13:34:33 -07:00
Krishna Srinivas
bb9be02228 minio-browser: do not redirect to /minio if MINIO_BROWSER=off (#2863) 
fixes #2837
 2016-10-06 08:30:32 -07:00
Harshavardhana
64f37bbf5b rpc: Add RPC client tests. (#2858) 2016-10-06 02:30:54 -07:00
Karthic Rao
0fc96fa25c Refactor bucket policy handler test to use API test initializer. (#2859) 2016-10-06 02:02:42 -07:00
Karthic Rao
2d8c6f8288 unit test for bucketPolicyConditionMatch function. (#2857) 2016-10-06 00:23:46 -07:00
Harshavardhana
b94211bd66 api: ListObjectsV1 compliance with AWS S3. (#2856) 
XSD - xml schema definition for SOAP operations
on S3 provides positional restrictions on XML
output.

Fix the response by re-arranging the positions in
accordance with S3 behavior.

Fixes #2849
 2016-10-05 20:12:47 -07:00
Harshavardhana
6494b77d41 server: Add more elaborate startup messages. (#2731) 
These messages based on our prep stage during XL
and prints more informative message regarding
drive information.

This change also does a much needed refactoring.
 2016-10-05 12:48:07 -07:00
Bala FA
63a7ca1af0 web: fix jwt token expiry set to one day by default. (#2819) 
Fixes #2818
 2016-10-05 10:18:55 -07:00
Krishna Srinivas
95f544657a Signature-V2: use raw resource/query from the request for signature calculation. (#2833) 2016-10-05 09:18:53 -07:00
Harshavardhana
740a919e25 config: Use migrateV8 to v9 function properly. (#2852) 2016-10-05 02:28:04 -07:00
Krishnan Parthasarathi
402c92beda Add listObjectParts test w/ unknown request signature type (#2847) 2016-10-04 07:57:35 -07:00
Karthic Rao
6a9013b97c misspell fixes. (#2835) 2016-10-04 00:09:21 -07:00
Krishnan Parthasarathi
73b50aea2d Add preSign auth type tests for ListObjectPartsHandler and PutObjectPartHandler (#2834) 2016-10-03 22:05:33 -07:00
Aditya Manthramurthy
315e66858c Add PostgreSQL notifier (#2739) (#2824) 
* The user is required to specify a table name and database connection
  information in the configuration file.

* INSERTs and DELETEs are done via prepared statements for speed.

* Assumes a table structure, and requires PostgreSQL 9.5 or above due to
  the use of UPSERT.

* Creates the table if it does not exist with the given table name using
  a query like:

    CREATE TABLE myminio (
        key varchar PRIMARY KEY,
        value JSONB
    );

* Vendors some required libraries.
 2016-10-03 17:29:55 -07:00
Krishnan Parthasarathi
4f902d42b2 Add unit-tests for ListObjectParts API handler (#2826) 
* Add missing uploadID test
... make variables in test code unexported.
* Add ServerNotInitialized test for ListObjectPartsHandler
* Add tests for ListObjectParts with signatureV2 and Anonymous requests
* Add failure test cases for ListObjectParts
 2016-10-03 08:54:57 -07:00
Krishna Srinivas
61a18ed48f sha256: Verify sha256 along with md5sum, signature is verified on the request early. (#2813) 2016-10-02 15:51:49 -07:00
Anis Elleuch
b5a6dd1395 Avoid path-cleaning policy resources for a better compliance with S3 (#2823) 2016-10-01 21:30:25 -07:00
Krishnan Parthasarathi
83e6e1060e Layer LimitReader responsibly allowing sign verification to work (#2821) 2016-10-01 09:37:40 -07:00
Krishnan Parthasarathi
ddeb8242d8 PutObjectPartHandler unit-tests (#2810) 2016-10-01 08:23:26 -07:00
Harshavardhana
5ecba587f7 api: Relax object name validation. (#2814) 
Fixes #2812
 2016-09-30 16:56:36 -07:00
Harshavardhana
db3da97a50 signature/v2: Fix presigned requests. 2016-09-30 15:22:00 -07:00
Harshavardhana
5885ffc8ae signature: Add legacy signature v2 support transparently. (#2811) 
Add new tests as well.
 2016-09-30 14:32:13 -07:00
Anis Elleuch
9fb1c89f81 Add TLS encryption capability to RPC clients (#2789) 2016-09-29 23:42:37 -07:00
Anis Elleuch
1e6afac3bd Add NATS notifier (#2795) 2016-09-29 23:42:10 -07:00
Harshavardhana
64083b9227 signature: Region changes should be handled just like AWS. (#2805) 
- PutBucket happens with 'us-east-1'.
- ListBuckets happens with any region.
- GetBucketLocation happens with 'us-east-1' and location is returned.
 2016-09-29 15:51:00 -07:00
Krishnan Parthasarathi
5fdd768903 Make addition of TopicConfig to globalEventNotifier go-routine safe (#2806) 2016-09-28 22:46:19 -07:00
Krishnan Parthasarathi
428629f577 Add unit tests for server-main.go (#2802) 2016-09-28 11:19:07 -07:00
Harshavardhana
1edd74dda2 update: Deprecate the usage of update=yes query param. (#2801) 
Fixes #2799
 2016-09-28 02:41:21 -07:00
Krishnan Parthasarathi
740ecf530c Add PutBucketNotification, ListenBucketNotification handler unit-tests. (#2787) 2016-09-28 01:08:03 -07:00
Aditya Manthramurthy
10d2ef5449 Remove comments relating to deprecated MINIO_DEBUG envvar (#2797) 2016-09-27 18:28:46 -07:00
Aditya Manthramurthy
8ea571c7f7 Remove MINIO_DEBUG environment variable (#2794) 
Removes the unimplemented settings of MINIO_DEBUG=mem and makes
MINIO_DEBUG=lock the default behaviour.
 2016-09-27 14:35:43 -07:00
Harshavardhana
ca3022d545 api: Change ListenBucketNotification with new API format. (#2791) 
Take prefix, suffix and events as query params.
 2016-09-27 13:17:43 -07:00
Anis Elleuch
9417614a8e Recalculate free minimum disk space (#2788) 
* Fix calculating free space disk by using blocks available for unprivileged user

* Use fixed minimal free disk space instead of percentage
 2016-09-27 12:46:38 -07:00
Aditya Manthramurthy
70d52bbc4c Add unit test for rate-limit-handler (#2661) (#2784) 2016-09-26 21:31:12 -07:00
Harshavardhana
6aa2fc95c0 Revert "bucket: refactor policies and fix bugs related to enforcing policies. (#2766)" 
This reverts commit ca5ca8332b.
 2016-09-26 19:32:33 -07:00
Harshavardhana
cfbab22237 web: Remove bucket policy when we have no more statements. (#2779) 2016-09-26 03:11:22 -07:00
Harshavardhana
be0e06c0aa web: Simplify and rename GetAllBucketPolicy --> ListAllBucketPolicies. (#2778) 2016-09-25 21:53:19 -07:00
Harshavardhana
1c941fd787 rpc: Should validate server versions. (#2775) 
Fixes #2764
 2016-09-24 03:34:45 -07:00
Krishnan Parthasarathi
669783f875 Purge stale object cache entry (#2770) 2016-09-23 19:55:28 -07:00
Krishnan Parthasarathi
27e474b3d2 Improve code coverage in bucket-notification-handlers.go (#2759) 
* Fix incorrect test cases for bucket-notification handler

* Add tests covering failure cases for bucket notification
 2016-09-23 13:32:51 -07:00
Krishna Srinivas
1e53316241 Add tests for presigned-get (#2767) 
* web-handlers: support for presigned-get json-rpc call for MinioBrowser's "share" feature.

* Add tests for presigned-get
 2016-09-23 01:25:49 -07:00
Harshavardhana
ca5ca8332b bucket: refactor policies and fix bugs related to enforcing policies. (#2766) 
This patch also addresses the problem of double caching at
object layer once at XL and another at handler layer.
 2016-09-22 23:47:48 -07:00
Bala FA
aa579bbc20 web: add method to get all policies for given bucket name. (#2756) 
Refer #1858
 2016-09-22 23:06:45 -07:00
Harshavardhana
e375d822da bucket: SetBucketPolicy should save a valid Version and validate. (#2762) 2016-09-22 22:27:21 -07:00
Anis Elleuch
fc783f8407 More tests for web handlers (#2755) 
* Return negative values of Total and Free in StorageInfo() when we fail to get disk info

* Return consistent messages in web handlers when the server is not initialized
 2016-09-22 16:35:12 -07:00
Anis Elleuch
ef22330563 Require go 1.7.1 to build Minio server (#2727) 2016-09-22 10:33:52 -07:00
Karthic Rao
1148f95292 ineffassign fixes (#2758) 2016-09-21 23:03:54 -07:00
Karthic Rao
f7430ec09c use runtime/debug.Stack() in leak detect test (#2757) 2016-09-21 22:04:35 -07:00
Karthic Rao
b8903d842c api/complete-multipart: fixes and tests. (#2719) 
* api/complete-multipart: tests and simplification.

- Removing the logic of sending white space characters.
- Fix for incorrect HTTP response status for certain cases.
- Tests for New Multipart Upload and Complete Multipart Upload.

* tests: test for Delelete Object API handler
 2016-09-21 20:08:08 -07:00
Aditya Manthramurthy
32f097b4d6 Controller rpc tests (#2709) 
* Test code for controller-handler operations:

* Heal operations
* List operation
* Switch to "testing" lib, moving away from gocheck
* Minor refactors

* Remove extra call to initGracefulShutdown

* Remove dead code in mainControl:

Dead code found by the TestControlMain() test function that always
passes.

* Add tests for control-*-main.go
 2016-09-21 19:58:50 -07:00
Krishnan Parthasarathi
559ad38b8c Add bucket-notification-handler tests (#2750) 2016-09-21 17:41:34 -07:00
Anis Elleuch
90417d2dd6 Check for bucket existence in Set/Get/Remove bucket policy workflow + tests (#2745) 2016-09-21 16:38:50 -07:00
Anis Elleuch
e66fb4bd7b configMigrate() returns errors + tests (#2735) 2016-09-21 09:44:57 -07:00
Harshavardhana
018c90dae7 events: ElasticSearch doesnt support objects with '/' in them. (#2747) 
Fix this by using a unique sha256 generated for each unique key.
 2016-09-20 16:36:18 -07:00
Anis Elleuch
a5066e8f76 Better code coverage of handler-utils.go (#2746) 2016-09-20 10:40:46 -07:00
Harshavardhana
0a3448c8b6 events: Change event notifiers to delete and update keys. (#2742) 
ElasticSearch and Redis are both treated like a database.
Each indexs are based on the object names uniquely indentifying
the event. Upon each delete event of the named object deletes
the index on elasticsearch and redis respectively.
 2016-09-20 02:11:17 -07:00
Harshavardhana
c4964232eb config: Fail to start for config mistakes. (#2740) 2016-09-19 15:23:49 -07:00
Harshavardhana
ef3c807b4a policies: Parser should handle Principals with various forms. (#2733) 
Handles cases for these three combinations

  - "Principal": "*",
  - "Principal": { "AWS" : "*" }
  - "Principal": { "AWS" : [ "*" ]}

Fixes #2732
 2016-09-19 13:52:28 -07:00
Harshavardhana
113b93346b lock: Make some cleanup and moving the code around. (#2718) 
This patch just avoids lot of ifs and inverts some logic.
 2016-09-19 13:14:55 -07:00
Krishna Srinivas
a955676986 Signature-V4: Dump the request with error message on signature mismatch. (#2734) 
fixes #2691
 2016-09-19 10:17:46 -07:00
Harshavardhana
725df557b5 tests: Add tests for bucket-notification-utils (#2726) 
Part - 2 final fix #2711
 2016-09-17 03:19:39 -07:00
Harshavardhana
490056eee3 tests: Add tests for bucket-notification-utils (#2723) 
Part fix - 1 for #2711
 2016-09-16 17:26:27 -07:00
Harshavardhana
797d749322 tests: Add tests for filterRuleMatch (#2722) 
Part-1 fix for #2418
 2016-09-16 16:44:44 -07:00
Harshavardhana
79888bfff7 tests: Add auth-handler. (#2721) 
Fixes #2658
 2016-09-16 15:17:49 -07:00
Anis Elleuch
010f61e91f Add more tests for event-notifier code (#2716) 2016-09-16 14:26:05 -07:00
Harshavardhana
9216981262 tests: Add test for diskCount. (#2717) 
Fixes #2312
 2016-09-16 13:44:52 -07:00
Anis Elleuch
b89a1cd482 tests: Implemented more tests for fs-v1*.go (#2686) 2016-09-16 13:06:49 -07:00
Harshavardhana
7d37dea449 tests: Add more streaming signature tests. (#2713) 
Part fix for #2621
 2016-09-16 02:45:42 -07:00
Frank
df2ef64d20 Upgrade to new dsync version incl. stale lock detection (#2708) 2016-09-16 00:30:55 -07:00
Anis Elleuch
7a549096de XL and FS use different tree walk ignored errors (#2707) 2016-09-15 13:43:40 -07:00
Harshavardhana
a1ff351f21 tests: Fix ListMultipartUploadsHandler tests. (#2705) 2016-09-15 01:44:19 -07:00
Harshavardhana
03430d0db8 tests: Add ListBucketHandler tests. (#2701) 
part-3 final fix for #2412
 2016-09-14 23:53:42 -07:00
Anis Elleuch
6f73d597e0 Fix tracing twice an error in fs Complete Multipart Upload (#2703) 2016-09-14 21:24:54 -07:00
Anis Elleuch
a84548d7ea Fix FS remove bucket regression bug (#2693) 2016-09-14 16:41:39 -07:00
Harshavardhana
1e6d67b16d server: Remove deadcode. (#2699) 2016-09-14 13:43:08 -07:00
Aditya Manthramurthy
6533927237 Lock-free rate-limit algorithm + bug-fix (#2694) 2016-09-14 11:27:37 -07:00
Harshavardhana
da9ae574df server: We should fail properly during server startup. (#2689) 
Fixes #2688
 2016-09-14 01:11:03 -07:00
Harshavardhana
ee7e70c992 tests: Add tests for ListMultipartUploads, DeleteMultipleObjects. (#2649) 
Additionally adds PostPolicyHandler tests.
 2016-09-13 21:22:31 -07:00
Krishna Srinivas
54a9f59a13 Init: Print SQS ARNs after globalEventNotifier is inited. (#2682) 
fixes #2681
 2016-09-13 21:18:30 -07:00
Harshavardhana
e6fd664331 tests: Fix format-config tests. 2016-09-13 21:18:30 -07:00
Karthic Rao
b247ec9352 tests: refactor object-handler tests. (#2656) 
- Move the initialization to a common executor for object Layer API
  tests.d
 2016-09-13 21:18:30 -07:00
Harshavardhana
43befab8ef Change distributed server wording. 2016-09-13 21:18:30 -07:00
Harshavardhana
eae0281c64 tests: Add GetBucketLocation, HeadBucket tests. (#2644) 2016-09-13 21:18:30 -07:00
Karthic Rao
8bd78fbdfb performance: gjson parsing for readXLMeta, listParts, getObjectInfo. (#2631) 
- Using gjson for constructing xlMetaV1{} in realXLMeta.
- Test for parsing constructing xlMetaV1{} using gjson.
- Changes made since benchmarks showed 30-40% improvement in speed.
- Follow up comments in issue https://github.com/minio/minio/issues/2208
  for more details.
- gjson parsing of parts from xl.json for listParts.
- gjson parsing of statInfo from xl.json for getObjectInfo.
- Vendorizing gjson dependency.
 2016-09-13 21:18:30 -07:00
Krishnan Parthasarathi
66459a4ce0 Add unit-tests for formatting disks during initialization (#2635) 
* Add unit-tests for formatting disks during initialization

- Fixed corresponding code at places where it was deviating from the
  tabular spec.

* Added more test cases and simplified algo

... based on feedback from ``go test -coverprofile``.
 2016-09-13 21:18:30 -07:00
Harshavardhana
182109f0de xl: Heal format.json properly on fresh disks. 2016-09-13 21:18:30 -07:00
Harshavardhana
9998e9ea19 api: Response timeFormat do not need to have nano-second precision. 
Fixes an error reported by s3verify.
 2016-09-13 21:18:30 -07:00
Harshavardhana
ba2ba328da server: Fixes for various conditions 
- Fix distributed branch to be able to run FS version.
- Fix distributed branch to be able to run XL local disks.
- Ignore initialization failures of notification and bucket
  policies, the codepath should load whatever is possible.
 2016-09-13 21:18:30 -07:00
Anis Elleuch
67b8080144 Fix control lock rpc name in control lock cmd (#2627) 2016-09-13 21:18:30 -07:00
Anis Elleuch
239a34ca97 Add tests for regular and streaming v4 PutObject Handler (#2618) 2016-09-13 21:18:30 -07:00
Krishna Srinivas
81d8263ae2 binary-update: Do not fetch update info for minio binary compiled from source. 
fixes #2494
 2016-09-13 21:18:30 -07:00
Krishna Srinivas
b4e4846e9f PutObject: object layer now returns ObjectInfo instead of md5sum to avoid extra GetObjectInfo call. (#2599) 
From the S3 layer after PutObject we were calling GetObjectInfo for bucket notification. This can
be avoided if PutObjectInfo returns ObjectInfo.

fixes #2567
 2016-09-13 21:18:30 -07:00
Krishna Srinivas
92e49eab5a FS/Multipart: Do not rename append files to another tmp file as the append files are already in tmp location. (#2612) 2016-09-13 21:18:30 -07:00
Harshavardhana
c4a7b950a0 fs: Fix asynchronous multipart bug. 
Construct part path properly.
 2016-09-13 21:18:30 -07:00
Karthic Rao
1ce339abeb Fixing ineffssaign errors (#2608) 2016-09-13 21:18:30 -07:00
Anis Elleuch
3e284162d7 Add global flags to all commands and subcommands (#2605) 2016-09-13 21:18:30 -07:00
Anis Elleuch
ff99392102 Enhance minio server help template (#2603) 2016-09-13 21:18:30 -07:00
Krishna Srinivas
9358ee011b logging: Print stack trace in case of errors. 
fixes #1827
 2016-09-13 21:18:30 -07:00
Harshavardhana
37cbcae6ba xl: Remove an unecessary lock with isBucketExist() (#2593) 
Fixes #2566
 2016-09-13 21:18:30 -07:00
Harshavardhana
ae64b7fac8 XL: Handle object layer initialization properly. 
Initialization when disk was down the network disk
reported an incorrect error rather than errDiskNotFound.

This resulted in incorrect error handling during
prepInitStorage() stage.

Fixes #2577
 2016-09-13 21:18:30 -07:00
Anis Elleuch
d936ed90ae Avoid testing on system errors strings in posix (#2583) 2016-09-13 21:18:30 -07:00
Krishna Srinivas
7cc77eba45 XL/Healing: errDiskNotFound is the only pardonable error in xlShouldHeal. (#2586) 
This is so that we try to heal a file for all the "bad" cases except when the disk is down.
 2016-09-13 21:18:30 -07:00
Karthic Rao
07d232c7b4 instrumentation: instrumentation for locks. (#2584) 
- Instrumentation for locks.
- Detailed test coverage.
- Adding RPC control handler to fetch lock instrumentation.
- RPC control handlers suite tests with a test RPC server.
 2016-09-13 21:18:30 -07:00
Krishnan Parthasarathi
de67bca211 Move formatting of disks out of object layer initialization (#2572) 2016-09-13 21:18:30 -07:00
Harshavardhana
bca1385683 lock: Fix support single node XL locking as well. 2016-09-13 21:18:30 -07:00
Harshavardhana
cbe87cb2ed Fix fd-leak in rpcClient close it pro-actively. 2016-09-13 21:18:30 -07:00
Anis Elleuch
0513b3ed07 Add Heal Disk Metadata RPC API + tests (#2556) 2016-09-13 21:18:30 -07:00
Frank
7f92165c79 Single use DRWMutex, RPCClient refactor and added missing cases to lock-rpc-server (#2560) 
This PR contains various fixes for the distributed release:
- Use DRWMutex in namespace-lock only for a single Lock()/RLock() call in conformance to server-side rw-locking as implemented in minio/dsync
- Implement missing cases in lock-rpc-server to catch Unlock() for active read locks and RUnlock() for an active write lock
- Refactor RPCClient to release local mutex while making actual RPC.Call()
 2016-09-13 21:18:30 -07:00
Harshavardhana
780ccc26f7 server: Validate server arguments for duplicates. (#2554) 
- Validates invalid format inputs.
- Validates duplicate entries.
- Validates sufficient amount of disks.

Partially fixes #2502
 2016-09-13 21:18:30 -07:00
Harshavardhana
339425fd52 server: Fetch StorageInfo() from underlying disks transparently. (#2549) 
Fixes #2511
 2016-09-13 21:18:30 -07:00
Harshavardhana
fa6e9540a8 server: We shouldn't exit the server in lazy init. (#2548) 
Avoid fatalIf instead these are non-critical errors,
continue running the server.

 - initializing bucket notifications
 - initializing bucket policies.
 - migrating bucket policies failure.

Fixes #2547
 2016-09-13 21:18:30 -07:00
Harshavardhana
9605fde04d controller/auth: Implement JWT based authorization for controller. (#2544) 
Fixes #2474
 2016-09-13 21:18:30 -07:00
Anis Elleuch
200d327737 List only objects that need healing (#2546) 2016-09-13 21:18:30 -07:00
Harshavardhana
e1b0985b5b rpc: Refactor authentication and login changes. (#2543) 
- Cache login requests.
- Converge validating distributed setup.
 2016-09-13 21:18:30 -07:00
Krishnan Parthasarathi
c8dfc4cda4 auth-rpc: Reset token on disconnect (#2542) 2016-09-13 21:18:30 -07:00
Bala FA
7922a54c9a rpc-client: remove unwanted nil check of rpcClient. (#2538) 2016-09-13 21:18:30 -07:00
Krishnan Parthasarathi
bda6bcd5be Layered rpc-client implementation (#2512) 2016-09-13 21:18:30 -07:00
Harshavardhana
7e3e24b394 rpc: client login should ignore server versions. 2016-09-13 21:18:30 -07:00
Harshavardhana
bb0466f4ce control: Fix controller CLI handling with distributed server object layer. 
Object layer initialization is done lazily fix it.
 2016-09-13 21:18:30 -07:00
Harshavardhana
8797952409 server: Add server command line for running in distributed mode 2016-09-13 21:18:30 -07:00
Krishnan Parthasarathi
a4691611a7 Move initGracefulShutdown before objectLayer initialization (#2468) 2016-09-13 21:18:30 -07:00
awwalker
7c7eb1475d splitNetPath: Add support for windows paths including volumeNames e.g ip:C:\network\path 2016-09-13 21:18:30 -07:00
Harshavardhana
0bce3d6d63 Fix web handler initialize with distributed lazy init. 2016-09-13 21:18:30 -07:00
Krishnan Parthasarathi
804d91ef61 storage/rpc-client: Reconnect on network disconnect (#2436) 2016-09-13 21:18:30 -07:00
Harshavardhana
43098df9d8 rpc: Re-factor ReadFile behavior client <--> server. 
Current code did not marshal/unmarshal buffers properly from
the server, The reason being buffers have to be allocated and
sent properly back to client to be consumable.
 2016-09-13 21:18:30 -07:00
Harshavardhana
6908a0dcd4 Extract rpc server wrapped errors and translate to storage error. 2016-09-13 21:18:30 -07:00
Harshavardhana
cae5761f16 rpc/client: Add missing rpcTokens for each rpc calls. 2016-09-13 21:18:30 -07:00
Harshavardhana
61af764f8a Add rpc layer authentication. 2016-09-13 21:18:30 -07:00
Harshavardhana
b4172ad3c8 Rename rpc-{client,server} storage-rpc-{client,server} 2016-09-13 21:18:30 -07:00
Harshavardhana
4917038f55 Move the ObjectAPI() resource to be beginning of each handlers. 
We should return back proper errors so that the clients can
retry until server has been initialized.
 2016-09-13 21:18:30 -07:00