17e49a9ed2
signature-v2 fix. ( #2918 )
...
- Return errors similar to V4 Sign processsing.
- Return ErrMissing fields when Auth Header fields are missing.
- Return InvalidAccessID when accessID doesn't match.
* tests: Adding V2 signature tests for bucket handler API's.
2016-10-13 09:25:56 -07:00
0aabc1d8d9
Use Peer RPC to propagate bucket policy changes ( #2891 )
2016-10-13 09:19:04 -07:00
55f6828750
Do not print update message unless there is an update. ( #2919 )
2016-10-13 09:17:08 -07:00
6303f26330
Protect map from concurrent access ( Fixes #2915 ) ( #2916 )
...
Protects the Peers RPC clients map from concurrent access to fix a data race condition.
2016-10-13 01:33:50 -07:00
b59bac670a
Handle err returned by rpc.Server.RegisterName ( #2910 )
2016-10-12 23:13:24 -07:00
84acc820c7
Fix free drive space calculation in XL mode ( #2917 )
2016-10-12 20:22:15 -07:00
fdaa129a5b
Fix dockerfile container image. ( #2892 )
2016-10-12 18:09:08 -07:00
df59967f59
Avoid checking date header of web requests by properly applying generic handlers ( #2914 )
2016-10-12 12:58:36 -07:00
73982c8cb6
Listen bucket notification for multiple prefixes/suffixes ( #2911 )
...
* Listen bucket notification for multiple prefixes/suffixes
* After review fixes
2016-10-12 11:02:15 -07:00
6199aa0707
Peer RPCs for bucket notifications ( #2877 )
...
* Implements a Peer RPC router that sends info to all Minio servers in the cluster.
* Bucket notifications are propagated to all nodes via this RPC router.
* Bucket listener configuration is persisted to separate object layer
file (`listener.json`) and peer RPCs are used to communicate changes
throughout the cluster.
* When events are generated, RPC calls to send them to other servers
where bucket listeners may be connected is implemented.
* Some bucket notification tests are now disabled as they cannot work in
the new design.
* Minor fix in `funcFromPC` to use `path.Join`
2016-10-12 01:03:50 -07:00
a5921b5743
Use same timestamp for all chunks in chunked signature ( #2908 )
2016-10-11 23:46:51 -07:00
ff91ecb177
tests: Adding unknown signature type test for API handlers. ( #2905 )
2016-10-11 20:38:10 -07:00
a6357502c1
Correct typo in error string ( #2902 )
2016-10-11 08:56:02 -07:00
fa8ea41cd9
lock/instrumentation: Cleanup and print in user friendly form. ( #2807 )
2016-10-11 00:50:27 -07:00
3ac6790ca2
tests: Add Object Layer nil test for bucket-handler API's ( #2899 )
2016-10-11 00:00:02 -07:00
268b96058f
ns-lock: lock namespace during FS object operations. ( #2896 )
2016-10-10 10:20:04 -07:00
0d031c432b
Fix typo in serverVersion ( #2894 )
2016-10-10 10:11:56 -07:00
6e8f3224c5
Test coverage for lock rpc server ( #2893 )
...
* Add test coverage for removeEntry and removeEntryIfExists
* Initial test framework for Lock/Unlock functionality
* Add clarification comments
* Add test coverage code for RLock() and RUnlock()
* Add test coverage for Expired() function
* Have all lock-rpc-server test functions start with the same prefix
* Properly initialize JWT security token
2016-10-10 10:11:29 -07:00
9c53e9f4c3
tests: Enhance coverage for bucket policy handlers. ( #2895 )
2016-10-10 09:29:56 -07:00
2d5e988a6d
Refactor streaming signatureV4 w/ state machine ( #2862 )
...
* Refactor streaming signatureV4 w/ state machine
- Used state machine to make transitions between reading chunk header,
chunk data and trailer explicit.
* debug: add print/panic statements to gather more info on CI failure
* Persist lastChunk status between Read() on ChunkReader
... remove panic() which was added as interim aid for debugging.
* Add unit-tests to cover v4 streaming signature
2016-10-10 01:42:32 -07:00
3cfb23750a
control: Implement service command 'stop,restart,status'. ( #2883 )
...
- stop - stops all the servers.
- restart - restart all the servers.
- status - prints status of storage info about the cluster.
2016-10-09 23:03:10 -07:00
57f75b1d9b
Ignore copy conditions when ETag is not available ( #2888 )
2016-10-09 16:21:42 -07:00
e213172431
tests: Missing anonymous tests for bucket-handlers. ( #2885 )
2016-10-09 09:21:37 -07:00
f5f007e183
Test: Add test case for xl.HealObject() ( #2884 )
...
fixes #2842
2016-10-08 17:08:17 -07:00
09463265ce
tests: Adding anonymous requests tests for bucket policy handlers. ( #2882 )
2016-10-08 01:04:26 -07:00
8f4cf2a7d0
tests: anonymous/unsigned tests for object handler API's . ( #2881 )
2016-10-07 23:28:50 -07:00
30183c4a9a
tests: cleanup and unsigned request test. ( #2880 )
...
- Cleaning up of ListMultipartUpload API test for improving readability,
code maintainance and extensibility.
- Moving ListMultipartUploads to Go 1.7 sub tests.
- Using the new Anonymous request helper function for
ListMultipartUploads.
2016-10-07 20:16:57 -07:00
d1df5e0ae1
tests: Add helper function for API handler anonymous request tests. ( #2876 )
...
- Add helper function for API handler anonymous request tests.
- Add PutObject Part Anonymous request case using the new helper
function to validate its functionality.
2016-10-07 11:16:11 -07:00
f1bc9343a1
prep: Initialization should wait instead of exit the servers. ( #2872 )
...
- Servers do not exit for invalid credentials instead they print and wait.
- Servers do not exit for version mismatch instead they print and wait.
- Servers do not exit for time differences between nodes they print and wait.
2016-10-07 11:15:55 -07:00
97f4989945
tests: cleaning up. ( #2875 )
...
- Clean up PutObjectPart and ListObjectPart API handler tests.
- Add more comments, make the tests more readable.
- Add verification for HTTP response status code.
- Initialize the test using object Layer.
- Move to Go 1.7 sub tests.
2016-10-07 08:02:37 -07:00
1e5e213d24
auth: Make sure we initialize or change config before RPC requests. ( #2867 )
2016-10-06 13:35:56 -07:00
a8105ec068
- Test utility function for easy asserting of cases wherein objectLayer ( #2865 )
...
is `nil` in API handlers.
- Remove the existing tests for the `nil` check and use the new method
to test for object layer being `nil`.
2016-10-06 13:34:33 -07:00
bb9be02228
minio-browser: do not redirect to /minio if MINIO_BROWSER=off ( #2863 )
...
fixes #2837
2016-10-06 08:30:32 -07:00
64f37bbf5b
rpc: Add RPC client tests. ( #2858 )
2016-10-06 02:30:54 -07:00
0fc96fa25c
Refactor bucket policy handler test to use API test initializer. ( #2859 )
2016-10-06 02:02:42 -07:00
2d8c6f8288
unit test for bucketPolicyConditionMatch function. ( #2857 )
2016-10-06 00:23:46 -07:00
b94211bd66
api: ListObjectsV1 compliance with AWS S3. ( #2856 )
...
XSD - xml schema definition for SOAP operations
on S3 provides positional restrictions on XML
output.
Fix the response by re-arranging the positions in
accordance with S3 behavior.
Fixes #2849
2016-10-05 20:12:47 -07:00
6494b77d41
server: Add more elaborate startup messages. ( #2731 )
...
These messages based on our prep stage during XL
and prints more informative message regarding
drive information.
This change also does a much needed refactoring.
2016-10-05 12:48:07 -07:00
63a7ca1af0
web: fix jwt token expiry set to one day by default. ( #2819 )
...
Fixes #2818
2016-10-05 10:18:55 -07:00
95f544657a
Signature-V2: use raw resource/query from the request for signature calculation. ( #2833 )
2016-10-05 09:18:53 -07:00
740a919e25
config: Use migrateV8 to v9 function properly. ( #2852 )
2016-10-05 02:28:04 -07:00
402c92beda
Add listObjectParts test w/ unknown request signature type ( #2847 )
2016-10-04 07:57:35 -07:00
6a9013b97c
misspell fixes. ( #2835 )
2016-10-04 00:09:21 -07:00
73b50aea2d
Add preSign auth type tests for ListObjectPartsHandler and PutObjectPartHandler ( #2834 )
2016-10-03 22:05:33 -07:00
315e66858c
Add PostgreSQL notifier ( #2739 ) ( #2824 )
...
* The user is required to specify a table name and database connection
information in the configuration file.
* INSERTs and DELETEs are done via prepared statements for speed.
* Assumes a table structure, and requires PostgreSQL 9.5 or above due to
the use of UPSERT.
* Creates the table if it does not exist with the given table name using
a query like:
CREATE TABLE myminio (
key varchar PRIMARY KEY,
value JSONB
);
* Vendors some required libraries.
2016-10-03 17:29:55 -07:00
4f902d42b2
Add unit-tests for ListObjectParts API handler ( #2826 )
...
* Add missing uploadID test
... make variables in test code unexported.
* Add ServerNotInitialized test for ListObjectPartsHandler
* Add tests for ListObjectParts with signatureV2 and Anonymous requests
* Add failure test cases for ListObjectParts
2016-10-03 08:54:57 -07:00
61a18ed48f
sha256: Verify sha256 along with md5sum, signature is verified on the request early. ( #2813 )
2016-10-02 15:51:49 -07:00
b5a6dd1395
Avoid path-cleaning policy resources for a better compliance with S3 ( #2823 )
2016-10-01 21:30:25 -07:00
83e6e1060e
Layer LimitReader responsibly allowing sign verification to work ( #2821 )
2016-10-01 09:37:40 -07:00
ddeb8242d8
PutObjectPartHandler unit-tests ( #2810 )
2016-10-01 08:23:26 -07:00
5ecba587f7
api: Relax object name validation. ( #2814 )
...
Fixes #2812
2016-09-30 16:56:36 -07:00
db3da97a50
signature/v2: Fix presigned requests.
2016-09-30 15:22:00 -07:00
5885ffc8ae
signature: Add legacy signature v2 support transparently. ( #2811 )
...
Add new tests as well.
2016-09-30 14:32:13 -07:00
9fb1c89f81
Add TLS encryption capability to RPC clients ( #2789 )
2016-09-29 23:42:37 -07:00
1e6afac3bd
Add NATS notifier ( #2795 )
2016-09-29 23:42:10 -07:00
64083b9227
signature: Region changes should be handled just like AWS. ( #2805 )
...
- PutBucket happens with 'us-east-1'.
- ListBuckets happens with any region.
- GetBucketLocation happens with 'us-east-1' and location is returned.
2016-09-29 15:51:00 -07:00
5fdd768903
Make addition of TopicConfig to globalEventNotifier go-routine safe ( #2806 )
2016-09-28 22:46:19 -07:00
428629f577
Add unit tests for server-main.go ( #2802 )
2016-09-28 11:19:07 -07:00
1edd74dda2
update: Deprecate the usage of update=yes query param. ( #2801 )
...
Fixes #2799
2016-09-28 02:41:21 -07:00
740ecf530c
Add PutBucketNotification, ListenBucketNotification handler unit-tests. ( #2787 )
2016-09-28 01:08:03 -07:00
10d2ef5449
Remove comments relating to deprecated MINIO_DEBUG envvar ( #2797 )
2016-09-27 18:28:46 -07:00
8ea571c7f7
Remove MINIO_DEBUG environment variable ( #2794 )
...
Removes the unimplemented settings of MINIO_DEBUG=mem and makes
MINIO_DEBUG=lock the default behaviour.
2016-09-27 14:35:43 -07:00
ca3022d545
api: Change ListenBucketNotification with new API format. ( #2791 )
...
Take prefix, suffix and events as query params.
2016-09-27 13:17:43 -07:00
9417614a8e
Recalculate free minimum disk space ( #2788 )
...
* Fix calculating free space disk by using blocks available for unprivileged user
* Use fixed minimal free disk space instead of percentage
2016-09-27 12:46:38 -07:00
70d52bbc4c
Add unit test for rate-limit-handler ( #2661 ) ( #2784 )
2016-09-26 21:31:12 -07:00
6aa2fc95c0
Revert "bucket: refactor policies and fix bugs related to enforcing policies. ( #2766 )"
...
This reverts commit ca5ca8332b
2016-09-26 19:32:33 -07:00
cfbab22237
web: Remove bucket policy when we have no more statements. ( #2779 )
2016-09-26 03:11:22 -07:00
be0e06c0aa
web: Simplify and rename GetAllBucketPolicy --> ListAllBucketPolicies. ( #2778 )
2016-09-25 21:53:19 -07:00
1c941fd787
rpc: Should validate server versions. ( #2775 )
...
Fixes #2764
2016-09-24 03:34:45 -07:00
669783f875
Purge stale object cache entry ( #2770 )
2016-09-23 19:55:28 -07:00
27e474b3d2
Improve code coverage in bucket-notification-handlers.go ( #2759 )
...
* Fix incorrect test cases for bucket-notification handler
* Add tests covering failure cases for bucket notification
2016-09-23 13:32:51 -07:00
1e53316241
Add tests for presigned-get ( #2767 )
...
* web-handlers: support for presigned-get json-rpc call for MinioBrowser's "share" feature.
* Add tests for presigned-get
2016-09-23 01:25:49 -07:00
ca5ca8332b
bucket: refactor policies and fix bugs related to enforcing policies. ( #2766 )
...
This patch also addresses the problem of double caching at
object layer once at XL and another at handler layer.
2016-09-22 23:47:48 -07:00
aa579bbc20
web: add method to get all policies for given bucket name. ( #2756 )
...
Refer #1858
2016-09-22 23:06:45 -07:00
e375d822da
bucket: SetBucketPolicy should save a valid Version and validate. ( #2762 )
2016-09-22 22:27:21 -07:00
fc783f8407
More tests for web handlers ( #2755 )
...
* Return negative values of Total and Free in StorageInfo() when we fail to get disk info
* Return consistent messages in web handlers when the server is not initialized
2016-09-22 16:35:12 -07:00
ef22330563
Require go 1.7.1 to build Minio server ( #2727 )
2016-09-22 10:33:52 -07:00
1148f95292
ineffassign fixes ( #2758 )
2016-09-21 23:03:54 -07:00
f7430ec09c
use runtime/debug.Stack() in leak detect test ( #2757 )
2016-09-21 22:04:35 -07:00
b8903d842c
api/complete-multipart: fixes and tests. ( #2719 )
...
* api/complete-multipart: tests and simplification.
- Removing the logic of sending white space characters.
- Fix for incorrect HTTP response status for certain cases.
- Tests for New Multipart Upload and Complete Multipart Upload.
* tests: test for Delelete Object API handler
2016-09-21 20:08:08 -07:00
32f097b4d6
Controller rpc tests ( #2709 )
...
* Test code for controller-handler operations:
* Heal operations
* List operation
* Switch to "testing" lib, moving away from gocheck
* Minor refactors
* Remove extra call to initGracefulShutdown
* Remove dead code in mainControl:
Dead code found by the TestControlMain() test function that always
passes.
* Add tests for control-*-main.go
2016-09-21 19:58:50 -07:00
559ad38b8c
Add bucket-notification-handler tests ( #2750 )
2016-09-21 17:41:34 -07:00
90417d2dd6
Check for bucket existence in Set/Get/Remove bucket policy workflow + tests ( #2745 )
2016-09-21 16:38:50 -07:00
e66fb4bd7b
configMigrate() returns errors + tests ( #2735 )
2016-09-21 09:44:57 -07:00
018c90dae7
events: ElasticSearch doesnt support objects with '/' in them. ( #2747 )
...
Fix this by using a unique sha256 generated for each unique key.
2016-09-20 16:36:18 -07:00
a5066e8f76
Better code coverage of handler-utils.go ( #2746 )
2016-09-20 10:40:46 -07:00
0a3448c8b6
events: Change event notifiers to delete and update keys. ( #2742 )
...
ElasticSearch and Redis are both treated like a database.
Each indexs are based on the object names uniquely indentifying
the event. Upon each delete event of the named object deletes
the index on elasticsearch and redis respectively.
2016-09-20 02:11:17 -07:00
c4964232eb
config: Fail to start for config mistakes. ( #2740 )
2016-09-19 15:23:49 -07:00
ef3c807b4a
policies: Parser should handle Principals with various forms. ( #2733 )
...
Handles cases for these three combinations
- "Principal": "*",
- "Principal": { "AWS" : "*" }
- "Principal": { "AWS" : [ "*" ]}
Fixes #2732
2016-09-19 13:52:28 -07:00
113b93346b
lock: Make some cleanup and moving the code around. ( #2718 )
...
This patch just avoids lot of ifs and inverts some logic.
2016-09-19 13:14:55 -07:00
a955676986
Signature-V4: Dump the request with error message on signature mismatch. ( #2734 )
...
fixes #2691
2016-09-19 10:17:46 -07:00
725df557b5
tests: Add tests for bucket-notification-utils ( #2726 )
...
Part - 2 final fix #2711
2016-09-17 03:19:39 -07:00
490056eee3
tests: Add tests for bucket-notification-utils ( #2723 )
...
Part fix - 1 for #2711
2016-09-16 17:26:27 -07:00
797d749322
tests: Add tests for filterRuleMatch ( #2722 )
...
Part-1 fix for #2418
2016-09-16 16:44:44 -07:00
79888bfff7
tests: Add auth-handler. ( #2721 )
...
Fixes #2658
2016-09-16 15:17:49 -07:00
010f61e91f
Add more tests for event-notifier code ( #2716 )
2016-09-16 14:26:05 -07:00
9216981262
tests: Add test for diskCount. ( #2717 )
...
Fixes #2312
2016-09-16 13:44:52 -07:00
b89a1cd482
tests: Implemented more tests for fs-v1*.go ( #2686 )
2016-09-16 13:06:49 -07:00
7d37dea449
tests: Add more streaming signature tests. ( #2713 )
...
Part fix for #2621
2016-09-16 02:45:42 -07:00
df2ef64d20
Upgrade to new dsync version incl. stale lock detection ( #2708 )
2016-09-16 00:30:55 -07:00
7a549096de
XL and FS use different tree walk ignored errors ( #2707 )
2016-09-15 13:43:40 -07:00
a1ff351f21
tests: Fix ListMultipartUploadsHandler tests. ( #2705 )
2016-09-15 01:44:19 -07:00
03430d0db8
tests: Add ListBucketHandler tests. ( #2701 )
...
part-3 final fix for #2412
2016-09-14 23:53:42 -07:00
6f73d597e0
Fix tracing twice an error in fs Complete Multipart Upload ( #2703 )
2016-09-14 21:24:54 -07:00
a84548d7ea
Fix FS remove bucket regression bug ( #2693 )
2016-09-14 16:41:39 -07:00
1e6d67b16d
server: Remove deadcode. ( #2699 )
2016-09-14 13:43:08 -07:00
6533927237
Lock-free rate-limit algorithm + bug-fix ( #2694 )
2016-09-14 11:27:37 -07:00
da9ae574df
server: We should fail properly during server startup. ( #2689 )
...
Fixes #2688
2016-09-14 01:11:03 -07:00
ee7e70c992
tests: Add tests for ListMultipartUploads, DeleteMultipleObjects. ( #2649 )
...
Additionally adds PostPolicyHandler tests.
2016-09-13 21:22:31 -07:00
54a9f59a13
Init: Print SQS ARNs after globalEventNotifier is inited. ( #2682 )
...
fixes #2681
2016-09-13 21:18:30 -07:00
e6fd664331
tests: Fix format-config tests.
2016-09-13 21:18:30 -07:00
b247ec9352
tests: refactor object-handler tests. ( #2656 )
...
- Move the initialization to a common executor for object Layer API
tests.d
2016-09-13 21:18:30 -07:00
43befab8ef
Change distributed server wording.
2016-09-13 21:18:30 -07:00
eae0281c64
tests: Add GetBucketLocation, HeadBucket tests. ( #2644 )
2016-09-13 21:18:30 -07:00
8bd78fbdfb
performance: gjson parsing for readXLMeta, listParts, getObjectInfo. ( #2631 )
...
- Using gjson for constructing xlMetaV1{} in realXLMeta.
- Test for parsing constructing xlMetaV1{} using gjson.
- Changes made since benchmarks showed 30-40% improvement in speed.
- Follow up comments in issue https://github.com/minio/minio/issues/2208
for more details.
- gjson parsing of parts from xl.json for listParts.
- gjson parsing of statInfo from xl.json for getObjectInfo.
- Vendorizing gjson dependency.
2016-09-13 21:18:30 -07:00
66459a4ce0
Add unit-tests for formatting disks during initialization ( #2635 )
...
* Add unit-tests for formatting disks during initialization
- Fixed corresponding code at places where it was deviating from the
tabular spec.
* Added more test cases and simplified algo
... based on feedback from ``go test -coverprofile``.
2016-09-13 21:18:30 -07:00
182109f0de
xl: Heal `format.json` properly on fresh disks.
2016-09-13 21:18:30 -07:00
9998e9ea19
api: Response timeFormat do not need to have nano-second precision.
...
Fixes an error reported by s3verify.
2016-09-13 21:18:30 -07:00
ba2ba328da
server: Fixes for various conditions
...
- Fix distributed branch to be able to run FS version.
- Fix distributed branch to be able to run XL local disks.
- Ignore initialization failures of notification and bucket
policies, the codepath should load whatever is possible.
2016-09-13 21:18:30 -07:00
67b8080144
Fix control lock rpc name in control lock cmd ( #2627 )
2016-09-13 21:18:30 -07:00
239a34ca97
Add tests for regular and streaming v4 PutObject Handler ( #2618 )
2016-09-13 21:18:30 -07:00
81d8263ae2
binary-update: Do not fetch update info for minio binary compiled from source.
...
fixes #2494
2016-09-13 21:18:30 -07:00
b4e4846e9f
PutObject: object layer now returns ObjectInfo instead of md5sum to avoid extra GetObjectInfo call. ( #2599 )
...
From the S3 layer after PutObject we were calling GetObjectInfo for bucket notification. This can
be avoided if PutObjectInfo returns ObjectInfo.
fixes #2567
2016-09-13 21:18:30 -07:00
92e49eab5a
FS/Multipart: Do not rename append files to another tmp file as the append files are already in tmp location. ( #2612 )
2016-09-13 21:18:30 -07:00
c4a7b950a0
fs: Fix asynchronous multipart bug.
...
Construct part path properly.
2016-09-13 21:18:30 -07:00
1ce339abeb
Fixing ineffssaign errors ( #2608 )
2016-09-13 21:18:30 -07:00
3e284162d7
Add global flags to all commands and subcommands ( #2605 )
2016-09-13 21:18:30 -07:00
ff99392102
Enhance minio server help template ( #2603 )
2016-09-13 21:18:30 -07:00
9358ee011b
logging: Print stack trace in case of errors.
...
fixes #1827
2016-09-13 21:18:30 -07:00
37cbcae6ba
xl: Remove an unecessary lock with isBucketExist() ( #2593 )
...
Fixes #2566
2016-09-13 21:18:30 -07:00
ae64b7fac8
XL: Handle object layer initialization properly.
...
Initialization when disk was down the network disk
reported an incorrect error rather than errDiskNotFound.
This resulted in incorrect error handling during
prepInitStorage() stage.
Fixes #2577
2016-09-13 21:18:30 -07:00
d936ed90ae
Avoid testing on system errors strings in posix ( #2583 )
2016-09-13 21:18:30 -07:00
7cc77eba45
XL/Healing: errDiskNotFound is the only pardonable error in xlShouldHeal. ( #2586 )
...
This is so that we try to heal a file for all the "bad" cases except when the disk is down.
2016-09-13 21:18:30 -07:00
07d232c7b4
instrumentation: instrumentation for locks. ( #2584 )
...
- Instrumentation for locks.
- Detailed test coverage.
- Adding RPC control handler to fetch lock instrumentation.
- RPC control handlers suite tests with a test RPC server.
2016-09-13 21:18:30 -07:00
de67bca211
Move formatting of disks out of object layer initialization ( #2572 )
2016-09-13 21:18:30 -07:00
bca1385683
lock: Fix support single node XL locking as well.
2016-09-13 21:18:30 -07:00
cbe87cb2ed
Fix fd-leak in rpcClient close it pro-actively.
2016-09-13 21:18:30 -07:00
0513b3ed07
Add Heal Disk Metadata RPC API + tests ( #2556 )
2016-09-13 21:18:30 -07:00
7f92165c79
Single use DRWMutex, RPCClient refactor and added missing cases to lock-rpc-server ( #2560 )
...
This PR contains various fixes for the distributed release:
- Use DRWMutex in namespace-lock only for a single Lock()/RLock() call in conformance to server-side rw-locking as implemented in minio/dsync
- Implement missing cases in lock-rpc-server to catch Unlock() for active read locks and RUnlock() for an active write lock
- Refactor RPCClient to release local mutex while making actual RPC.Call()
2016-09-13 21:18:30 -07:00
780ccc26f7
server: Validate server arguments for duplicates. ( #2554 )
...
- Validates invalid format inputs.
- Validates duplicate entries.
- Validates sufficient amount of disks.
Partially fixes #2502
2016-09-13 21:18:30 -07:00
339425fd52
server: Fetch StorageInfo() from underlying disks transparently. ( #2549 )
...
Fixes #2511
2016-09-13 21:18:30 -07:00
fa6e9540a8
server: We shouldn't exit the server in lazy init. ( #2548 )
...
Avoid fatalIf instead these are non-critical errors,
continue running the server.
- initializing bucket notifications
- initializing bucket policies.
- migrating bucket policies failure.
Fixes #2547
2016-09-13 21:18:30 -07:00
9605fde04d
controller/auth: Implement JWT based authorization for controller. ( #2544 )
...
Fixes #2474
2016-09-13 21:18:30 -07:00
200d327737
List only objects that need healing ( #2546 )
2016-09-13 21:18:30 -07:00
e1b0985b5b
rpc: Refactor authentication and login changes. ( #2543 )
...
- Cache login requests.
- Converge validating distributed setup.
2016-09-13 21:18:30 -07:00
c8dfc4cda4
auth-rpc: Reset token on disconnect ( #2542 )
2016-09-13 21:18:30 -07:00
7922a54c9a
rpc-client: remove unwanted nil check of rpcClient. ( #2538 )
2016-09-13 21:18:30 -07:00
bda6bcd5be
Layered rpc-client implementation ( #2512 )
2016-09-13 21:18:30 -07:00
7e3e24b394
rpc: client login should ignore server versions.
2016-09-13 21:18:30 -07:00
bb0466f4ce
control: Fix controller CLI handling with distributed server object layer.
...
Object layer initialization is done lazily fix it.
2016-09-13 21:18:30 -07:00
8797952409
server: Add server command line for running in distributed mode
2016-09-13 21:18:30 -07:00
a4691611a7
Move initGracefulShutdown before objectLayer initialization ( #2468 )
2016-09-13 21:18:30 -07:00
7c7eb1475d
splitNetPath: Add support for windows paths including volumeNames e.g ip:C:\network\path
2016-09-13 21:18:30 -07:00
0bce3d6d63
Fix web handler initialize with distributed lazy init.
2016-09-13 21:18:30 -07:00
804d91ef61
storage/rpc-client: Reconnect on network disconnect ( #2436 )
2016-09-13 21:18:30 -07:00
43098df9d8
rpc: Re-factor ReadFile behavior client <--> server.
...
Current code did not marshal/unmarshal buffers properly from
the server, The reason being buffers have to be allocated and
sent properly back to client to be consumable.
2016-09-13 21:18:30 -07:00
6908a0dcd4
Extract rpc server wrapped errors and translate to storage error.
2016-09-13 21:18:30 -07:00
cae5761f16
rpc/client: Add missing rpcTokens for each rpc calls.
2016-09-13 21:18:30 -07:00
61af764f8a
Add rpc layer authentication.
2016-09-13 21:18:30 -07:00
b4172ad3c8
Rename rpc-{client,server} storage-rpc-{client,server}
2016-09-13 21:18:30 -07:00
4917038f55
Move the ObjectAPI() resource to be beginning of each handlers.
...
We should return back proper errors so that the clients can
retry until server has been initialized.
2016-09-13 21:18:30 -07:00
b0f3f94163
unify single-node and distributed namespace locking ( #2401 )
2016-09-13 21:18:30 -07:00
b7c169d71d
object layer initialization using distributed locking ( #2397 )
...
* vendorized latest minio/dsync
* wip - object layer initialization using distributed locking
2016-09-13 21:18:30 -07:00
3939c75345
Added distributed RWMutex ( #2369 )
2016-09-13 21:18:30 -07:00
e55926e8cf
distribute: Make server work with multiple remote disks
...
This change initializes rpc servers associated with disks that are
local. It makes object layer initialization on demand, namely on the
first request to the object layer.
Also adds lock RPC service vendorized minio/dsync
2016-09-13 21:18:30 -07:00
f82f535509
Fix possible race in shutdown callbacks process and simplify shuttting down profiler ( #2684 )
2016-09-13 18:43:45 -07:00
51e337228e
Avoid hiding disk errors in some cases in FS Shutdown ( #2668 )
2016-09-13 11:01:10 -07:00
241c56e6d7
Use Set instead of Add in the event that the request already contains the content-length ( #2683 )
2016-09-13 10:59:40 -07:00
895471afa1
Fixes #2678 ( #2679 )
...
Refactor `ratelimit.acquire()` to properly enforce the *globalMaxConn*
limit.
2016-09-12 15:53:54 -07:00
85e2d886bc
bucketPolicy: Trim incoming paths properly for policy verification.
2016-09-11 10:36:54 -07:00
8acf4d112a
Return Owner information in ListObjectV2 only when fetch-owner is specified ( #2654 )
2016-09-10 10:44:38 -07:00
e6abfb3b67
More tests for format-config for a better coverage ( #2655 )
2016-09-10 09:54:22 -07:00
040af08473
server: Startup message now prints configured ARNs. ( #2653 )
...
Fixes #2628
2016-09-10 02:23:28 -07:00
e3de237eb8
Move config to v7 to fix previous v6 bug on null Notification entries. ( #2650 )
...
Thanks to @koolhead17 for reporting this.
2016-09-10 00:51:25 -07:00
11785529fc
Add Request Not Ready error when presigned date request is not valid ( #2646 )
2016-09-10 00:38:07 -07:00
32201a18ab
Forbid requests generated at least 15 min in the past or in the future ( #2648 )
2016-09-09 15:33:35 -07:00
421cccb1d7
Add naughty disk to simulate disk errors ( #2645 )
2016-09-09 12:53:09 -07:00
400e9309f1
Signature-V4: Include content-length for signature calculation. ( #2643 )
...
This is to be compatible with clients which includes content-length for signature calculation
(and hence deviating from AWS Signature-v4 spec)
2016-09-09 09:38:07 -07:00
a4afb312d4
S3-layer: Multiple-delete should return success for non-existent objects. ( #2633 )
...
Fixes #2630
2016-09-07 11:49:12 -07:00
afe874f15a
api: CopyObject pipe should be closed pro-actively upon error. ( #2625 )
...
Fixes a deadlock reproduced while running s3verify during
RemoveObject(). Previously held lock by GetObject() inside
the go-routine was never relenquished.
2016-09-06 02:23:32 -07:00
b291dbe9c5
s3: Fix compliance bugs reported by s3verify. ( #2624 )
...
- Fixes couple of error strings reported are mismatching.
- Fixes a error HTTP status which was wrong fixed.
- Remove usage of an deprecated PostResponse, au contraire
to their documentation there is no response body in
PostPolicy.
2016-09-06 02:18:02 -07:00
bc8f34bfe7
server/mux: Remove unused waitgroup from listenerMux. ( #2614 )
...
Change struct names to be more meaningful.
2016-09-02 19:07:42 -07:00
2dc7ecc59b
notifications: Fix bucket notifications for DeleteMultipleObjects. ( #2609 )
...
Now reports bucket notifications for DeleteMultipleObjects API.
Also deletes multiple objects in parallel.
2016-09-02 01:59:08 -07:00
7398d737b5
profiler: Fix it properly and generate/save profiles even failure situations. ( #2607 )
...
Fixes #2594
2016-09-01 20:13:11 -07:00
511a8cbe04
profiler: Fix how we generate profiles. ( #2596 )
...
Fixes #2594
2016-09-01 03:50:31 -07:00
e79d2381fc
Fix rare 'go test -race' failure in ListenServe{Plain,TLS} ( #2588 )
2016-08-31 16:07:44 -07:00
a3b4199e9b
FS/Multipart: Allow for parallel uploads of parts with same part number. ( #2587 )
...
CBL client does not close connection when the backup process is stopped, this causes
read() on the stream on the server side to block and hence the lock held on the part
is not released. When the backup process is restarted, we again try to lock on the
part and this will block. Using a unique tmp name and not locking it fixes the problem.
2016-08-31 16:01:32 -07:00
3aa0574c6b
FS/multipart: Append the parts to a file as and when the parts arrive. ( #2513 )
2016-08-31 13:42:57 -07:00
5716f1c199
Fix incorrect error strings. ( #2580 )
2016-08-30 13:47:09 -07:00
2020afd200
server: http.Server do not add deadlines causes issues. ( #2581 )
...
Adding deadlines is a no go since Golang doesn't back off
the timers if there is an active i/o in progress.
It is meant to be for applications to handle this themselves
and manually progress the deadlines.
Fixes #2561
2016-08-30 12:52:19 -07:00
7431acb2c4
web: add handlers for set/get bucket policy. ( #2486 )
...
Refer #1858
2016-08-30 10:04:50 -07:00
2f9975c76c
notifications: GetObjectInfo() later if bucket notifications are set. ( #2575 )
2016-08-30 03:42:37 -07:00
7270ca4157
pkg/wildcard: Simplify the wildcard logic further. ( #2555 )
2016-08-27 00:27:17 -07:00
b67c8970c9
bucket-handlers: fix missing return statement ( #2553 )
...
If the location was invalid, it would write an error response but then
continue to attempt to make the bucket. Whether or not it would succeed,
it would attempt to call response.WriteHeaders twice in a row, which
would cause a message to be logged to the server console (bad).
Here is the relevant Go code:
c80e0d374b/src/net/http/server.go (L878-L881)
2016-08-25 20:00:47 -07:00
418921de89
minor cleanup
...
- Reused contains() from utils.go at a couple of places
- Cleanup in return statements and boolean checks
2016-08-24 22:54:34 -07:00
ec4260d260
api: BucketNotification should disallow duplicate notification. ( #2539 )
...
Added checks to look for duplicated notification configs.
Fixes #2472
2016-08-23 18:42:30 -07:00
c39d3db7a0
server/tls: allocate tls.Config{} properly. ( #2537 )
...
There is a golang bug which crashes the server, allocate tls.Config properly to avoid
this problem.
```
2016/08/22 20:21:03 http: panic serving 127.0.0.1:40744: runtime error: invalid memory address or nil pointer dereference
goroutine 38 [running]:
net/http.(*conn).serve.func1(0xc820526680)
/home/harsha/.gimme/versions/go1.6.2.linux.amd64/src/net/http/server.go:1389 +0xc1
panic(0xbefa80, 0xc820010140)
/home/harsha/.gimme/versions/go1.6.2.linux.amd64/src/runtime/panic.go:443 +0x4e9
crypto/tls.(*Conn).serverHandshake(0xc820368300, 0x0, 0x0)
/home/harsha/.gimme/versions/go1.6.2.linux.amd64/src/crypto/tls/handshake_server.go:43 +0x4d6
crypto/tls.(*Conn).Handshake(0xc820368300, 0x0, 0x0)
/home/harsha/.gimme/versions/go1.6.2.linux.amd64/src/crypto/tls/conn.go:1035 +0x169
net/http.(*conn).serve(0xc820526680)
/home/harsha/.gimme/versions/go1.6.2.linux.amd64/src/net/http/server.go:1405 +0x382
created by net/http.(*Server).Serve
/home/harsha/.gimme/versions/go1.6.2.linux.amd64/src/net/http/server.go:2137 +0x44e
```
Fixes #2536
2016-08-23 00:20:15 -07:00
2da0cfc904
signature: Use the client sent region if region is properly validated. ( #2534 )
...
A properly validated region string from the client should be once
validated properly.
Fixes #2521
2016-08-22 19:43:36 -07:00
9dd0e3dc44
config: Initialize notify for fresh configs. ( #2531 )
2016-08-22 16:04:04 -07:00
45c928e2f5
boot: checkPortAvailability() should fail only for EADDRINUSE error and ignore other errors. ( #2527 )
...
fixes #2510
2016-08-22 10:20:01 -07:00
Karthic Rao
Aditya Manthramurthy
Harshavardhana
Krishnan Parthasarathi
Anis Elleuch
Mateusz Gajewski
Frank
Krishna Srinivas
Bala FA
Krishna Srinivas
awwalker
Krishnan Parthasarathi
Kevin Qiu
Brendan Ashworth
Mohit Agarwal