Commit Graph

1394 Commits

Author SHA1 Message Date
Anis Elleuch
7432b5c9b2
Use user CAs in checkEndpoint() call (#8911)
The server info handler makes a http connection to other
nodes to check if they are up but does not load the custom
CAs in ~/.minio/certs/CAs.

This commit fix it.

Co-authored-by: Harshavardhana <harsha@minio.io>
2020-02-02 07:15:29 +05:30
poornas
881e983ed9
Fix Retention, ObjectLock, LegalHold struct namespaces correctly. (#8909)
Reverts #8903 to allow structs to be unmarshalled 
even if the namespace is missing.
2020-01-30 09:58:05 +05:30
Ashish Kumar Sinha
5bd0e95eef
Set default namespace for necessary structs (#8903) 2020-01-29 10:19:38 +05:30
Harshavardhana
0cbebf0f57 Rename pkg/{tagging,lifecycle} to pkg/bucket sub-directory (#8892)
Rename to allow for more such features to come in a more
proper hierarchical manner.
2020-01-27 14:12:34 -08:00
poornas
2232e095d5 Make admin permissions more granular for admin handlers. (#8888) 2020-01-26 20:47:52 -06:00
Klaus Post
c7178d2066 Profiling: Add base, fix memory profiling (#8850)
For 'snapshot' type profiles, record a 'before' profile that can be used 
as `go tool pprof -base=before ...` to compare before and after.

"Before" profiles are included in the zipped package.

[`runtime.MemProfileRate`](https://golang.org/pkg/runtime/#pkg-variables) 
should not be updated while the application is running, so we set it at startup.

Co-authored-by: Harshavardhana <harsha@minio.io>
2020-01-21 15:49:25 -08:00
Harshavardhana
e2b3c083aa
fix: close and drain the response body always (#8847) 2020-01-21 02:46:58 -08:00
Nitish Tiwari
61c17c8933 Add ObjectTagging Support (#8754)
This PR adds support for AWS S3 ObjectTagging API as explained here
https://docs.aws.amazon.com/AmazonS3/latest/dev/object-tagging.html
2020-01-20 08:45:59 -08:00
poornas
60e60f68dd Add support for object locking with legal hold. (#8634) 2020-01-16 15:41:56 -08:00
Harshavardhana
5aa5dcdc6d
lock: improve locker initialization at init (#8776)
Use reference format to initialize lockers
during startup, also handle `nil` for NetLocker
in dsync and remove *errorLocker* implementation

Add further tuning parameters such as

 - DialTimeout is now 15 seconds from 30 seconds
 - KeepAliveTimeout is not 20 seconds, 5 seconds
   more than default 15 seconds
 - ResponseHeaderTimeout to 10 seconds
 - ExpectContinueTimeout is reduced to 3 seconds
 - DualStack is enabled by default remove setting
   it to `true`
 - Reduce IdleConnTimeout to 30 seconds from
   1 minute to avoid idleConn build up

Fixes #8773
2020-01-10 02:35:06 -08:00
Harshavardhana
0a70bc24ac
Disallow only policy statements which are exactly same (#8785) 2020-01-09 19:29:57 -08:00
Praveen raj Mani
4cd1bbb50a This PR fixes two things (#8772)
- Stop spawning store replay routines when testing the notification targets
- Properly honor the target.Close() to clean the resources used

Fixes #8707

Co-authored-by: Harshavardhana <harsha@minio.io>
2020-01-09 19:45:44 +05:30
Harshavardhana
c2cde6beb5
policy: Allow duplicate statements with different effects (#8775)
This allows "Allow" and "Deny" conflicting statements,
where we evaluate to implicit "Deny".
2020-01-08 23:00:54 -08:00
Harshavardhana
abc1c1070a Add custom policy claim name (#8764)
In certain organizations policy claim names
can be not just 'policy' but also things like
'roles', the value of this field might also
be *string* or *[]string* support this as well

In this PR we are still not supporting multiple
policies per STS account which will require a
more comprehensive change.
2020-01-08 17:21:58 -08:00
Harshavardhana
60813bef29
Allow proper setCount SLAs across zones (#8752)
Fixes scenario where zones are appropriately
handled, along with supporting overriding set
count. The new fix also ensures that we handle
the various setup types properly.

Update documentation to properly indicate the
behavior.

Fixes #8750

Co-authored-by: Nitish Tiwari <nitish@minio.io>
2020-01-07 09:13:44 -08:00
Harshavardhana
933c60bc3a Add crypto context errors (#8740)
Currently when connections to vault fail, client
perpetually retries this leads to assumptions that
the server has issues and masks the problem.

Re-purpose *crypto.Error* type to send appropriate
errors back to the client.
2020-01-06 16:15:22 -08:00
Harshavardhana
6695fd6a61
Add more context aware error for policy parsing errors (#8726)
In existing functionality we simply return a generic
error such as "MalformedPolicy" which indicates just
a generic string "invalid resource" which is not very
meaningful when there might be multiple types of errors
during policy parsing. This PR ensures that we send
these errors back to client to indicate the actual
error, brings in two concrete types such as

 - iampolicy.Error
 - policy.Error

Refer #8202
2020-01-03 11:28:52 -08:00
Harshavardhana
0b7bd024fb Fix dependencies graph for minio source compilation (#8717)
We had messy cyclical dependency problem with `mc`
due to dependencies in pkg/console, moved the pkg/console
to minio for more control and also to avoid any further
cyclical dependencies of `mc` clobbering up the
dependencies on server.

Fixes #8659
2019-12-31 09:36:13 +05:30
Haines Chan
b2044dd22f Fix typo in comments (#8704) 2019-12-27 15:51:32 +05:30
kannappanr
cd59a945d8
Fix: Policy action name for Get/Put BucketLifeCycle (#8698)
S3:GetBucketLifeCycle should be changed to S3:GetLifeCycleConfiguration
S3:PutBucketLifeCycle should be changed to S3:PutLifeCycleConfiguration
2019-12-25 22:04:46 -08:00
Bruce Wang
c476b27a65 Comment typo "index max" to "index map" (#8700) 2019-12-24 21:57:43 -08:00
Harshavardhana
586614c73f fix: temp credentials shouldn't allow policy/group changes (#8675)
This PR fixes the issue where we might allow policy changes
for temporary credentials out of band, this situation allows
privilege escalation for those temporary credentials. We
should disallow any external actions on temporary creds
as a practice and we should clearly differentiate which
are static and which are temporary credentials.

Refer #8667
2019-12-19 14:21:21 -08:00
Harshavardhana
471a3a650a
fix: Don't allow to set unconfigured notification ARNs (#8643)
Fixes #8642
2019-12-13 12:36:45 -08:00
Ashish Kumar Sinha
abc266caa1 Add bucket and object count along with total object size (#8639) 2019-12-12 09:58:59 -08:00
Harshavardhana
c364f0af6c Start using custom HTTP transport for webhook endpoints (#8630)
Use a more performant http transport for webhook
endpoints with proper connection pooling, appropriate
timeouts etc.
2019-12-12 06:53:50 -08:00
Anis Elleuch
555969ee42 Add data usage collect with its new admin API (#8553)
Admin data usage info API returns the following

(Only FS & XL, for now)

- Number of buckets
- Number of objects
- The total size of objects
- Objects histogram
- Bucket sizes
2019-12-12 06:02:37 -08:00
Ashish Kumar Sinha
e2c5d29017 Bucket,Object count & Usage removed if set to default (#8638) 2019-12-11 21:56:47 -08:00
kannappanr
d266b3a066
Admin Info: Modify Uptime to return seconds (#8635) 2019-12-11 17:56:02 -08:00
Ashish Kumar Sinha
24fb1bf258 New Admin Info (#8497) 2019-12-11 14:27:03 -08:00
Klaus Post
bf3a97d3aa S3 Select: Concurrent LINES delimited json parsing (#8610)
The speedup is ~5x on a 6 core CPU
2019-12-09 06:55:31 -08:00
Harshavardhana
476111968a Update help messages with new wording (#8616)
Final update to all messages across sub-systems
after final review, the only change here is that
NATS now has TLS and TLSSkipVerify to be consistent
for all other notification targets.
2019-12-06 13:53:51 -08:00
Klaus Post
f1e2e1cc9e S3 Select: Mismatched types don't match (#8608)
When comparing for equality, if types cannot be matched, they don't match.
2019-12-06 07:24:41 -08:00
Aleksandr Petruhin
d2dc964cb5 Support TLS auth for Kafka notification target (#8609) 2019-12-05 15:31:46 -08:00
Harshavardhana
d8e3de0cae Ensure comment is always a valid key (#8604)
Also fix LDAP leaky connection
2019-12-05 18:17:42 +05:30
Harshavardhana
c9940d8c3f Final changes to config sub-system (#8600)
- Introduces changes such as certain types of
  errors that can be ignored or which need to 
  go into safe mode.
- Update help text as per the review
2019-12-04 15:32:37 -08:00
Harshavardhana
794eb54da8 Export command prints turned-off sub-sys as comments (#8594)
This PR also tries to

- Preserve the order of keys printed in export command
- Fix cache to be enabled with _STATE env to keep
  backward compatibility
2019-12-03 10:50:20 -08:00
Harshavardhana
5d3d57c12a
Start using error wrapping with fmt.Errorf (#8588)
Use fatih/errwrap to fix all the code to use
error wrapping with fmt.Errorf()
2019-12-02 09:28:01 -08:00
Harshavardhana
b21835f195 Honor DurationSeconds properly for WebIdentity (#8581)
Also cleanup code to add various constants for
verbatim strings across the code base.

Fixes #8482
2019-11-29 18:57:54 +05:30
Klaus Post
87443af49e Cache Windows mount point info (#8572)
It seems like every upload creates a new StorageInfo object.

This makes it very sensitive to slow code paths.

A particular slow function is `IsLikelyMountPoint` on Windows since `windows.GetVolumePathName` is slow.

We cache the result of this operation.

Before:

```
Operation: PUT - Concurrency: 12
* Average: 70.46 MB/s, 70.46 obj/s, 70.46 ops ended/s (59.58s)

Aggregated, split into 59 x 1s time segments:
* Fastest: 77.67 MB/s, 77.67 obj/s, 79.00 ops ended/s (1s)
* 50% Median: 70.84 MB/s, 70.84 obj/s, 70.00 ops ended/s (1s)
* Slowest: 61.51 MB/s, 61.51 obj/s, 61.00 ops ended/s (1s)
```

After:
```
Operation: PUT - Concurrency: 12
* Average: 166.80 MB/s, 166.80 obj/s, 166.74 ops ended/s (59.8s)

Aggregated, split into 59 x 1s time segments:
* Fastest: 183.83 MB/s, 183.83 obj/s, 186.00 ops ended/s (1s)
* 50% Median: 167.47 MB/s, 167.47 obj/s, 166.00 ops ended/s (1s)
* Slowest: 139.30 MB/s, 139.30 obj/s, 141.00 ops ended/s (1s)
```
2019-11-27 20:10:13 +05:30
Harshavardhana
5d65428b29
Handle localhost distributed setups properly (#8577)
Fixes an issue reported by @klauspost and @vadmeste

This PR also allows users to expand their clusters
from single node XL deployment to distributed mode.
2019-11-26 11:42:10 -08:00
Harshavardhana
720442b1a2
Add lock expiry handler to expire state locks (#8562) 2019-11-25 16:39:43 -08:00
Harshavardhana
e542084c37
Add etcd path prefix for all IAM assets (#8569)
Currently, we use the top-level prefix "config/"
for all our IAM assets, instead of to provide
tenant-level separation bring 'path_prefix'
to namespace the access properly.

Fixes #8567
2019-11-25 16:33:34 -08:00
Harshavardhana
4e9de58675 Avoid pointer based copy, instead use Clone() (#8547)
This PR adds functional test to test expanded
cluster syntax.
2019-11-21 17:54:51 +05:30
poornas
a9b87c0a16 Fix typo in event name (#8545) 2019-11-20 16:35:01 -08:00
Harshavardhana
5ac4b517c9
Order all keys in config (#8541)
New changes

- return default values when sub-sys is
  not configured.
- state is hidden parameter now
- remove worm mode to be saved in config
2019-11-20 15:10:24 -08:00
poornas
ca96560d56 Add object retention at the per object (#8528)
level - this PR builds on #8120 which
added PutBucketObjectLockConfiguration and
GetBucketObjectLockConfiguration APIS

This PR implements PutObjectRetention,
GetObjectRetention API and enhances
PUT and GET API operations to display
governance metadata if permissions allow.
2019-11-20 13:18:09 -08:00
Harshavardhana
347b29d059 Implement bucket expansion (#8509) 2019-11-19 17:42:27 -08:00
Harshavardhana
7cdb67680e
Add help with order of keys (#8535) 2019-11-19 13:48:13 -08:00
poornas
929951fd49 Add support for multiple admins (#8487)
Also define IAM policies for administering
MinIO server
2019-11-19 02:03:18 -08:00
Harshavardhana
13a3d17321
Do not add comments after migration (#8530)
Also filter out empty comments from being
printed.
2019-11-16 14:57:36 -08:00