Commit Graph

1559 Commits

Author SHA1 Message Date
Krishnan Parthasarathi
76b6dc0112
Add licverifier package (#10237)
license verification package implements a simple library to 
verify MinIO Subnet license keys.
2020-08-10 13:30:12 -07:00
Harshavardhana
1e2ebc9945
feat: time to bring back http2.0 support (#10230)
Bonus move our CI/CD to go1.14
2020-08-10 09:02:29 -07:00
Harshavardhana
6c6137b2e7
add cluster maintenance healthcheck drive heal affinity (#10218) 2020-08-07 13:22:53 -07:00
Anis Elleuch
433c2831ae
fix: typo in parsing non remote env variables (#10223) 2020-08-07 09:57:20 -07:00
Harshavardhana
77509ce391
Support looking up environment remotely (#10215)
adds a feature where we can fetch the MinIO
command-line remotely, this
is primarily meant to add some stateless
nature to the MinIO deployment in k8s
environments, MinIO operator would run a
webhook service endpoint
which can be used to fetch any environment
value in a generalized approach.
2020-08-06 18:03:16 -07:00
poornas
adcaa6f9de
fix: Change ListBucketTargets handler (#10217)
to list all targets across a tenant.
Also fixing some validations.
2020-08-06 17:10:21 -07:00
poornas
121164db56
fix: relax some replication validations (#10210)
Also inherit storage class from source object
if replication configuration does not have a storage
class specified for destination bucket.
2020-08-05 20:01:20 -07:00
poornas
3acc0ebb81
fix: Change service name in Arn for replication (#10205) 2020-08-05 00:43:18 -07:00
Harshavardhana
d61eac080b
fix: connection_string should override other params (#10180)
closes #9965
2020-08-03 09:16:00 -07:00
poornas
a8dd7b3eda
Refactor replication target management. (#10154)
Generalize replication target management so
that remote targets for a bucket can be
managed with ARNs. `mc admin bucket remote`
command will be used to manage targets.
2020-07-30 19:55:22 -07:00
Harshavardhana
fe157166ca
fix: Pass context all the way down to the network call in lockers (#10161)
Context timeout might race on each other when timeouts are lower
i.e when two lock attempts happened very quickly on the same resource
and the servers were yet trying to establish quorum.

This situation can lead to locks held which wouldn't be unlocked
and subsequent lock attempts would fail.

This would require a complete server restart. A potential of this
issue happening is when server is booting up and we are trying
to hold a 'transaction.lock' in quick bursts of timeout.
2020-07-29 23:15:34 -07:00
poornas
b46ab7e921
Rename replication target handler (#10142)
Rename replication target handler to a generic bucket target handler
2020-07-28 11:50:47 -07:00
Harshavardhana
f200a7fb6a
fix: speed up OBD tests avoid unnecessary memory allocation (#10141)
replace dummy buffer with nullReader{} instead,
to avoid large memory allocations in memory
constrainted environments. allows running
obd tests in such environments.
2020-07-27 14:51:59 -07:00
Praveen raj Mani
b800541fbe
fix: a type in NSQ notification target environment key (#10118)
fixes #10100
2020-07-23 12:19:36 -07:00
Anis Elleuch
1340281cb8
Fix marshaling expiration field in lifecycle (#10117) 2020-07-23 08:01:25 -07:00
Anis Elleuch
456b2ef6eb
Avoid healing to be stuck with many concurrent event listeners (#10111)
If there are many listeners to bucket notifications or to the trace
subsystem, healing fails to work properly since it suspends itself when
the number of concurrent connections is above a certain threshold.

These connections are also continuous and not costly (*no disk access*),
it is okay to just ignore them in waitForLowHTTPReq().
2020-07-22 13:16:55 -07:00
poornas
c43da3005a
Add support for server side bucket replication (#9882) 2020-07-21 17:49:56 -07:00
Bruce Wang
e464a5bfbc
Fix bug with fields that contain trimming spaces (#10079)
String x might contain trimming spaces. And it needs to be trimmed. For
example, in csv files, there might be trimming spaces in a field that
ought to meet a query condition that contains the value without
trimming spaces. This applies to both intCast and floatCast functions.
2020-07-21 12:57:09 -07:00
findmyname666
f9648d3976
add tests lifecycle rules with empty prefix (#10093) 2020-07-20 13:12:50 -07:00
Harshavardhana
2955aae8e4
feat: Add notification support for bucketCreates and removal (#10075) 2020-07-20 12:52:49 -07:00
findmyname666
aa6468932b
make sure lifecycle rule ID is present (#10084) 2020-07-19 15:10:05 -07:00
Anis Elleuch
44c8af66ad
fs: Fix expiry regression after versioning refactor (#10083)
Do not ignore non-versioned objects in lifecycle compute 
action function.
2020-07-18 15:43:13 -07:00
Harshavardhana
14b1c9f8e4
fix: return Range errors after If-Matches (#10045)
closes #7292
2020-07-17 13:01:22 -07:00
Anis Elleuch
4a447a439a
Fix lifecycle rules not applied in some cases (#10072)
HasActiveRules was not behaving as expected, this commit fixes it
and adds more unit tests.
2020-07-17 09:48:00 -07:00
findmyname666
48aebf2d9d
allow lifecycle rules with overlapping prefixes (#10053) 2020-07-16 07:39:41 -07:00
Harshavardhana
9c66812b99
Add missing action stringer for DeleteVersionAction (#10049) 2020-07-15 17:28:49 +05:30
Harshavardhana
cfc5681b36 fix: policy_test to use minio-go/v7 2020-07-14 10:30:00 -07:00
Harshavardhana
369a876ebe
fix: handle array policies in JWT claim (#10041)
PR #10014 was not complete as only handled
policy claims partially.
2020-07-14 10:26:47 -07:00
Anis Elleuch
778e9c864f
Move dependency from minio-go v6 to v7 (#10042) 2020-07-14 09:38:05 -07:00
Harshavardhana
e7d7d5232c
fix: admin info output and improve overall performance (#10015)
- admin info node offline check is now quicker
- admin info now doesn't duplicate the code
  across doing the same checks for disks
- rely on StorageInfo to return appropriate errors
  instead of calling locally.
- diskID checks now return proper errors when
  disk not found v/s format.json missing.
- add more disk states for more clarity on the
  underlying disk errors.
2020-07-13 09:51:07 -07:00
Harshavardhana
3b9fbf80ad
fix: make sure to use new restClient for healthcheck (#10026)
Without instantiating a new rest client we can
have a recursive error which can lead to
healthcheck returning always offline, this can
prematurely take the servers offline.
2020-07-11 22:19:38 -07:00
Harshavardhana
ba756cf366
fix: extract array type for policy claim if present (#10014) 2020-07-10 14:48:44 -07:00
Klaus Post
968342c732
Remove usage of go-ieproxy for windows (#10009)
There is a potential for deadlock on Windows 10
refer https://github.com/mattn/go-ieproxy/issues/17 

remove this dependency for now.
2020-07-10 12:08:14 -07:00
Andreas Auernhammer
a317a2531c
admin: new API for creating KMS master keys (#9982)
This commit adds a new admin API for creating master keys.
An admin client can send a POST request to:
```
/minio/admin/v3/kms/key/create?key-id=<keyID>
```

The name / ID of the new key is specified as request
query parameter `key-id=<ID>`.

Creating new master keys requires KES - it does not work with
the native Vault KMS (deprecated) nor with a static master key
(deprecated).

Further, this commit removes the `UpdateKey` method from the `KMS`
interface. This method is not needed and not used anymore.
2020-07-08 18:50:43 -07:00
Harshavardhana
72e0745e2f
fix: migrate to go.etcd.io import path (#9987)
with the merge of https://github.com/etcd-io/etcd/pull/11823
etcd v3.5.0 will now have a properly imported versioned path

this fixes our pending migration to newer repo
2020-07-07 19:04:29 -07:00
Anis Elleuch
4cf80f96ad
fix: lifecycle XML parsing errors with Versioning (#9974) 2020-07-05 09:08:42 -07:00
Anis Elleuch
d4af132fc4
lifecycle: Expiry should not delete versions (#9972)
Currently, lifecycle expiry is deleting all object versions which is not
correct, unless noncurrent versions field is specified.

Also, only delete the delete marker if it is the only version of the
given object.
2020-07-04 20:56:02 -07:00
Anis Elleuch
2be20588bf
Reroute requests based token heal/listing (#9939)
When manual healing is triggered, one node in a cluster will 
become the authority to heal. mc regularly sends new requests 
to fetch the status of the ongoing healing process, but a load 
balancer could land the healing request to a node that is not 
doing the healing request.

This PR will redirect a request to the node based on the node 
index found described as part of the client token. A similar
technique is also used to proxy ListObjectsV2 requests
by encoding this information in continuation-token
2020-07-03 11:53:03 -07:00
Harshavardhana
174f428571
add additional fdatasync before close() on writes (#9947) 2020-07-01 10:57:23 -07:00
Klaus Post
2e338e84cb
fix owhanging crashes in parquet S3 select (#9921) 2020-07-01 08:15:41 -07:00
kannappanr
5089a7167d
Handle empty retention in get/put object retention (#9948)
Fixes #9943
2020-06-30 16:44:24 -07:00
Anis Elleuch
90f36c1389
Update documentation for Accounting API (#9909) 2020-06-30 08:34:08 -07:00
Harshavardhana
a38ce29137
fix: simplify background heal and trigger heal items early (#9928)
Bonus fix during versioning merge one of the PR was missing
the offline/online disk count fix from #9801 port it correctly
over to the master branch from release.

Additionally, add versionID support for MRF

Fixes #9910
Fixes #9931
2020-06-29 13:07:26 -07:00
Anis Elleuch
7cea3f7da4
madmin: Strip 80/443 from the endpoint when http/https (#9937)
Users having endpoints with this format http://url:80 or http://url:443
will face signature mismatch error.

The reason is that  S3 spec ignores :80 or :443 port in the endpoint
when calculating the signature, so this PR will just strip them.
2020-06-29 12:31:07 -07:00
Klaus Post
2d0f65a5e3
Add archived parquet as int. package (#9912)
Since github.com/minio/parquet-go is archived add it as internal package.
2020-06-25 07:31:16 -07:00
kannappanr
b460b5967f
fix IAM policy action name for Get/PutBucketLifeCycle (#9893) 2020-06-23 10:18:32 -07:00
Harshavardhana
b8cb21c954
allow more than N number of locks in TopLocks (#9883) 2020-06-20 06:33:01 -07:00
Michael Mayr
53f0cc1340
Implement CLIENT SETNAME for Redis connections (#9876)
Add note about CLIENT SETNAME needing auth
2020-06-19 13:28:28 -07:00
Harshavardhana
7c061fa3b6
on darwin fallback to maximum possible rlimit (#9859)
fixes #9857
2020-06-17 07:47:42 -07:00
Harshavardhana
d55f4336ae
preserve context per request for local locks (#9828)
In the Current bug we were re-using the context
from previously granted lockers, this would
lead to lock timeouts for existing valid
read or write locks, leading to premature
timeout of locks.

This bug affects only local lockers in FS
or standalone erasure coded mode. This issue
is rather historical as well and was present
in lsync for some time but we were lucky to
not see it.

Similar changes are done in dsync as well
to keep the code more familiar

Fixes #9827
2020-06-14 07:43:10 -07:00