Commit Graph

7243 Commits

Author SHA1 Message Date
Benjamin Sodenkamp c00d410e61
Added bucket name param to ToJSONError call (#9961)
when called with InvalidBucketName error.
The user is shown a more specific error
when the param is present.
2020-07-10 12:10:39 -07:00
Klaus Post 968342c732
Remove usage of go-ieproxy for windows (#10009)
There is a potential for deadlock on Windows 10
refer https://github.com/mattn/go-ieproxy/issues/17 

remove this dependency for now.
2020-07-10 12:08:14 -07:00
Harshavardhana 5c15656c55
support bootstrap client to use healthcheck restClient (#10004)
- reduce locker timeout for early transaction lock
  for more eagerness to timeout
- reduce leader lock timeout to range from 30sec to 1minute
- add additional log message during bootstrap phase
2020-07-10 09:26:21 -07:00
Harshavardhana 2e8fc6ebfe
cleanup STS docs (#10003) 2020-07-10 09:07:12 -07:00
kannappanr efe9fe6124
azure: Return success when deleting non-existent object (#9981) 2020-07-10 08:30:23 -07:00
Nitish Tiwari 30c251efd3
Add Grafana dashboard (#10000) 2020-07-09 12:01:58 -07:00
Klaus Post c850905e43
fix: threadwalk lockup under high load (#9992)
Main issue is that `t.pool[params]` should be `t.pool[oldest]`.

We add a bit more safety features for the code.

* Make writes to the endTimerCh non-blocking in all cases
   so multiple releases cannot lock up.
* Double check expectations.
* Shift down deletes with copy instead of truncating slice.
* Actually delete the oldest if we are above total limit.
* Actually delete the oldest found and not the current.
* Unexport the mutex so nobody from the outside can meddle with it.
2020-07-09 07:02:18 -07:00
Andreas Auernhammer a317a2531c
admin: new API for creating KMS master keys (#9982)
This commit adds a new admin API for creating master keys.
An admin client can send a POST request to:
```
/minio/admin/v3/kms/key/create?key-id=<keyID>
```

The name / ID of the new key is specified as request
query parameter `key-id=<ID>`.

Creating new master keys requires KES - it does not work with
the native Vault KMS (deprecated) nor with a static master key
(deprecated).

Further, this commit removes the `UpdateKey` method from the `KMS`
interface. This method is not needed and not used anymore.
2020-07-08 18:50:43 -07:00
Ravind Kumar ee20ebe07a Remove dead link related to DC/OS Deployment Guide (#9996) 2020-07-08 17:38:20 -07:00
Harshavardhana 2743d4ca87
fix: Add support for preserving mtime for replication (#9995)
This PR is needed for bucket replication support
2020-07-08 17:36:56 -07:00
Harshavardhana 6136a963c8
fix: bump the response header timeout for forwarder as well (#9994)
continuation of #9986, add more place where the lower timeout
comes into effect.
2020-07-08 10:55:24 -07:00
Harshavardhana 60417950c7
fix: the versioning/object lock documentation appropriately (#9988)
- Move the bucket level features into `docs/bucket` directory
- fix issue template and simplify some of them
2020-07-08 08:44:43 -07:00
Anis Elleuch fa211f6a10
heal: Fix healing delete markers (#9989) 2020-07-07 20:54:09 -07:00
Harshavardhana 72e0745e2f
fix: migrate to go.etcd.io import path (#9987)
with the merge of https://github.com/etcd-io/etcd/pull/11823
etcd v3.5.0 will now have a properly imported versioned path

this fixes our pending migration to newer repo
2020-07-07 19:04:29 -07:00
Klaus Post aa4d1021eb
Remove timeout from putobject and listobjects (#9986)
Use a separate client for these calls that can take a long time.

Add request context to these so they are canceled when the client 
disconnects instead except for ListObject which doesn't have any equivalent.
2020-07-07 12:19:57 -07:00
Harshavardhana 93e7e4a0e5
fix: cors handling after gorilla mux update (#9980)
fixes #9979
2020-07-06 20:55:19 -07:00
Anis Elleuch c2f7cd1104
Consider errFileVersionNotFound during healing assessment (#9977)
Healing an object which has multiple versions was not working because
the healing code forgot to consider errFileVersionNotFound error as a
use case that needs healing
2020-07-06 08:09:48 -07:00
Harshavardhana 38eef5ce4c
fix: documentation fixes for docker ENV settings (#9975)
- update CREDITS file
- fix markdown links
- talk a bit more about upgrades
2020-07-06 06:42:34 -07:00
Anis Elleuch 4cf80f96ad
fix: lifecycle XML parsing errors with Versioning (#9974) 2020-07-05 09:08:42 -07:00
Anis Elleuch d4af132fc4
lifecycle: Expiry should not delete versions (#9972)
Currently, lifecycle expiry is deleting all object versions which is not
correct, unless noncurrent versions field is specified.

Also, only delete the delete marker if it is the only version of the
given object.
2020-07-04 20:56:02 -07:00
Harshavardhana c087a05b43
fix: simplify data structure before release (#9968)
- additionally upgrade to msgp@v1.1.2
- change StatModTime,StatSize fields as
  simple Size/ModTime
- reduce 50000 entries per List batch to 10000
  as client needs to wait too long to see the
  first batch some times which is not desired
  and it is worth we write the data as soon
  as we have it.
2020-07-04 12:25:53 -07:00
Harshavardhana cdb0e6ffed
support proper values for listMultipartUploads/listParts (#9970)
object KMS is configured with auto-encryption,
there were issues when using docker registry -
this has been left unnoticed for a while.

This PR fixes an issue with compatibility.

Additionally also fix the continuation-token implementation
infinite loop issue which was missed as part of #9939

Also fix the heal token to be generated as a client
facing value instead of what is remembered by the
server, this allows for the server to be stateless 
regarding the token's behavior.
2020-07-03 19:27:13 -07:00
Harshavardhana 03b84091fc
auto enable versioning with object locking (#9967)
this is to preserve versioning for object-locked
buckets from current release code.
2020-07-03 15:30:06 -07:00
Anis Elleuch 2be20588bf
Reroute requests based token heal/listing (#9939)
When manual healing is triggered, one node in a cluster will 
become the authority to heal. mc regularly sends new requests 
to fetch the status of the ongoing healing process, but a load 
balancer could land the healing request to a node that is not 
doing the healing request.

This PR will redirect a request to the node based on the node 
index found described as part of the client token. A similar
technique is also used to proxy ListObjectsV2 requests
by encoding this information in continuation-token
2020-07-03 11:53:03 -07:00
Harshavardhana e59ee14f40
Tune tcp keep-alives with new kernel timeout options (#9963)
For more deeper understanding https://blog.cloudflare.com/when-tcp-sockets-refuse-to-die/
2020-07-03 10:03:41 -07:00
Anis Elleuch 21a37e3393
fix: ListObjectVersions should return ordered Version & DeleteMarker (#9959)
The S3 specification says that versions are ordered in the response of
list object versions.

mc snapshot needs this to know which version comes first especially when
two versions have the same exact last-modified field.
2020-07-03 09:15:44 -07:00
Harshavardhana 810a4f0723
fix: return proper errors Get/HeadObject for deleteMarkers (#9957) 2020-07-02 16:17:27 -07:00
Krishna Srinivas 4c266df863
fix: proxy ListObjects request to one of the server based on hash(bucket) (#9881) 2020-07-02 10:56:22 -07:00
Klaus Post abd999f64a
fix: list object versions in distributed setup (#9958)
Remove calls to `WalkVersions` was calling the wrong endpoint, 
so unless quorum could be reached with local disks no results 
would ever be returned.
2020-07-02 10:29:50 -07:00
Minio Trusted 04de19c870 Update yaml files to latest version RELEASE.2020-07-02T00-15-09Z 2020-07-01 17:36:52 -07:00
Benjamin Sodenkamp 648cb13e02
Added 'close' to results channel in Walk() (#9956) 2020-07-01 14:29:45 -07:00
Harshavardhana 174f428571
add additional fdatasync before close() on writes (#9947) 2020-07-01 10:57:23 -07:00
Harshavardhana 5388ae4acb
make sure to delete data-usage cache upon bucket deletes (#9952) 2020-07-01 10:55:28 -07:00
Klaus Post 2e338e84cb
fix owhanging crashes in parquet S3 select (#9921) 2020-07-01 08:15:41 -07:00
kannappanr 5089a7167d
Handle empty retention in get/put object retention (#9948)
Fixes #9943
2020-06-30 16:44:24 -07:00
Harshavardhana c0ac25bfff
fix: readiness needs to be like liveness (#9941)
Readiness as no reasoning to be cluster scope
because that is not how the k8s networking works
for pods, all the pods to a deployment are not
sharing the network in a singleton. Instead they
are run as local scopes to themselves, with
readiness failures the pod is potentially taken
out of the network to be resolvable - this
affects the distributed setup in myriad of
different ways.

Instead readiness should behave like liveness
with local scope alone, and should be a dummy
implementation.

This PR all the startup times and overal k8s
startup time dramatically improves.

Added another handler called as `/minio/health/cluster`
to understand the cluster scope health.
2020-06-30 11:28:27 -07:00
Klaus Post 27a1f3ed2b
fs: Check if cache root was added (#9945)
Fixes #9942
2020-06-30 09:32:36 -07:00
Anis Elleuch 90f36c1389
Update documentation for Accounting API (#9909) 2020-06-30 08:34:08 -07:00
Harshavardhana 91817d0d1a
fix: implement generic Walk for gateway (#9938)
Walk() functionality was missing on gateway
implementations leading to missing functionality
for the browser UI such as remove multiple objects,
download as zip file etc.

This PR brings a generic implementation across
all gateway's, it is not required to repeat the
same code in all gateway's
2020-06-29 17:07:23 -07:00
poornas 55a3b071ea
Allow optionally to disable range caching. (#9908)
The default behavior is to cache each range requested
to cache drive. Add an environment variable
`MINIO_RANGE_CACHE` - when set to off, it disables
range caching and instead downloads entire object
in the background.

Fixes #9870
2020-06-29 13:25:29 -07:00
Harshavardhana a38ce29137
fix: simplify background heal and trigger heal items early (#9928)
Bonus fix during versioning merge one of the PR was missing
the offline/online disk count fix from #9801 port it correctly
over to the master branch from release.

Additionally, add versionID support for MRF

Fixes #9910
Fixes #9931
2020-06-29 13:07:26 -07:00
Anis Elleuch 7cea3f7da4
madmin: Strip 80/443 from the endpoint when http/https (#9937)
Users having endpoints with this format http://url:80 or http://url:443
will face signature mismatch error.

The reason is that  S3 spec ignores :80 or :443 port in the endpoint
when calculating the signature, so this PR will just strip them.
2020-06-29 12:31:07 -07:00
Harshavardhana dcffd87e08
upgrade docker images to alpine 3.12 (#9934) 2020-06-29 09:36:29 +05:30
Harshavardhana 3b813148b3
update gorilla deps for query parsing performance improvements (#9929) 2020-06-28 11:37:56 -07:00
Harshavardhana 4bba2cd034
fix: disallow versioning to be suspended with object lock (#9930) 2020-06-28 08:15:15 -07:00
Harshavardhana 28a1a17187
docs update versioning images (#9925) 2020-06-27 10:27:23 -07:00
Eco a5e879fbe4
doc: typo fix in versioning README (#9926)
requets -> requests
2020-06-27 05:06:31 -07:00
Harshavardhana f7f12b8604
fix: crash in storage rest client due to spurious query params (#9924)
regression got introduced in dee3cf2d7f
when the DeleteVersion API was changed, but the corresponding query
params were left in-tact.
2020-06-26 16:49:49 -07:00
Praveen raj Mani cf5d051afc
update notification rulesMap when reloading bucketMetadata (#9917) 2020-06-26 13:17:31 -07:00
Harshavardhana 2f681bed57
fix: pop entries from each drives in parallel (#9918) 2020-06-25 23:20:12 -07:00