Commit Graph

6100 Commits

Author SHA1 Message Date
poornas
3adc311c1c Fix regression in admin router when no route matches (#8409)
When `mc admin user add` is attempted in gateway mode without
etcd setup, NoSuchBucket error is returned instead of MethodNotAllowed.
Regression from commit - e48005ddc7
2019-10-16 20:39:23 -07:00
Anis Elleuch
ee05280721 fs: Remove stale background append temporary file (#8404)
Background append creates a temporary file which appends
uploaded parts as long as they are available, but when a
client stops the upload, the temporary file is not removed
by any way.

This commit removes the temporary file when the server does
its regular removing stale multipart uploads.
2019-10-17 00:27:52 +05:30
poornas
c4e2af8ca3 Remove cache env from server help message (#8405) 2019-10-16 23:22:57 +05:30
Harshavardhana
5afb1b6747
Add support for {jwt:sub} substitutions for policies (#8393)
Fixes #8345
2019-10-16 08:59:59 -07:00
Harshavardhana
f2cc97a44c
Improve MQTT token registration retry (#8397) 2019-10-15 19:39:14 -07:00
Harshavardhana
d48fd6fde9
Remove unusued params and functions (#8399) 2019-10-15 18:35:41 -07:00
Andreas Auernhammer
b7ee0bbbc9 madmin: Migrate to provable secure channel construction (#8395)
This commit replaces the currently used channel construction
for en/decrypting config data with a provable secure scheme.
2019-10-15 02:36:04 +05:30
Harshavardhana
68a519a468
Use errgroups instead of sync.WaitGroup as needed (#8354) 2019-10-14 09:44:51 -07:00
Harshavardhana
c33bae057f Fix build issues with event target logger 2019-10-11 19:07:09 -07:00
poornas
d7060c4c32 Allow logging targets to be configured to receive minio (#8347)
specific errors, `application` errors or `all` by default.

console logging on server by default lists all logs -
enhance admin console API to accept `type` as query parameter to
subscribe to application/minio logs.
2019-10-11 18:50:54 -07:00
Minio Trusted
8964ef821f Update yaml files to latest version RELEASE.2019-10-12T01-39-57Z 2019-10-12 01:44:42 +00:00
Harshavardhana
bd10640846 Fix logger and audit http endpoint config lookup 2019-10-11 18:33:25 -07:00
Harshavardhana
175b07d6e4
Fix queueStore stops working with concurrent PUT/DELETE requests (#8381)
- This PR fixes situation to avoid underflow, this is possible
  because of disconnected operations in replay/sendEvents
- Hold right locks if Del() operation is performed in Get()
- Remove panic in the code and use loggerOnce
- Remove Timer and instead use Ticker instead for proper ticks
2019-10-11 17:46:03 -07:00
Harshavardhana
f16df2a4e7 Avoid crash in PopulatePublicKey() by re-initializing the args (#8388)
This is to avoid nil pointer dereference when method by pointer
reference and method by value reference are implemented.

Fixes #8387
2019-10-12 03:37:08 +05:30
Ashish Kumar Sinha
1c90485b56 Remove duplicate cpu hardware info (#8384) 2019-10-12 00:15:43 +05:30
Harshavardhana
2320a877bc
Add CREDITS license from all dependent projects (#8382) 2019-10-11 11:24:06 -07:00
Minio Trusted
ae752ed1fa Update yaml files to latest version RELEASE.2019-10-11T00-38-09Z 2019-10-11 00:45:13 +00:00
Aditya Manthramurthy
c566cc6b61 Fix errors in ldap doc (#8374) 2019-10-10 12:28:39 +05:30
Harshavardhana
36e12a6038 Assume local endpoints appropriately in k8s deployments (#8375)
On Kubernetes/Docker setups DNS resolves inappropriately
sometimes where there are situations same endpoints with
multiple disks come online indicating either one of them
is local and some of them are not local. This situation
can never happen and its only a possibility in orchestrated
deployments with dynamic DNS. Following code ensures that we
treat if one of the endpoint says its local for a given host
it is true for all endpoints for the same host. Following code
ensures that this assumption is true and it works in all
scenarios and it is safe to assume for a given host.

This PR also adds validation such that we do not crash the
server if there are bugs in the endpoints list in dsync
initialization.

Thanks to Daniel Valdivia <hola@danielvaldivia.com> for
reproducing this, this fix is needed as part of the
https://github.com/minio/m3 project.
2019-10-10 10:14:17 +05:30
Kaan Kabalak
42531db37e Update browser UI assets (#8373) 2019-10-10 08:22:47 +05:30
Harshavardhana
6a4ef2e48e Initialize configs correctly, move notification config (#8367)
This PR also removes deprecated tests, adds checks
to avoid races reproduced on CI/CD.
2019-10-09 11:41:15 +05:30
Harshavardhana
d2a8be6fc2 gateway/hdfs: Fix isObjectDir to behave correctly (#8368) 2019-10-09 04:20:43 +05:30
Harshavardhana
290ad0996f Move etcd, logger, crypto into their own packages (#8366)
- Deprecates _MINIO_PROFILER, `mc admin profile` does the job
- Move ENVs to common location in cmd/config/
2019-10-08 11:17:56 +05:30
Mark Felder
bffc378a4f Secret key field is not for creating a new password (#8362) 2019-10-07 10:55:50 -07:00
Harshavardhana
3b8adf7528 Move storageclass config handling into cmd/config/storageclass (#8360)
Continuation of the changes done in PR #8351 to refactor,
add tests and move global handling into a more idiomatic
style for Go as packages.
2019-10-07 11:20:24 +05:30
Klaus Post
002ac82631 S3 Select: Add parser support for lists. (#8329) 2019-10-06 07:52:45 -07:00
Harshavardhana
e85df07518 Add prometheus auth-type to turn-off authentication (#8356)
Also this PR moves the original doc from cookbook to
MinIO repo under docs/metrics/prometheus/

Fixes #8323
2019-10-04 23:48:59 +05:30
Harshavardhana
589e32a4ed Refactor config and split them in packages (#8351)
This change is related to larger config migration PR
change, this is a first stage change to move our
configs to `cmd/config/` - divided into its subsystems
2019-10-04 23:05:33 +05:30
Ashish Kumar Sinha
74008446fe CPU hardware info (#8187) 2019-10-03 20:18:38 +05:30
Praveen raj Mani
e48005ddc7 Add more context to rpc version mismatch errors (#8271)
Fixes #5665
2019-10-03 00:08:12 -07:00
Minio Trusted
83066f953c Update yaml files to latest version RELEASE.2019-10-02T21-19-38Z 2019-10-02 21:25:57 +00:00
Harshavardhana
90bfa6260a Fix LDAP TLS support to use custom CAs (#8352) 2019-10-03 01:44:57 +05:30
Harshavardhana
8b80eca184 List buckets only once per sub-system initialization (#8333)
Current master repeatedly calls ListBuckets() during
initialization of multiple sub-systems

Use single ListBuckets() call for each sub-system as
follows

- LifeCycle
- Policy
- Notification
2019-10-02 05:35:02 +05:30
Harshavardhana
fb1374f2f7 Rename iam/validator -> iam/openid and add tests (#8340)
Refactor as part of config migration
2019-10-02 03:37:20 +05:30
Harshavardhana
ff5bf51952 admin/heal: Fix deep healing to heal objects under more conditions (#8321)
- Heal if the part.1 is truncated from its original size
- Heal if the part.1 fails while being verified in between
- Heal if the part.1 fails while being at a certain offset

Other cleanups include make sure to flush the HTTP responses
properly from storage-rest-server, avoid using 'defer' to
improve call latency. 'defer' incurs latency avoid them
in our hot-paths such as storage-rest handlers.

Fixes #8319
2019-10-02 01:42:15 +05:30
Anis Elleuch
61927d228c listV2: Continuation and NextContinuation tokens are encoded with base64 (#8337)
Minio V2 listing uses object names/prefixes as continuation tokens. This
is problematic when object names contain some characters that are forbidden
in XML documents. This PR will use base64 encoded form of continuation
and next continuation tokens to address that corner case.
2019-10-02 01:39:29 +05:30
Kanagaraj M
82b9f2c931 remove yarn.lock from browser (#8348)
We already have package-lock.json and yarn.lock is not required
2019-10-01 12:19:40 -07:00
Yao Zongyou
6a19d7b25a skip checking error also on Mac in TestCheckPortAvailability (#8343) 2019-10-01 23:12:18 +05:30
Harshavardhana
ac2e0596bd Fix core-js build failure on node version 12.11.0 (#8341) 2019-10-01 23:11:31 +05:30
Ashish Kumar Sinha
42c821e164 Replace proxy with forward in coredns example (#8344) 2019-10-01 22:23:08 +05:30
Harshavardhana
20b907d8fb
Update hadoop docs to indicate new committers (#8060) 2019-09-30 21:34:06 -07:00
Harshavardhana
f45977d371
Fix error handling in DeleteFileBulk storage handler (#8327)
errors.errorString() cannot be marshalled by gob
encoder, so using a slice of []error would fail
to be encoded. This leads to no errors being
generated instead gob.Decoder on the storage-client
would see an io.EOF

To avoid such bugs introduce a typed error for
handling such translations and register this type
for gob encoding support.
2019-09-30 19:01:28 -07:00
Harshavardhana
4ec9b349d0
azure: Fix upload corruption with PutObject() on certain sizes (#8330)
On objects bigger than 100MiB can have a corrupted object
stored due to partial blockListing attempted right after
each blocks uploaded. Simplify this code to ensure that
all the blocks successfully uploaded are committed right
away.

This PR also updates the azure-sdk-go to latest release.
2019-09-30 18:42:18 -07:00
Kanagaraj M
65ac7c5671 add dynamic scrolling to bucket list in browser (#8336)
Fixes #8181
2019-09-30 18:14:51 -07:00
poornas
5c2af3f792 Add more context to error messages in STS handlers(#8304) 2019-10-01 02:35:19 +05:30
Andreas Auernhammer
8771e83545 Update klauspost/reedsolomon dependency to 1.9.3 (#8328)
This commit updates the reedsolomon dependency
since it contains an fix for an unexpected property
of the `Split` function.

See: klauspost/reedsolomon#109
2019-10-01 02:31:00 +05:30
Ashish Kumar Sinha
fa5a1cebd9 support space character in access key (#8335) 2019-10-01 02:25:37 +05:30
Ashish Kumar Sinha
4f981a0b42 Change YAML files to reference the newer APIs (#8299) 2019-09-30 00:04:55 -07:00
Harshavardhana
127641731a
Parallelize initialization of storageDisks (#8288) 2019-09-27 16:47:12 -07:00
Klaus Post
c1a17c2561 S3 Select: Aggregate AVG/SUM as float (#8326)
Force sum/average to be calculated as a float.

As noted in #8221

> run SELECT AVG(CAST (Score as int)) FROM S3Object on

```
Name,Score
alice,80
bob,81
```

> AWS S3 gives 80.5 and MinIO gives 80.

This also makes overflows much more unlikely.
2019-09-27 16:12:03 -07:00