Commit Graph

2087 Commits

Author SHA1 Message Date
poornas
3adc311c1c Fix regression in admin router when no route matches (#8409)
When `mc admin user add` is attempted in gateway mode without
etcd setup, NoSuchBucket error is returned instead of MethodNotAllowed.
Regression from commit - e48005ddc7
2019-10-16 20:39:23 -07:00
Anis Elleuch
ee05280721 fs: Remove stale background append temporary file (#8404)
Background append creates a temporary file which appends
uploaded parts as long as they are available, but when a
client stops the upload, the temporary file is not removed
by any way.

This commit removes the temporary file when the server does
its regular removing stale multipart uploads.
2019-10-17 00:27:52 +05:30
poornas
c4e2af8ca3 Remove cache env from server help message (#8405) 2019-10-16 23:22:57 +05:30
Harshavardhana
5afb1b6747
Add support for {jwt:sub} substitutions for policies (#8393)
Fixes #8345
2019-10-16 08:59:59 -07:00
Harshavardhana
f2cc97a44c
Improve MQTT token registration retry (#8397) 2019-10-15 19:39:14 -07:00
Harshavardhana
d48fd6fde9
Remove unusued params and functions (#8399) 2019-10-15 18:35:41 -07:00
Harshavardhana
68a519a468
Use errgroups instead of sync.WaitGroup as needed (#8354) 2019-10-14 09:44:51 -07:00
poornas
d7060c4c32 Allow logging targets to be configured to receive minio (#8347)
specific errors, `application` errors or `all` by default.

console logging on server by default lists all logs -
enhance admin console API to accept `type` as query parameter to
subscribe to application/minio logs.
2019-10-11 18:50:54 -07:00
Harshavardhana
bd10640846 Fix logger and audit http endpoint config lookup 2019-10-11 18:33:25 -07:00
Harshavardhana
175b07d6e4
Fix queueStore stops working with concurrent PUT/DELETE requests (#8381)
- This PR fixes situation to avoid underflow, this is possible
  because of disconnected operations in replay/sendEvents
- Hold right locks if Del() operation is performed in Get()
- Remove panic in the code and use loggerOnce
- Remove Timer and instead use Ticker instead for proper ticks
2019-10-11 17:46:03 -07:00
Ashish Kumar Sinha
1c90485b56 Remove duplicate cpu hardware info (#8384) 2019-10-12 00:15:43 +05:30
Harshavardhana
36e12a6038 Assume local endpoints appropriately in k8s deployments (#8375)
On Kubernetes/Docker setups DNS resolves inappropriately
sometimes where there are situations same endpoints with
multiple disks come online indicating either one of them
is local and some of them are not local. This situation
can never happen and its only a possibility in orchestrated
deployments with dynamic DNS. Following code ensures that we
treat if one of the endpoint says its local for a given host
it is true for all endpoints for the same host. Following code
ensures that this assumption is true and it works in all
scenarios and it is safe to assume for a given host.

This PR also adds validation such that we do not crash the
server if there are bugs in the endpoints list in dsync
initialization.

Thanks to Daniel Valdivia <hola@danielvaldivia.com> for
reproducing this, this fix is needed as part of the
https://github.com/minio/m3 project.
2019-10-10 10:14:17 +05:30
Harshavardhana
6a4ef2e48e Initialize configs correctly, move notification config (#8367)
This PR also removes deprecated tests, adds checks
to avoid races reproduced on CI/CD.
2019-10-09 11:41:15 +05:30
Harshavardhana
d2a8be6fc2 gateway/hdfs: Fix isObjectDir to behave correctly (#8368) 2019-10-09 04:20:43 +05:30
Harshavardhana
290ad0996f Move etcd, logger, crypto into their own packages (#8366)
- Deprecates _MINIO_PROFILER, `mc admin profile` does the job
- Move ENVs to common location in cmd/config/
2019-10-08 11:17:56 +05:30
Harshavardhana
3b8adf7528 Move storageclass config handling into cmd/config/storageclass (#8360)
Continuation of the changes done in PR #8351 to refactor,
add tests and move global handling into a more idiomatic
style for Go as packages.
2019-10-07 11:20:24 +05:30
Harshavardhana
e85df07518 Add prometheus auth-type to turn-off authentication (#8356)
Also this PR moves the original doc from cookbook to
MinIO repo under docs/metrics/prometheus/

Fixes #8323
2019-10-04 23:48:59 +05:30
Harshavardhana
589e32a4ed Refactor config and split them in packages (#8351)
This change is related to larger config migration PR
change, this is a first stage change to move our
configs to `cmd/config/` - divided into its subsystems
2019-10-04 23:05:33 +05:30
Ashish Kumar Sinha
74008446fe CPU hardware info (#8187) 2019-10-03 20:18:38 +05:30
Praveen raj Mani
e48005ddc7 Add more context to rpc version mismatch errors (#8271)
Fixes #5665
2019-10-03 00:08:12 -07:00
Harshavardhana
90bfa6260a Fix LDAP TLS support to use custom CAs (#8352) 2019-10-03 01:44:57 +05:30
Harshavardhana
8b80eca184 List buckets only once per sub-system initialization (#8333)
Current master repeatedly calls ListBuckets() during
initialization of multiple sub-systems

Use single ListBuckets() call for each sub-system as
follows

- LifeCycle
- Policy
- Notification
2019-10-02 05:35:02 +05:30
Harshavardhana
fb1374f2f7 Rename iam/validator -> iam/openid and add tests (#8340)
Refactor as part of config migration
2019-10-02 03:37:20 +05:30
Harshavardhana
ff5bf51952 admin/heal: Fix deep healing to heal objects under more conditions (#8321)
- Heal if the part.1 is truncated from its original size
- Heal if the part.1 fails while being verified in between
- Heal if the part.1 fails while being at a certain offset

Other cleanups include make sure to flush the HTTP responses
properly from storage-rest-server, avoid using 'defer' to
improve call latency. 'defer' incurs latency avoid them
in our hot-paths such as storage-rest handlers.

Fixes #8319
2019-10-02 01:42:15 +05:30
Anis Elleuch
61927d228c listV2: Continuation and NextContinuation tokens are encoded with base64 (#8337)
Minio V2 listing uses object names/prefixes as continuation tokens. This
is problematic when object names contain some characters that are forbidden
in XML documents. This PR will use base64 encoded form of continuation
and next continuation tokens to address that corner case.
2019-10-02 01:39:29 +05:30
Yao Zongyou
6a19d7b25a skip checking error also on Mac in TestCheckPortAvailability (#8343) 2019-10-01 23:12:18 +05:30
Harshavardhana
f45977d371
Fix error handling in DeleteFileBulk storage handler (#8327)
errors.errorString() cannot be marshalled by gob
encoder, so using a slice of []error would fail
to be encoded. This leads to no errors being
generated instead gob.Decoder on the storage-client
would see an io.EOF

To avoid such bugs introduce a typed error for
handling such translations and register this type
for gob encoding support.
2019-09-30 19:01:28 -07:00
Harshavardhana
4ec9b349d0
azure: Fix upload corruption with PutObject() on certain sizes (#8330)
On objects bigger than 100MiB can have a corrupted object
stored due to partial blockListing attempted right after
each blocks uploaded. Simplify this code to ensure that
all the blocks successfully uploaded are committed right
away.

This PR also updates the azure-sdk-go to latest release.
2019-09-30 18:42:18 -07:00
poornas
5c2af3f792 Add more context to error messages in STS handlers(#8304) 2019-10-01 02:35:19 +05:30
Ashish Kumar Sinha
fa5a1cebd9 support space character in access key (#8335) 2019-10-01 02:25:37 +05:30
Harshavardhana
127641731a
Parallelize initialization of storageDisks (#8288) 2019-09-27 16:47:12 -07:00
Harshavardhana
4155f4e49b
trace: Print either Transfer-Encoding or Content-Length (#8314)
If Transfer-Encoding is set client would have
never set Content-Length as its considered
malformed HTTP request
2019-09-27 10:19:27 -07:00
Bala FA
2a2ff96ee1 change ReadPerf into ReadThroughput in NetPerfInfo. (#8316)
Previously `ReadPerf` was in time.Duration is changed to `ReadThroughput` in uint64.
2019-09-27 00:01:18 +05:30
Harshavardhana
fd53057654 Add InfoCannedPolicy API to fetch only necessary policy (#8307)
This PR adds
- InfoCannedPolicy() API for efficiency in fetching policies
- Send group memberships for LDAPUser if available
2019-09-26 23:53:13 +05:30
Klaus Post
ff726969aa Switch to Snappy -> S2 compression (#8189) 2019-09-25 23:08:24 -07:00
Harshavardhana
c8fbc94329
Fix writing 'format.json' and make it atomic (#8296)
- Choose a unique uuid such that under situations of duplicate
  mounts we do not append to an existing json entry.
- Avoid AppendFile instead use WriteAll() to write the entire
  byte array atomically.
2019-09-24 18:47:26 -07:00
Anis Elleuch
a790877c01 s3: Encode continuation & next continuation tokens when asked (#8292)
When url encoding is passed in v2 listing handler, continuationToken
and nextContinuationToken needs to be encoded. The reason is that
both represents an object name/prefix in Minio server and it could
contain a character unsupported by XML specification.
2019-09-24 05:30:53 +05:30
Harshavardhana
77dc2031a2 Fix LDAP responseXML to be named appropriately (#8285)
This PR additionally also adds support for missing

- Session policy support for AD/LDAP
- Add API request/response parameters detail
- Update example to take ldap username,
  password input from the command line
- Fixes session policy handling for
  ClientGrants and WebIdentity
2019-09-24 03:51:16 +05:30
Harshavardhana
975134e42b
Add checks in DiskInfo() to protect against changing mounts (#8286) 2019-09-23 15:16:55 -07:00
Andreas Auernhammer
cb7d23cb17 remove SSE-S3 key rotation in CopyObject (#8278)
This commit removes the SSE-S3 key rotation functionality
from CopyObject since there will be a dedicated Admin-API
for this purpose.

Also update the security documentation to link to mc and
the admin documentation.
2019-09-24 02:05:04 +05:30
poornas
2e02e1889b Cleanup ResponseWriter function for audit and trace (#8283) 2019-09-24 02:04:28 +05:30
ebozduman
dbf7b1e573 starts-with policy condition support issue (#7937) 2019-09-22 14:20:49 -07:00
Harshavardhana
26985ac632 Fix all failing tests with -race 2019-09-22 11:01:46 -07:00
Praveen raj Mani
ad75683bde Authorize prometheus endpoint with bearer token (#7640) 2019-09-22 20:27:12 +05:30
poornas
4925bc3e80 log server startup messages to admin console api (#8264) 2019-09-22 13:54:32 +05:30
Andreas Auernhammer
ffded5a930 make the crypto error type a native go type (#8267)
This commit makes the `crypto.Error` type a native go (string)
type. That allows us to define error values as constants instead
of variables.

For reference see:
 - https://twitter.com/_aead_/status/1118170258215514115?s=20
 - https://dave.cheney.net/2016/04/07/constant-errors
2019-09-22 01:12:51 -07:00
Andreas Auernhammer
2b51fe9f26 make SSE request header check comprehensive (#8276)
This commit refactors the SSE header check
by moving it into the `crypto` package, adds
a unit test for it and makes the check comprehensive.
2019-09-21 03:26:12 +05:30
Harshavardhana
4780fa5a58 Remove setting net.Conn Deadlines as its not needed anymore (#8269)
This commit fixes a bug introduced in af6c6a2b35.

Setting deadlines in Go results in arbitrary hangs as reported here
https://github.com/golang/go/issues/34385

Fixes https://github.com/minio/minio/issues/7852
2019-09-20 23:37:24 +05:30
Andreas Auernhammer
b823d6d7bd remove the unused code for decrypting io.Writer (#8277)
This commit removes unused code for decrypting
`io.Writer` since the actual implementation only
decrypts `io.Reader`
2019-09-20 14:51:07 +05:30
Andreas Auernhammer
a9d724120f remove TLS 1.3 opt-in code (#8275)
This commit removes the TLS 1.3 opt-in code.
Since TLS 1.3 is opt-out for >= Go 1.13 this
code is not needed anymore.
2019-09-20 01:51:44 +05:30