Commit Graph

5890 Commits

Author SHA1 Message Date
Aditya Manthramurthy 4101d4917c Fix IAM users migration regression (#8008) 2019-08-01 12:31:04 -07:00
maihde d966d29fed fix: add integer code for Windows Subsystem for Linux filesystem (#8010) 2019-08-01 06:00:57 -07:00
Minio Trusted c301f5882d Update yaml files to latest version RELEASE.2019-07-31T18-57-56Z 2019-07-31 19:06:20 +00:00
Harshavardhana 123cccaed1 Honor connection pooling while tracing (#7979)
This PR fixes relying on r.Context().Done()
by setting

```
Connection: "close"
```

HTTP Header, this has detrimental issues for
client side connection pooling. Since this
header explicitly tells clients to turn-off
connection pooling. This causing pro-active
connections to be closed leaving many conn's
in TIME_WAIT state. This can be observed with
`mc admin trace -a` when running distributed
setup.

This PR also fixes tracing filtering issue
when bucket names have `minio` as prefixes,
trace was erroneously ignoring them.
2019-07-31 11:08:39 -07:00
Anis Elleuch cbd02c58be federation: Avoid printing context canceled error (#7997)
Golang proactively prints this error
        `http: proxy error: context canceled`

when a request arrived to the current deployment and
redirected to another deployment in a federated setup.

Since this error can confuse users, this commit will
just hide it.
2019-07-31 11:08:10 -07:00
Aditya Manthramurthy c71895f225 Listen for PolicyDB events from etcd and fix etcd watch handling (#7992) 2019-07-30 18:50:49 -07:00
Harshavardhana b83413b167 Use GOPROXY to speed up builds (#7984)
Read more here https://proxy.golang.org proposal 
for go1.13
2019-07-30 22:27:11 +05:30
Praveen raj Mani 63e0a81760 Ignore stale notification queues in notification.xml (#7673)
Allow renaming/editing a notification config. By replying with 
a successful GetBucketNotification response, without checking 
for any missing config ARN in targetList.

Fixes #7650
2019-07-30 14:19:06 +05:30
Harshavardhana 8d47ef503c Fix crash observed in OPA initialization (#7990)
Related to #7982, this PR refactors the code
such that we validate the OPA or JWKS in a
common place.

This is also a refactor which is already done
in the new config migration change. Attempt
to avoid any network I/O during Unmarshal of
JSON from disk, instead do it later when
updating the in-memory data structure.
2019-07-29 15:58:25 -07:00
Harshavardhana 54eded2e6f Do not assume all HTTP errors as Network errors (#7983)
In situations such as when client uploading data,
prematurely disconnects from server such as pressing
ctrl-c before uploading all the data. Under this
situation in distributed setup we prematurely
disconnect disks causing a reconnect loop. This has
an adverse affect we end up leaving a lot of files
in temporary location which ideally should have been
cleaned up when Put() prematurely fails.

This is also a regression which got introduced in #7610
2019-07-29 14:48:18 -07:00
Harshavardhana 94c88890b8 Add additional logging for OPA connections (#7982) 2019-07-28 08:33:25 +05:30
Harshavardhana e871e27562 Refactor and simplify etcd helpers used in IAM subsystem (#7980) 2019-07-26 13:42:54 -07:00
Harshavardhana 007a52b546
Add common validation for compression and encryption (#7978) 2019-07-26 02:41:16 -07:00
Praveen raj Mani efb8b00db0 Preserve tailing backslash in URL paths (#7678)
Fixes #7649
2019-07-25 20:55:09 -07:00
Harshavardhana d744865dc6 Enable config for NAS gateway mode (#7948)
Starting with #7751 we don't store config
in etcd anymore, allow NAS to honor config
on disk.
2019-07-25 17:41:25 -07:00
Harshavardhana e40c29e834 Fail appropriately if the disk has I/O errors (#7972)
If the disk has I/O errors, we should simply ignore
such a disk and not be bothered about it - until
it is replaced.
2019-07-25 13:35:27 -07:00
Praveen raj Mani b0cea1c0f3 Enable event persistence in AMQP (#7565) 2019-07-25 11:20:24 -07:00
Harshavardhana 6f2b4675fa
Add krb5 support for HDFS gateway (#7933) 2019-07-24 18:05:48 -07:00
Harshavardhana a4ce1daf99 docs: Use --user to start container in non-root (#7966) 2019-07-24 17:35:52 -07:00
Aditya Manthramurthy 7bdaf9bc50 Update on-disk storage format for users system (#7949)
- Policy mapping is now at `config/iam/policydb/users/myuser1.json`
  and includes version.

- User identity file is now versioned.

- Migrate old data to the new format.
2019-07-24 17:34:23 -07:00
Praveen raj Mani 55d4eee6f1 Enable event persistence in MySQL and PostgreSQL (#7629) 2019-07-24 10:18:29 -07:00
Harshavardhana ac82798d0a Remove uneeded calls on FS (#7967) 2019-07-24 15:59:13 +05:30
Minio Trusted 5b71c21330 Update yaml files to latest version RELEASE.2019-07-24T02-02-23Z 2019-07-24 02:09:19 +00:00
kannappanr 3e3fbdf8e6
Remove file added inadvertently (#7968) 2019-07-23 18:51:54 -07:00
Praveen raj Mani c9349747ca Enable event-persistence in NATS and NATS-Streaming (#7612) 2019-07-23 10:37:25 -07:00
Praveen raj Mani 2b9b907f9c Enable event persistence in Redis (#7601) 2019-07-23 10:22:08 -07:00
Daryl Finlay 9389a55e5d Cancel PutObjectPart on upload abort (#7940)
Calling ListMultipartUploads fails if an upload is aborted while a
part is being uploaded because the directory for the upload exists
(since fsRenameFile ends up calling os.MkdirAll) but the meta JSON file
doesn't. To fix this we make sure an upload hasn't been aborted during
PutObjectPart by checking the existence of the directory for the upload
while moving the temporary part file into it.
2019-07-22 22:36:15 -07:00
Harshavardhana 87e6533cf3 Add some design docs for distributed setup (#7950) 2019-07-23 07:48:10 +05:30
Christian Muehlhaeuser 38bc3a45db Fixed tautological conditions (#7959)
We already check for err being equal to nil above, no need
to check again.
2019-07-22 17:06:08 -07:00
Christian Muehlhaeuser c5faba55c1 Comment: Typo Fix (#7958) 2019-07-21 05:55:09 +01:00
Harshavardhana 8b5e6e338c Fix: Only add SRV records that match the bucket name exactly (#7957)
Problem: MinIO incorrectly appends DNS SRV records of buckets that have a prefix match with a given bucket. E.g bucket1 would incorrectly get bucket's DNS records too.
Solution: This fix ensures that we only add SRV records that match the key exactly
2019-07-20 11:29:05 +01:00
poornas 0373a1699b Add error filter to admin trace API (#7923)
This allows MinIO to have the ability to send back only error trace
2019-07-20 01:38:26 +01:00
Krishnan Parthasarathi 559a59220e Add initial support for bucket lifecycle (#7563)
This PR is based off @sinhaashish's PR for object lifecycle
management, which includes support only for,
- Expiration of object
- Filter using object prefix (_not_ object tags)

N B the code for actual expiration of objects will be included in a
subsequent PR.
2019-07-19 21:20:33 +01:00
Yao Zongyou 59e1763816 doc: use make instead of go command to test changes (#7951) 2019-07-19 14:40:39 +01:00
poornas 041a812ba0 trace api: add call stats to trace (#7915)
Stats such as call latency, bytes received and sent have been added
2019-07-18 23:29:17 +01:00
Krishnan Parthasarathi fbfc9a61ec Add node address information to logs (#7941) 2019-07-18 09:58:37 -07:00
Philipp Dallig be9baa1464 Fix startup without MINIO_USERNAME and MINIO_GROUPNAME (#7944) 2019-07-18 21:49:49 +05:30
Minio Trusted b058e32348 Update yaml files to latest version RELEASE.2019-07-17T22-54-12Z 2019-07-17 22:59:33 +00:00
Lucas ea66a52ed1 Add KMS master key from Docker secret (#7825) 2019-07-17 20:55:26 +01:00
Harshavardhana 55dd017e62 Deprecate auto detection of container user (#7930)
There is no reliable way to handle fallbacks for
MinIO deployments, due to various command line
options and multiple locations which require
access inside container.

Parsing command line options is tricky to figure
out which is the backend disk etc, we did try
to fix this in implementations of check-user.go
but it wasn't complete and introduced more bugs.

This PR simplifies the entire approach to rather
than running Docker container as non-root by default
always, it allows users to opt-in. Such that they
are aware that that is what they are planning to do.

In-fact there are other ways docker containers can
be run as regular users, without modifying our
internal behavior and adding more complexities.
2019-07-17 19:20:55 +01:00
Kanagaraj M 12353caf35 Fix: Support Unicode delimiters in s3 select (#7931) 2019-07-17 19:10:17 +01:00
Harshavardhana a57c747667 Document vault in prod mode instead of dev mode (#7928) 2019-07-16 01:32:15 +01:00
Anis Elleuch 28661c0413 heal: Trigger auto-heal once each month instead of 24 hours (#7934) 2019-07-16 00:03:42 +01:00
Harshavardhana 04a152be12 Redirect to browser only if browser is enabled (#7914) 2019-07-15 20:01:17 +01:00
Harshavardhana bce3f8237d Allow users to give anonymous access (#7926)
Current code already allows users to GetPolicy/SetPolicy
there was a missing code in ListAllBucketPolicies to allow
access, this fixes this behavior.

Fixes #7913
2019-07-15 20:00:41 +01:00
Harshavardhana 16a45e5aff
Fix dynamic help vars for sub-commands (#7925)
The fix in #7646 introduced a regression which
was left unnoticed, the fix didn't work for
sub-commands unfortunately. This fixes it
by moving v1.21.0 version of the minio/cli
package.

Fixes #7924
2019-07-12 23:32:27 -07:00
Anis Elleuch 000a60f238 xl: Heal empty parts (#7860)
posix.VerifyFile() doesn't know how to check if a file
is corrupted if that file is empty. We do have the part
size in xl.json so we pass it to VerifyFile to return
an error so healing empty parts can work properly.
2019-07-13 00:29:44 +01:00
Praveen raj Mani bf278ca36f Enable event persistence in NSQ (#7579) 2019-07-12 10:41:57 +01:00
Ashish Kumar Sinha 97f2bc26b9 Add validations for object name length and prefix (#7746)
fixes #7717
2019-07-12 10:08:12 +05:30
Praveen raj Mani bba562235b Enable persistent event store in elasticsearch (#7564) 2019-07-12 08:23:20 +05:30