Commit Graph

5809 Commits

Author SHA1 Message Date
kannappanr f409f10d18 Fix SimpleCI to use different data directory than mint (#7520)
Currently, the backend minio server uses the same data directory
as the mint test itself, causing `s3 sync` to fail often.

Now `minio` backend will use a different data directory `/data`
instead of `/mint/data`
2019-04-12 12:51:36 -07:00
Anis Elleuch 60d6887992 Make Encoding URL more compliant to S3 spec (#7360)
There is no written specification about how to encode key names
when url encoding type is passed.

However, this change will encode URLs as url.QueryEscape() does
while considering AWS S3 exceptions.
2019-04-12 12:02:37 -07:00
Andreas Auernhammer 012e4b42f9 http: opt-in to TLS 1.3 (#7483)
This commit enables TLS 1.3 on the server. For Go 1.12 TLS 1.3 is
enabled by an explicit opt-in.
2019-04-11 20:46:15 -07:00
poornas a74cb93666 Worm: Permit key-rotation of S3 encrypted objects (#7429)
Fixes : #7399
2019-04-10 11:31:50 -07:00
Andreas Auernhammer 849e06a316 crypto: add unit test for vault config verification (#7413)
This commit adds a unit test for the vault
config verification (which covers also `IsEmpty()`).

Vault-related code is hard to test with unit tests
since a Vault service would be necessary. Therefore
this commit only adds tests for a fraction of the code.

Fixes #7409
2019-04-10 11:05:53 -07:00
Praveen raj Mani 47ca411163 Enhance the event store interface to support channeling (#7343)
- Avoids code duplication across the other targets. By having a
  centralized function call.

- Reduce the room for race.
2019-04-10 18:16:01 +05:30
Aditya Manthramurthy ddb0d646aa Use passed lock-type in GetObjectNInfo cache implementation (#7505) 2019-04-09 14:49:45 -07:00
kannappanr 5ecac91a55
Replace Minio refs in docs with MinIO and links (#7494) 2019-04-09 11:39:42 -07:00
kannappanr 188ac8e369 Browser: Allow users to do s3 operations, if policy allows (#7487)
Fixes #7472
2019-04-09 20:47:41 +05:30
Minio Trusted ea052a2a00 Update yaml files to latest version RELEASE.2019-04-09T01-22-30Z 2019-04-09 01:30:27 +00:00
Harshavardhana a2e344bf30 Preserve ETag case for S3 compatibility (#7498)
Most hadoop distributions hortonworks, cloudera all
depend on aws-sdk-java 1.7.x to 1.10.x - the releases
which have bugs related case sensitive check for
ETag header. Go changes the case of the headers set
to be canonical but only preserves them when set
through a direct map.

This fixes most compatibility issues we have had
in the past supporting older hadoop distributions.
2019-04-08 16:54:46 -07:00
poornas 10a607154d Fix ListObjectsV2 for gateway encryption mode (#7491)
Fixes #7468 by setting NextContinuationToken only if list is
truncated
2019-04-08 15:12:00 -07:00
Harshavardhana 720ed3f5e8 Add go-mod cache for .travis to speed up builds (#7477) 2019-04-05 12:00:31 +05:30
Harshavardhana 0188009c7e Expose total and available disk space (#7453) 2019-04-05 09:51:50 +05:30
Harshavardhana 979309148e
Add doc updates for new go1.12 installation (#7486) 2019-04-04 13:36:31 -07:00
Minio Trusted 082bd5f31f Update yaml files to latest version RELEASE.2019-04-04T18-31-46Z 2019-04-04 18:39:37 +00:00
Harshavardhana 0146a9d592 Fix docker files to enable GO111MODULE 2019-04-04 11:06:50 -07:00
Aditya Manthramurthy b1b1d77893 Set S3 Select record message length to 128KiB (#7475)
- Previously this limit was a little more than 1MiB, and it broke
  compatibility with AWS SDK Java causing a buffer overflow error.
2019-04-04 00:41:52 -07:00
Harshavardhana c90999df98 Valid if bucket names are internal (#7476)
This commit fixes a privilege escalation issue against
the S3 and web handlers. An authenticated IAM user
can:

- Read from or write to the internal '.minio.sys'
bucket by simply sending a properly signed
S3 GET or PUT request. Further, the user can
- Read from or write to the internal '.minio.sys'
bucket using the 'Upload'/'Download'/'DownloadZIP'
API by sending a "browser" request authenticated
with its JWT token.
2019-04-03 23:10:37 -07:00
Andreas Auernhammer 9a740736a4 fix privilege escalation against inter-node communication (#7474)
This commit fixes another privilege escalation issue
abusing the inter-node communication of distributed
servers to obtain/modify the server configuration.

The inter-node communication is authenticated using
JWT-Tokens. Further, IAM users accessing the cluster
via the web UI also get a JWT token and the browser
will add this "user" JWT token to each the request.

Now, a user can extract that JWT token an can craft
HTTP POST requests for the inter-node communication
API endpoint. Since the server accepts ANY valid
JWT token it also accepts inter-node commands from
an authenticated user such that the user can execute
arbitrary commands bypassing the IAM policy engine
and impersonate other users, change its own IAM policy
or extract the admin access/secret key.

This is fixed by only accepting "admin" JWT tokens
(tokens containing the admin access key - and therefore
were generated with the admin secret key). Consequently,
only the admin user can execute such inter-node commands.
2019-04-03 12:16:19 -07:00
Harshavardhana 313a3a286a Migrate to go1.12 to simplify our cmd/http package (#7302)
Simplify the cmd/http package overall by removing
custom plain text v/s tls connection detection, by
migrating to go1.12 and choose minimum version
to be go1.12

Also remove all the vendored deps, since they
are not useful anymore.
2019-04-02 18:28:39 -07:00
Anis Elleuch 4c23e6fa55 rpc: Avoid using Pool since it conflicts with http2 (#7467)
A race is detected between a bytes.Buffer generated with cmd/rpc.Pool
and http2 module. An issue is raised in golang (https://github.com/golang/go/issues/31192).

Meanwhile, this commit disables Pool in RPC code and it generates a
new 1kb of bytes.Buffer for each RPC call.
2019-04-02 13:34:21 -07:00
Krishna Srinivas ef791764e0 Do no access nsLockMap.lockMap when using dsync (#7464)
There is no need to access nsLockMap.lockMap when using dsync
2019-04-02 12:27:20 -07:00
Anis Elleuch 53011606a5 Show 401 unauthorized msg when nodes are started with different creds (#7433)
Before this commit, nodes wait indefinitely without showing any
indicate error message when a node is started with different access
and secret keys.

This PR will show '401 Unauthorized' in this case.
2019-04-02 12:25:34 -07:00
Krishnan Parthasarathi 93a9078b23 Assign deploymentID for first minio server in distributed setup (#7427)
- Pass local endpoints to functions fixing formatXL during startup
2019-04-02 10:50:13 -07:00
Ashish Kumar Sinha a4bdcba503 Add check for extra input field (#7437)
fixes #6559
2019-04-02 12:15:32 +05:30
poornas 023866642c canonicalize ETag correctly (#7442)
Fixes #7441 
Trim extra quotes prefixing/suffixing ETag in
CompleteMultipartUpload request.
2019-04-01 12:19:52 -07:00
Harshavardhana 619611933a
Remove policy nesting errors (#7449)
Policy nesting has been supported for a while
now, we should remove references of code and
docs indicating nesting is not allowed anymore.
2019-03-31 08:42:43 -07:00
poornas dd1d69ab5c fix mysql config for native password authentication (#7436)
fixes #7430
2019-03-28 10:45:22 -07:00
Harshavardhana 16ba679103 Fix markdown table formatting (#7435) 2019-03-27 16:16:58 -07:00
Anis Elleuch e13c99ed82 Makefile: Fix getting misspell code when already installed (#7434)
Grouping misspell installation code so it won't executed when
misspell is found in the current system.
2019-03-27 16:16:17 -07:00
Minio Trusted 8cf707779c Update yaml files to latest version RELEASE.2019-03-27T22-35-21Z 2019-03-27 22:41:57 +00:00
Harshavardhana 6df05e489d Set Read/Write timeouts only for net.Conn not http.Server (#7431)
Fixes #7425
2019-03-27 22:10:06 +05:30
Harshavardhana 4a698c731b HealObjects should remove objects without quorum (#7407)
This PR adds a way to list objects without quorum
such that they can purged by `mc admin heal --remove`
2019-03-26 14:57:44 -07:00
Harshavardhana 9629de8230 Add proper context based logging when bitrot stream calls fail (#7415) 2019-03-26 13:59:33 -07:00
Harshavardhana 0250f7de67 Cleanup stale multipart uploads older than 3 days (#7424)
Fixes #6627
2019-03-25 13:41:05 -07:00
kannappanr 7154b8a568
Error log: Correct error type in anonymous mode (#7414)
Currently message is set to error type value.
Message field is not used in error logs. it is used only in the case of info logs.

This PR sets error message field to store error type correctly.
2019-03-25 13:40:08 -07:00
Anis Elleuch 8689ec258b Don't decrypt ETag in validation when source is SSEC multipart (#7423)
Copying an encrypted SSEC object when this latter is uploaded using
multipart mechanism was failing because ETag in case of encrypted
multipart upload is not encrypted.

This PR fixes the behavior.
2019-03-25 12:17:31 -07:00
Krishnan Parthasarathi aac9e2a7dd Return deploymentID in ServerInfo REST call (#7422)
Makes deploymentID information uniform in distributed setup
2019-03-25 11:55:28 -07:00
Harshavardhana 719d21efd8 Generate coverage across all sub-dirs (#7416) 2019-03-25 11:54:14 -07:00
Harshavardhana e0a87e96de
Populate host value from GetSourceIP directly (#7417) 2019-03-25 11:45:42 -07:00
Harshavardhana 0a44e70177 Change findEndpoint to return errors for Scanlines (#7390) 2019-03-25 13:31:02 +05:30
Rushan 22b4fe0a51 Update browser UI with new logo and colors (#7408) 2019-03-23 09:57:09 -07:00
Praveen raj Mani 89e45d0695 Restart process should use the current process' pid (#7373)
This fixes varying pids for server-respawns. And avoids duplicate process
creating multiple pids when the server restart signal is triggered with
service restart enabled.

Fixes #7350
2019-03-20 22:20:30 -07:00
poornas 8e1e701d35 Azure:ETag returned by ListObjects to be consistent with GetObjectInfo (#7301) 2019-03-20 18:11:46 -07:00
kannappanr 87cf51d5ab
unused code: Remove LoadCredentials function (#7369)
It is required to set the environment variable in the case of distributed
minio. LoadCredentials is used to notify peers of the change and will not work if
environment variable is set. so, this function will never be called.
2019-03-20 18:09:57 -07:00
Minio Trusted 8af45f0135 Update yaml files to latest version RELEASE.2019-03-20T22-38-47Z 2019-03-20 22:49:06 +00:00
Harshavardhana a9032b52b8 Change storageRESTTimeout to 1minute (#7398) 2019-03-20 13:20:09 -07:00
Harshavardhana c184038b6a Add proper custom errors object creations (#7387)
In scenario 1

```
- bucket/object-prefix
- bucket/object-prefix/object
```

Server responds with `XMinioParentIsObject`

In scenario 2

```
- bucket/object-prefix/object
- bucket/object-prefix
```

Server responds with `XMinioObjectExistsAsDirectory`

Fixes #6566
2019-03-20 13:06:53 -07:00
poornas 12b79d9f3b Remove duplicate error in switch case. (#7381)
Fixes: #7380

crypto.ErrInvalidCustomerKey was being handled twice in toAPIErrorCode()
2019-03-19 17:21:05 -07:00