Commit Graph

5479 Commits

Author SHA1 Message Date
Harshavardhana 55ef51a99d Vendorize all recent changes to minio-go (#7135)
- Default support for S3 dualstack endpoints (IPv6 support)
- Support granular policy conditionals in List operations
- Support proxy cookies for stickiness
2019-01-23 19:22:09 +05:30
Anis Elleuch dc2348daa5 heal: Preserve deployment ID from reference format.json (#7126)
Deployment ID is not copied into new formats after healing format. Although,
this is not critical since a new deployment ID will be generated and set in the
next cluster restart, it is still much better if we don't change the deployment
id of a cluster for a better tracking.
2019-01-22 18:32:06 -08:00
Aditya Manthramurthy 042d7f25e4 Fix regexp matcher of browser assets and paths (#7083)
Fix regexp matcher for special assets for the browser to clash with
less of the object namespace.

Assets should now be loaded with the /minio/ prefix. Previously,
favicon.ico (and others) could be loaded at any path matching
/minio/*/favicon.ico. This clashes with a large part of the object
namespace. With this change, /minio/favicon.ico will serve the favicon
but not /minio/mybucket/favicon.ico

Fixes #7077
2019-01-22 10:58:28 -08:00
Andreas Auernhammer 8c1b649b2d load system CAs before trying to load custom CAs (#7133)
This changes causes `getRootCAs` to always load system-wide CAs.
Any additional custom CAs (at `certs/CA/`) are added to the certificate pool
of system CAs.

The previous behavior was incorrect since all no system-wide CAs were
loaded if either there were CAs under `certs/CA` or the `certs/CA`
directory didn't exist at all.
2019-01-22 09:18:06 -08:00
Kumar Sukhani f03ccec912 Fix slack Link (#7131) 2019-01-22 19:53:50 +05:30
Nitish Tiwari 0bb65f84bb
Add example for IPv6 for address flag (#7127) 2019-01-22 15:55:27 +05:30
Harshavardhana 8e0910ab3e Fix build issues on BSDs in pkg/cpu (#7116)
Also add a cross compile script to test always cross
compilation for some well known platforms and architectures
, we support out of box compilation of these platforms even
if we don't make an official release build.

This script is to avoid regressions in this area when we
add platform dependent code.
2019-01-22 09:27:23 +05:30
Harshavardhana 5353edcc38
Support policy variable replacement (#7085)
This PR supports iam and bucket policies to have
policy variable replacements in resource and
condition key values.

For example
- ${aws:username}
- ${aws:userid}
2019-01-21 10:27:14 +05:30
Harshavardhana 3265112d04 Remove gateway implementations for manta, sia and b2 (#7115) 2019-01-20 08:10:58 -08:00
Harshavardhana 4fdacb8b14
Add policy conditions support for Listing operations on browser (#7106)
Fixes https://github.com/minio/minio/issues/7095
2019-01-20 12:50:01 +05:30
Krishna Srinivas 267f183fc8 Do not do StorageInfo() and ListBuckets() for FS/Erasure in health check handler (#7090)
Health checking programs very frequently use /minio/health/live 
to check health, hence we can avoid doing StorageInfo() and 
ListBuckets() for FS/Erasure backend.
2019-01-20 10:28:36 +05:30
Harshavardhana 3d22a9d84f Support rootCAs for notification targets (#7108)
Add support for RootCAs for notification targets
mqtt and webhook
2019-01-20 09:57:18 +05:30
Krishna Srinivas 51ec61ee94 Fix healing whole file bitrot (#7123)
* Use 0-byte file for bitrot verification of whole-file-bitrot files

Also pass the right checksum information for bitrot verification

* Copy xlMeta info from latest meta except []checksums and []Parts while healing
2019-01-20 07:58:40 +05:30
Harshavardhana 74c2048ea9 Add proper contexts with timeouts for etcd operations (#7097)
This fixes an issue of perceived hang when incorrect
unreachable URLs are specified in MINIO_ETCD_ENDPOINTS
variable.

Fixes #7096
2019-01-18 09:36:45 -08:00
Krishna Srinivas 730ac5381c Simplify parallelReader.Read() (#7109)
Simplify parallelReader.Read() which also fixes previous 
implementation where it was returning before all the parallel 
reading go-routines had terminated which caused race conditions.
2019-01-18 21:18:24 +05:30
Alex Simenduev 6dd8a83c5a change credential chain order in s3 gateway to mimic official docs (#7091) 2019-01-17 10:31:51 -08:00
Harshavardhana 1a7e6d4768 Handle multiple conditions in policies (#7079)
Fixes #7078
2019-01-17 10:28:24 -08:00
Krishna Srinivas 98c950aacd Streaming bitrot verification support (#7004) 2019-01-17 18:28:18 +05:30
Minio Trusted 94c52e3816 Update yaml files to latest version RELEASE.2019-01-16T21-44-08Z 2019-01-16 21:51:40 +00:00
Harshavardhana 8766c5eb22 Add version as part of Server: header (#7100)
This was agreed after discussing with @abperiasamy, we
borrowed the idea from Apache's own documentation.
2019-01-16 13:38:41 -08:00
kannappanr e0d22359e7
Fix lint warnings (#7099) 2019-01-16 12:49:20 -08:00
Harshavardhana 6dd13e68c2 Support V2 signatures when autoencryption is enabled (#7084)
When auto-encryption is turned on, we pro-actively add SSEHeader
for all PUT, POST operations. This is unusual for V2 signature
calculation because V2 signature doesn't have a pre-defined set
of signed headers in the request like V4 signature. According to
V2 we should canonicalize all incoming supported HTTP headers.

Make sure to validate signatures before we mutate http headers
2019-01-16 12:12:06 -08:00
Harshavardhana 633001c8ba Inherit certsDir from configDir if latter is set (#7098)
This is to ensure backward compatibility for all existing
deployments which use custom config dir to point to their
certs directory.
2019-01-16 12:04:32 -08:00
Bala FA e23a42305c Rebase minio/parquet-go and fix null handling. (#7067) 2019-01-16 21:52:04 +05:30
Krishna Srinivas 63d2583e91 Avoid holding write lock on config in situations where it is not needed (#7082)
This is to allow the cluster to come up when N/2 number of disks is available.
2019-01-16 13:59:21 +05:30
Harshavardhana a2f66abbe8
Update STS API docs with Version query param (#7071) 2019-01-16 09:38:32 +05:30
Andreas Auernhammer b28661b673 doc: add security documentation to provide some background info (#7028)
This commit adds some documentation about the design of the
SSE-C and SSE-S3 implementation. It describes how the Minio server
encrypt objects and manages keys.
2019-01-15 14:27:57 -08:00
Harshavardhana e8791ae274 Remove Minio server arch, version from `Server:` header (#7074) 2019-01-15 13:16:11 +05:30
Scott Dunlop 309975d477 Add missing time import to counter_darwin.go (#7081) 2019-01-14 17:21:27 -08:00
Praveen raj Mani 6571641735 Persist offline mqtt events in the `queueDir` and replay (#7037) 2019-01-14 12:39:00 +05:30
Harshavardhana 8757c963ba
Migrate all Peer communication to common Notification subsystem (#7031)
Deprecate the use of Admin Peers concept and migrate all peer
communication to Notification subsystem. This finally allows
for a common subsystem for all peer notification in case of
distributed server deployments.
2019-01-14 12:14:20 +05:30
Praveen raj Mani 9a71f2fdfa link to ppc64le binary added (#7065)
Fixes #7063
2019-01-11 20:16:19 +05:30
Nick Craig-Wood 9c26fe47b0 Fix server side copy of files with `?` in - fixes #7058 (#7059)
Before this change the CopyObjectHandler and the CopyObjectPartHandler
both looked for a `versionId` parameter on the `X-Amz-Copy-Source` URL
for the version of the object to be copied on the URL unescaped version
of the header.  This meant that files that had question marks in were
truncated after the question mark so that files with `?` in their
names could not be server side copied.

After this change the URL unescaping is done during the parsing of the
`versionId` parameter which fixes the problem.

This change also introduces the same logic for the
`X-Amz-Copy-Source-Version-Id` header field which was previously
ignored, namely returning an error if it is present and not `null`
since minio does not currently support versions.

S3 Docs:
- https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectCOPY.html
- https://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadUploadPartCopy.html
2019-01-10 13:10:10 -08:00
Sidhartha Mani f3f47d8cd3 Add ServerCPULoadInfo() and ServerMemUsageInfo() admin API (#7038) 2019-01-09 19:04:19 -08:00
Minio Trusted de1d39e436 Update yaml files to latest version RELEASE.2019-01-10T00-21-20Z 2019-01-10 00:28:50 +00:00
poornas ed1275a063 Fix copy from encrypted multipart to single encrypted part (#7056)
When source is encrypted multipart object and the parts are not
evenly divisible by DARE package block size, target encrypted size
will not necessarily be the same as encrypted source object.
2019-01-09 15:17:21 -08:00
kannappanr a7d407fa42
Display message on failure to get lock on format.json in fs mode on startup (#6538)
Retry to see if the lock is free. Retry time will increase binomially.
2019-01-09 10:13:04 -08:00
Anis Elleuch 4e6e05f8e0 virtual host: Fix making new buckets (#7054)
This commit removes old code preventing PUT requests with '/' as a path,
because this is not needed anymore after the introduction of the virtual
host style in Minio server code.

'PUT /' when global domain is not configured already returns 405 Method
Not Allowed http error.
2019-01-09 11:59:41 +05:30
Bala FA b0deea27df Refactor s3select to support parquet. (#7023)
Also handle pretty formatted JSON documents.
2019-01-08 16:53:04 -08:00
Kaan Kabalak e98d89274f Upgrade to Webpack 4 (#7045) 2019-01-08 11:04:59 -08:00
kannappanr c59206bcd3
GCS ListMultipartUploads: Don't return on first uploadid (#7014)
ListMultipartUploads code returns only the first uploadid.

Fixes #7011
2019-01-08 11:03:28 -08:00
Harshavardhana 7f2d439baa Avoid printing in S3 tests (#7043) 2019-01-07 22:32:30 +05:30
poornas 5a80cbec2a Add double encryption at S3 gateway. (#6423)
This PR adds pass-through, single encryption at gateway and double
encryption support (gateway encryption with pass through of SSE
headers to backend).

If KMS is set up (either with Vault as KMS or using
MINIO_SSE_MASTER_KEY),gateway will automatically perform
single encryption. If MINIO_GATEWAY_SSE is set up in addition to
Vault KMS, double encryption is performed.When neither KMS nor
MINIO_GATEWAY_SSE is set, do a pass through to backend.

When double encryption is specified, MINIO_GATEWAY_SSE can be set to
"C" for SSE-C encryption at gateway and backend, "S3" for SSE-S3
encryption at gateway/backend or both to support more than one option.

Fixes #6323, #6696
2019-01-05 14:16:42 -08:00
Harshavardhana 2d19011a1d Add support for AssumeRoleWithWebIdentity (#6985) 2019-01-04 13:48:12 -08:00
Harshavardhana e82dcd195c Deprecate config-dir bring in certs-dir for TLS configuration (#7033)
This PR is to provide indication that config-dir will be removed
in future and all users should migrate to new --certs-dir option

Fixes #7016
Fixes #7032
2019-01-02 10:05:16 -08:00
Nitish Tiwari fcb56d864c Add ServerDrivesPerfInfo() admin API (#6969)
This is part of implementation for mc admin health command. The
ServerDrivesPerfInfo() admin API returns read and write speed
information for all the drives (local and remote) in a given Minio
server deployment.

Part of minio/mc#2606
2018-12-31 09:46:44 -08:00
Krishnan Parthasarathi 75cd4201b0 Update go-sql-driver/mysql package (#7019) 2018-12-29 21:59:03 +05:30
Harshavardhana f24c017e9a Move docker edge to latest Go as well (#7030) 2018-12-28 17:24:24 -08:00
Harshavardhana b5280ba243
Migrate to Go version 1.11.4 (#7026) 2018-12-28 14:04:39 -08:00
Harshavardhana 2a0e4b6f58
Add boolean function condition support (#7027) 2018-12-28 12:18:58 -08:00