Commit Graph

1970 Commits

Author SHA1 Message Date
Harshavardhana
6dd13e68c2 Support V2 signatures when autoencryption is enabled (#7084)
When auto-encryption is turned on, we pro-actively add SSEHeader
for all PUT, POST operations. This is unusual for V2 signature
calculation because V2 signature doesn't have a pre-defined set
of signed headers in the request like V4 signature. According to
V2 we should canonicalize all incoming supported HTTP headers.

Make sure to validate signatures before we mutate http headers
2019-01-16 12:12:06 -08:00
Harshavardhana
633001c8ba Inherit certsDir from configDir if latter is set (#7098)
This is to ensure backward compatibility for all existing
deployments which use custom config dir to point to their
certs directory.
2019-01-16 12:04:32 -08:00
Bala FA
e23a42305c Rebase minio/parquet-go and fix null handling. (#7067) 2019-01-16 21:52:04 +05:30
Krishna Srinivas
63d2583e91 Avoid holding write lock on config in situations where it is not needed (#7082)
This is to allow the cluster to come up when N/2 number of disks is available.
2019-01-16 13:59:21 +05:30
Harshavardhana
e8791ae274 Remove Minio server arch, version from Server: header (#7074) 2019-01-15 13:16:11 +05:30
Praveen raj Mani
6571641735 Persist offline mqtt events in the queueDir and replay (#7037) 2019-01-14 12:39:00 +05:30
Harshavardhana
8757c963ba
Migrate all Peer communication to common Notification subsystem (#7031)
Deprecate the use of Admin Peers concept and migrate all peer
communication to Notification subsystem. This finally allows
for a common subsystem for all peer notification in case of
distributed server deployments.
2019-01-14 12:14:20 +05:30
Nick Craig-Wood
9c26fe47b0 Fix server side copy of files with ? in - fixes #7058 (#7059)
Before this change the CopyObjectHandler and the CopyObjectPartHandler
both looked for a `versionId` parameter on the `X-Amz-Copy-Source` URL
for the version of the object to be copied on the URL unescaped version
of the header.  This meant that files that had question marks in were
truncated after the question mark so that files with `?` in their
names could not be server side copied.

After this change the URL unescaping is done during the parsing of the
`versionId` parameter which fixes the problem.

This change also introduces the same logic for the
`X-Amz-Copy-Source-Version-Id` header field which was previously
ignored, namely returning an error if it is present and not `null`
since minio does not currently support versions.

S3 Docs:
- https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectCOPY.html
- https://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadUploadPartCopy.html
2019-01-10 13:10:10 -08:00
Sidhartha Mani
f3f47d8cd3 Add ServerCPULoadInfo() and ServerMemUsageInfo() admin API (#7038) 2019-01-09 19:04:19 -08:00
poornas
ed1275a063 Fix copy from encrypted multipart to single encrypted part (#7056)
When source is encrypted multipart object and the parts are not
evenly divisible by DARE package block size, target encrypted size
will not necessarily be the same as encrypted source object.
2019-01-09 15:17:21 -08:00
kannappanr
a7d407fa42
Display message on failure to get lock on format.json in fs mode on startup (#6538)
Retry to see if the lock is free. Retry time will increase binomially.
2019-01-09 10:13:04 -08:00
Anis Elleuch
4e6e05f8e0 virtual host: Fix making new buckets (#7054)
This commit removes old code preventing PUT requests with '/' as a path,
because this is not needed anymore after the introduction of the virtual
host style in Minio server code.

'PUT /' when global domain is not configured already returns 405 Method
Not Allowed http error.
2019-01-09 11:59:41 +05:30
Bala FA
b0deea27df Refactor s3select to support parquet. (#7023)
Also handle pretty formatted JSON documents.
2019-01-08 16:53:04 -08:00
kannappanr
c59206bcd3
GCS ListMultipartUploads: Don't return on first uploadid (#7014)
ListMultipartUploads code returns only the first uploadid.

Fixes #7011
2019-01-08 11:03:28 -08:00
Harshavardhana
7f2d439baa Avoid printing in S3 tests (#7043) 2019-01-07 22:32:30 +05:30
poornas
5a80cbec2a Add double encryption at S3 gateway. (#6423)
This PR adds pass-through, single encryption at gateway and double
encryption support (gateway encryption with pass through of SSE
headers to backend).

If KMS is set up (either with Vault as KMS or using
MINIO_SSE_MASTER_KEY),gateway will automatically perform
single encryption. If MINIO_GATEWAY_SSE is set up in addition to
Vault KMS, double encryption is performed.When neither KMS nor
MINIO_GATEWAY_SSE is set, do a pass through to backend.

When double encryption is specified, MINIO_GATEWAY_SSE can be set to
"C" for SSE-C encryption at gateway and backend, "S3" for SSE-S3
encryption at gateway/backend or both to support more than one option.

Fixes #6323, #6696
2019-01-05 14:16:42 -08:00
Harshavardhana
2d19011a1d Add support for AssumeRoleWithWebIdentity (#6985) 2019-01-04 13:48:12 -08:00
Harshavardhana
e82dcd195c Deprecate config-dir bring in certs-dir for TLS configuration (#7033)
This PR is to provide indication that config-dir will be removed
in future and all users should migrate to new --certs-dir option

Fixes #7016
Fixes #7032
2019-01-02 10:05:16 -08:00
Nitish Tiwari
fcb56d864c Add ServerDrivesPerfInfo() admin API (#6969)
This is part of implementation for mc admin health command. The
ServerDrivesPerfInfo() admin API returns read and write speed
information for all the drives (local and remote) in a given Minio
server deployment.

Part of minio/mc#2606
2018-12-31 09:46:44 -08:00
Harshavardhana
b5280ba243
Migrate to Go version 1.11.4 (#7026) 2018-12-28 14:04:39 -08:00
Harshavardhana
4e4f855b30
Add support for new policy conditions (#7024)
This PR implements following condition types

- StringEqualsIgnoreCase and StringNotEqualsIgnoreCase
- BinaryEquals
2018-12-26 17:39:30 -08:00
Harshavardhana
fb8d0d7cf7
Add support for hostname lookups instead of IPs in MINIO_PUBLIC_IPS (#7018)
DNS names will be resolved to their respective IPs if specified
in MINIO_PUBLIC_IPS.

Fixes #6862
2018-12-23 03:08:21 -08:00
Harshavardhana
a536cf5dc0 Buffconn should buffer upto maxHeaderBytes to avoid ErrBufferFull (#7017)
It can happen with erroneous clients which do not send `Host:`
header until 4k worth of header bytes have been read. This can lead
to Peek() method of bufio to fail with ErrBufferFull.

To avoid this we should make sure that Peek buffer is as large as
our maxHeaderBytes count.
2018-12-23 12:03:04 +05:30
Anis Elleuch
632022971b s3: Don't set NextMarker when listing is not truncated (#7012)
Setting NextMarker when IsTruncated is not set seems to be confusing
AWS C++ SDK, this commit will avoid setting any string in NextMarker.
2018-12-20 13:30:25 -08:00
kannappanr
7881791a91
CopyObject:Set Content-Type to application/octet-stream if it is not set (#6958) 2018-12-19 14:31:45 -08:00
Harshavardhana
d2f8f8c7ee Fix ETag handling with auto-encryption with CopyObject conditions (#7000)
minio-java tests were failing under multiple places when
auto encryption was turned on, handle all the cases properly

This PR fixes

 - CopyObject should decrypt ETag before it does if-match
 - CopyObject should not try to preserve metadata of source
   when rotating keys, unless explicitly asked by the user.
 - We should not try to decrypt Compressed object etag, the
   potential case was if user sets encryption headers along
   with compression enabled.
2018-12-19 14:12:53 -08:00
kannappanr
8c32311b80 Change lock name to include names instead of hash. (#6886)
Previously lockname included the hash of the bucket, object
and uploadid.

This is a part of fix required to list oldest locks on the server.
2018-12-19 13:57:51 -08:00
Ashish Kumar Sinha
9bb88e610e Deletion of subfolders of multipart (#6961)
Delete subfolders under multipart folder upon completion of CompleteMultipartUpload, AbortMultipartUpload and cleanupStaleMultipartUploads functions
2018-12-19 11:27:10 -08:00
Harshavardhana
d1e41695fe Add support for federation on browser (#6891) 2018-12-19 18:43:47 +05:30
Anis Elleuch
99b843a64e Add anonymous flag to prevent logging sensitive information (#6899) 2018-12-18 16:08:11 -08:00
Harshavardhana
4f31a9a33b Reload users upon AddUser on peers (#6975)
Also migrate ReloadFormat to notification subsystem,
remove GetConfig() we do not use this API anymore
2018-12-18 14:39:21 -08:00
Harshavardhana
e7c902bbbc
Return proper errors when admin API is not initialized (#6988)
Especially in gateway IAM admin APIs are not enabled
if etcd is not enabled, we should enable admin API though
but only enable IAM and Config APIs with etcd configured.
2018-12-18 13:03:26 -08:00
poornas
7da0336ac8 Make sure env are loaded before gateway layer initialization (#6989) 2018-12-18 10:42:09 -08:00
Harshavardhana
3be616de3f Send deployment ID in notification event response elements (#6991) 2018-12-18 10:05:26 -08:00
Harshavardhana
c5bf22fd90 Turn off printing IPv6 endpoints when listening on all interfaces (#6986)
By default when we listen on all interfaces, we print all the
endpoints that at local to all interfaces including IPv6
addresses. Remove IPv6 addresses in endpoint list to be
printed in endpoints unless explicitly specified with '--address'
2018-12-18 21:56:30 +05:30
poornas
7c9f934875 Disallow SSE requests when object layer has encryption disabled (#6981) 2018-12-14 21:39:59 -08:00
Andreas Auernhammer
d264d2c899 add auto-encryption feature (#6523)
This commit adds an auto-encryption feature which allows
the Minio operator to ensure that uploaded objects are
always encrypted.

This change adds the `autoEncryption` configuration option
as part of the KMS conifguration and the ENV. variable
`MINIO_SSE_AUTO_ENCRYPTION:{on,off}`.

It also updates the KMS documentation according to the
changes.

Fixes #6502
2018-12-14 13:35:48 -08:00
Harshavardhana
bebaff269c Support IPv6 in minio command line (#6947)
Fixes #6946
2018-12-14 13:07:46 +05:30
Harshavardhana
52b159b1db Allow versionId to be null for Delete,CopyObjectPart (#6972) 2018-12-14 11:34:37 +05:30
Nitish Tiwari
324834e4da Remove duplicate switch case (#6966)
Fixes #6948
2018-12-13 21:58:48 -08:00
Harshavardhana
c2ed1347d9 Do not list objects unless specified in policy (#6970)
Currently we use GetObject to check if we are allowed to list,
this might be a security problem since there are many users now
who actively disable a publicly readable listing, anyone who
can guess the browser URL can list the objects.

This PR turns off this behavior and provides a more expected way
based on the policies.

This PR also additionally improves the Download() object
implementation to use a more streamlined code.

These are precursor changes to facilitate federation and web
identity support in browser.
2018-12-14 09:45:09 +05:30
Anis Elleuch
50f6f9fe58 S3 api: Ignore encoding in xml body (#6953)
One user reported having discovered the following error:

API: SYSTEM()
Time: 20:06:17 UTC 12/06/2018
Error: xml: encoding "US-ASCII" declared but Decoder.CharsetReader is nil
1: cmd/handler-utils.go:43:cmd.parseLocationConstraint()
2: cmd/auth-handler.go:250:cmd.checkRequestAuthType()
3: cmd/bucket-handlers.go:411:cmd.objectAPIHandlers.PutBucketHandler()
4: cmd/api-router.go100cmd.(objectAPIHandlers).PutBucketHandler-fm()
5: net/http/server.go:1947:http.HandlerFunc.ServeHTTP()

Hence, adding support of different xml encoding. Although there
is no clear specification about it, even setting "GARBAGE" as an xml
encoding won't change the behavior of AWS, hence the encoding seems
to be ignored.

This commit will follow that behavior and will ignore encoding field
and consider all xml as utf8 encoded.
2018-12-13 12:09:50 -08:00
Harshavardhana
6f7c99a333 Allow versionId to be null for Copy,Get,Head API calls (#6942)
Fixes #6935
2018-12-12 11:43:44 -08:00
Andreas Auernhammer
21d8c0fd13 refactor vault configuration and add master-key KMS (#6488)
This refactors the vault configuration by moving the
vault-related environment variables to `environment.go`
(Other ENV should follow in the future to have a central
place for adding / handling ENV instead of magic constants
and handling across different files)

Further this commit adds master-key SSE-S3 support.
The operator can specify a SSE-S3 master key using
`MINIO_SSE_MASTER_KEY` which will be used as master key
to derive and encrypt per-object keys for SSE-S3
requests.

This commit is also a pre-condition for SSE-S3
auto-encyption support.

Fixes #6329
2018-12-12 12:20:29 +05:30
Kale Blankenship
79b9a9ce46 Provide actual size in events instead of compressed size. (#6950)
Previous behavior did not check if the object was compressed and
incorrectly reported the stored size rather than the actual object
size.
2018-12-11 17:30:15 -08:00
Harshavardhana
b9b353db4b Add env to support synchronous ops for all calls (#6877) 2018-12-11 16:22:56 -08:00
poornas
11a9b317a3 Disable ListenBucket notifications for NAS gateway (#6954) 2018-12-11 16:16:09 -08:00
Praveen raj Mani
9af7d627ac Preserve the compression headers while copying (#6952)
Fixes #6951
2018-12-11 12:05:41 -08:00
Harshavardhana
76d9d54603 Filter listing buckets based on user level access (#6940)
Fixes #6701
2018-12-10 22:57:22 +05:30
Harshavardhana
4c7c571875 Support JSON to CSV and CSV to JSON output format conversion (#6910)
This PR implements one of the pending items in issue #6286
in S3 API a user can request CSV output for a JSON document
and a JSON output for a CSV document. This PR refactors
the code a little bit to bring this feature.
2018-12-07 14:55:32 -08:00
Harshavardhana
3e124315c8 Increase the keep alive timeout to 30 secs (#6924)
Go by default uses a 3 * minute, we should
atleast use 30 secs as 10 secs is too aggressive.
2018-12-06 22:56:16 +05:30
Anis Elleuch
40852801ea fix: Better check of RPC type requests (#6927)
guessIsRPCReq() considers all POST requests as RPC but doesn't
check if this is an object operation API or not, which is actually
confusing bucket forwarder handler when it receives a new multipart
upload API which is a POST http request.

Due to this bug, users having a federated setup are not able to
upload a multipart object using an endpoint which doesn't actually
contain the specified bucket that will store the object.

Hence this commit will fix the described issue.
2018-12-05 14:28:48 -08:00
poornas
f6980c4630 fix ConfigSys and NotificationSys initialization for NAS (#6920) 2018-12-05 14:03:42 -08:00
Harshavardhana
8fcc787cba Register notFound handler only once per root router (#6926)
registering notFound handler more than once causes
gorilla mux to return error for all registered paths
greater than > 8. This might be a bug in the gorilla/mux
but we shouldn't be using it this way. NotFound handler
should be only registered once per root router.

Fixes #6915
2018-12-05 11:54:12 -08:00
Praveen raj Mani
4e6d3c093f Errors in notification config should not crash the server (#6881)
Fixes #6870
2018-12-04 18:27:12 -08:00
Anis Elleuch
61145361fd Fallback to non-loopback IF addresses for Domain IPs (#6918)
When MINIO_PUBLIC_IPS is not specified and no endpoints are passed
as arguments, fallback to the address of non loop-back interfaces.

This is useful so users can avoid setting MINIO_PUBLIC_IPS in docker
or orchestration scripts, ince users naturally setup an internal
network that connects all instances.
2018-12-04 17:35:22 -08:00
Harshavardhana
6add646130 Fix logging of initialization errors in distributed mode (#6914) 2018-12-04 10:25:56 -08:00
Harshavardhana
20e61fb362 Redirect browser requests only if browser is enabled (#6909)
This PR fixes an issue introduced in PR #6848, when
browser is disabled we shouldn't re-direct the requests
returning AccessDenied.

Fixes #6907
2018-12-04 13:08:24 +05:30
Bala FA
18ced1102c handle post policy only if it is set. (#6852)
Previously policy in post form is assumed to be set always.  This is
fixed by doing the check when policy is set.
2018-12-03 12:01:28 -08:00
Harshavardhana
d6af3c1237 Add bucket notification support for NAS gateway (#6908)
Fixes #6885
2018-12-03 14:02:14 +05:30
Andreas Auernhammer
5549a44566 rename vault namespace env variable to be more idiomatic (#6905)
This commit renames the env variable for vault namespaces
such that it begins with `MINIO_SSE_`. This is the prefix
for all Minio SSE related env. variables (like KMS).
2018-12-01 05:28:49 -08:00
Praveen raj Mani
e7af31c2ff Removed clientID from NATS-Streaming Config (#6391)
clientID must be a unique `UUID` for each connections. Now, the
server generates it, rather considering the config.

Removing it as it is non-beneficial right now.

Fixes #6364
2018-11-30 10:46:17 +05:30
Harshavardhana
26120d7838 Ignore permission errors on config-dir (#6894) 2018-11-29 18:14:05 -08:00
Harshavardhana
bef7c01c58 Choose right users in federation mode for CopyObject (#6895) 2018-11-29 17:35:11 -08:00
kannappanr
d85199e9de
Vendorize minio-go (#6883)
Fixes #6873
2018-11-29 11:13:03 -08:00
Harshavardhana
8608a84c23 Ignore hidden directory .snapshot for NetApp volumes (#6889) 2018-11-29 11:39:21 +05:30
Harshavardhana
b7226f4c82 Add transaction lock when migrating configs (#6878)
When migrating configs it happens often that some
servers fail to start due to version mismatch etc.

Hold a transaction lock such that all servers get
serialized.
2018-11-28 17:38:23 -08:00
Pontus Leitzler
f930ffe9e2 Fix gcs context (#6869) 2018-11-28 13:49:51 +05:30
poornas
45bb11e020 Set namespace on vault client if VAULT_NAMESPACE env is set (#6867) 2018-11-27 14:42:32 -08:00
Harshavardhana
e1164103d4 Reject if tokens are missing for temp credentials (#6860) 2018-11-27 13:24:04 -08:00
Harshavardhana
83fe70f710 Fix CopyObject regression calculating md5sum (#6868)
CopyObject() failed to calculate proper md5sum
when without encryption headers. This is a regression
fix perhaps introduced in commit 5f6d717b7a

Fixes https://github.com/minio/minio-go/issues/1044
2018-11-27 13:23:32 -08:00
Harshavardhana
12a6523fb2 Do not delete parts in multipart if 0 bytes (#6855)
This can create inconsistencies i.e Parts might have
lesser number of parts than ChecksumInfos. This will
result in object to be not readable.

This PR also allows for deleting previously created
corrupted objects.
2018-11-26 13:20:21 -08:00
Harshavardhana
dba61867e8 Redirect browser requests returning AccessDenied (#6848)
Anonymous requests from S3 resources returning
AccessDenied should be auto redirected to browser
for login.
2018-11-26 12:15:12 -08:00
Anis Elleuch
dd092f6c2b gateway: Properly set globalMinioPort (#6859)
globalMinioPort is used in federation which stores the address
and the port number of the server hosting the specified bucket,
this latter uses globalMinioPort but this latter is not set in
startup of the gateway mode.

This commit fixes the behavior.
2018-11-26 23:19:38 +05:30
Nitish Tiwari
2a810c7da2
Improve du thread performance (#6849) 2018-11-26 10:35:14 +05:30
Harshavardhana
9e3fce441e Audit log claims from token (#6847) 2018-11-22 09:33:24 +05:30
Harshavardhana
d4265f9a13 Simplify OPA to use rootCAs custom transport (#6843)
Also close the connections properly to use the
connection pooling properly for HTTP clients.
2018-11-22 08:31:05 +05:30
Harshavardhana
a9de303d8b
Update command line docs (#6839) 2018-11-20 17:35:33 -08:00
Anis Elleuch
69bd6df464 storage: Implement Close() in REST client (#6826)
Calling /minio/prometheuses/metrics calls xlSets.StorageInfo() which creates a new
storage REST client and closes it. However, currently, closing does nothing
to the underlying opened http client.

This commit introduces a closing behavior by calling CloseIdleConnections
provided by http.Transport upon the initialization of this latter.
2018-11-20 11:07:19 -08:00
Harshavardhana
bfb505aa8e Refactor logging in more Go idiomatic style (#6816)
This refactor brings a change which allows
targets to be added in a cleaner way and also
audit is now moved out.

This PR also simplifies logger dependency for auditing
2018-11-19 14:47:03 -08:00
poornas
d732b1ff9d Fix to cache objects on downloads (#6828)
fixes #6817
2018-11-19 11:00:46 -08:00
Igor K
ef517bd0da update for build on DilOS (#6770) 2018-11-19 19:40:02 +05:30
Anis Elleuch
7b579caf68 heal: Fix heal sequences cleanup process (#6780)
The current code triggers a timeout to cleanup a heal seq from
healSeqMap, but we don't know if the user did or not launch a new
healing sequence with the same path.

Add endTime to healSequence struct and add a periodic heal-sequence
cleaner to remove heal sequences only if this latter is older than
10 minutes.
2018-11-16 14:59:51 -08:00
Harshavardhana
272b8003d6 Honor header only when requested for use (#6815) 2018-11-16 10:27:48 -08:00
Anis Elleuch
1c24c93f73 storage: Upgrade REST version after adding WriteAll API (#6819)
Rolling update doesn't work properly because Storage REST API has
a new API WriteAll() but without API version number increase.

Also be sure to return 404 for unknown http paths.
2018-11-15 12:18:58 -08:00
poornas
5f6d717b7a Fix: Preserve MD5Sum for SSE encrypted objects (#6680)
To conform with AWS S3 Spec on ETag for SSE-S3 encrypted objects,
encrypt client sent MD5Sum and store it on backend as ETag.Extend
this behavior to SSE-C encrypted objects.
2018-11-14 17:36:41 -08:00
Harshavardhana
7e1661f4fa Performance improvements to SELECT API on certain query operations (#6752)
This improves the performance of certain queries dramatically,
such as 'count(*)' etc.

Without this PR
```
~ time mc select --query "select count(*) from S3Object" myminio/sjm-airlines/star2000.csv.gz
2173762

real	0m42.464s
user	0m0.071s
sys	0m0.010s
```

With this PR
```
~ time mc select --query "select count(*) from S3Object" myminio/sjm-airlines/star2000.csv.gz
2173762

real	0m17.603s
user	0m0.093s
sys	0m0.008s
```

Almost a 250% improvement in performance. This PR avoids a lot of type
conversions and instead relies on raw sequences of data and interprets
them lazily.

```
benchcmp old new
benchmark                        old ns/op       new ns/op       delta
BenchmarkSQLAggregate_100K-4     551213          259782          -52.87%
BenchmarkSQLAggregate_1M-4       6981901985      2432413729      -65.16%
BenchmarkSQLAggregate_2M-4       13511978488     4536903552      -66.42%
BenchmarkSQLAggregate_10M-4      68427084908     23266283336     -66.00%

benchmark                        old allocs     new allocs     delta
BenchmarkSQLAggregate_100K-4     2366           485            -79.50%
BenchmarkSQLAggregate_1M-4       47455492       21462860       -54.77%
BenchmarkSQLAggregate_2M-4       95163637       43110771       -54.70%
BenchmarkSQLAggregate_10M-4      476959550      216906510      -54.52%

benchmark                        old bytes       new bytes      delta
BenchmarkSQLAggregate_100K-4     1233079         1086024        -11.93%
BenchmarkSQLAggregate_1M-4       2607984120      557038536      -78.64%
BenchmarkSQLAggregate_2M-4       5254103616      1128149168     -78.53%
BenchmarkSQLAggregate_10M-4      26443524872     5722715992     -78.36%
```
2018-11-14 15:55:10 -08:00
Pontus Leitzler
f9779b24ad Enable default vet flags (#6810)
Enable default vet flags except experimental
2018-11-14 10:23:44 -08:00
Harshavardhana
f1f23f6f11 Add sync mode for 'xl.json' (#6798)
xl.json is the source of truth for all erasure
coded objects, without which we won't be able to
read the objects properly. This PR enables sync
mode for writing `xl.json` such all writes go hit
the disk and are persistent under situations such
as abrupt power failures on servers running Minio.
2018-11-14 19:48:35 +05:30
Harshavardhana
cf26c937e4 Remove UA worm and cache (#6809) 2018-11-14 13:18:55 +05:30
Harshavardhana
d6572879a8 Check for STS Action first to allow browser requests (#6796) 2018-11-13 15:53:06 -08:00
Anis Elleuch
b6ab8f50fa azure: Support non standard Azure cloud environments (#6712)
This change will allow users to enter the endpoint of the
storage account if this latter belongs to a different Azure
cloud environment, such as US gov cloud.

e.g:

  `MINIO_ACCESS_KEY=testaccount \
     MINIO_SECRET_KEY=accountsecretkey \
     minio gateway azure https://testaccount.blob.usgovcloudapi.net`
2018-11-13 15:51:49 -08:00
Harshavardhana
c82acc599a Treat empty xl.json as file not found (#6804)
If the buffer is empty we can avoid parsing
it and treat it essentially as `xl.json`
is effectively missing.
2018-11-13 11:57:03 -08:00
Andreas Auernhammer
2447bb58dd change received system signal output to upper case (#6761)
This commit slightly improves the output of the minio server
in case of a received system signal.
2018-11-12 15:07:16 -08:00
Harshavardhana
a55a298e00 Make sure to log unhandled errors always (#6784)
In many situations, while testing we encounter
ErrInternalError, to reduce logging we have
removed logging from quite a few places which
is acceptable but when ErrInternalError occurs
we should have a facility to log the corresponding
error, this helps to debug Minio server.
2018-11-12 11:07:43 -08:00
kannappanr
df2d75a2a3
Cleanup unnecessary logs (#6788) 2018-11-09 14:03:37 -08:00
Harshavardhana
b24b320807 Set notification namespace for NotificationConfiguration (#6789) 2018-11-09 10:40:03 -08:00
kannappanr
c872c1f1dc
Return default ETag if fs.json is empty (#6787) 2018-11-09 10:34:59 -08:00
Harshavardhana
38978eb2aa Avoid decrypting encrypted multipart final size (#6776)
Multipart object final size is not a contiguous
encrypted object representation, so trying to
decrypt this size will lead to an error in some
cases. The multipart object should be detected first
and then decoded with its respective parts instead.

This PR handles this situation properly, added a
test as well to detect these in the future.
2018-11-08 10:27:21 -08:00
Matthias Schneider
71c66464c1 feature: added nsq as broker for events (#6740) 2018-11-07 10:23:13 -08:00
Harshavardhana
bf414068a3 Parse and return proper errors with x-amz-security-token (#6766)
This PR also simplifies the token and access key validation
across our signature handling.
2018-11-07 20:10:03 +05:30
Andreas Auernhammer
d07fb41fe8 add key-rotation for SSE-S3 objects (#6755)
This commit adds key-rotation for SSE-S3 objects.
To execute a key-rotation a SSE-S3 client must
 - specify the `X-Amz-Server-Side-Encryption: AES256` header
   for the destination
 - The source == destination for the COPY operation.

Fixes #6754
2018-11-05 10:26:10 -08:00
Harshavardhana
a9cda850ca Add forceStop flag to provide facility to stop healing (#6718)
This PR also makes sure that we deal with HTTP request
count by ignoring the on-going heal operation, i.e
do not wait on itself.
2018-11-04 19:24:16 -08:00
Harshavardhana
bef0318c36 Support audit logs with additional fields (#6738)
This PR adds support

- Request query params
- Request headers
- Response headers

AuditLogEntry is exported and versioned as well
starting with this PR.
2018-11-02 18:40:08 -07:00
Harshavardhana
3f19ea98bb Honor certs from config-dir (#6757)
This was a regression introduced in 9fe51e392b
2018-11-02 11:53:45 -07:00
Praveen raj Mani
c96073f985 Test for multiple values for x-amz-meta header added (#6753) 2018-11-02 11:32:18 -07:00
Harshavardhana
d2f240c791 Ignore windows hidden folders (#6735)
On Windows erasure coding setup if

```
~ minio server V:\ W:\ X:\ Z:\
```

is not possible due to NTFS creating couple of
hidden folders, this PR allows minio to use
the entire drive.
2018-11-02 11:31:55 -07:00
Harshavardhana
6491dfbbd6 Fix etcd TLS handling (#6748)
etcd fails to connect if TLS config is set, make TLS
conditional to input arguments instead
2018-11-01 21:41:11 -07:00
kannappanr
d9cfa5fcd3
Shouldn't require space in HTTP host header (#6743)
Fixes #6741
2018-10-31 15:31:42 -07:00
kannappanr
9ed7fb4916
Do not call multiple response.WriteHeader calls (#6733)
Execute method in s3Select package makes a response.WriteHeader call.

Not calling it again in SelectObjectContentHandler function in case of
error in s3Select.Execute call.
2018-10-31 14:09:26 -07:00
Harshavardhana
f162d7bd97 Performance improvements by re-using record buffer (#6622)
Avoid unnecessary pointer reference allocations
when not needed, for example

- *SelectFuncs{}
- *Row{}
2018-10-31 08:48:01 +05:30
Harshavardhana
36990aeafd Avoid double bucket validation in DeleteObjectHandler (#6720)
On a heavily loaded server, getBucketInfo() becomes slow,
one can easily observe deleting an object causes many
additional network calls.

This PR is to let the underlying call return the actual
error and write it back to the client.
2018-10-30 16:07:57 -07:00
Harshavardhana
9fe51e392b Support etcd TLS certficates (#6719)
This PR supports two models for etcd certs

- Client-to-server transport security with HTTPS
- Client-to-server authentication with HTTPS client certificates
2018-10-29 11:14:12 -07:00
Harshavardhana
7e879a45d5 Add policy claim support for JWT (#6660)
This way temporary credentials can use canned
policies on the server without configuring OPA.
2018-10-29 11:08:59 -07:00
poornas
1c911c5f40 Fix: Validate copy-part encryption header and metadata (#6725)
Otherwise CopyObjectPart would continue to upload part with incorrect
encryption option and fail when upload is finalized
2018-10-29 06:40:34 -07:00
poornas
bd8dc17b7a gateway s3:Make sure to convert s3 errors to ObjectLayer errors (#6717) 2018-10-28 22:11:20 -07:00
Harshavardhana
8491a29ec3 Fix healing bucket properly (#6716)
Bucket should be healed properly if it partially exists
on only one set, since bucket is common for all sets.

Fixes #6710
2018-10-28 14:13:17 -07:00
Pontus Leitzler
81d21850ec Root CAs can be used for backend without TLS (#6711) 2018-10-28 06:21:00 +05:30
kannappanr
c6ec3fdfba Return error response header only for HEAD method (#6709) 2018-10-26 18:03:17 -07:00
Anis Elleuch
88c3dd49c6 copy: Ensure that the user has GET access to the src object (#6715) 2018-10-26 16:12:44 -07:00
kannappanr
6869f6d9dd
Remove unwanted logs (#6708) 2018-10-26 14:41:25 -07:00
Harshavardhana
3f643acb99 HealBucket was double counting endpoints (#6707)
Endpoint comparisons blindly without looking
if its local is wrong because the actual drive
for a local disk is always going to provide just
the path without the HTTP endpoint.

Add code such that this is taken care properly in
all situations. Without this PR HealBucket() would
wrongly conclude that the healing doesn't have quorum
when there are larger number of local disks involved.

Fixes #6703
2018-10-26 10:25:52 -07:00
Pontus Leitzler
ea73accefd Remove h2 from NextProtos since it doesn't work (#6705) 2018-10-26 12:48:39 +05:30
Harshavardhana
555d54371c Fix CopyObjectPart broken source encryption support (#6699)
Current master didn't support CopyObjectPart when source
was encrypted, this PR fixes this by allowing range
CopySource decryption at different sequence numbers.

Fixes #6698
2018-10-25 08:50:06 -07:00
Harshavardhana
bf66e9a529 Reload etcd users and policies properly (#6694)
Currently there was a bug in how we reload users and policies
which leads to users/policies going missing due to wrong path
construction.

Fixes #6693
2018-10-24 17:40:06 -07:00
Harshavardhana
fde8c38638 Add default canned policies (#6690) 2018-10-24 17:14:27 -07:00
Praveen raj Mani
ecb042aa1c Copy and CopyPart changes for compression (#6669)
This PR fixes

- The target object should be compressed even if the
  source object is not compressed.

- The actual size for an encrypted object should be the
  `decryptedSize`
2018-10-23 11:46:20 -07:00
Pontus Leitzler
9631d65552 Fix goroutine test fatalf (#6682)
Use t.Error/t.ErrorF instead if t.Fatal/t.Fatalf

Add returns to achieve same behaviour as earlier
2018-10-23 09:44:20 -07:00
Andreas Auernhammer
586466584f fix wrong actual part size assignment in CopyObjectPart (#6652)
This commit fixes a wrong assignment to `actualPartSize`.
The `actualPartSize` for an encrypted src object is not `srcInfo.Size`
because that's the encrypted object size which is larger than the
actual object size. So the actual part size for an encrypted
object is the decrypted size of `srcInfo.Size`.
2018-10-22 14:23:23 -07:00
Ashish Kumar Sinha
c0b4bf0a3e SQL select query for CSV/JSON (#6648)
select * , select column names have been implemented for CSV.
select * is implemented for JSON.
2018-10-22 12:12:22 -07:00
Praveen raj Mani
acf46cc3b5 SetConfigHandler should avoid setting an invalid notification config (#6679)
Fixes #6642 
Fixes #6641
2018-10-22 11:51:26 -07:00
Aarushi Arya
7c2ae4eaf7 Remove tmp file and multipart folder in FS mode. (#6677)
Fixes #6588
2018-10-22 07:36:30 -07:00
Harshavardhana
989d7af9ac Handle corrupted files correctly by fixing quorum (#6670)
This PR completes the missing functionality from #6592
2018-10-19 11:00:09 -07:00
Andreas Auernhammer
8a6c3aa3cd crypto: add RemoveInternalEntries function (#6616)
This commit adds a function for removing crypto-specific
internal entries from the object metadata.

See #6604
2018-10-19 10:50:52 -07:00
Harshavardhana
62b560510b Fix SSE-C source decryption handling (#6671)
Without this fix we have room for two different type of
errors.
- Source is encrypted and we didn't provide any source encryption keys

This results in Incomplete body error to be returned back to the client
since source is encrypted and we gave the reader as is to the object
layer which was of a decrypted value leading to "IncompleteBody"

- Source is not encrypted and we provided source encryption keys.

This results in a corrupted object on the destination which is
considered encrypted but cannot be read by the server and returns
the following error.

```
<Error><Code>XMinioObjectTampered</Code><Message>The requested object
was modified and may be compromised</Message><Resource>/id-platform-gamma/
</Resource><RequestId>155EDC3E86BFD4DA</RequestId><HostId>3L137</HostId>
</Error>
```
2018-10-19 10:41:13 -07:00
poornas
7e0f1eb8b5 Fix: verify client sent md5sum in encrypted PutObjectPart request (#6668)
This PR also removes check for SSE-S3 headers as this
is not required by S3 specification.
2018-10-18 16:05:05 -07:00
Pontus Leitzler
b43e8337b1 Add error handling in api-resource.go (#6651) 2018-10-18 07:31:46 -07:00
poornas
3b55357045 fix:Init globalIAMSys for ExecObjectLayerAPITest (#6636) 2018-10-17 17:25:50 -07:00
Harshavardhana
18d9a20ff6 Enable admin users API on gateway (#6659)
This is only enabled when etcd is enabled, healing is only
enabled for erasure coded backend.
2018-10-17 17:25:16 -07:00
Anis Elleuch
6590aba6d2 xl: PUT an empty dir on an existing prefix succeed (#6658)
This commit fixes a regression introduced in f187a16962
the regression returned AccessDenied when a client is trying to create an empty
directory on a existing prefix, though it should return 200 OK to be close as
much as possible to S3 specification.
2018-10-17 16:37:02 -07:00
Harshavardhana
2e81f27d27 Allow all browser calls to honor multi-users (#6645)
- GetAuth
- SetAuth
- GenerateAuth

Disallow changing bucket level metadata or creating/deleting buckets.
2018-10-17 16:23:09 -07:00
Anis Elleuch
ae3c05aa37 Avoid printing i/o closed pipe error message (#6654)
Since refactoring to GetObjectNInfo style, there are many cases
when i/o closed pipe is printed like, downloading an object
with wrong encryption key. This PR removes the log.
2018-10-17 15:52:18 -07:00
Praveen raj Mani
cef044178c Treat columns with spaces inbetween [s3Select] (#6597)
replace the double/single quotes with backticks for the xwb1989/sqlparser
to recognise such queries.

Fixes #6589
2018-10-17 11:01:26 -07:00
Pontus Leitzler
c998d1ac8c Add missing error check (#6632) 2018-10-17 10:57:12 -07:00
Wenjie
088c595e01 handle exception InvalidPart (#6649) 2018-10-17 21:50:58 +05:30
Andreas Auernhammer
fdf691fdcc move SSE-C TLS enforcement into generic handler (#6639)
This commit moves the check that SSE-C requests
must be made over TLS into a generic HTTP handler.

Since the HTTP server uses custom TCP connection handling
it is not possible to use `http.Request.TLS` to check
for TLS connections. So using `globalIsSSL` is the only
option to detect whether the request is made over TLS.
By extracting this check into a separate handler it's possible
to refactor other parts of the SSE handling code further.
2018-10-16 19:22:09 -07:00
Harshavardhana
88c8c2d6cd Fix browser login with multi users (#6644) 2018-10-16 18:44:58 -07:00
Anis Elleuch
362ebdcbab xl: Add '.CORRUPTED' prefix to corrupted objects (#6592)
getObjectInfo() checks if a given object has enough xl.json in disks,
returns 404 if not and added .CORRUPTED suffix to object.
2018-10-16 15:49:35 -07:00
Harshavardhana
b251454dd6 Fix toggling users status (#6640) 2018-10-16 14:55:23 -07:00
Harshavardhana
21c8693d9c Disable printing access/secrets in systemd (#6621)
Minio when run as a service in `systemd` should
avoid printing access/secret keys.
2018-10-16 13:19:12 -07:00
Harshavardhana
1e7e5e297c
Add canned policy support (#6637)
This PR adds an additional API where we can create
a new set of canned policies which can be used with one
or many users.
2018-10-16 12:48:19 -07:00
kannappanr
c7f180ffa9 Add code to translate errInvalidEncryptionParameters to APIErrcode (#6625)
Fixes #6623
2018-10-16 12:27:34 -07:00
kannappanr
b8bd8d6a03 Validate user provided SSE-C key on Head Object API (#6600)
Fixes #6598
2018-10-16 12:24:27 -07:00
Andreas Auernhammer
baec331e84 crypto: add functions for sealing/unsealing the etag for SSE (#6618)
This commit adds two functions for sealing/unsealing the
etag (a.k.a. content MD5) in case of SSE single-part upload.

Sealing the ETag is neccessary in case of SSE-S3 to preserve
the security guarantees. In case of SSE-S3 AWS returns the
content-MD5 of the plaintext object as ETag. However, we
must not store the MD5 of the plaintext for encrypted objects.
Otherwise it becomes possible for an attacker to detect
equal/non-equal encrypted objects. Therefore we encrypt
the ETag before storing on the backend. But we only need
to encrypt the ETag (content-MD5) if the client send it -
otherwise the client cannot verify it anyway.
2018-10-16 10:02:19 -07:00
poornas
557f382477 cache: remove cache space constraint (#6635)
relax cache constraint of requiring 100 times size of object
being cached for better cache utilization.
2018-10-16 11:06:42 +05:30
Harshavardhana
23b166b318 Remove applying custom policies with STS access keys (#6626)
Move away from allowing custom policies, all policies in
STS come from OPA otherwise they fail.
2018-10-15 12:44:03 -07:00
Anis Elleuch
81a481e098 profiling: Fix downloading tracing profiling data (#6599)
pkg/pprof saves tracing profiling data in a different file name
(trace.out) in contrary to other profiling mode.
2018-10-15 11:13:19 -07:00
Anis Elleuch
5b3090dffc encryption: Fix copy from encrypted multipart to single part (#6604)
CopyObject handler forgot to remove multipart encryption flag in metadata
when source is an encrypted multipart object and the target is also encrypted
but single part object.

This PR also simplifies the code to facilitate review.
2018-10-15 11:07:36 -07:00
Harshavardhana
3ef3fefd54 Add ListUsers API to list all configured users in IAM (#6619) 2018-10-13 12:48:43 +05:30
Andreas Auernhammer
28e25eac78 crypto: add helper functions for unsealing object keys (#6609)
This commit adds 3 helper functions for SSE-C and SSE-S3
to simplify object key unsealing in server code.

See #6600
2018-10-12 18:06:38 -07:00
Harshavardhana
b0c9ae7490 Add audit logging for S3 and Web handlers (#6571)
This PR brings an additional logger implementation
called AuditLog which logs to http targets

The intention is to use AuditLog to log all incoming
requests, this is used as a mechanism by external log
collection entities for processing Minio requests.
2018-10-12 12:25:59 -07:00
Harshavardhana
143e7fe300 Add etcd support to support STS on gateway mode (#6531) 2018-10-12 11:32:18 -07:00
poornas
83d8e01c81 fix: Close cacheReader if cache entry has expired (#6610)
prevent locking issues

Fixes #6602
2018-10-10 23:01:24 -07:00
poornas
110458cd10 Fix: Disallow requests with SSE-KMS headers (#6587)
Addresses issue #6582. Minio server currently does not
have SSE-KMS support. Reject requests with SSE-KMS headers
with NotImplementedErr
2018-10-09 15:04:53 -07:00
Harshavardhana
54ae364def Introduce STS client grants API and OPA policy integration (#6168)
This PR introduces two new features

- AWS STS compatible STS API named AssumeRoleWithClientGrants

```
POST /?Action=AssumeRoleWithClientGrants&Token=<jwt>
```

This API endpoint returns temporary access credentials, access
tokens signature types supported by this API

  - RSA keys
  - ECDSA keys

Fetches the required public key from the JWKS endpoints, provides
them as rsa or ecdsa public keys.

- External policy engine support, in this case OPA policy engine

- Credentials are stored on disks
2018-10-09 14:00:01 -07:00
Anis Elleuch
cbc5d78a09 Handle read/quorum errors when initializing all subsystems (#6585)
- Only require len(disks)/2 to initialize the cluster
- Fix checking of read/write quorm in subsystems init
- Add retry mechanism in policy and notification to avoid aborting in case of read/write quorums errors
2018-10-08 15:47:13 -07:00
Anis Elleuch
66d911653f xl: Fix typo in PutObjectPart when part size is 10Mb (#6574)
PutObjectPart forgot to allocate buffer memory when the size
of the uploaded part is exactly equal to blockSizeV1 = 10 Mb.
2018-10-05 17:09:50 -07:00
Anis Elleuch
ca6f795504 xl: check for correct err variable when prepareFile fails in PutObjectPart (#6573)
in xl.PutObjectPart call, prepareFile detected an error when the storage
is exhausted but we were returning the wrong error.

With this commit, users can see the correct error message when their disks
become full.
2018-10-05 17:09:20 -07:00
Harshavardhana
c3408f4f04 Send correct bucket notifications from web handlers (#6572)
Upload, Download, DownloadZip were incomplete
2018-10-05 11:20:00 -07:00
Krishna Srinivas
81bee93b8d Move remote disk StorageAPI abstraction from RPC to REST (#6464) 2018-10-04 17:44:06 -07:00
Anis Elleuch
f187a16962 xl: Require full write quorum in rename() as general rule (#6535)
Simplify the logic of using rename() in xl. Currently, renaming
doesn't require the source object/dir to be existent in at least
read quorum disks, since there is no apparent reason for it
to be as a general rule, this commit will just simplify the
logic to avoid possible inconsistency in the backend in the future.
2018-10-04 17:22:49 -07:00
Anis Elleuch
e031f2b614 xl: Fix typo in PutObject when uploading a 10Mb file (#6567)
PutObject forgot to allocate memory when the size of the uploaded
file is exactly equal to blockSizeV1 = 10 Mb.
2018-10-04 15:37:15 -07:00
Harshavardhana
c2b7b82ef4 Verify bitrot in ReadFile correctly when verifier is set (#6563)
This is a major regression introduced in this commit

ce02ab613d is the first bad commit
commit ce02ab613d
Author: Krishna Srinivas <634494+krishnasrinivas@users.noreply.github.com>
Date:   Mon Aug 6 15:14:08 2018 -0700

    Simplify erasure code by separating bitrot from erasure code (#5959)

:040000 040000 794f58d82a d2201ebfc8 M	cmd

This effects all distributed server deployments since this commit

All the following releases are affected

- RELEASE.2018-09-25T21-34-43Z
- RELEASE.2018-09-12T18-49-56Z
- RELEASE.2018-09-11T01-39-21Z
- RELEASE.2018-09-01T00-38-25Z
- RELEASE.2018-08-25T01-56-38Z
- RELEASE.2018-08-21T00-37-20Z
- RELEASE.2018-08-18T03-49-57Z

Thanks to Anis for reproducing the issue
2018-10-04 10:16:38 -07:00
Harshavardhana
02ad9d6072 Avoid unsolicited response over idle http client (#6562)
Without this PR minio server is writing an erroneous
response to clients on an idle connections which ends
up printing following message

```
Unsolicited response received on idle HTTP channel
```

This PR would avoid sending responses on idle connections
.i.e routine network errors.
2018-10-04 10:13:12 -07:00
Anis Elleuch
ea9408ccbb worm: when enabled, avoid renaming the existing object in tmp directory (#6560)
In XL PutObject & CompleteMultipartUpload, the existing object is renamed
to the temporary directory before checking if worm is enabled or not.

Most of the times, this doesn't cause an issue unless two uploads to the
same location occurs at the same time. Since there is no locking in object
handlers, both uploads will reach XL layer. The second client acquiring
write lock in put object or complete upload in XL will rename the object
to the temporary directory before doing the check and returning the error (wrong!).

This commit fixes then the behavior: no rename to temporary directory if
worm is enabled.
2018-10-03 20:39:24 -07:00
Pontus Leitzler
307765591d Use GetObjectInfo instead of GetObjectNInfo before cache decision (#6553) 2018-10-03 11:02:32 -07:00
Harshavardhana
5d859b2178
gateway/azure: allow putObject to support block based upload (#6552)
Current implementation simply uses all the memory locally
and crashes when a large upload is initiated using Minio
browser UI.

This PR uploads stream in blocks and finally commits the blocks
allowing for low memory footprint while uploading large objects
through Minio browser UI.

This PR also adds ETag compatibility for single PUT operations.

Fixes #6542
Fixes #6550
2018-10-02 23:08:16 -07:00
Harshavardhana
223967fd32 Return always a default heal item upon unexpected error (#6556)
Never return an empty result item even upon error,
choose all the default values and based on the errors
make sure to send right result reply.
2018-10-02 17:13:51 -07:00
Pontus Leitzler
274b35154c Fix go1.11 vet shadow errors (#6555) 2018-10-02 15:21:34 -07:00
Harshavardhana
c05ced08bb
Handle Range requests on empty objects (#6557)
Return a proper error on empty objects, S3 returns
416 Invalid Range on objects which are empty.

We should return the same.
2018-10-02 12:48:51 -07:00
Praveen raj Mani
c7722fbb1b Simplify pkg mimedb (#6549)
Content-Type resolution can now use a function `TypeByExtension(extension)` 
to resolve to the respective content-type.
2018-10-02 11:48:17 +05:30
Harshavardhana
f163bed40d
Add Vault support for custom CAs directory (#6527) 2018-10-01 13:49:10 -07:00
Harshavardhana
b4772849f9 Heal recursively all entries in config/ prefix (#6545)
This to ensure that we heal all entries in config/
prefix, we will have IAM and STS related files which
are being introduced in #6168 PR

This is a change to ensure that we heal all of them
properly, not just `config.json`
2018-10-01 22:24:26 +05:30
Harshavardhana
839a758a36
Fix a crash due to race between Abort/CompleteMultipart (#6544)
Fixes #6429
2018-10-01 09:50:09 -07:00
Harshavardhana
aebfceeafb Heal backend configuration file (#6532)
Fixes #6461
2018-09-29 13:47:01 +05:30
Anis Elleuch
83d7ec09c1 Disable restarting server after setting a new config (#6521)
Also disable listening to service restart event in tests since
we don't do this anymore.
2018-09-28 12:10:51 -07:00
Harshavardhana
8c29f69b00 Fix racy error communication inside go-routine (#6539)
Use CloseWithError to communicate errors in pipe,
this PR also fixes potential shadowing of error
2018-09-28 13:14:59 +05:30
Praveen raj Mani
ce9d36d954 Add object compression support (#6292)
Add support for streaming (golang/LZ77/snappy) compression.
2018-09-28 09:06:17 +05:30
Anis Elleuch
5c765bc63e Fix one possible data race in admin tests (#6537)
go test shows the following warning:
```
WARNING: DATA RACE
Write at 0x000002909e18 by goroutine 276:
  github.com/minio/minio/cmd.testAdminCmdRunnerSignalService()
      /home/travis/gopath/src/github.com/minio/minio/cmd/admin-rpc_test.go:44 +0x94

 Previous read at 0x000002909e18 by goroutine 194:
  github.com/minio/minio/cmd.testServiceSignalReceiver()
      /home/travis/gopath/src/github.com/minio/minio/cmd/admin-handlers_test.go:467 +0x70
```

The reason for this data race is that some admin tests are not waiting for go routines
that they created to be properly exited, which triggers the race detector.
2018-09-27 17:16:30 -07:00
Anis Elleuch
6c7c6bec91 profiler: Download API returns error when all nodes fail (#6525)
When download profiling data API fails to gather profiling data
from all nodes for any reason (including profiler not enabled),
return 400 http code with the appropriate json message.
2018-09-27 10:34:37 -07:00
poornas
ed703c065d Add ObjectOptions to GetObjectNInfo (#6533) 2018-09-27 15:36:45 +05:30
Aditya Manthramurthy
387584356f Remove unused range parsing code and update tests (#6530) 2018-09-27 15:24:07 +05:30
Harshavardhana
1111419d4a Add debugging for mutex, tracing (#6522) 2018-09-27 09:32:05 +05:30
Anis Elleuch
aa4e2b1542 Use GetObjectNInfo in CopyObject and CopyObjectPart (#6489) 2018-09-25 12:39:46 -07:00
Harshavardhana
1e5ac39ff3 Return HTTP response error for malformed requests (#6517) 2018-09-25 11:07:26 -07:00
Andreas Auernhammer
8cf7b88cc5 add functions to remove confidential information (#6516)
This commit adds two functions for removing
confidential information - like SSE-C keys -
from HTTP headers / object metadata.

This creates a central point grouping all
headers/entries which must be filtered / removed.

See also https://github.com/minio/minio/pull/6489#discussion_r219797993
of #6489
2018-09-24 21:02:51 +05:30
Harshavardhana
48bfebe442 HEAD on an object should mimic GET without body (#6508)
Add "Range" header support etc.
2018-09-23 22:54:10 +05:30
Pontus Leitzler
df60b3c733 Remove unnecessary contexts passed as data to FatalIf. No need to log an empty context. (#6487) 2018-09-21 16:04:11 -07:00
Aditya Manthramurthy
584cb61bb8 Switch back to GetObjectInfo for HEAD requests (#6513) 2018-09-21 13:48:58 -07:00
kannappanr
c1798cc89a
Update GCS storage library to support GetObject API on gzipped objects (#6506)
Fixes #6280
2018-09-21 11:43:10 -07:00
Aditya Manthramurthy
3c8fabd116 Fix cleanup of pipe in GetObjectNInfo handlers (#6509) 2018-09-21 11:42:06 -07:00
Aditya Manthramurthy
36e51d0cee Add GetObjectNInfo to object layer (#6449)
The new call combines GetObjectInfo and GetObject, and returns an
object with a ReadCloser interface.

Also adds a number of end-to-end encryption tests at the handler
level.
2018-09-20 19:22:09 -07:00
Harshavardhana
7d0645fb3a Deprecate domain, browser as config entries (#6498) 2018-09-20 14:56:32 -07:00
Harshavardhana
b62ed5dc90 select API CSV may not be specified (#6493)
This should be present until we support JSON
2018-09-20 15:04:26 +05:30
Harshavardhana
f0641a0406 Avoid changing creds in streaming signature (#6495)
This PR fixes a potential issue where credentials can be
changed in middle of different chunks during data transfer
which can lead to unexpected bugs.
2018-09-19 16:52:05 -07:00
Harshavardhana
3d060f8b64 Peer/rpc should never honor RPC calls without object-layer (#6486)
Fixes #6484
2018-09-19 21:32:56 +05:30
Jay Mundrawala
052a7b8eec Allow minio s3 gateway to use different AWS auth mechanisms (#6422)
Allow minio s3 gateway to use aws environment credentials,
IAM instance credentials, or AWS file credentials.

If AWS_ACCESS_KEY_ID, AWS_SECRET_ACCSES_KEY are set, 
or minio is running on an ec2 instance with IAM instance credentials, 
or there is a file $HOME/.aws/credentials, minio running as an S3
gateway will authenticate with AWS S3 using those one of credentials.

The lookup order:
1. AWS environment varaibles
2. IAM instance credentials
3. $HOME/.aws/credentials
4. minio environment variables

To authenticate with the minio gateway, you will always use the
minio environment variables MINIO_ACCESS_KEY MINIO_SECRET_KEY.
2018-09-19 18:05:30 +05:30
Anis Elleuch
9531cddb06 Add Profiler Admin API (#6463)
Two handlers are added to admin API to enable profiling and disable
profiling of a server in a standalone mode, or all nodes in the
distributed mode.

/minio/admin/profiling/start/{cpu,block,mem}:
  - Start profiling and return starting JSON results, e.g. one
    node is offline.

/minio/admin/profiling/download:
  - Stop the on-going profiling task
  - Stream a zip file which contains all profiling files that can
    be later inspected by go tool pprof
2018-09-18 16:46:35 -07:00
Harshavardhana
6fe9a613c0 Prioritize HTTP requests over Heal (#6468)
Additionally also heal 256 objects at any given
time in parallel.

Fixes #6196
Fixes #6241
2018-09-17 18:28:34 -07:00
Andreas Auernhammer
b729a4e83c Remove brittle TestServerTLSCiphers unit test (#5982)
The test TestServerTLSCiphers seems to fail sometimes for
no obvious reason. Actually the test is not needed
(as unit test) since minio/mint tests the server's TLS ciphers
as part of its security tests.

Fixes #5977
2018-09-17 15:55:09 +05:30
Harshavardhana
a0683d3c1f Send progress only when requested by client in SelectObject (#6467) 2018-09-17 11:52:46 +05:30
Anis Elleuch
66fda7a37f Use retry mechanism when initializing configuration (#6475)
Currently, one node in a cluster can fail to boot with the following error message:

```
ERROR Unable to initialize config system: Storage resources are insufficient for the write operation
```

  This happens when disks are formatted, read quorum is met but write
quorum is not met. In checkServerConfig(), a insufficient read quorum
error is replaced by errConfigNotFound, the code will generate a
new config json and try to save it, but it will fail because write
quorum is not met.

  Replacing read quorum with errConfigNotFound is also wrong because it
can lead, in rare cases, to overwrite the config set by the user.

  So, this commit adds a retry mechanism in configuration initialization
to retry only with read or write quorum errors.

  This commit will also fix the following cases:
 - Read quorum is lost just after the initialization of the object layer.
 - Write quorum not met when upgrading configuration version.
2018-09-15 22:09:51 -07:00
Harshavardhana
a63bc9254d Add 'disk' tag to log output to enhance 'disk not found' errors (#6460) 2018-09-13 21:42:50 -07:00
poornas
14fa0097b0 fix: UploadPart,CopyObjectPart does not need sse-s3 header (#6386)
S3 API spec for UploadPart requires encryption headers to be
specified only for SSE-C
2018-09-13 14:53:03 -07:00
Harshavardhana
e3777b1dd9 Combine obtaining resource, host, method into one operation (#6465)
This also adds a reduced timeout for errant connections, to
be quickly closed if they can't even send HTTP headers properly.

Fixes #6459
2018-09-13 18:17:03 +05:30
Anis Elleuch
ce419c9835 Bump RPC version after ReadFile RPC arguments change (#6457)
ReadFile RPC input argument has been changed in commit a8f5939452959d27674560c6b803daa9,
however, RPC doesn't detect such a change when it calls other nodes with older versions.

Hence, bumping RPC version.

Fixes #6458
2018-09-12 10:45:28 -07:00
Harshavardhana
166e998788
Fix healthcheck for NAS gateway (#6452)
It was expected that in gateway mode, we do not know
the backend types whereas in NAS gateway since its
an extension of FS mode (standalone) this leads to
an issue in LivenessCheckHandler() which would perpetually
return 503, this would affect all kubernetes, openshift
deployments of NAS gateway.
2018-09-11 13:44:10 -07:00
Andreas Auernhammer
267a0a3dfa fix X-Amz-Credential parsing for V4 policy signature (#6451)
This commit fixes an AWS S3 incompatibility issue.
The AccessKeyID may contain one or more `/` which caused
the server to interpret parts of the AccessKeyID as
other `X-Amz-Credential` parameters (like date, region, ...)

This commit fixes this by allowing 5 or more
`X-Amz-Credential` parameter strings and only interpreting
the last 5.

Fixes #6443
2018-09-11 11:17:23 -07:00
Harshavardhana
5479a6e33e Ignore migration if config files don't exist (#6448) 2018-09-10 18:24:57 -07:00
Anis Elleuch
7571582000 Print storage errors during distributed initialization (#6441)
This commit will print connection failures to other disks in other nodes
after 5 retries. It is useful for users to understand why the
distribued cluster fails to boot up.
2018-09-10 16:21:59 -07:00
Harshavardhana
12b4971b70 Rename config.json in config-dir with '.deprecated' extension (#6446)
Fixes #6444
2018-09-10 16:15:47 -07:00
poornas
5c0b98abf0 Add ObjectOptions to ObjectLayer calls (#6382) 2018-09-10 09:42:43 -07:00
Praveen raj Mani
30d4a2cf53 s3select should honour custom record delimiter (#6419)
Allow custom delimiters like `\r\n`, `a`, `\r` etc in input csv and 
replace with `\n`.

Fixes #6403
2018-09-10 21:50:28 +05:30
Anis Elleuch
92bc7caf7a Reword missing credentials error msg (#6418)
Enhance a little bit the error message that is showing
when access & secret keys are not specified in the
environment when running Minio in gateway and server mode.

This commit also removes a redundant check of access/secret keys.
2018-09-09 22:51:48 +05:30
Harshavardhana
19202bae81 Allow backward compatible way to load creds from config.json (#6435)
Print warning message for users to migrate to newer style of distributed
deployment by always setting credentials as ENVs.

Fixes #6434
2018-09-07 11:18:49 -07:00
Annanay Agarwal
7cb87f863e Kafka (sarama) authentication with user/pass (#6291) 2018-09-07 00:01:58 -07:00
Nitish Tiwari
67d8396af4
Fix Manta gateway client creation flow (#6425)
This commit fixes the Manta gateway client creation flow. We now affix
the endpoint scheme with endpoint URL while creating the Manta client
for gateway.

Also add steps in Manta gateway docs on how to run with custom Manta
endpoint.

Fixes #6408
2018-09-07 08:41:42 +05:30
Krishnan Parthasarathi
9e5c4df106 Gateway should honour --address flag while checking port availability (#6428)
Fixes #6426
2018-09-06 16:42:33 -07:00
Andreas Auernhammer
fd8749f42a return Access Denied for invalid SSE keys (#6432)
This commit fixes are regression in the server regarding
handling SSE requests with wrong SSE-C keys.

The server now returns an AWS S3 compatable API error (access denied)
in case of the SSE key does not match the secret key used during upload.

Fixes #6431
2018-09-06 12:31:12 -07:00
Anis Elleuch
3099af70a3 Add admin get/set config keys API (#6113)
This PR adds two new admin APIs in Minio server and madmin package:
- GetConfigKeys(keys []string) ([]byte, error)
- SetConfigKeys(params map[string]string) (err error)

A key is a path in Minio configuration file, (e.g. notify.webhook.1)

The user will always send a string value when setting it in the config file,
the API will know how to convert the value to the appropriate type. The user
is also able to set a raw json.

Before setting a new config, Minio will validate all fields and try to connect
to notification targets if available.
2018-09-06 20:33:18 +05:30
Harshavardhana
fd1b8491db
Drain response body properly for http connection pool (#6415)
Currently Go http connection pool was not being properly
utilized leading to degrading performance as the number
of concurrent requests increased.

As recommended by Go implementation, we have to drain the
response body and close it.
2018-09-05 16:47:14 -07:00
Anis Elleuch
1961f2ef54 xl: Fix removing an empty directory (#6421)
Removing an empty directory is not working because of xl.DeleteObject()
was only checking if the passed prefix is an actual object but it
should also check if it is an empty directory.
2018-09-05 16:38:03 -07:00
Harshavardhana
631c78e655
Bump up soMaxConn backlog for listener to 2048 (#6416)
soMaxConn value is 128 on almost all linux systems,
this value is too low for Minio at times when used
against large concurrent workload e.g: spark applications
this causes a sort of SYN flooding observed by the kernel
to allow for large backlog increase this value to 2048.

With this value we do not see anymore SYN flooding
kernel messages.
2018-09-05 13:16:19 -07:00
Harshavardhana
e0f8b767ba Fail for critical errors early on during prepare storage (#6404) 2018-09-05 10:20:54 -07:00
Harshavardhana
d0d015361c Fix config subsystem to wait on quorum number of formatted disks (#6407) 2018-09-05 20:55:55 +05:30
Krishna Srinivas
81b7e5c7a8 Send length instead of empty buffer for ReadFile() (#6414) 2018-09-04 23:22:05 -07:00
Krishnan Parthasarathi
1126410e62 Implement ListMultipartUploads, ListObjectParts for GCS gateway (#6377)
ListMultipartUploads implementation is meant for docker-registry
use-case only. It lists only the first upload with a prefix matching
the object being uploaded.
2018-09-04 13:11:33 -07:00
ebozduman
882a1a1ccc Stops listing objects for write-only access (#6396) 2018-08-31 13:20:27 -07:00
Harshavardhana
8690d62146 Allow fallback listen if first listener fails (#6380)
On linux listen() uses kernel features TCP_FASTOPEN, DEFER_ACCEPT

Fixes #6379
2018-08-31 13:17:05 -07:00
Anis Elleuch
85117d554f xl: Avoid removing a directory if it is not an object dir (#6395)
DeleteObject should not remove any directory unless the latter
is an empty directory.

Fixes #6394
2018-08-31 13:16:35 -07:00
Harshavardhana
4487f70f08 Revert all GetObjectNInfo related PRs (#6398)
* Revert "Encrypted reader wrapped in NewGetObjectReader should be closed (#6383)"

This reverts commit 53a0bbeb5b.

* Revert "Change SelectAPI to use new GetObjectNInfo API (#6373)"

This reverts commit 5b05df215a.

* Revert "Implement GetObjectNInfo object layer call (#6290)"

This reverts commit e6d740ce09.
2018-08-31 13:10:12 -07:00
Harshavardhana
fb27388101 HTTP headers are case insensitive handle them appropriately (#6390)
An issue was reproduced when minio-js client functional
tests are setting lower case http headers, in our current
master branch we specifically look for canonical host header
which may be not necessarily true for all http clients.
This leads to a perpetual hang on the *net.Conn*.

This PR fixes regression caused by #6206 by handling the
case insensitivity.
2018-08-31 17:00:32 +05:30
Bala FA
72fa2b4537 Add RPC counters for HTTP stats. (#6206)
This patch introduces separate counters for HTTP stats for minio
reserved bucket.

Fixes #6158
2018-08-30 14:17:58 +05:30
Harshavardhana
53a0bbeb5b
Encrypted reader wrapped in NewGetObjectReader should be closed (#6383) 2018-08-29 19:18:00 -07:00
Harshavardhana
384a862940
Return quorum error based on disks in abortMultipartUpload (#6362)
Fixes #4980
2018-08-29 13:36:19 -07:00
Harshavardhana
029f52880b
With no read quorum config should be treated as notFound (#6374)
This will allow the config subsystem to initialize properly
in situations where many servers are coming up in a rolling
fashion.
2018-08-28 14:23:22 -07:00
Harshavardhana
5b05df215a
Change SelectAPI to use new GetObjectNInfo API (#6373)
This PR also removes some double checks
2018-08-28 13:08:30 -07:00
Aditya Manthramurthy
e6d740ce09 Implement GetObjectNInfo object layer call (#6290)
This combines calling GetObjectInfo and GetObject while returning a
io.ReadCloser for the object's body. This allows the two operations to
be under a single lock, fixing a race between getting object info and
reading the object body.
2018-08-27 15:28:23 +05:30
Anis Elleuch
06d2dfa31c Fix WORM and BROWSER status calculation (#6360)
One typo introduced in a recent commit miscalculates if worm and browser
are enabled or not. A simple test is also added to detect this issue
in the future if it ever happens again.
2018-08-24 14:36:14 -07:00
poornas
d547873b17 webhandler - display encryption errors properly (#6339)
For encrypted objects, download errors need to be
displayed in web response format instead of xml format.

Fixes #6327
2018-08-24 07:56:24 -07:00
Krishna Srinivas
52f6d5aafc Rename of structs and methods (#6230)
Rename of ErasureStorage to Erasure (and rename of related variables and methods)
2018-08-23 23:35:37 -07:00
Harshavardhana
2211a5f1b8 Avoid ListenBucket targets to be listed in ServerInfo (#6340)
In current master when you do `mc watch` you can see a
dynamic ARN being listed which exposes the remote IP as well

```
mc watch play/airlines
```

On another terminal
```
mc admin info play
●  play.minio.io:9000
   Uptime : online since 11 hours ago
  Version : 2018-08-22T07:50:45Z
   Region :
 SQS ARNs : arn:minio:sqs::httpclient+51c39c3f-131d-42d9-b212-c5eb1450b9ee+73.222.245.195:33408
    Stats : Incoming 30GiB, Outgoing 7.6GiB
  Storage : Used 7.7GiB
```

SQS ARNs listed as part of ServerInfo should be only external targets,
since listing an ARN here is not useful and it cannot be re-purposed in
any manner.

This PR fixes this issue by filtering out httpclient from the ARN list.

This is a regression introduced in #5294 0e4431725c
2018-08-23 23:31:14 -07:00
Harshavardhana
1ffa6adcd4 Ignore io.EOF returned by ReadFrom for zero byte fs.json (#6346)
Fixes #6256
2018-08-24 11:34:21 +05:30
kannappanr
add57a6938
Add content-length as part of event notification structure (#6341)
Fixes #6321
2018-08-23 14:40:54 -07:00
Praveen raj Mani
65e05a06fb Remove notifications Fix (#6082)
Remove all the notifications for an empty rulesMap

Fixes #6053
2018-08-23 22:53:18 +05:30
Harshavardhana
b01e69e08f
Initialize global object layer after all subsystems have initialized (#6333)
This is to ensure that object API operations are not performed
on a server on which subsystems are yet to be initialized.
2018-08-22 23:11:17 -07:00
Harshavardhana
beb6d40ce6
Avoid crash when policy subsystem is not initialized (#6326)
Fixes #6324
2018-08-21 15:38:51 -07:00
poornas
19db921555 CopyObject: fix regression in key rotation (#6331)
After key rotation, metadata was not being replaced with new sealed key.
Regression introduced in commit e71ef905f9
2018-08-21 15:12:00 -07:00
kannappanr
2d84b02bc4 Check for absence of checksum field and attributes. (#6298)
Fixes #6295
2018-08-20 16:58:47 -07:00
Harshavardhana
7d7e21aebb Merge initConfig logic to ConfigSys (#6312) 2018-08-19 13:57:18 -07:00
Andreas Auernhammer
d531080b7e add SSE-KMS not-implemented error handling (#6234)
This commit adds error handling for SSE-KMS requests to
HEAD, GET, PUT and COPY operations. The server responds
with `not implemented` if a client sends a SSE-KMS
request.
2018-08-17 21:07:19 -07:00
Harshavardhana
a6b8a5487a Fail gateway properly with an error on port conflicts (#6303) 2018-08-17 21:06:36 -07:00
Harshavardhana
9f14433cbd Ensure that setConfig uses latest functionality (#6302) 2018-08-17 18:51:34 -07:00
Harshavardhana
50a817e3d3 Use new listener which implements enhanced tcp features (#6289)
This package provide customizable TCP net.Listener with various
performance-related options:

 * SO_REUSEPORT. This option allows linear scaling server performance
   on multi-CPU servers.
   See https://www.nginx.com/blog/socket-sharding-nginx-release-1-9-1/ for details.
 * TCP_DEFER_ACCEPT. This option expects the server reads from the accepted
   connection before writing to them.
 * TCP_FASTOPEN. See https://lwn.net/Articles/508865/ for details.
2018-08-17 18:44:02 -07:00
Harshavardhana
5a4a57700b Add select docs and fix return values for Select API (#6300) 2018-08-17 17:11:39 -07:00
Harshavardhana
3de5a3157f Enhance picking valid xlMeta based on quorum (#6297)
This PR borrows the idea from getFormatXLQuorum()
2018-08-17 14:42:04 -07:00
poornas
e71ef905f9 Add support for SSE-S3 server side encryption with vault (#6192)
Add support for sse-s3 encryption with vault as KMS.

Also refactoring code to make use of headers and functions defined in
crypto package and clean up duplicated code.
2018-08-17 12:52:14 -07:00
Harshavardhana
65de2d68c0 Allow for proper garbage collection pooling bytes.Buffer (#6266) 2018-08-16 18:37:43 -07:00
Harshavardhana
eab947cf42 Make sure to update modTime in erasure metadata (#6296)
This is to ensure that when we update xl.json with new
parts have the latest modtime, in-turn avoids consistency
issues when the disk is offline.
2018-08-16 17:55:01 -07:00
Harshavardhana
0fe9e95250 Validate prefixes on all sets (#6294)
This PR fixes a regression introduced in 8eb838bf91
where hashing technique was used on prefixes to get the right set
to perform the operation, this is not correct since prefixes and
their corresponding keys might hash to a different value depending
on the key length.

For prefixes/directories we should look everywhere to support proper
quorum based listing.

Fixes #6293
2018-08-16 16:49:38 -07:00
kannappanr
c7946ab9ab
Remove unnecessary error log messages (#6186) 2018-08-16 12:57:49 -07:00
Harshavardhana
f26325c988 Support supplying custom drives per set count (#6261) 2018-08-15 16:35:21 -07:00
Arjun Mishra
7c14cdb60e S3 Select API Support for CSV (#6127)
Add support for trivial where clause cases
2018-08-15 03:30:19 -07:00
Harshavardhana
0e02328c98 Migrate config.json from config-dir to backend (#6195)
This PR is the first set of changes to move the config
to the backend, the changes use the existing `config.json`
allows it to be migrated such that we can save it in on
backend disks.

In future releases, we will slowly migrate out of the
current architecture.

Fixes #6182
2018-08-15 10:11:47 +05:30
Harshavardhana
380524ae27
Unlock read lock on uploadID upon errors (#6283) 2018-08-14 18:35:30 -07:00
kannappanr
0286e61aee Log disk not found error just once (#6059)
Modified the LogIf function to log only if the error passed
is not on the ignored errors list.

Currently, only disk not found error is added to the list.
Added a new function in logger package called LogAlwaysIf, 
which will print on any error.

Fixes #5997
2018-08-14 13:58:48 -07:00
wd256
ff29aed05d gcs: Translate S3 user-defined metadata prefix to/from GCS custom metadata prefix (#6270) 2018-08-14 11:53:39 -07:00
Harshavardhana
64f2c61813 Implement memory efficient readdir for windows (#6247)
Fixes #6164
2018-08-09 14:52:29 -07:00
Andreas Auernhammer
525c04fd07 crypto: add SSE-KMS HTTP header detection (#6228)
This commit adds support for detecting SSE-KMS headers.
The server should be able to detect SSE-KMS headers to
at least fail such S3 requests with not implemented.
2018-08-09 13:02:57 -07:00
Anis Elleuch
5a1ae862a7 Avoid sending an error after 206 HTTP code (#6264)
When a S3 client sends a GET Object with a range header, 206 http
code is returned indicating success, however the call of the object
layer's GetObject() inside the handler can return an error and will lead
to writing an XML error message, which is obviously wrong since
we already sent 206 http code. So in the case, we just stop sending
data to the S3 client, this latter can still detect if there is no
error when comparing received data with Content-Length header
in the Get Object response.
2018-08-08 15:39:47 -07:00
Harshavardhana
2f1756489e Add tracing capabilities for internode rpc Servers (#6254) 2018-08-07 15:21:30 +05:30
Harshavardhana
9719640e34 Use sha256-simd instead of crypto/sha256 (#6252) 2018-08-06 18:00:10 -07:00
Krishna Srinivas
ce02ab613d Simplify erasure code by separating bitrot from erasure code (#5959) 2018-08-06 15:14:08 -07:00
Oleg Kovalov
37de2dbd3b simplifying if-else chains to switches (#6208) 2018-08-06 10:26:40 -07:00
Harshavardhana
a82500f162 Support dumb terminals by turning off color (#6246)
ANSI colors do not work on dumb terminals, in situations
when minio is running as a service under systemd.

This PR ensures we turn off color in those situations.
2018-08-06 18:16:49 +05:30
Harshavardhana
2dede2fdc2 Add reliable RemoveAll to handle racy situations (#6227) 2018-08-06 09:45:28 +05:30
Harshavardhana
13fbb96736 Hold locks granularly in nslockMap (#6242)
With benchmarks increases the performance for small files
by almost 4x times the previous releases.
2018-08-06 08:55:25 +05:30
Harshavardhana
eabfcea34e
Add granular locking in retryTicker (#6236)
This is to avoid serializing RPC contention on ongoing
parallel operations, the blocking profile indicating
all calls were being serialized through setRetryTicker.
2018-08-03 18:57:00 -07:00
Andreas Auernhammer
a078703214 catch crypto.* errors and add SSE-S3 invalid algorithm err (#6229)
This commit adds the crypto.* errors to the
`toAPIErrorCode` switch. Further this commit adds an S3
API error code returned whenever the client specifes a
SSE-S3 request with an invalid algorithm parameter.
2018-08-03 16:55:45 -07:00
Harshavardhana
bd2b22572f
Increase max idle connections from 100 to 4096 (#6244)
This is to be inline with our benchmarking results
2018-08-03 16:29:28 -07:00
poornas
5f69f04909 nas gateway: fix regression in global bucket policy initialization (#6243)
Fixes #6238
globalPolicySys used to be initialized in fs/xl layer. The referenced
commit moved this logic to server/gateway initialization,but a check
to avoid double initialization prevented globalPolicySys to be loaded
from disk for NAS.

fixes regression from commit be1700f595
2018-08-03 15:12:18 -07:00
Harshavardhana
556a51120c Deprecate ListLocks and ClearLocks (#6233)
No locks are ever left in memory, we also
have a periodic interval of clearing stale locks
anyways. The lock instrumentation was not complete
and was seldom used.

Deprecate this for now and bring it back later if
it is really needed. This also in-turn seems to improve
performance slightly.
2018-08-02 23:09:42 +05:30
poornas
eb391a53c1 check for syscall errors in posix-errors helper functions (#6232) 2018-08-02 10:38:51 +05:30
Harshavardhana
e17e09ea3c Handle POST object upload without filename param (#6221)
POST mime/multipart upload style can have filename value optional
which leads to implementation issues in Go releases in their
standard mime/multipart library.

When `filename` doesn't exist Go doesn't update `form.File` which
we rely on to extract the incoming file data, strangely when `filename`
is not specified this data is buffered in memory and is now part of
`form.Value` instead of `form.File` which creates an inconsistent
behavior.

This PR tries to fix this in our code for the time being, but ideal PR
would be to fix the upstream mime/multipart library to handle the
above situation consistently.
2018-08-01 14:19:11 -07:00
Andreas Auernhammer
76c423392a crypto: add GenerateIV from random IV generation (#6215)
This commit adds a `GenerateIV` function to simplify
the generation of random IVs.

It adds some unit tests for `GenerateIV` in key_test.go
2018-08-01 01:02:07 -07:00
Aarushi Arya
8e6d756e3a Appropriate error message on unsuccessful update. (#6203) 2018-08-01 01:01:37 -07:00
Andreas Auernhammer
a7c9058375 crypto: implement Stringer for S3 and SSEC (#6216)
This commit adds a `fmt.Stringer` implementation for
SSE-S3 and SSE-C. The string representation is the
domain used for object key sealing.
See: `ObjectKey.Seal(...)` and `ObjectKey.Unseal(...)`
2018-07-31 11:15:12 -07:00
Nitish Tiwari
197af49c99 Fix healthcheck handler to verify gateway backend liveness (#6218)
Fixes #6217
2018-07-31 10:55:34 -07:00
kannappanr
264cc4020f Return 503 instead of 404 if more than half of disks are not found (#6207)
Fixes #6163
2018-07-31 00:23:29 -07:00
Harshavardhana
dbd89bbae3 Remove double RLocks for GetBucketInfo (#6209) 2018-07-30 17:38:52 -07:00
Aditya Manthramurthy
224a272cf2 Fix type of bitrot mismatch error (#6205)
The error type `hashMismatchError` is lost when the error is received
from a remote disk.

Fixes #6201
2018-07-29 15:00:37 +05:30
Harshavardhana
ad86454580 Make sure to handle FaultyDisks in listing ops (#6204)
Continuing from PR 157ed65c35

Our posix.go implementation did not handle  I/O errors
properly on the disks, this led to situations where
top-level callers such as ListObjects might return early
without even verifying all the available disks.

This commit tries to address this in Kubernetes, drbd/nbd based
persistent volumes which can disconnect under load and
result in the situations with disks return I/O errors.

This commit also simplifies listing operation, listing
never returns any error. We can avoid this since we pretty
much ignore most of the errors anyways. When objects are
accessed directly we return proper errors.
2018-07-27 15:32:19 -07:00
Andreas Auernhammer
644c2ce326 crypto: add support for parsing/creating SSE-C/SSE-S3 metadata (#6169)
* crypto: add support for parsing SSE-C/SSE-S3 metadata

This commit adds support for detecting and parsing
SSE-C/SSE-S3 object metadata. With the `IsEncrypted`
functions it is possible to determine whether an object
seems to be encrypted. With the `ParseMetadata` functions
it is possible to validate such metadata and extract the
SSE-C/SSE-S3 related values.

It also fixes some naming issues.

* crypto: add functions for creating SSE object metadata

This commit adds functions for creating SSE-S3 and
SSE-C metadata. It also adds a `CreateMultipartMetadata`
for creating multipart metadata.

For all functions unit tests are included.
2018-07-25 13:35:54 -07:00
Harshavardhana
2debe77586 Remove error returned when part sizes are un-equal (#6183)
Since implementing `pwrite` like implementation would
require a more complex code than background append
implementation, it is better to keep the current code
as is and not implement `pwrite` based functionality.

Closes #4881
2018-07-24 21:31:03 -07:00
Harshavardhana
20480ba3f7 Remove references to MINIO_ENDPOINTS (#6200)
MINIO_ENDPOINTS is a special case scenario
we don't need to document it for now.
2018-07-24 17:17:25 -07:00
kannappanr
4f52d22c36
Fix make test failure (#6185)
Fix shadowing errors.
2018-07-24 14:17:58 -07:00
Harshavardhana
157ed65c35 Fix healthcheck handler to check errors in local disks only (#6184)
Healthcheck handler in current implementation was
performing ListBuckets() to check for liveness of Minio
service. ListBuckets() implementation on the other hand
doesn't do quorum based listing and if one of the disks
returned error, an I/O error it would be lead to kubernetes
taking the minio pod down prematurely even if the disk
is not local to that minio server.

The reason is ListBuckets() call cannot be trusted to
provide us the valid information that we need, Minio is a
clustered application which is designed to handle disk
failures. Error on one of the disks doesn't mean the pod
should become fully non-operational.

This PR attempts to fix this by only checking for alive
disks which are local to each setup and also by simply
performing a Stat() operation, if the Stat() returned
error on all disks local to a particular server then
we can let kubernetes safely take it down, until then
we should be operational.
2018-07-23 12:21:25 -07:00
Harshavardhana
5acc2a6db1 S3 gateway signature probe use a unique bucket (#6190)
This fixes an issue because someone is
using `probe-bucket-sign` bucket name in
region 'eu-central-1'
2018-07-23 10:16:56 -07:00
kannappanr
2cd14f567c Do not set Key and BucketName in ErrorResponse, if empty (#6174) 2018-07-23 14:09:09 +05:30
Harshavardhana
f1be356cc6 Do not use parallel deletes to avoid random I/O (#6178)
The current code for deleting 1000 objects simultaneously
causes significant random I/O, which on slower drives
leads to servers disconnecting in a distributed setup.

Simplify this by serially deleting and reducing the
chattiness of this operation.
2018-07-20 21:21:01 -07:00
kannappanr
76ddf4d32f Log x-amz-request-id as log and XML error response (#6173)
Currently, requestid field in logEntry is not populated, as the
requestid field gets set at the very end.
It is now set before regular handler functions. This is also
useful in setting it as part of the XML error response.

Travis build for ppc64le has been quite inconsistent and stays queued
for most of the time. Removing this build as part of Travis.yml for
the time being.
2018-07-20 18:46:32 -07:00
Harshavardhana
36ab615518 Remove unused functions and constants (#6175) 2018-07-20 23:37:43 +05:30
kannappanr
963a70053b
Do not trace HTTP body of PostPolicyBuckethandler (#6177) 2018-07-20 07:37:46 -07:00
Anis Elleuch
9c5e971a58 Add new console/http loggers (#6066)
- Add console target logging, enabled by default.
- Add http target logging, which supports an endpoint
  with basic authentication (username/password are passed
  in the endpoint url itself)
- HTTP target logging is asynchronous and some logs can be
  dropped if channel buffer (10000) is full
2018-07-19 15:55:06 -07:00
Anis Elleuch
b1c9eb0e01 Disable splitting lines in pretty error messages (#6171)
In a small window, UI error tries to split lines for an eye candy
error message. However, since we show some docs.minio.io links in some
error messages, these links are actually broken and not easily selected
in a X terminal. This PR changes the behavior and won't split lines
anymore.
2018-07-19 15:49:02 -07:00
kannappanr
43cc0096fa
Add support for deployment ID (#6144)
deployment ID helps in identifying a minio deployment in the case of remote
logging targets.
2018-07-18 20:17:35 -07:00
Anis Elleuch
e8a008f5b5 Better validation of all config file fields (#6090)
Add Validate() to serverConfig to call it at server
startup and in Admin SetConfig handler to minimize
errors scenario after server restart.
2018-07-18 11:22:29 -07:00
Andreas Auernhammer
758a80e39b crypto: add basic functionality for parsing SSE-C headers (#6148)
This commit adds basic support for SSE-C / SSE-C copy.
This includes functions for determining whether SSE-C
is requested by the S3 client and functions for parsing
such HTTP headers.

All S3 SSE-C parsing errors are exported such that callers
can pattern-match to forward the correct error to S3
clients.

Further the SSE-C related internal metadata entry-keys
are added by this commit.
2018-07-18 10:49:26 -07:00
wd256
3ec4738955 gcs: Use Pager to iterate results in ListObjectsV1/V2 (#6162)
Fixes #6052
2018-07-18 21:49:16 +05:30
Andreas Auernhammer
6c93c60424 crypto: add a basic KMS implementation (#6161)
This commit adds a basic KMS implementation for an
operator-specified SSE-S3 master key. The master key
is wrapped as KMS such that using SSE-S3 with master key
and SSE-S3 with KMS can use the same code.

Bindings for a remote / true KMS (like hashicorp vault)
will be added later on.
2018-07-17 22:40:34 -07:00
Krishna Srinivas
0c9f4c9092 formatMetaV1 should be "inherited" by disk format structs (#6134) 2018-07-16 20:26:42 -07:00
Andreas Auernhammer
289d6ce1d7 crypto: update SSE-S3 and SSE-C key derivation (#6152)
This commit updates the key derivation to reflect the
latest change of crypto/doc.go. This includes handling
the insecure legacy KDF.

Since #6064 is fixed, the 3. test case for object key
generation is enabled again.
2018-07-16 07:49:50 -07:00
Harshavardhana
db26d3c9e2 Fix handling files at leaf attempting disk.ListDir() (#6155)
Return an ignorable error upon readDir() failure on
a file.
2018-07-14 12:11:48 +05:30
Krishnan Parthasarathi
a1ef90be52 gcs: Limit number of objects listed to max-keys (#6133) 2018-07-13 10:27:26 -07:00
Nitish Tiwari
2aa18cafc6 Update federation target to etcd/clientv3 (#6119)
With CoreDNS now supporting etcdv3 as the DNS backend, we
can update our federation target to etcdv3. Users will now be
able to use etcdv3 server as the federation backbone.

Minio will update bucket data to etcdv3 and CoreDNS can pick
that data up and serve it as bucket style DNS path.
2018-07-12 14:12:40 -07:00
Andreas Auernhammer
adf7340394 fix size computation for en/decrypted objects (#6147)
This PR fixes the size calculation for encrypted multipart
objects.
2018-07-12 11:23:32 -07:00
Harshavardhana
b11a8eb3f4
Support multiple values for x-amz-meta header (#6145)
Fixes #5595
2018-07-12 09:40:14 -07:00
Andreas Auernhammer
15771ebe8d Fix decrypted object size and key derivation in CopyObjectPart (#6141)
This commit fixes the size calculation for multipart
objects. The decrypted size of an encrypted multipart
object is the sum of the decrypted part sizes.

Also fixes the key derivation in CopyObjectPart.
Instead of using the same object-encryption-key for each
part now an unique per-part key is derived.

Updates #6139
2018-07-12 21:59:56 +05:30
Praveen raj Mani
44865596db SignatureV4 validation with Metadata in the presignedUrl (#5894)
The `X-Amz-Meta-`/`X-Minio-Meta-` will now be recognized in query string also.

Fixes #5857 #5950
2018-07-10 20:27:10 -07:00
Anis Elleuch
be1700f595 Avoid startup abort when a notify target is down (#6126)
Minio server was preventing itself to start when any notification
target is down and not running. The PR changes the behavior by
avoiding startup abort in that case, so the user will still
be able to access Minio server using mc admin commands after
a restart or set config commands.
2018-07-10 07:20:31 +05:30
Krishna Srinivas
40ed0d1f5d Support 1GB disk size (#6137)
Pivotal CF by default has 1GB disk option which causes minio to not start
2018-07-09 18:23:49 -07:00
Andreas Auernhammer
b181a693fb fix object rebinding SSE-C security guarantee violation (#6121)
This commit fixes a weakness of the key-encryption-key
derivation for SSE-C encrypted objects. Before this
change the key-encryption-key was not bound to / didn't
depend on the object path. This allows an attacker to
repalce objects - encrypted with the same
client-key - with each other.

This change fixes this issue by updating the
key-encryption-key derivation to include:
 - the domain (in this case SSE-C)
 - a canonical object path representation
 - the encryption & key derivation algorithm

Changing the object path now causes the KDF to derive a
different key-encryption-key such that the object-key
unsealing fails.
Including the domain (SSE-C) and encryption & key
derivation algorithm is not directly neccessary for this
fix. However, both will be included for the SSE-S3 KDF.
So they are included here to avoid updating the KDF
again when we add SSE-S3.

The leagcy KDF 'DARE-SHA256' is only used for existing
objects and never for new objects / key rotation.
2018-07-09 17:18:28 -07:00
Mike Scarlett
c310cbbe89 Update comments regarding GCS component count (#6131) 2018-07-06 17:07:11 -07:00
poornas
1da362538b cache: allow ellipse style entries for MINIO_CACHE_DRIVES (#6088)
Fixes #5863
2018-07-03 16:54:10 -07:00
Krishna Srinivas
e40a5e05e1 Do notification in background to not block S3 client REST calls (#6005) 2018-07-03 11:09:36 -07:00
kannappanr
726e75611e
Do not log BucketNotFound errors on minio console (#6114) 2018-07-03 11:04:55 -07:00
Harshavardhana
80b3e9cb03 use appropriate HTTP status for storage is full (#6117) 2018-07-03 09:40:14 -07:00
Harshavardhana
6c85706c24 Use GetSourceIP for source ip as request params (#6109)
Fixes #6108
2018-07-02 14:40:18 -07:00
Praveen raj Mani
360f3f9335 Checking the existence of the bucket in DeleteObjectHandler (#6085)
Fixes #6077
2018-06-30 22:35:43 -07:00
wd256
25f9b0bc3b Handle ListObjectsV2 start-after parameter in ObjectLayer (#6078) 2018-07-01 09:52:45 +05:30
Harshavardhana
92a6676a2f Avoid unnecessary logging for policy not found errors (#6104) 2018-06-29 06:30:10 -07:00
Harshavardhana
e5e522fc61
docs: fix all Chinese doc links for the new docs site (#6097)
Additionally fix typos, default to US locale words
2018-06-28 16:02:02 -07:00
Harshavardhana
de251483d1 Avoid ticker timer to simplify disk usage (#6101)
This PR simplifies the code to avoid tracking
any running usage events. This PR also brings
in an upper threshold of upto 1 minute suspend
the usage function after which the usage would
proceed without waiting any longer.
2018-06-28 15:05:45 -07:00
Andreas Auernhammer
805186ab97 new package: cmd/crypto (#6062)
This commit introduces a new crypto package providing
AWS S3 related cryptographic building blocks to implement
SSE-S3 (master key or KMS) and SSE-C.

This change only adds some basic functionallity esp.
related to SSE-S3 and documents the general approach
for SSE-S3 and SSE-C.
2018-06-28 12:47:42 -07:00
Praveen raj Mani
ea76e72054 Incorrect error message for insufficient volume fix (#6099)
Reply back with appropriate error message when the server is spawn
with volume of insufficient size (< 1GiB).

Fixes #5993.
2018-06-28 12:01:05 -07:00
Harshavardhana
25de775560 disable disk-usage when export is root mount path (#6091)
disk usage crawling is not needed when a tenant
is not sharing the same disk for multiple other
tenants. This PR adds an optimization when we
see a setup uses entire disk, we simply rely on
statvfs() to give us total usage.

This PR also additionally adds low priority
scheduling for usage check routine, such that
other go-routines blocked will be automatically
unblocked and prioritized before usage.
2018-06-27 18:59:38 -07:00
Harshavardhana
abf209b1dd load bucket policies using object layer API (#6084)
This PR fixes an issue during gateway mode
where underlying policies were not translated
into meaningful policies.
2018-06-27 12:29:48 +05:30
Nitish Tiwari
ad79c626c6
Throw 404 for head requests for prefixes without trailing "/" (#5966)
Minio server returns 403 (access denied) for head requests to prefixes
without trailing "/", this is different from S3 behaviour. S3 returns
404 in such cases.

Fixes #6080
2018-06-26 06:54:00 +05:30
Andreas Auernhammer
cd152f404a replace os.Exit with panic for logger.CriticalIf (#6065)
This commit prevents complete server failures caused by
`logger.CriticalIf` calls. Instead of calling `os.Exit(1)`
the function now executes a panic with a special value
indicating that a critical error happend. At the top HTTP
handler layer panics are recovered and if its a critical
error the client gets an InternalServerError status code.

Further this allows unit tests to cover critical-error code
paths.
2018-06-25 13:51:49 -07:00
Praveen raj Mani
5fbdd70de9 Minio handle for list_objects_v2 with a blank ContinuationToken (#5940)
minio will now return an error for a blank continuation token in list_objects_v2,so as in s3.

Fixes #5931
2018-06-25 12:35:43 -07:00