Commit Graph

5294 Commits

Author SHA1 Message Date
Anis Elleuch 5c765bc63e Fix one possible data race in admin tests (#6537)
go test shows the following warning:
```
WARNING: DATA RACE
Write at 0x000002909e18 by goroutine 276:
  github.com/minio/minio/cmd.testAdminCmdRunnerSignalService()
      /home/travis/gopath/src/github.com/minio/minio/cmd/admin-rpc_test.go:44 +0x94

 Previous read at 0x000002909e18 by goroutine 194:
  github.com/minio/minio/cmd.testServiceSignalReceiver()
      /home/travis/gopath/src/github.com/minio/minio/cmd/admin-handlers_test.go:467 +0x70
```

The reason for this data race is that some admin tests are not waiting for go routines
that they created to be properly exited, which triggers the race detector.
2018-09-27 17:16:30 -07:00
Anis Elleuch 6c7c6bec91 profiler: Download API returns error when all nodes fail (#6525)
When download profiling data API fails to gather profiling data
from all nodes for any reason (including profiler not enabled),
return 400 http code with the appropriate json message.
2018-09-27 10:34:37 -07:00
poornas ed703c065d Add ObjectOptions to GetObjectNInfo (#6533) 2018-09-27 15:36:45 +05:30
Aditya Manthramurthy 387584356f Remove unused range parsing code and update tests (#6530) 2018-09-27 15:24:07 +05:30
Harshavardhana 1111419d4a Add debugging for mutex, tracing (#6522) 2018-09-27 09:32:05 +05:30
Anis Elleuch 20378821cf madmin: close http response when returning an error (#6526)
httpRespToErrorResponse() usually reads the http response when
the http error code is not expected to parse the json error
response in the http body, however it was never properly closing
the connection. This PR fixes the behavior.
2018-09-26 11:03:35 -07:00
Pontus Leitzler ce1bfa6de8 Removed unused vendored dependencies (#6520) 2018-09-25 14:51:51 -07:00
Minio Trusted 6c26227081 Update yaml files to latest version RELEASE.2018-09-25T21-34-43Z 2018-09-25 21:39:03 +00:00
Anis Elleuch aa4e2b1542 Use GetObjectNInfo in CopyObject and CopyObjectPart (#6489) 2018-09-25 12:39:46 -07:00
Harshavardhana 1e5ac39ff3 Return HTTP response error for malformed requests (#6517) 2018-09-25 11:07:26 -07:00
Harshavardhana ec2295c3dc Quickly build dev docker images using 'make docker' (#6505)
This PR simplifies the process of developer build of local
docker containers using `make docker`.

You need to provide a TAG i.e

```
TAG=y4m4/minio:exp make docker
```
2018-09-25 10:33:25 -07:00
Andreas Auernhammer 8cf7b88cc5 add functions to remove confidential information (#6516)
This commit adds two functions for removing
confidential information - like SSE-C keys -
from HTTP headers / object metadata.

This creates a central point grouping all
headers/entries which must be filtered / removed.

See also https://github.com/minio/minio/pull/6489#discussion_r219797993
of #6489
2018-09-24 21:02:51 +05:30
Harshavardhana 48bfebe442 HEAD on an object should mimic GET without body (#6508)
Add "Range" header support etc.
2018-09-23 22:54:10 +05:30
Pontus Leitzler df60b3c733 Remove unnecessary contexts passed as data to FatalIf. No need to log an empty context. (#6487) 2018-09-21 16:04:11 -07:00
Aditya Manthramurthy 584cb61bb8 Switch back to GetObjectInfo for HEAD requests (#6513) 2018-09-21 13:48:58 -07:00
kannappanr c1798cc89a
Update GCS storage library to support GetObject API on gzipped objects (#6506)
Fixes #6280
2018-09-21 11:43:10 -07:00
Aditya Manthramurthy 3c8fabd116 Fix cleanup of pipe in GetObjectNInfo handlers (#6509) 2018-09-21 11:42:06 -07:00
Aditya Manthramurthy 36e51d0cee Add GetObjectNInfo to object layer (#6449)
The new call combines GetObjectInfo and GetObject, and returns an
object with a ReadCloser interface.

Also adds a number of end-to-end encryption tests at the handler
level.
2018-09-20 19:22:09 -07:00
Harshavardhana 7d0645fb3a Deprecate domain, browser as config entries (#6498) 2018-09-20 14:56:32 -07:00
Kanagaraj M 7c339e248a fix success alert not shown after object is uploaded (#6500) 2018-09-20 22:54:09 +05:30
Harshavardhana b62ed5dc90 select API CSV may not be specified (#6493)
This should be present until we support JSON
2018-09-20 15:04:26 +05:30
Harshavardhana f0641a0406 Avoid changing creds in streaming signature (#6495)
This PR fixes a potential issue where credentials can be
changed in middle of different chunks during data transfer
which can lead to unexpected bugs.
2018-09-19 16:52:05 -07:00
Harshavardhana 3d060f8b64 Peer/rpc should never honor RPC calls without object-layer (#6486)
Fixes #6484
2018-09-19 21:32:56 +05:30
Jay Mundrawala 052a7b8eec Allow minio s3 gateway to use different AWS auth mechanisms (#6422)
Allow minio s3 gateway to use aws environment credentials,
IAM instance credentials, or AWS file credentials.

If AWS_ACCESS_KEY_ID, AWS_SECRET_ACCSES_KEY are set, 
or minio is running on an ec2 instance with IAM instance credentials, 
or there is a file $HOME/.aws/credentials, minio running as an S3
gateway will authenticate with AWS S3 using those one of credentials.

The lookup order:
1. AWS environment varaibles
2. IAM instance credentials
3. $HOME/.aws/credentials
4. minio environment variables

To authenticate with the minio gateway, you will always use the
minio environment variables MINIO_ACCESS_KEY MINIO_SECRET_KEY.
2018-09-19 18:05:30 +05:30
Anis Elleuch 9531cddb06 Add Profiler Admin API (#6463)
Two handlers are added to admin API to enable profiling and disable
profiling of a server in a standalone mode, or all nodes in the
distributed mode.

/minio/admin/profiling/start/{cpu,block,mem}:
  - Start profiling and return starting JSON results, e.g. one
    node is offline.

/minio/admin/profiling/download:
  - Stop the on-going profiling task
  - Stream a zip file which contains all profiling files that can
    be later inspected by go tool pprof
2018-09-18 16:46:35 -07:00
Harshavardhana 6fe9a613c0 Prioritize HTTP requests over Heal (#6468)
Additionally also heal 256 objects at any given
time in parallel.

Fixes #6196
Fixes #6241
2018-09-17 18:28:34 -07:00
Andreas Auernhammer b729a4e83c Remove brittle TestServerTLSCiphers unit test (#5982)
The test TestServerTLSCiphers seems to fail sometimes for
no obvious reason. Actually the test is not needed
(as unit test) since minio/mint tests the server's TLS ciphers
as part of its security tests.

Fixes #5977
2018-09-17 15:55:09 +05:30
Harshavardhana a0683d3c1f Send progress only when requested by client in SelectObject (#6467) 2018-09-17 11:52:46 +05:30
Anis Elleuch 66fda7a37f Use retry mechanism when initializing configuration (#6475)
Currently, one node in a cluster can fail to boot with the following error message:

```
ERROR Unable to initialize config system: Storage resources are insufficient for the write operation
```

  This happens when disks are formatted, read quorum is met but write
quorum is not met. In checkServerConfig(), a insufficient read quorum
error is replaced by errConfigNotFound, the code will generate a
new config json and try to save it, but it will fail because write
quorum is not met.

  Replacing read quorum with errConfigNotFound is also wrong because it
can lead, in rare cases, to overwrite the config set by the user.

  So, this commit adds a retry mechanism in configuration initialization
to retry only with read or write quorum errors.

  This commit will also fix the following cases:
 - Read quorum is lost just after the initialization of the object layer.
 - Write quorum not met when upgrading configuration version.
2018-09-15 22:09:51 -07:00
Harshavardhana a63bc9254d Add 'disk' tag to log output to enhance 'disk not found' errors (#6460) 2018-09-13 21:42:50 -07:00
poornas 14fa0097b0 fix: UploadPart,CopyObjectPart does not need sse-s3 header (#6386)
S3 API spec for UploadPart requires encryption headers to be
specified only for SSE-C
2018-09-13 14:53:03 -07:00
Kanagaraj M d99c397746 fix public directory listing not working in browser (#6466)
Fixes #6436
2018-09-13 14:43:50 -07:00
Harshavardhana e3777b1dd9 Combine obtaining resource, host, method into one operation (#6465)
This also adds a reduced timeout for errant connections, to
be quickly closed if they can't even send HTTP headers properly.

Fixes #6459
2018-09-13 18:17:03 +05:30
Minio Trusted 63c03758e6 Update yaml files to latest version RELEASE.2018-09-12T18-49-56Z 2018-09-12 18:55:31 +00:00
Anis Elleuch ce419c9835 Bump RPC version after ReadFile RPC arguments change (#6457)
ReadFile RPC input argument has been changed in commit a8f5939452959d27674560c6b803daa9,
however, RPC doesn't detect such a change when it calls other nodes with older versions.

Hence, bumping RPC version.

Fixes #6458
2018-09-12 10:45:28 -07:00
Harshavardhana 0c2b708484 for O_RDONLY mode hold shared locks on windows (#6454)
Fixes #6401
2018-09-12 09:29:41 -07:00
Harshavardhana 166e998788
Fix healthcheck for NAS gateway (#6452)
It was expected that in gateway mode, we do not know
the backend types whereas in NAS gateway since its
an extension of FS mode (standalone) this leads to
an issue in LivenessCheckHandler() which would perpetually
return 503, this would affect all kubernetes, openshift
deployments of NAS gateway.
2018-09-11 13:44:10 -07:00
Andreas Auernhammer 267a0a3dfa fix `X-Amz-Credential` parsing for V4 policy signature (#6451)
This commit fixes an AWS S3 incompatibility issue.
The AccessKeyID may contain one or more `/` which caused
the server to interpret parts of the AccessKeyID as
other `X-Amz-Credential` parameters (like date, region, ...)

This commit fixes this by allowing 5 or more
`X-Amz-Credential` parameter strings and only interpreting
the last 5.

Fixes #6443
2018-09-11 11:17:23 -07:00
Minio Trusted 985fd7d4e7 Update yaml files to latest version RELEASE.2018-09-11T01-39-21Z 2018-09-11 01:43:45 +00:00
Harshavardhana 5479a6e33e Ignore migration if config files don't exist (#6448) 2018-09-10 18:24:57 -07:00
ebozduman fb4186f6b9 Adds missing info to docs for credentials and domain env. vars. (#6447)
* Adds missing information to documentation for credentials and domain environment variables for distributed minio server startup.
2018-09-10 17:14:40 -07:00
Anis Elleuch 7571582000 Print storage errors during distributed initialization (#6441)
This commit will print connection failures to other disks in other nodes
after 5 retries. It is useful for users to understand why the
distribued cluster fails to boot up.
2018-09-10 16:21:59 -07:00
Harshavardhana 12b4971b70 Rename config.json in config-dir with '.deprecated' extension (#6446)
Fixes #6444
2018-09-10 16:15:47 -07:00
poornas 5c0b98abf0 Add ObjectOptions to ObjectLayer calls (#6382) 2018-09-10 09:42:43 -07:00
Praveen raj Mani 30d4a2cf53 s3select should honour custom record delimiter (#6419)
Allow custom delimiters like `\r\n`, `a`, `\r` etc in input csv and 
replace with `\n`.

Fixes #6403
2018-09-10 21:50:28 +05:30
Anis Elleuch 92bc7caf7a Reword missing credentials error msg (#6418)
Enhance a little bit the error message that is showing
when access & secret keys are not specified in the
environment when running Minio in gateway and server mode.

This commit also removes a redundant check of access/secret keys.
2018-09-09 22:51:48 +05:30
Harshavardhana 19202bae81 Allow backward compatible way to load creds from config.json (#6435)
Print warning message for users to migrate to newer style of distributed
deployment by always setting credentials as ENVs.

Fixes #6434
2018-09-07 11:18:49 -07:00
Praveen raj Mani e7a4512a90 Redis documentation Fix (#6378)
Fixed few typos and missing `format` field in the example config provided.

Fixes #6356
2018-09-07 07:12:01 -07:00
Kaan Kabalak ff822e64ca Show pointer when user hovers over folder name (#6412)
To indicate that there is an action tied when the user clicks on the
folder name on the Minio Browser, the default cursor has been replaced
with a pointer on folder name hover. This is a regression from #6193.

Fixes #6411
2018-09-07 14:15:53 +05:30
Annanay Agarwal 7cb87f863e Kafka (sarama) authentication with user/pass (#6291) 2018-09-07 00:01:58 -07:00