Commit Graph

6440 Commits

Author SHA1 Message Date
Anis Elleuch 5c13765168 postgresql: Disable validation of connectionString field (#6397)
A bug concerning the validation of connectionString is found,
however there is no solution to fix it for now, postgresql API
doesn't help to do that hence disabling validation of that field.
2018-09-06 20:34:52 +05:30
Anis Elleuch 3099af70a3 Add admin get/set config keys API (#6113)
This PR adds two new admin APIs in Minio server and madmin package:
- GetConfigKeys(keys []string) ([]byte, error)
- SetConfigKeys(params map[string]string) (err error)

A key is a path in Minio configuration file, (e.g. notify.webhook.1)

The user will always send a string value when setting it in the config file,
the API will know how to convert the value to the appropriate type. The user
is also able to set a raw json.

Before setting a new config, Minio will validate all fields and try to connect
to notification targets if available.
2018-09-06 20:33:18 +05:30
Harshavardhana fd1b8491db
Drain response body properly for http connection pool (#6415)
Currently Go http connection pool was not being properly
utilized leading to degrading performance as the number
of concurrent requests increased.

As recommended by Go implementation, we have to drain the
response body and close it.
2018-09-05 16:47:14 -07:00
Anis Elleuch 1961f2ef54 xl: Fix removing an empty directory (#6421)
Removing an empty directory is not working because of xl.DeleteObject()
was only checking if the passed prefix is an actual object but it
should also check if it is an empty directory.
2018-09-05 16:38:03 -07:00
Harshavardhana 631c78e655
Bump up soMaxConn backlog for listener to 2048 (#6416)
soMaxConn value is 128 on almost all linux systems,
this value is too low for Minio at times when used
against large concurrent workload e.g: spark applications
this causes a sort of SYN flooding observed by the kernel
to allow for large backlog increase this value to 2048.

With this value we do not see anymore SYN flooding
kernel messages.
2018-09-05 13:16:19 -07:00
Harshavardhana e0f8b767ba Fail for critical errors early on during prepare storage (#6404) 2018-09-05 10:20:54 -07:00
Harshavardhana d0d015361c Fix config subsystem to wait on quorum number of formatted disks (#6407) 2018-09-05 20:55:55 +05:30
Krishna Srinivas 81b7e5c7a8 Send length instead of empty buffer for ReadFile() (#6414) 2018-09-04 23:22:05 -07:00
Harshavardhana 9e32cc283f Fix distributed docs to mention homogenous envs (#6405)
Also deprecate old syntax use only ellipses
2018-09-05 08:54:04 +05:30
Krishnan Parthasarathi 1126410e62 Implement ListMultipartUploads, ListObjectParts for GCS gateway (#6377)
ListMultipartUploads implementation is meant for docker-registry
use-case only. It lists only the first upload with a prefix matching
the object being uploaded.
2018-09-04 13:11:33 -07:00
Kanagaraj Mayilsamy 2c46214291 fix browser hang when listobjects is denied (#6385)
When Deny ListObjects policy is set on bucket, the browser
is hanging while trying to list the object. This is now
fixed by showing appropriate error message and an
empty object list.

Fixes #6371
2018-09-04 13:02:02 -07:00
Barnaby Keene d13bd5b9b5 Remove double backtick that was breaking docs (#6410)
On the documentation site, the double backtick with nothing in between was breaking the page render and making the text itself look quite awkward!
2018-09-04 12:06:57 -07:00
Minio Trusted c8c70a3750 Update yaml files to latest version RELEASE.2018-09-01T00-38-25Z 2018-09-01 00:43:06 +00:00
ebozduman 882a1a1ccc Stops listing objects for write-only access (#6396) 2018-08-31 13:20:27 -07:00
Harshavardhana 8690d62146 Allow fallback listen if first listener fails (#6380)
On linux listen() uses kernel features TCP_FASTOPEN, DEFER_ACCEPT

Fixes #6379
2018-08-31 13:17:05 -07:00
Anis Elleuch 85117d554f xl: Avoid removing a directory if it is not an object dir (#6395)
DeleteObject should not remove any directory unless the latter
is an empty directory.

Fixes #6394
2018-08-31 13:16:35 -07:00
Harshavardhana 4487f70f08 Revert all GetObjectNInfo related PRs (#6398)
* Revert "Encrypted reader wrapped in NewGetObjectReader should be closed (#6383)"

This reverts commit 53a0bbeb5b.

* Revert "Change SelectAPI to use new GetObjectNInfo API (#6373)"

This reverts commit 5b05df215a.

* Revert "Implement GetObjectNInfo object layer call (#6290)"

This reverts commit e6d740ce09.
2018-08-31 13:10:12 -07:00
Harshavardhana fb27388101 HTTP headers are case insensitive handle them appropriately (#6390)
An issue was reproduced when minio-js client functional
tests are setting lower case http headers, in our current
master branch we specifically look for canonical host header
which may be not necessarily true for all http clients.
This leads to a perpetual hang on the *net.Conn*.

This PR fixes regression caused by #6206 by handling the
case insensitivity.
2018-08-31 17:00:32 +05:30
Kaan Kabalak 5e7ccc983d UI: Handle policies listed in mc as 'none' by not showing them (#6353) 2018-08-30 15:20:27 -07:00
Bala FA 72fa2b4537 Add RPC counters for HTTP stats. (#6206)
This patch introduces separate counters for HTTP stats for minio
reserved bucket.

Fixes #6158
2018-08-30 14:17:58 +05:30
Praveen raj Mani 5399d91965 Add version for appveyor to use go1.10.4 (#6384) 2018-08-30 14:09:31 +05:30
Harshavardhana 53a0bbeb5b
Encrypted reader wrapped in NewGetObjectReader should be closed (#6383) 2018-08-29 19:18:00 -07:00
Harshavardhana 384a862940
Return quorum error based on disks in abortMultipartUpload (#6362)
Fixes #4980
2018-08-29 13:36:19 -07:00
Harshavardhana 029f52880b
With no read quorum config should be treated as notFound (#6374)
This will allow the config subsystem to initialize properly
in situations where many servers are coming up in a rolling
fashion.
2018-08-28 14:23:22 -07:00
Harshavardhana 5b05df215a
Change SelectAPI to use new GetObjectNInfo API (#6373)
This PR also removes some double checks
2018-08-28 13:08:30 -07:00
Harshavardhana a13cd7b7c4 Separate build functional tests from Makefile (#6351)
Recently travis seems to have issues with builds after
merge, so this PR is an experiment to fix this.

Failed: https://travis-ci.org/minio/minio/builds/419769285#L2310

Succeeded: https://travis-ci.org/minio/minio/builds/419565606#L2322

This PR was tested to be working properly on my fork to build
on travis, would need to do the same in minio/minio and see
how it goes through.
2018-08-28 13:57:01 +05:30
Anis Elleuch d524924b80 Fix gateway s3 doc to run custom S3 endpoint (#6369)
To pass a custom S3 endpoint in S3 gateway, the user needs
to specify it as an argument after 'minio gateway s3' and not
as '--address' option since this latter specifies the address
to which the gateway should listen.
2018-08-28 10:09:07 +05:30
Aditya Manthramurthy e6d740ce09 Implement GetObjectNInfo object layer call (#6290)
This combines calling GetObjectInfo and GetObject while returning a
io.ReadCloser for the object's body. This allows the two operations to
be under a single lock, fixing a race between getting object info and
reading the object body.
2018-08-27 15:28:23 +05:30
Harshavardhana cea4f82586
Revert "Remove curl after the usage (#6347)" (#6366)
This reverts commit dafa5073cb.

This commit revert also fixes #6365
2018-08-26 23:40:21 -07:00
Minio Trusted 1d6ce115da Update yaml files to latest version RELEASE.2018-08-25T01-56-38Z 2018-08-25 02:04:10 +00:00
Anis Elleuch 06d2dfa31c Fix WORM and BROWSER status calculation (#6360)
One typo introduced in a recent commit miscalculates if worm and browser
are enabled or not. A simple test is also added to detect this issue
in the future if it ever happens again.
2018-08-24 14:36:14 -07:00
poornas d547873b17 webhandler - display encryption errors properly (#6339)
For encrypted objects, download errors need to be
displayed in web response format instead of xml format.

Fixes #6327
2018-08-24 07:56:24 -07:00
Praveen raj Mani 01721a840a Build fix for go1.11rc1 (#6354)
Vendorized the following packages

- "github.com/rjeczalik/notify"
- "github.com/minio/highwayhash"

Fixes #6315
2018-08-24 13:29:17 +05:30
Krishna Srinivas 52f6d5aafc Rename of structs and methods (#6230)
Rename of ErasureStorage to Erasure (and rename of related variables and methods)
2018-08-23 23:35:37 -07:00
Harshavardhana 2211a5f1b8 Avoid ListenBucket targets to be listed in ServerInfo (#6340)
In current master when you do `mc watch` you can see a
dynamic ARN being listed which exposes the remote IP as well

```
mc watch play/airlines
```

On another terminal
```
mc admin info play
●  play.minio.io:9000
   Uptime : online since 11 hours ago
  Version : 2018-08-22T07:50:45Z
   Region :
 SQS ARNs : arn:minio:sqs::httpclient+51c39c3f-131d-42d9-b212-c5eb1450b9ee+73.222.245.195:33408
    Stats : Incoming 30GiB, Outgoing 7.6GiB
  Storage : Used 7.7GiB
```

SQS ARNs listed as part of ServerInfo should be only external targets,
since listing an ARN here is not useful and it cannot be re-purposed in
any manner.

This PR fixes this issue by filtering out httpclient from the ARN list.

This is a regression introduced in #5294 0e4431725c
2018-08-23 23:31:14 -07:00
Harshavardhana 1ffa6adcd4 Ignore io.EOF returned by ReadFrom for zero byte `fs.json` (#6346)
Fixes #6256
2018-08-24 11:34:21 +05:30
kannappanr add57a6938
Add content-length as part of event notification structure (#6341)
Fixes #6321
2018-08-23 14:40:54 -07:00
Manjunath A Kumatagi dafa5073cb Remove curl after the usage (#6347) 2018-08-23 10:48:29 -07:00
Praveen raj Mani 65e05a06fb Remove notifications Fix (#6082)
Remove all the notifications for an empty rulesMap

Fixes #6053
2018-08-23 22:53:18 +05:30
Dee Koder 5c9354894b Update Issue & PR Templates to have regression information when available (#6342) 2018-08-23 01:41:22 -07:00
Harshavardhana b01e69e08f
Initialize global object layer after all subsystems have initialized (#6333)
This is to ensure that object API operations are not performed
on a server on which subsystems are yet to be initialized.
2018-08-22 23:11:17 -07:00
Raphael Randschau 8601f29d95 select: fix int overflow of math.MaxInt64 on ARM (#6317) 2018-08-22 16:16:04 +05:30
Nitish Tiwari 0aee722e3f
Fix Minio browser screenshots in docs (#6334)
Fixes #6308
2018-08-22 13:15:36 +05:30
Harshavardhana beb6d40ce6
Avoid crash when policy subsystem is not initialized (#6326)
Fixes #6324
2018-08-21 15:38:51 -07:00
poornas 19db921555 CopyObject: fix regression in key rotation (#6331)
After key rotation, metadata was not being replaced with new sealed key.
Regression introduced in commit e71ef905f9
2018-08-21 15:12:00 -07:00
Minio Trusted 68b9e9e7e7 Update yaml files to latest version RELEASE.2018-08-21T00-37-20Z 2018-08-21 00:44:41 +00:00
kannappanr 2d84b02bc4 Check for absence of checksum field and attributes. (#6298)
Fixes #6295
2018-08-20 16:58:47 -07:00
poornas 8b2801bd46 Update documentation to show how to update minio config fields (#6301)
- with recent commit 1fb2e9ef95, config
can no longer be updated by editing config.json. This is because config
has been migrated inside the minio backend. Update documentation on
how to set/get configuration using mc admin config command.
2018-08-20 13:37:10 -07:00
Harshavardhana 7d7e21aebb Merge initConfig logic to ConfigSys (#6312) 2018-08-19 13:57:18 -07:00
Nitish Tiwari bf14e5ce1b Fix distributed doc as total storage capacity is not displayed anymore (#6309) 2018-08-18 10:01:24 -07:00