Commit Graph

1302 Commits

Author SHA1 Message Date
Nitish Tiwari f30c95a301 Add etcd handling for web-handler methods 2018-06-08 10:22:01 -07:00
Harshavardhana 481390d51a Converge etcd functionality as part of quick.Config 2018-06-08 10:22:01 -07:00
Nitish 6df1e4a529 Add functionality to add old buckets to etcd on startup
Buckets already present on a Minio server before it joins a
bucket federated deployment will now be added to etcd during
startup. In case of a bucket name collision, admin is informed
via Minio server console message.

Added configuration migration for configuration stored in etcd
backend.

Also, environment variables are updated and ListBucket path style
request is no longer forwarded.
2018-06-08 10:22:01 -07:00
Harshavardhana 853ea371ce Bring etcd support for bucket DNS federation
- Supports centralized `config.json`
- Supports centralized `bucket` service records
  for client lookups
- implement a new proxy forwarder
2018-06-08 10:22:01 -07:00
Harshavardhana 617a6d8e47 Add request tracing support for other REST APIs (#6013) 2018-06-07 10:41:13 -07:00
Praveen raj Mani c0cfe21c00 Ignore region in the case of admin API (#5919)
Admin API is not an S3 API and hence it is not required
to honor server region while validating admin API calls.

Fixes #2411
2018-06-07 10:37:31 -07:00
Bala FA 6a8bfcef1c remove separate file for posix utils. (#5948) 2018-06-07 12:31:40 +05:30
Harshavardhana 6138cae8e7 Persist MINIO_WORM as part of config.json (#6022) 2018-06-06 18:10:51 -07:00
Krishna Srinivas 113570b514 Refresh in-memory bucket policy cache every 5 minutes (#6007) 2018-06-06 12:52:56 -07:00
Bala FA 6a53dd1701 Implement HTTP POST based RPC (#5840)
Added support for new RPC support using HTTP POST.  RPC's 
arguments and reply are Gob encoded and sent as HTTP 
request/response body.

This patch also removes Go RPC based implementation.
2018-06-06 14:21:56 +05:30
Harshavardhana 3143454982 Deprecate and remove configurable disk usage check (#6016) 2018-06-05 18:53:44 -07:00
Harshavardhana eafc15cd47 Fix presigned URL for access key with special characters (#6012)
Fixes #6011
2018-06-05 10:48:51 -07:00
Harshavardhana 6fb0604502 Allow usage check to be configurable (#6006) 2018-06-04 18:35:41 -07:00
Harshavardhana df1b33013f Fix byte pool usage, use only one pool for all sets. (#5990) 2018-06-01 16:41:23 -07:00
Harshavardhana 537fd8c821 Bring nicer ui error if creds are missing in Gateway (#6003) 2018-06-01 15:45:11 -07:00
Krishna Srinivas 69b9d6fbee Add user-agent for azure blob requests (#6002) 2018-06-01 15:44:48 -07:00
Harshavardhana c22b9d5d4d
Fix s3cmd issue with ACL handling (#5999)
With the implementation of dummy GET ACL handlers,
tools like s3cmd perform few operations which causes
the ACL call to be invoked. Make sure that in our
router configuration GET?acl comes before actual
GET call to facilitate this dummy call.
2018-05-31 19:43:50 -07:00
Harshavardhana c247e603d2 terminate disk-usage routine in tests (#5998)
tests were written in the manner by editing internal
variables of fsObjects to mimic certain behavior from
APIs, but this is racy when an active go-routine is
reading from the same variable.

Make sure to terminate the go-routine if possible for
these tests.
2018-05-31 14:03:46 -07:00
Harshavardhana 74328c3061 Implement auto cert reloading (#5963) 2018-05-31 12:30:15 -07:00
Harshavardhana 487ecedc51 fix admin info peer to point to first endpoint (#5996)
The current problem is that when you invoke

```
mc admin info myminio | head -1
●  localhost:9000
```

This output is incorrect as the expected output should be
```
mc admin info myminio | head -1
●  192.168.1.17:9000
```
2018-05-31 08:52:21 +05:30
Andreas Auernhammer 9fb94e6aa8 ensure authenticated request bodies for Admin-API (#5984)
This commit adds a check to the server's admin-API such that it only
accepts Admin-API requests with authenticated bodies. Further this
commit updates the `madmin` package to always add the
`X-Amz-Content-Sha256` header.

This change improves the Admin-API security since the server does not
accept unauthenticated request bodies anymore.

After this commit `mc` must be updated to the new `madmin` api because
requests over TLS connections will fail.
2018-05-30 14:49:03 -07:00
Harshavardhana 5282639f3c Add prometheus endpoint to support total Used storageInfo (#5988)
Since we deprecated Total/Free we don't need to update
prometheus with those metrics. This PR also adds support
for caching implementation.
2018-05-30 11:30:14 -07:00
Arjun Mishra dd0db526d9 Remove quotes for XSS Protection Header (#5992) 2018-05-30 14:59:27 +05:30
Yaroslav Skopets a50cc7e937 Add Prometheus metrics for Minio gateway (#5987) 2018-05-30 10:13:46 +05:30
Harshavardhana 000e360196 Deprecate showing drive capacity and total free (#5976)
This addresses a situation that we shouldn't be
displaying Total/Free anymore, instead we should simply
show the total usage.
2018-05-23 17:30:25 -07:00
Harshavardhana e6ec645035 Implement support for calculating disk usage per tenant (#5969)
Fixes #5961
2018-05-23 15:41:29 +05:30
Harshavardhana 71c4ff9d10 Fix tabs/indents in help message (#5962) 2018-05-21 11:11:57 -07:00
Harshavardhana ac58283001 Support endpoints through env (#5952) 2018-05-18 17:51:03 -07:00
Andreas Auernhammer 9c8b7306f5 security: fix write-to-RAM DoS vulnerability (#5957)
This commit fixes a DoS vulnerability for certain APIs using
signature V4 by verifying the content-md5 and/or content-sha56 of
the request body in a streaming mode.

The issue was caused by reading the entire body of the request into
memory to verify the content-md5 or content-sha56 checksum if present.

The vulnerability could be exploited by either replaying a V4 request
(in the 15 min time frame) or sending a V4 presigned request with a
large body.
2018-05-18 11:27:25 -07:00
kannappanr ffa6b45d00 Remove treewalk abort log message (#5953)
Log doesn't add any value and can be removed.

Fixes #5943
2018-05-17 17:26:16 -07:00
Bala FA 4eb788df79 rename checkPathValid() to getValidPath() (#5949) 2018-05-17 07:27:07 -07:00
Nitish Tiwari e6ebcc4cb6 Remove redundant prometheus data points (#5934)
Removed field minio_http_requests_total as it was redundant with
minio_http_requests_duration_seconds_count

Also removed field minio_server_start_time_seconds as it was
redundant with process_start_time_seconds
2018-05-15 12:23:43 -07:00
Harshavardhana c872c30ea3 fix: introduce isLeafDir in healing to fix the crash (#5920)
This PR also supports healing directories.

Fixes #5917
2018-05-10 16:53:42 -07:00
Harshavardhana 5b74f918d4 Implement a dummy GetBucketACL for legacy reasons (#5906)
GetBucketACL call returns empty for all GET in ACL requests, 
the primary purpose of this PR is to provide legacy API support 
for legacy applications.

Fixes #5706
2018-05-10 09:32:26 +05:30
Krishna Srinivas cc8178cdc4 Log errors only once for event notification errors (#5905) 2018-05-09 15:59:45 -07:00
Harshavardhana 4886bfbc72 fix: Avoid more crashes due to concurrent map usage (#5912)
This PR fixes another situation where a crash occurs
thanks to @krishnasrinivas for reproducing this

Fixes #5897
2018-05-09 15:11:51 -07:00
Anis Elleuch c525424179 Return detailed msg for notification errors (#5913) 2018-05-09 15:11:24 -07:00
Anis Elleuch c1f1fa3cc8 More fixes for error messages (#5911) 2018-05-09 13:29:37 -07:00
Harshavardhana 98f81ced86 fix: Avoid concurrent map writes in go-routines (#5898)
Fixes #5897
2018-05-09 11:25:38 -07:00
Praveen raj Mani 3971483e18 Remove trailing/leading spaces in Certificates (#5885)
Fixes:  #5632
2018-05-09 14:26:38 +05:30
Ashish Kumar Sinha deb685c5b5 Enhancements in Minio Prometheus exporter (#5848)
Standardized Minio collectors based on Prometheus 
recommendations.
2018-05-09 01:38:27 -07:00
Krishna Srinivas bb34bd91f1 Fix unnecessary log messages to avoid flooding the logs (#5900) 2018-05-09 01:38:27 -07:00
Krishna Srinivas 9de8fefa90 Fix azure list-objects not to list minio.sys.tmp/ (#5895) 2018-05-09 01:38:27 -07:00
Anis Elleuch 6d5f2a4391 Better support of empty directories (#5890)
Better support of HEAD and listing of zero sized objects with trailing
slash (a.k.a empty directory). For that, isLeafDir function is added
to indicate if the specified object is an empty directory or not. Each
backend (xl, fs) has the responsibility to store that information.
Currently, in both of XL & FS, an empty directory is represented by
an empty directory in the backend.

isLeafDir() checks if the given path is an empty directory or not,
since dir listing is costly if the latter contains too many objects,
readDirN() is added in this PR to list only N number of entries.
In isLeadDir(), we will only list one entry to check if a directory
is empty or not.
2018-05-09 01:38:21 -07:00
Anis Elleuch 32700fca52 Enhance fatal errors printing of common issues seen by users (#5878) 2018-05-08 19:04:36 -07:00
Harshavardhana 54cd29b51c Add new S3 APIs which are not implemented (#5899)
Fixes #5896
2018-05-08 21:45:10 +05:30
Harshavardhana c98d8cb1c7 fs: fix a regression allow reading of existing files (#5889) 2018-05-07 17:00:44 -07:00
Andreas Auernhammer c5a00e513c fix DoS vulnerability in request authentication (#5887)
This commit fixes a DoS vulnerability in the
request authentication. The root cause is an 'unlimited'
read-into-RAM from the request body.

Since this read happens before the request authentication
is verified the vulnerability can be exploit without any
access privileges.

This commit limits the size of the request body to 3 MB.
This is about the same size as AWS. The limit seems to be
between 1.6 and 3.2 MB - depending on the AWS machine which
is handling the request.
2018-05-04 11:16:14 -07:00
Anis Elleuch 9439dfef64 Use defer style to stop tickers to avoid current/possible misuse (#5883)
This commit ensures that all tickers are stopped using defer ticker.Stop()
style. This will also fix one bug seen when a client starts to listen to
event notifications and that case will result a leak in tickers.
2018-05-04 10:43:20 -07:00
Krishna Srinivas 0f746a14a3 Do not use crypto.SHA3_256 as placeholder for HighwayHash256 (#5847) 2018-05-04 10:42:22 -07:00