Commit Graph

1372 Commits

Author SHA1 Message Date
Andreas Auernhammer
cd152f404a replace os.Exit with panic for logger.CriticalIf (#6065)
This commit prevents complete server failures caused by
`logger.CriticalIf` calls. Instead of calling `os.Exit(1)`
the function now executes a panic with a special value
indicating that a critical error happend. At the top HTTP
handler layer panics are recovered and if its a critical
error the client gets an InternalServerError status code.

Further this allows unit tests to cover critical-error code
paths.
2018-06-25 13:51:49 -07:00
Praveen raj Mani
5fbdd70de9 Minio handle for list_objects_v2 with a blank ContinuationToken (#5940)
minio will now return an error for a blank continuation token in list_objects_v2,so as in s3.

Fixes #5931
2018-06-25 12:35:43 -07:00
Ashish Kumar Sinha
0bbdd02a57 Updating disk storage for FS/Erasure mode (#6081)
Updating the disk storage stats for FS/Erasure coded backend
2018-06-25 10:46:48 -07:00
Annanay Agarwal
78abe5234e Add functionality to make cache limit configurable (#5991) 2018-06-25 10:24:12 -07:00
Nitish Tiwari
eb0e56ccf6 Update content-language field for Azure gateway (#6061)
Fixes https://github.com/minio/minio-go/issues/996
2018-06-21 09:46:45 -07:00
Aditya Manthramurthy
670b538dde Fix server crash when no system certificates are found (#6055) 2018-06-19 13:38:22 -07:00
Nitish Tiwari
186000328e Update StorageClass from Objectinfo (#6054)
See #6051
2018-06-19 11:22:08 -07:00
Harshavardhana
2575f4198a Auto-probe backend signature support (#6044) 2018-06-18 15:27:14 -07:00
kannappanr
577d10674d
Cleanup stack trace in error logs (#6045)
Add compile time GOROOT path to the list of prefix
of file paths to be removed.

Add webhandler function names to the slice that
stores function names to terminate logging.
2018-06-18 12:04:46 -07:00
kannappanr
81ee79b042
Do not log browser login failures (#6049) 2018-06-18 11:49:49 -07:00
kannappanr
001d9a4ae7
Remove unwanted volume access denied log (#6042)
Fixes #6039
2018-06-18 11:43:41 -07:00
Harshavardhana
c3a5146422 Presign content-sha256 should look for query and header (#6046) 2018-06-15 14:21:17 -07:00
Nitish Tiwari
36c39d04da Update vendorized minio-go to support start-after param (#6043)
Fixes #6032
2018-06-14 15:08:02 -07:00
Harshavardhana
28d526bc68 Change CriticalIf to FatalIf for proper error message (#6040)
During startup until the object layer is initialized
logger is disabled to provide for a cleaner UI error
message. CriticalIf is disabled, use FatalIf instead.

Also never call os.Exit(1) on running servers where
you can return error to client in handlers.
2018-06-14 10:17:07 -07:00
Harshavardhana
05f96f3956 Avoid allocating larger buffer if the incoming body is small (#6035) 2018-06-13 11:55:12 -07:00
Harshavardhana
cb9ee1584a Fix TestHealStartNStatusHandler sporadic failure (#6015)
Fixes #5818
2018-06-12 16:36:31 -07:00
Andreas Auernhammer
9f4c120731 limit memory allocations during multiple object deletion (#6033)
This commit limits the amount of memory allocated by the
S3 Multi-Object-Delete-API. The server used to allocate as
many bytes as provided by the client using Content-Length.

S3 specifies that the S3 Multi-Object-Delete-API can delete
at most 1000 objects using a single request.
(See: https://docs.aws.amazon.com/AmazonS3/latest/API/multiobjectdeleteapi.html)
Since the maximum S3 object name is limited to 1024 bytes the
XML body sent by the client can only contain up to 1000 * 1024
bytes (excluding XML format overhead).

This commit limits the size of the parsed XML for the S3
Multi-Object-Delete-API to 2 MB. This fixes a DoS
vulnerability since (auth.) clients, MitM-adversaries
(without TLS) and un-auth. users accessing buckets allowing
multi-delete by policy can kill the server.

This behavior is similar to the AWS-S3 implementation.
2018-06-11 10:51:38 -07:00
Harshavardhana
371349787f Remove region requirement for Healing (#6031) 2018-06-08 17:54:57 -07:00
Nitish Tiwari
3dc13323e5 Use random host from among multiple hosts to create requests
Also use hosts passed to Minio startup command to populate IP
addresses if MINIO_PUBLIC_IPS is not set.
2018-06-08 10:22:01 -07:00
Nitish Tiwari
6ce7265c8c Add support for CopyObject across regions and multiple Minio IPs
This PR adds CopyObject support for objects residing in buckets
in different Minio instances (where Minio instances are part of
a federated setup).

Also, added support for multiple Minio domain IPs. This is required
for distributed deployments, where one deployment may have multiple
nodes, each with a different public IP.
2018-06-08 10:22:01 -07:00
Nitish Tiwari
f30c95a301 Add etcd handling for web-handler methods 2018-06-08 10:22:01 -07:00
Harshavardhana
481390d51a Converge etcd functionality as part of quick.Config 2018-06-08 10:22:01 -07:00
Nitish
6df1e4a529 Add functionality to add old buckets to etcd on startup
Buckets already present on a Minio server before it joins a
bucket federated deployment will now be added to etcd during
startup. In case of a bucket name collision, admin is informed
via Minio server console message.

Added configuration migration for configuration stored in etcd
backend.

Also, environment variables are updated and ListBucket path style
request is no longer forwarded.
2018-06-08 10:22:01 -07:00
Harshavardhana
853ea371ce Bring etcd support for bucket DNS federation
- Supports centralized `config.json`
- Supports centralized `bucket` service records
  for client lookups
- implement a new proxy forwarder
2018-06-08 10:22:01 -07:00
Harshavardhana
617a6d8e47 Add request tracing support for other REST APIs (#6013) 2018-06-07 10:41:13 -07:00
Praveen raj Mani
c0cfe21c00 Ignore region in the case of admin API (#5919)
Admin API is not an S3 API and hence it is not required
to honor server region while validating admin API calls.

Fixes #2411
2018-06-07 10:37:31 -07:00
Bala FA
6a8bfcef1c remove separate file for posix utils. (#5948) 2018-06-07 12:31:40 +05:30
Harshavardhana
6138cae8e7 Persist MINIO_WORM as part of config.json (#6022) 2018-06-06 18:10:51 -07:00
Krishna Srinivas
113570b514 Refresh in-memory bucket policy cache every 5 minutes (#6007) 2018-06-06 12:52:56 -07:00
Bala FA
6a53dd1701 Implement HTTP POST based RPC (#5840)
Added support for new RPC support using HTTP POST.  RPC's 
arguments and reply are Gob encoded and sent as HTTP 
request/response body.

This patch also removes Go RPC based implementation.
2018-06-06 14:21:56 +05:30
Harshavardhana
3143454982 Deprecate and remove configurable disk usage check (#6016) 2018-06-05 18:53:44 -07:00
Harshavardhana
eafc15cd47 Fix presigned URL for access key with special characters (#6012)
Fixes #6011
2018-06-05 10:48:51 -07:00
Harshavardhana
6fb0604502 Allow usage check to be configurable (#6006) 2018-06-04 18:35:41 -07:00
Harshavardhana
df1b33013f Fix byte pool usage, use only one pool for all sets. (#5990) 2018-06-01 16:41:23 -07:00
Harshavardhana
537fd8c821 Bring nicer ui error if creds are missing in Gateway (#6003) 2018-06-01 15:45:11 -07:00
Krishna Srinivas
69b9d6fbee Add user-agent for azure blob requests (#6002) 2018-06-01 15:44:48 -07:00
Harshavardhana
c22b9d5d4d
Fix s3cmd issue with ACL handling (#5999)
With the implementation of dummy GET ACL handlers,
tools like s3cmd perform few operations which causes
the ACL call to be invoked. Make sure that in our
router configuration GET?acl comes before actual
GET call to facilitate this dummy call.
2018-05-31 19:43:50 -07:00
Harshavardhana
c247e603d2 terminate disk-usage routine in tests (#5998)
tests were written in the manner by editing internal
variables of fsObjects to mimic certain behavior from
APIs, but this is racy when an active go-routine is
reading from the same variable.

Make sure to terminate the go-routine if possible for
these tests.
2018-05-31 14:03:46 -07:00
Harshavardhana
74328c3061 Implement auto cert reloading (#5963) 2018-05-31 12:30:15 -07:00
Harshavardhana
487ecedc51 fix admin info peer to point to first endpoint (#5996)
The current problem is that when you invoke

```
mc admin info myminio | head -1
●  localhost:9000
```

This output is incorrect as the expected output should be
```
mc admin info myminio | head -1
●  192.168.1.17:9000
```
2018-05-31 08:52:21 +05:30
Andreas Auernhammer
9fb94e6aa8 ensure authenticated request bodies for Admin-API (#5984)
This commit adds a check to the server's admin-API such that it only
accepts Admin-API requests with authenticated bodies. Further this
commit updates the `madmin` package to always add the
`X-Amz-Content-Sha256` header.

This change improves the Admin-API security since the server does not
accept unauthenticated request bodies anymore.

After this commit `mc` must be updated to the new `madmin` api because
requests over TLS connections will fail.
2018-05-30 14:49:03 -07:00
Harshavardhana
5282639f3c Add prometheus endpoint to support total Used storageInfo (#5988)
Since we deprecated Total/Free we don't need to update
prometheus with those metrics. This PR also adds support
for caching implementation.
2018-05-30 11:30:14 -07:00
Arjun Mishra
dd0db526d9 Remove quotes for XSS Protection Header (#5992) 2018-05-30 14:59:27 +05:30
Yaroslav Skopets
a50cc7e937 Add Prometheus metrics for Minio gateway (#5987) 2018-05-30 10:13:46 +05:30
Harshavardhana
000e360196 Deprecate showing drive capacity and total free (#5976)
This addresses a situation that we shouldn't be
displaying Total/Free anymore, instead we should simply
show the total usage.
2018-05-23 17:30:25 -07:00
Harshavardhana
e6ec645035 Implement support for calculating disk usage per tenant (#5969)
Fixes #5961
2018-05-23 15:41:29 +05:30
Harshavardhana
71c4ff9d10 Fix tabs/indents in help message (#5962) 2018-05-21 11:11:57 -07:00
Harshavardhana
ac58283001 Support endpoints through env (#5952) 2018-05-18 17:51:03 -07:00
Andreas Auernhammer
9c8b7306f5 security: fix write-to-RAM DoS vulnerability (#5957)
This commit fixes a DoS vulnerability for certain APIs using
signature V4 by verifying the content-md5 and/or content-sha56 of
the request body in a streaming mode.

The issue was caused by reading the entire body of the request into
memory to verify the content-md5 or content-sha56 checksum if present.

The vulnerability could be exploited by either replaying a V4 request
(in the 15 min time frame) or sending a V4 presigned request with a
large body.
2018-05-18 11:27:25 -07:00
kannappanr
ffa6b45d00 Remove treewalk abort log message (#5953)
Log doesn't add any value and can be removed.

Fixes #5943
2018-05-17 17:26:16 -07:00
Bala FA
4eb788df79 rename checkPathValid() to getValidPath() (#5949) 2018-05-17 07:27:07 -07:00
Nitish Tiwari
e6ebcc4cb6 Remove redundant prometheus data points (#5934)
Removed field minio_http_requests_total as it was redundant with
minio_http_requests_duration_seconds_count

Also removed field minio_server_start_time_seconds as it was
redundant with process_start_time_seconds
2018-05-15 12:23:43 -07:00
Harshavardhana
c872c30ea3 fix: introduce isLeafDir in healing to fix the crash (#5920)
This PR also supports healing directories.

Fixes #5917
2018-05-10 16:53:42 -07:00
Harshavardhana
5b74f918d4 Implement a dummy GetBucketACL for legacy reasons (#5906)
GetBucketACL call returns empty for all GET in ACL requests, 
the primary purpose of this PR is to provide legacy API support 
for legacy applications.

Fixes #5706
2018-05-10 09:32:26 +05:30
Krishna Srinivas
cc8178cdc4 Log errors only once for event notification errors (#5905) 2018-05-09 15:59:45 -07:00
Harshavardhana
4886bfbc72 fix: Avoid more crashes due to concurrent map usage (#5912)
This PR fixes another situation where a crash occurs
thanks to @krishnasrinivas for reproducing this

Fixes #5897
2018-05-09 15:11:51 -07:00
Anis Elleuch
c525424179 Return detailed msg for notification errors (#5913) 2018-05-09 15:11:24 -07:00
Anis Elleuch
c1f1fa3cc8 More fixes for error messages (#5911) 2018-05-09 13:29:37 -07:00
Harshavardhana
98f81ced86 fix: Avoid concurrent map writes in go-routines (#5898)
Fixes #5897
2018-05-09 11:25:38 -07:00
Praveen raj Mani
3971483e18 Remove trailing/leading spaces in Certificates (#5885)
Fixes:  #5632
2018-05-09 14:26:38 +05:30
Ashish Kumar Sinha
deb685c5b5 Enhancements in Minio Prometheus exporter (#5848)
Standardized Minio collectors based on Prometheus 
recommendations.
2018-05-09 01:38:27 -07:00
Krishna Srinivas
bb34bd91f1 Fix unnecessary log messages to avoid flooding the logs (#5900) 2018-05-09 01:38:27 -07:00
Krishna Srinivas
9de8fefa90 Fix azure list-objects not to list minio.sys.tmp/ (#5895) 2018-05-09 01:38:27 -07:00
Anis Elleuch
6d5f2a4391 Better support of empty directories (#5890)
Better support of HEAD and listing of zero sized objects with trailing
slash (a.k.a empty directory). For that, isLeafDir function is added
to indicate if the specified object is an empty directory or not. Each
backend (xl, fs) has the responsibility to store that information.
Currently, in both of XL & FS, an empty directory is represented by
an empty directory in the backend.

isLeafDir() checks if the given path is an empty directory or not,
since dir listing is costly if the latter contains too many objects,
readDirN() is added in this PR to list only N number of entries.
In isLeadDir(), we will only list one entry to check if a directory
is empty or not.
2018-05-09 01:38:21 -07:00
Anis Elleuch
32700fca52 Enhance fatal errors printing of common issues seen by users (#5878) 2018-05-08 19:04:36 -07:00
Harshavardhana
54cd29b51c Add new S3 APIs which are not implemented (#5899)
Fixes #5896
2018-05-08 21:45:10 +05:30
Harshavardhana
c98d8cb1c7 fs: fix a regression allow reading of existing files (#5889) 2018-05-07 17:00:44 -07:00
Andreas Auernhammer
c5a00e513c fix DoS vulnerability in request authentication (#5887)
This commit fixes a DoS vulnerability in the
request authentication. The root cause is an 'unlimited'
read-into-RAM from the request body.

Since this read happens before the request authentication
is verified the vulnerability can be exploit without any
access privileges.

This commit limits the size of the request body to 3 MB.
This is about the same size as AWS. The limit seems to be
between 1.6 and 3.2 MB - depending on the AWS machine which
is handling the request.
2018-05-04 11:16:14 -07:00
Anis Elleuch
9439dfef64 Use defer style to stop tickers to avoid current/possible misuse (#5883)
This commit ensures that all tickers are stopped using defer ticker.Stop()
style. This will also fix one bug seen when a client starts to listen to
event notifications and that case will result a leak in tickers.
2018-05-04 10:43:20 -07:00
Krishna Srinivas
0f746a14a3 Do not use crypto.SHA3_256 as placeholder for HighwayHash256 (#5847) 2018-05-04 10:42:22 -07:00
Harshavardhana
d69ba7d085 fs: fix logging by not logging common scenarios (#5882)
Fixes #5881
2018-05-03 08:29:38 +05:30
Krishna Srinivas
d6df9b16ac Return NoSuchKey for anonReqs with s3:ListBucket policy (#5876) 2018-05-02 12:13:27 +05:30
Krishna Srinivas
0ad0969d61 Remove s3:ListObjects policy action to be in sync with AWS-S3 (#5875) 2018-05-02 12:13:17 +05:30
wd256
7f207156ec Copy an object without user-defined metadata for Azure gateway (#5872) 2018-05-02 10:20:00 +05:30
Harshavardhana
5f9041571f Heal only when atleast one of the disk is unformatted (#5866)
Current healing has an issue when disks are healed
even when they are offline without knowing if disk
is unformatted. This can lead to issues of pre-maturely
removing the disk from the set just because it was
temporarily offline.

There is an increasing number of `mc admin heal` usage
on a cron or regular basis. It is possible that if healing
code saw disk is offline it might prematurely take it down,
this causes availability issues.

Fixes #5826
2018-05-01 09:07:39 +05:30
cbenien
e43cd0d5c8 Increased MaxIdleConnsPerHost to prevent excessive re-connections and TIME_WAIT when more than 100 clients are using minio (#5860) 2018-04-30 11:59:41 -07:00
Krishna Srinivas
9816264eed Support for ListObjectsV1 style marker for Azure gateway (#5856)
fixes #4948
2018-04-27 16:08:46 -07:00
Krishna Srinivas
11b8e292a7 Disable logging during unit tests (#5861) 2018-04-27 15:03:19 -07:00
Harshavardhana
b6ca39ea48 Support migrating inconsistent bucket policies (#5855)
Previously we used allow bucket policies without
`Version` field to be set to any given value, but
this behavior is inconsistent with AWS S3.

PR #5790 addressed this by making bucket policies
stricter and cleaner, but this causes a breaking
change causing any existing policies perhaps without
`Version` field or the field to be empty to fail upon
server startup.

This PR brings a code to migrate under these scenarios
as a one time operation.
2018-04-27 15:02:54 -07:00
poornas
0dc3d7ac18 fix error checks when cache is offline/missing. (#5850) 2018-04-26 10:39:05 +05:30
Krishna Srinivas
9aace6d36d Continue healing other objects even if objects without quorum exist (#5851)
fixes #5815
2018-04-25 11:56:39 -07:00
Krishna Srinivas
6831177394 Do not log errFileNotFound error (#5853) 2018-04-25 11:46:49 -07:00
kannappanr
fe126de98b
Regenerate fs.json if it is corrupted in FS mode (#5778)
Also return a default e-tag for pre-existing objects.
Fixes #5712
2018-04-24 17:36:43 -07:00
Bala FA
0d52126023 Enhance policy handling to support SSE and WORM (#5790)
- remove old bucket policy handling
- add new policy handling
- add new policy handling unit tests

This patch brings support to bucket policy to have more control not
limiting to anonymous.  Bucket owner controls to allow/deny any rest
API.

For example server side encryption can be controlled by allowing
PUT/GET objects with encryptions including bucket owner.
2018-04-24 15:53:30 -07:00
Andreas Auernhammer
21a3c0f482 disable elliptic curves P-384 and P-521 for TLS. (#5845)
This change disables the non-constant-time implementations of P-384 and P-521.
As a consequence a client using just these curves cannot connect to the server.
This should be no real issues because (all) clients at least support P-256.

Further this change also rejects ECDSA private keys of P-384 and P-521.
While non-constant-time implementations for the ECDHE exchange don't expose an
obvious vulnerability, using P-384 or P-521 keys for the ECDSA signature may allow
pratical timing attacks.

Fixes #5844
2018-04-24 15:47:30 -07:00
Harshavardhana
ccdb7bc286 Fix s3 compatibility fixes for getBucketLocation,headBucket,deleteBucket (#5842)
- getBucketLocation
- headBucket
- deleteBucket

Should return 404 or NoSuchBucket even for invalid bucket names, invalid
bucket names are only validated during MakeBucket operation
2018-04-24 08:57:33 +05:30
Harshavardhana
954142a98f Cleanup and make a safer code (#5794) 2018-04-21 20:51:53 -07:00
Bala FA
76cc65531c use package name correctly (#5827) 2018-04-21 19:23:54 -07:00
ebozduman
f16bfda2f2 Remove panic() and handle it appropriately (#5807)
This is an effort to remove panic from the source. 
Add a new call called CriticialIf, that calls LogIf and exits. 
Replace panics with one of CriticalIf, FatalIf and a return of error.
2018-04-19 17:24:43 -07:00
wd256
846f3e8f59 Disable "chunked" uploading by the GCS client for objects smaller than the chunk size. (#5835)
By disabling chunked uploading when the object size is less than the chunk size,
memory is not allocated unnecessarily.
2018-04-19 16:15:14 -07:00
Andreas Auernhammer
5a16671f72 do not set response header X-Frame-Options for S3 requests (#5838)
This change removes the X-Frame-Options header - It should
not be set for S3 requests since it can break CORS.

Fixes #5813
2018-04-19 11:46:37 -07:00
Ashish Kumar Sinha
9ebb72aa99 Introduce new unauthenticated endpoint /metric (#5723) (#5829)
/metric exposes Promethus compatible data for scraping metrics

Fixes: #5723
2018-04-18 16:01:42 -07:00
wd256
9f31da5d57 Fix PutObject/CopyObject with metadata for GCS gateway (#5828)
Make sure to apply standard headers such as Content-Type,
Content-Disposition and Content-Language to the correct
GCS object attributes during object upload and copy operations.

Fixes: #5800
2018-04-18 21:40:48 +05:30
Harshavardhana
c332b4fb5e fix shadowing issue in logger (#5830) 2018-04-16 13:36:45 -07:00
Harshavardhana
adf9a9d300 Remove all unused variables and functions (#5823) 2018-04-15 19:26:04 +05:30
Harshavardhana
97a8d856b6 Vendorize new changes from minio-go repo (#5821)
- When policy is empty delete the bucket policy (#966) (04/13/18) <Harshavardhana>
- Add tests to check if ListObjects/V2 returns expected StorageClass (#963) (04/10/18) <Nitish Tiwari>
- Update get/setBucketPolicy methods to use files instead of pkg/policy (#959) (04/10/18) <Nitish Tiwari>
- avoid unnecessary stat call during single copy (#962) (04/06/18) <Andreas Auernhammer>
- avoid sending SSE-S3 header during GET requests. (#965) (04/05/18) <Andreas Auernhammer>
- Fix stream SSE uploads with S3 encrypt type (#960) (04/02/18) <Jesús Espino>
- Fix xml parsing error for RemoveObjects API (#949) (03/29/18) <poornas>
- Allow to upload empty files in stream based uploads (#958) (03/26/18) <Jesús Espino>
- Add missing doneCh in the example for removeobjects (#955) (03/26/18) <Alexandr Korsak>
- tests: Remove partial related tests (#957) (03/26/18) <Anis Elleuch>
- Add transport connection broken error to retry list (#956) (03/19/18) <poornas>
- [refactor]: simplify client encryption examples (#952) (03/19/18) <Andreas Auernhammer>
- Add tests for putObjectContentLanguage (#950) (03/15/18) <Harshavardhana>
- Add putObject/getObject() client side encryption examples (#948) (03/13/18) <Harshavardhana>
2018-04-15 11:39:48 +05:30
Nitish Tiwari
638f01f9e4 Generalize loadConfig method to avoid reading from disk (#5819)
As we move to multiple config backends like local disk and etcd,
config file should not be read from the disk, instead the quick
package should load and verify for duplicate entries.
2018-04-13 15:14:19 -07:00
Harshavardhana
57b8db2088 Cleanup help to bring concise self-help message (#5788) 2018-04-13 11:57:05 -07:00
Harshavardhana
bcd54632f5 Fix broken 'minio update' command on Windows (#5816) 2018-04-13 11:51:03 -07:00
Nitish Tiwari
42f775af4f
Add cache config for configuration comparison (#5812) 2018-04-13 23:45:19 +05:30