Commit Graph

1271 Commits

Author SHA1 Message Date
Nitish Tiwari
e6ebcc4cb6 Remove redundant prometheus data points (#5934)
Removed field minio_http_requests_total as it was redundant with
minio_http_requests_duration_seconds_count

Also removed field minio_server_start_time_seconds as it was
redundant with process_start_time_seconds
2018-05-15 12:23:43 -07:00
Harshavardhana
c872c30ea3 fix: introduce isLeafDir in healing to fix the crash (#5920)
This PR also supports healing directories.

Fixes #5917
2018-05-10 16:53:42 -07:00
Harshavardhana
5b74f918d4 Implement a dummy GetBucketACL for legacy reasons (#5906)
GetBucketACL call returns empty for all GET in ACL requests, 
the primary purpose of this PR is to provide legacy API support 
for legacy applications.

Fixes #5706
2018-05-10 09:32:26 +05:30
Krishna Srinivas
cc8178cdc4 Log errors only once for event notification errors (#5905) 2018-05-09 15:59:45 -07:00
Harshavardhana
4886bfbc72 fix: Avoid more crashes due to concurrent map usage (#5912)
This PR fixes another situation where a crash occurs
thanks to @krishnasrinivas for reproducing this

Fixes #5897
2018-05-09 15:11:51 -07:00
Anis Elleuch
c525424179 Return detailed msg for notification errors (#5913) 2018-05-09 15:11:24 -07:00
Anis Elleuch
c1f1fa3cc8 More fixes for error messages (#5911) 2018-05-09 13:29:37 -07:00
Harshavardhana
98f81ced86 fix: Avoid concurrent map writes in go-routines (#5898)
Fixes #5897
2018-05-09 11:25:38 -07:00
Praveen raj Mani
3971483e18 Remove trailing/leading spaces in Certificates (#5885)
Fixes:  #5632
2018-05-09 14:26:38 +05:30
Ashish Kumar Sinha
deb685c5b5 Enhancements in Minio Prometheus exporter (#5848)
Standardized Minio collectors based on Prometheus 
recommendations.
2018-05-09 01:38:27 -07:00
Krishna Srinivas
bb34bd91f1 Fix unnecessary log messages to avoid flooding the logs (#5900) 2018-05-09 01:38:27 -07:00
Krishna Srinivas
9de8fefa90 Fix azure list-objects not to list minio.sys.tmp/ (#5895) 2018-05-09 01:38:27 -07:00
Anis Elleuch
6d5f2a4391 Better support of empty directories (#5890)
Better support of HEAD and listing of zero sized objects with trailing
slash (a.k.a empty directory). For that, isLeafDir function is added
to indicate if the specified object is an empty directory or not. Each
backend (xl, fs) has the responsibility to store that information.
Currently, in both of XL & FS, an empty directory is represented by
an empty directory in the backend.

isLeafDir() checks if the given path is an empty directory or not,
since dir listing is costly if the latter contains too many objects,
readDirN() is added in this PR to list only N number of entries.
In isLeadDir(), we will only list one entry to check if a directory
is empty or not.
2018-05-09 01:38:21 -07:00
Anis Elleuch
32700fca52 Enhance fatal errors printing of common issues seen by users (#5878) 2018-05-08 19:04:36 -07:00
Harshavardhana
54cd29b51c Add new S3 APIs which are not implemented (#5899)
Fixes #5896
2018-05-08 21:45:10 +05:30
Harshavardhana
c98d8cb1c7 fs: fix a regression allow reading of existing files (#5889) 2018-05-07 17:00:44 -07:00
Andreas Auernhammer
c5a00e513c fix DoS vulnerability in request authentication (#5887)
This commit fixes a DoS vulnerability in the
request authentication. The root cause is an 'unlimited'
read-into-RAM from the request body.

Since this read happens before the request authentication
is verified the vulnerability can be exploit without any
access privileges.

This commit limits the size of the request body to 3 MB.
This is about the same size as AWS. The limit seems to be
between 1.6 and 3.2 MB - depending on the AWS machine which
is handling the request.
2018-05-04 11:16:14 -07:00
Anis Elleuch
9439dfef64 Use defer style to stop tickers to avoid current/possible misuse (#5883)
This commit ensures that all tickers are stopped using defer ticker.Stop()
style. This will also fix one bug seen when a client starts to listen to
event notifications and that case will result a leak in tickers.
2018-05-04 10:43:20 -07:00
Krishna Srinivas
0f746a14a3 Do not use crypto.SHA3_256 as placeholder for HighwayHash256 (#5847) 2018-05-04 10:42:22 -07:00
Harshavardhana
d69ba7d085 fs: fix logging by not logging common scenarios (#5882)
Fixes #5881
2018-05-03 08:29:38 +05:30
Krishna Srinivas
d6df9b16ac Return NoSuchKey for anonReqs with s3:ListBucket policy (#5876) 2018-05-02 12:13:27 +05:30
Krishna Srinivas
0ad0969d61 Remove s3:ListObjects policy action to be in sync with AWS-S3 (#5875) 2018-05-02 12:13:17 +05:30
wd256
7f207156ec Copy an object without user-defined metadata for Azure gateway (#5872) 2018-05-02 10:20:00 +05:30
Harshavardhana
5f9041571f Heal only when atleast one of the disk is unformatted (#5866)
Current healing has an issue when disks are healed
even when they are offline without knowing if disk
is unformatted. This can lead to issues of pre-maturely
removing the disk from the set just because it was
temporarily offline.

There is an increasing number of `mc admin heal` usage
on a cron or regular basis. It is possible that if healing
code saw disk is offline it might prematurely take it down,
this causes availability issues.

Fixes #5826
2018-05-01 09:07:39 +05:30
cbenien
e43cd0d5c8 Increased MaxIdleConnsPerHost to prevent excessive re-connections and TIME_WAIT when more than 100 clients are using minio (#5860) 2018-04-30 11:59:41 -07:00
Krishna Srinivas
9816264eed Support for ListObjectsV1 style marker for Azure gateway (#5856)
fixes #4948
2018-04-27 16:08:46 -07:00
Krishna Srinivas
11b8e292a7 Disable logging during unit tests (#5861) 2018-04-27 15:03:19 -07:00
Harshavardhana
b6ca39ea48 Support migrating inconsistent bucket policies (#5855)
Previously we used allow bucket policies without
`Version` field to be set to any given value, but
this behavior is inconsistent with AWS S3.

PR #5790 addressed this by making bucket policies
stricter and cleaner, but this causes a breaking
change causing any existing policies perhaps without
`Version` field or the field to be empty to fail upon
server startup.

This PR brings a code to migrate under these scenarios
as a one time operation.
2018-04-27 15:02:54 -07:00
poornas
0dc3d7ac18 fix error checks when cache is offline/missing. (#5850) 2018-04-26 10:39:05 +05:30
Krishna Srinivas
9aace6d36d Continue healing other objects even if objects without quorum exist (#5851)
fixes #5815
2018-04-25 11:56:39 -07:00
Krishna Srinivas
6831177394 Do not log errFileNotFound error (#5853) 2018-04-25 11:46:49 -07:00
kannappanr
fe126de98b
Regenerate fs.json if it is corrupted in FS mode (#5778)
Also return a default e-tag for pre-existing objects.
Fixes #5712
2018-04-24 17:36:43 -07:00
Bala FA
0d52126023 Enhance policy handling to support SSE and WORM (#5790)
- remove old bucket policy handling
- add new policy handling
- add new policy handling unit tests

This patch brings support to bucket policy to have more control not
limiting to anonymous.  Bucket owner controls to allow/deny any rest
API.

For example server side encryption can be controlled by allowing
PUT/GET objects with encryptions including bucket owner.
2018-04-24 15:53:30 -07:00
Andreas Auernhammer
21a3c0f482 disable elliptic curves P-384 and P-521 for TLS. (#5845)
This change disables the non-constant-time implementations of P-384 and P-521.
As a consequence a client using just these curves cannot connect to the server.
This should be no real issues because (all) clients at least support P-256.

Further this change also rejects ECDSA private keys of P-384 and P-521.
While non-constant-time implementations for the ECDHE exchange don't expose an
obvious vulnerability, using P-384 or P-521 keys for the ECDSA signature may allow
pratical timing attacks.

Fixes #5844
2018-04-24 15:47:30 -07:00
Harshavardhana
ccdb7bc286 Fix s3 compatibility fixes for getBucketLocation,headBucket,deleteBucket (#5842)
- getBucketLocation
- headBucket
- deleteBucket

Should return 404 or NoSuchBucket even for invalid bucket names, invalid
bucket names are only validated during MakeBucket operation
2018-04-24 08:57:33 +05:30
Harshavardhana
954142a98f Cleanup and make a safer code (#5794) 2018-04-21 20:51:53 -07:00
Bala FA
76cc65531c use package name correctly (#5827) 2018-04-21 19:23:54 -07:00
ebozduman
f16bfda2f2 Remove panic() and handle it appropriately (#5807)
This is an effort to remove panic from the source. 
Add a new call called CriticialIf, that calls LogIf and exits. 
Replace panics with one of CriticalIf, FatalIf and a return of error.
2018-04-19 17:24:43 -07:00
wd256
846f3e8f59 Disable "chunked" uploading by the GCS client for objects smaller than the chunk size. (#5835)
By disabling chunked uploading when the object size is less than the chunk size,
memory is not allocated unnecessarily.
2018-04-19 16:15:14 -07:00
Andreas Auernhammer
5a16671f72 do not set response header X-Frame-Options for S3 requests (#5838)
This change removes the X-Frame-Options header - It should
not be set for S3 requests since it can break CORS.

Fixes #5813
2018-04-19 11:46:37 -07:00
Ashish Kumar Sinha
9ebb72aa99 Introduce new unauthenticated endpoint /metric (#5723) (#5829)
/metric exposes Promethus compatible data for scraping metrics

Fixes: #5723
2018-04-18 16:01:42 -07:00
wd256
9f31da5d57 Fix PutObject/CopyObject with metadata for GCS gateway (#5828)
Make sure to apply standard headers such as Content-Type,
Content-Disposition and Content-Language to the correct
GCS object attributes during object upload and copy operations.

Fixes: #5800
2018-04-18 21:40:48 +05:30
Harshavardhana
c332b4fb5e fix shadowing issue in logger (#5830) 2018-04-16 13:36:45 -07:00
Harshavardhana
adf9a9d300 Remove all unused variables and functions (#5823) 2018-04-15 19:26:04 +05:30
Harshavardhana
97a8d856b6 Vendorize new changes from minio-go repo (#5821)
- When policy is empty delete the bucket policy (#966) (04/13/18) <Harshavardhana>
- Add tests to check if ListObjects/V2 returns expected StorageClass (#963) (04/10/18) <Nitish Tiwari>
- Update get/setBucketPolicy methods to use files instead of pkg/policy (#959) (04/10/18) <Nitish Tiwari>
- avoid unnecessary stat call during single copy (#962) (04/06/18) <Andreas Auernhammer>
- avoid sending SSE-S3 header during GET requests. (#965) (04/05/18) <Andreas Auernhammer>
- Fix stream SSE uploads with S3 encrypt type (#960) (04/02/18) <Jesús Espino>
- Fix xml parsing error for RemoveObjects API (#949) (03/29/18) <poornas>
- Allow to upload empty files in stream based uploads (#958) (03/26/18) <Jesús Espino>
- Add missing doneCh in the example for removeobjects (#955) (03/26/18) <Alexandr Korsak>
- tests: Remove partial related tests (#957) (03/26/18) <Anis Elleuch>
- Add transport connection broken error to retry list (#956) (03/19/18) <poornas>
- [refactor]: simplify client encryption examples (#952) (03/19/18) <Andreas Auernhammer>
- Add tests for putObjectContentLanguage (#950) (03/15/18) <Harshavardhana>
- Add putObject/getObject() client side encryption examples (#948) (03/13/18) <Harshavardhana>
2018-04-15 11:39:48 +05:30
Nitish Tiwari
638f01f9e4 Generalize loadConfig method to avoid reading from disk (#5819)
As we move to multiple config backends like local disk and etcd,
config file should not be read from the disk, instead the quick
package should load and verify for duplicate entries.
2018-04-13 15:14:19 -07:00
Harshavardhana
57b8db2088 Cleanup help to bring concise self-help message (#5788) 2018-04-13 11:57:05 -07:00
Harshavardhana
bcd54632f5 Fix broken 'minio update' command on Windows (#5816) 2018-04-13 11:51:03 -07:00
Nitish Tiwari
42f775af4f
Add cache config for configuration comparison (#5812) 2018-04-13 23:45:19 +05:30
Andreas Auernhammer
f60765ac93 add some security HTTP headers (#5814)
This change adds some security headers like Content-Security-Policy.
It does not set the HSTS header because Content-Security-Policy prevents
mixed HTTP and HTTPS content and the server does not use cookies.
However it is a header which could be added later on.

It also moves some header added by #5805 from a vendored file
to a generic handler.

Fixes ##5813
2018-04-12 15:57:41 -07:00