Commit Graph

1125 Commits

Author SHA1 Message Date
kannappanr
6f7c6fc560 Honor browser enabled config value in startup message (#5313)
Currently, browser access information is displayed without checking
if browser enabled flag is turned off in config.json. Fixing it to
hide the information if the flag is turned off.

Fixes #5312
2018-01-04 11:00:52 +05:30
Harshavardhana
c0721164be Automatically set goroutines based on shardSize (#5346)
Update reedsolomon library to enable feature to automatically
set number of go-routines based on the input shard size,
since shard size is sort of a constant in Minio for
objects > 10MiB (default blocksize)

klauspost reported around 15-20% improvement in performance
numbers on older systems such as AVX and SSE3

```
name                  old speed      new speed      delta
Encode10x2x10000-8    5.45GB/s ± 1%  6.22GB/s ± 1%  +14.20%    (p=0.000 n=9+9)
Encode100x20x10000-8  1.44GB/s ± 1%  1.64GB/s ± 1%  +13.77%  (p=0.000 n=10+10)
Encode17x3x1M-8       10.0GB/s ± 5%  12.0GB/s ± 1%  +19.88%  (p=0.000 n=10+10)
Encode10x4x16M-8      7.81GB/s ± 5%  8.56GB/s ± 5%   +9.58%   (p=0.000 n=10+9)
Encode5x2x1M-8        15.3GB/s ± 2%  19.6GB/s ± 2%  +28.57%   (p=0.000 n=9+10)
Encode10x2x1M-8       12.2GB/s ± 5%  15.0GB/s ± 5%  +22.45%  (p=0.000 n=10+10)
Encode10x4x1M-8       7.84GB/s ± 1%  9.03GB/s ± 1%  +15.19%    (p=0.000 n=9+9)
Encode50x20x1M-8      1.73GB/s ± 4%  2.09GB/s ± 4%  +20.59%   (p=0.000 n=10+9)
Encode17x3x16M-8      10.6GB/s ± 1%  11.7GB/s ± 4%  +10.12%   (p=0.000 n=8+10)
```
2018-01-03 13:47:22 -08:00
Andreas Auernhammer
a6318dbdaf fix timing oracle attack against signature V2/V4 verification (#5335)
This change replaces the non-constant time comparison of
request signatures with a constant time implementation. This
prevents a timing attack which can be used to learn a valid 
signature for a request without knowing the secret key.

Fixes #5334
2018-01-02 12:00:02 +05:30
Harshavardhana
e39d7ddb0f Fix PostPolicy form tests without hardcoded dates (#5337)
Fixes #5336
2018-01-01 07:28:10 +05:30
Kaan Kabalak
659f724f4c Integrate existing remove bucket functionality from newux to current UI (#5289)
This commit takes the existing remove bucket functionality written by
brendanashworth, integrates it to the current UI with a dropdown for
each bucket, and fixes small issues that were present, like the dropdown
not disappearing after the user clicks on 'Delete' for certain buckets.
This feature only deletes a bucket that is empty (that has no objects).

Fixes #4166
2017-12-29 18:45:44 +05:30
A. Elleuch
2244adff07 fix: Better printing of XL config init error (#5284) 2017-12-28 23:02:48 +05:30
Nitish Tiwari
e3d841ffd1
Fix config.json parsing to fetch correct storage class (#5327) 2017-12-28 14:19:45 +05:30
Nitish Tiwari
545a9e4a82 Fix storage class related issues (#5322)
- Add storage class metadata validation for request header
- Change storage class header values to be consistent with AWS S3
- Refactor internal method to take only the reqd argument
2017-12-27 10:06:16 +05:30
Harshavardhana
f25ec31565 Set maxResources appropriately for gateway like server (#5321) 2017-12-24 20:09:30 +05:30
A. Elleuch
6ef0161835 fix: Restore empty files when healing (#5257)
HealFile() does not process the case when an empty file is lost in
some disks. Since, Reedsolomon erasure doesn't handle restoring empty
data, HealFile will create empty files similarly to CreateFile().
2017-12-22 14:57:57 -08:00
Nitish Tiwari
1a3dbbc9dd
Add x-amz-storage-class support (#5295)
This adds configurable data and parity options on a per object
basis. To use variable parity

- Users can set environment variables to cofigure variable
parity

- Then add header x-amz-storage-class to putobject requests
with relevant storage class values

Fixes #4997
2017-12-22 16:58:13 +05:30
Aditya Manthramurthy
f1355da72e Add base64 encoded MD5 output for Hash Reader (#5315)
- Use it to send the Content-MD5 header correctly encoded to S3
  Gateway

- Fixes a bug in PutObject (including anonymous PutObject) and
  PutObjectPart with S3 Gateway found when testing with Mint.
2017-12-21 17:27:33 -08:00
Krishnan Parthasarathi
bbe521ffec ReInitDisk RPC handler should use retryStorage (#5310) 2017-12-21 12:28:01 +05:30
Paul Stack
7d75d61621 Add Support for Manta Object Storage as a Gateway (#5025)
Manta is an Object Storage by [Joyent](https://www.joyent.com/)

This PR adds initial support for Manta. It is intended as non-production 
ready so that feedback can be obtained.
2017-12-20 13:37:56 +05:30
Harshavardhana
1f77708a30 Limit number of connections upto system maxlimit (#5109) 2017-12-20 13:30:14 +05:30
Timon Wong
84fc78d60f Implement Alibaba Cloud OSS gateway support (#5103) 2017-12-19 13:55:17 +05:30
poornas
a182fe8c15 update steps to make changes to config.json (#5292) 2017-12-17 21:00:12 -08:00
Harshavardhana
819d1e80c6 Add more delays on distributed startup for slow network (#5240)
Refer #5237
2017-12-16 08:25:29 -08:00
Harshavardhana
eb7c690ea9 Support in-place upgrades of new minio binary and releases. (#4961)
This PR allows 'minio update' to not only shows update banner
but also allows for in-place upgrades.

Updates are done safely by validating the downloaded
sha256 of the binary.

Fixes #4781
2017-12-15 12:33:42 -08:00
kannappanr
a1c1a18dc5 Remove "logger" field from config.json (#5268)
File logging removed as part of improvement to server logging.

config.json format updated to version 21.

Fixes #5176
2017-12-06 12:48:29 +05:30
Harshavardhana
eb2894233c Convert gateways into respective packages (#5200)
- Make azure gateway a package
- Make b2 gateway a package
- Make gcs gateway a package
- Make s3 gateway a package
- Make sia gateway a package
2017-12-05 17:58:09 -08:00
Harshavardhana
2755a0b763 Check if SSL is configured to validate input arguments (#5252)
This PR handles following situations

- secure endpoints provided, server should fail to start
  if TLS is not configured

- insecure endpoints provided, server starts ignoring
  if TLS is configured or not.

Fixes #5251
2017-12-04 12:17:12 +05:30
Aditya Manthramurthy
043e030a4a Add CopyObjectPart support to gateway S3 (#5213)
- Adds a metadata argument to the CopyObjectPart API to facilitate
  implementing encryption for copying APIs too.

- Update vendored minio-go - this version implements the
  CopyObjectPart client API for use with the S3 gateway.

Fixes #4885
2017-12-02 08:33:59 +05:30
Harshavardhana
490c30f853
erasure: Support cleaning up of stale multipart objects (#5250)
Just like our single directory/disk setup, this PR brings
the functionality to cleanup stale multipart objects
older > 2 weeks.
2017-11-30 18:11:42 -08:00
Harshavardhana
59749a2b85
erasure: Remove prefix based listing support on ListMultipartUploads (#5248)
Previously we have removed this support under FS on #4996,
deprecate this on erasure coded backend as well to simplify
our multipart support.
2017-11-30 15:58:46 -08:00
Michael Lynch
fc3cf97b81 Removing isValidObjectName from Sia gateway (#5243)
This check incorrectly rejects most valid filenames. The only filenames Sia
forbids are leading forward slashes and path traversal characters, but it's
better to simply allow Sia to reject invalid names on its own rather than try
to anticipate errors from Sia:

https://github.com/NebulousLabs/Sia/blob/master/doc/api/Renter.md#path-parameters-4
2017-11-30 14:43:21 -08:00
Harshavardhana
d45a8784fc
Fix hash order to generate more even distribution (#5247)
The problem in existing code was the following line

```
start := int(keyCrc%uint32(cardinality)) | 1
```

A given a value of N cardinality the ending result
because of the the bitwise '|' would lead to always
higher affinity to odd sequences.

As can be seen from the test cases that this can
lead to many objects being allocated the same set
of disks or atleast the first disk is an odd disk
always.  This introduces a performance problem
for majority of the objects under concurrent load.

Remove `| 1` to provide a more cleaner distribution
and the new code will be.
```
start := int(keyCrc % uint32(cardinality))
```

Thanks to Krishna Srinivas for pointing out the bitwise
situation here.
2017-11-30 12:57:03 -08:00
Krishna Srinivas
14e6c5ec08 Simplify the steps to make changes to config.json (#5186)
This change introduces following simplified steps to follow 
during config migration.

```
 // Steps to move from version N to version N+1
 // 1. Add new struct serverConfigVN+1 in config-versions.go
 // 2. Set configCurrentVersion to "N+1"
 // 3. Set serverConfigCurrent to serverConfigVN+1
 // 4. Add new migration function (ex. func migrateVNToVN+1()) in config-migrate.go
 // 5. Call migrateVNToVN+1() from migrateConfig() in config-migrate.go
 // 6. Make changes in config-current_test.go for any test change
```
2017-11-29 13:12:47 -08:00
Harshavardhana
a46b640da3 gateway/sia: Support proper {make,get}Bucket operations (#5229)
Current implementation we faked the makeBucket operations
to allow for s3 clients to behave properly. But instead
we can create a placeholder zero byte file instead, which
is a hexadecimal representation of the bucket name itself.
2017-11-28 13:40:44 +05:30
Krishna Srinivas
71f9d2beff Increase maximum size of PUT request to 5TB (#5241)
fixes #5148
2017-11-28 12:59:02 +05:30
Michael Lynch
cf414a6053 Fixing Sia file uploads (#5233)
The Sia gateway had a bug with uploading that prevented the user's uploads
from reaching the Sia backend. The PutObject function called fsRemoveFile at
the end of the function, which didn't give the Sia backend enough time to
upload the file to the Sia network.

This adds a goroutine that watches the file upload progress and doesn't delete
the file until the upload reaches 100% complete.

Note that this solution has the limitation where if the minio process dies in
the middle of upload, it will leave orphaned files in the SIA_TEMP directory
that the user will need to remove manually.
2017-11-28 12:25:15 +05:30
Harshavardhana
05b395e81d
Add more unit tests for azure/gcs/b2 gateway (#5236)
Also adds a blazer SDK update exposing
error response headers.
2017-11-27 18:29:22 -08:00
Paul Nicholls
6a2d7ae808 gateway/azure: ListParts return an empty list if no parts uploaded yet (#5230)
This makes azure ListParts implementation behave more like S3 
by returning an empty list rather than an error when no parts have
been uploaded yet.
2017-11-27 17:42:27 -08:00
Harshavardhana
8efa82126b
Convert errors tracer into a separate package (#5221) 2017-11-25 11:58:29 -08:00
David G
f4d4ea5c36 Implement Sia Gateway (#5114) 2017-11-22 12:12:10 -08:00
Krishna Srinivas
bbd05a8f1c gateway-gcs: Close the writer with error in case of any errors. (#5217)
fixes #5216
2017-11-21 14:45:37 -08:00
Krishna Srinivas
4393afb7e2 Remove checkGCSProjectID() as it needs extra permission setting (#5210)
fixes #5209
2017-11-21 10:43:39 -08:00
Krishna Srinivas
1a53734477 Rename UserDefined to UserMetadata for events (#5206)
fixes #5165
2017-11-20 15:32:25 -08:00
Andreas Auernhammer
e95c0bb913 return AWS compliant error if SSE-C key is wrong (#5203)
This PR changes the behavior of DecryptRequest.
Instead of returning `object-tampered` if the client provided
key is wrong DecryptRequest will return `access-denied`.

This is AWS S3 behavior.

Fixes #5202
2017-11-20 14:04:10 -08:00
Krishna Srinivas
fce556b8a0 Support for ListObjectParts in azure-gateway (#5198)
fixes #5169
2017-11-20 14:03:20 -08:00
Andreas Auernhammer
b97f99766f add benchmarks for erasure backend (#5084)
This change adds benchmarks for erasure read/write in different setups.
2017-11-17 14:57:04 -08:00
Nitish Tiwari
f7b6f7b22f Update getObjectInfo to stat for objects with trailing / (#5179)
Apache Spark sends getObject requests with trailing "/".
This PR updates the getObjectInfo to stat for files
even if they are sent with trailing "/".

Fixes #2965
2017-11-16 16:00:27 -08:00
Krishnan Parthasarathi
2a0a62b78d Return ErrContentSHA256Mismatch when sha256sum is invalid (#5188) 2017-11-16 11:13:04 -08:00
Krishnan Parthasarathi
67f66c40c1 Fix ListenBucketNotification deadlock (#5028)
Previously ListenBucketNotificationHandler could deadlock with
PutObjectHandler's eventNotify call when a client closes its
connection. This change removes the cyclic dependency between the
channel and map of ARN to channels by using a separate done channel to
signal that the client has quit.
2017-11-16 10:56:06 -08:00
Krishna Srinivas
5a2bdf6959 Handle Path validation inside the PostPolicy handler (#5192) 2017-11-15 14:10:45 -08:00
Harshavardhana
0827a2747b api: CopyObject should return NotImplemented for now (#5183)
Commit ca6b4773ed introduces SSE-C
support for HEAD, GET, PUT operations but since we do not
implement CopyObject() we should return NotImplemented.
2017-11-14 16:57:19 -08:00
Krishna Srinivas
e7a724de0d Virtual host style S3 requests (#5095) 2017-11-14 16:56:24 -08:00
Harshavardhana
a4d6195244 Add public data-types for easier external loading (#5170)
This change brings public data-types such that
we can ask projects to implement gateway projects
externally than maintaining in our repo.

All publicly exported structs are maintained in object-api-datatypes.go

completePart --> CompletePart
uploadMetadata --> MultipartInfo

All other exported errors are at object-api-errors.go
2017-11-14 13:55:10 +05:30
Krishna Srinivas
7d3eaf79ff Set Minio user-agent for GCS calls (#5154) 2017-11-13 19:06:51 -08:00
kannappanr
b63c37b28e Return MethodNotAllowed error in PostPolicyBucketHandler if URL contains object name (#5142)
S3 spec requires that MethodNotAllowed error be return if object name is part
of the URL.

Fix postpolicy related unit tests to not set object name as part of target URL.

Fixes #5141
2017-11-13 16:30:20 -08:00
kannappanr
f460eceb6d Check for value > 7 days in X-Amz-Expires header. (#5163)
Add a check to see if the X-Amz-Expires header in the presigned URL is less than 7 days.

Fixes #5162
2017-11-13 12:54:03 -08:00
Harshavardhana
d10679866c Fix minio distributed setup to properly work on windows (#5152)
On windows having a preceding "/" will cause problems, if the
command line already has C:/<export-folder/ in it. Final resulting
path on windows might become C:/C:/ this will cause problems
of starting minio server properly in distributed mode on windows.
As a special case make sure to trim off the separator.

NOTE: It is also perfectly fine for windows users to have a path
without C:/ since at that point we treat it as relative path
and obtain the full filesystem path as well. Providing C:/
style is necessary to provide paths other than C:/,
such as F:/, D:/ etc.

Another additional benefit here is that this style also
supports providing UNC paths as well.

Fixes #5136
2017-11-12 08:09:53 +05:30
Andreas Auernhammer
a79a7e570c replace SSE-C key derivation scheme (#5168)
This chnage replaces the current SSE-C key derivation scheme. The 'old'
scheme derives an unique object encryption key from the client provided key.
This key derivation was not invertible. That means that a client cannot change
its key without changing the object encryption key.
AWS S3 allows users to update there SSE-C keys by executing a SSE-C COPY with
source == destination. AWS probably updates just the metadata (which is a very
cheap operation). The old key derivation scheme would require a complete copy
of the object because the minio server would not be able to derive the same
object encryption key from a different client provided key (without breaking
the crypto. hash function).

This change makes the key derivation invertible.
2017-11-10 17:21:23 -08:00
Harshavardhana
16ecaac4fc Help message should prioritize gateway after server (#5153)
Currently gateway is listed as a command after {version, update}
which is incorrect, fix it.
2017-11-08 13:38:53 -08:00
Andreas Auernhammer
ca6b4773ed add SSE-C support for HEAD, GET, PUT (#4894)
This change adds server-side-encryption support for HEAD, GET and PUT
operations. This PR only addresses single-part PUTs and GETs without
HTTP ranges.

Further this change adds the concept of reserved object metadata which is required
to make encrypted objects tamper-proof and provide API compatibility to AWS S3.
This PR adds the following reserved metadata entries:
- X-Minio-Internal-Server-Side-Encryption-Iv          ('guarantees' tamper-proof property)
- X-Minio-Internal-Server-Side-Encryption-Kdf         (makes Key-MAC computation negotiable in future)
- X-Minio-Internal-Server-Side-Encryption-Key-Mac     (provides AWS S3 API compatibility)

The prefix `X-Minio_Internal` specifies an internal metadata entry which must not
send to clients. All client requests containing a metadata key starting with `X-Minio-Internal`
must also rejected. This is implemented by a generic-handler.

This PR implements SSE-C separated from client-side-encryption (CSE). This cannot decrypt
server-side-encrypted objects on the client-side. However, clients can encrypted the same object
with CSE and SSE-C.

This PR does not address:
 - SSE-C Copy and Copy part
 - SSE-C GET with HTTP ranges
 - SSE-C multipart PUT
 - SSE-C Gateway

Each point must be addressed in a separate PR.

Added to vendor dir:
 - x/crypto/chacha20poly1305
 - x/crypto/poly1305
 - github.com/minio/sio
2017-11-07 15:18:59 -08:00
Krishna Srinivas
7e7ae29d89 browser: Remove hardcoding of minioBrowserPrefix=/minio (#5048)
This enable reverse proxy of minio-browser. Fixes #5040
2017-11-06 15:59:37 -08:00
Krishnan Parthasarathi
7d18f00116 Make GCS multipart upload failures S3-compatible (#5138) 2017-11-06 10:09:21 -08:00
Harshavardhana
719f8c258a fix content-sha256 verification for presigned PUT (#5137)
It is possible that x-amz-content-sha256 is set through
the query params in case of presigned PUT calls, make sure
that we validate the incoming x-amz-content-sha256 properly.

Current code simply just allows this without honoring the
set x-amz-content-sha256, fix it.
2017-11-05 16:32:19 +05:30
Harshavardhana
dcdb07433a Support conditions for ListMultipartUploads and ListParts (#5130) 2017-11-02 11:39:48 -07:00
kannappanr
26e9f78a86 Display help when access/secret key is not set (#5132)
Display help message, when access and secret keys are not set in
any of the gateway.

Fixes #5124
2017-11-01 11:45:27 -07:00
Bala FA
32c6b62932 move credentials as separate package (#5115) 2017-10-31 11:54:32 -07:00
Harshavardhana
8d584bd819 Remove unused value from retry-storage (#5129) 2017-10-29 15:43:16 +05:30
Harshavardhana
203ac8edaa Bucket policies should use minio-go/pkg/policy instead. (#5090) 2017-10-27 16:14:06 -07:00
Harshavardhana
8bbfb1b714 Allow event notifications to work without region (#5119)
Fixes #5101
2017-10-27 15:09:55 -07:00
Timon Wong
6400f506da Simplify gateway backend registration (#5111) 2017-10-27 15:07:46 -07:00
Frank Wessels
7195ac7f14 Add space to error message (#5108) 2017-10-27 15:07:14 -07:00
Krishna Srinivas
03df692ae2 Support for bosh/pcf user-agent when querying for updates. (#5116) 2017-10-26 18:53:45 -07:00
kannappanr
a011fe8450 "0" offset is ignored in GetObject method in Azure Gateway code (#5118)
In GetObject method, Check if startoffset is a non-negative value.
Ignore check for startOffset > and check for only length > 0.

Fixes minio/mint#191
2017-10-26 18:01:46 -07:00
kannappanr
95d97c2d6d GCS gateway to return error in getBucketPolicy, when no policy is set (#5117)
Return NoSuchBucketPolicy error when there is no policy set.
Fixes minio/mint#199
2017-10-26 18:01:00 -07:00
Bala FA
bc8b936d4b convert ETag properly for all gateways (#5099)
Previously ID/ETag from backend service is used as is which causes
failure on s3cmd like tools where those tools use ETag as checksum to
validate data.  This is fixed by prepending "-1".

Refer minio/mint#193 minio/mint#201
2017-10-26 10:17:07 -07:00
Aditya Manthramurthy
d23ded0d83 Use retryableStorage after healing format.json (#5105)
- Previously networkStorage was being used and this lead to errors
  when listing with a down server/disk

Fixes #5089
2017-10-26 09:52:23 -07:00
Julien Maitrehenry
db3fed2279 Fix s3MetaToAzureProperties Content-Md5 key (#5068) 2017-10-25 11:00:07 -07:00
A. Elleuch
866dffcd62 log: Store http request/responses in a log file (#4804)
When MINIO_TRACE_DIR is provided, create a new log file and store all
HTTP requests + responses data, body are excluded to reduce memory
consumption. MINIO_HTTP_TRACE=1 enables logging. Use non mem
consuming http req/resp recorders, the maximum is about 32k per request.
This logs to STDOUT, body logging is disabled for PutObject PutObjectPart
GetObject.
2017-10-25 10:59:53 -07:00
Harshavardhana
5eb210dd2e Set etag properly to calculated value if available (#5106)
Fixes #5100
2017-10-24 12:25:42 -07:00
Harshavardhana
1d8a8c63db Simplify data verification with HashReader. (#5071)
Verify() was being called by caller after the data
has been successfully read after io.EOF. This disconnection
opens a race under concurrent access to such an object.
Verification is not necessary outside of Read() call,
we can simply just do checksum verification right inside
Read() call at io.EOF.

This approach simplifies the usage.
2017-10-22 11:00:34 +05:30
Julien Maitrehenry
1256b0b818 Fix multipart upload etag on azure gateway (#5055) 2017-10-20 14:00:18 -07:00
Krishna Srinivas
7e05b826fa Figure out projectID for GCS automatically from credentials.json (#5029)
fixes #5027
2017-10-20 13:59:12 -07:00
Harshavardhana
d82a1da511 Fix notification unmarshalling, unmarshal only when size is > 0 (#5087)
Fixes #5085
2017-10-20 13:57:57 -07:00
Frank Wessels
f598f4fd1b Fix typo in comment (#5088) 2017-10-20 15:08:15 +05:30
A. Elleuch
b919462610 fix: Avoid teeing data into a null cache buffer (#5070)
In some cases, Cache manager returns ErrCacheFull error when creating a
new cache buffer but the code still sends object data to nil cache buffer data.
2017-10-18 14:42:10 -07:00
Nitish Tiwari
8287ab091c Ignore file not found error for multipart-uploads (#5065)
Dont print the error errFileNotFound, as it is expected that concurrent
complete-multipart-uploads or abort-multipart-uploads would have deleted
the file, and the file may not be found

Fixes: https://github.com/minio/minio/issues/5056
2017-10-18 14:26:20 -07:00
Harshavardhana
f25bec6bf1 Do not attempt to generate URLToken for anonymous downloads (#5078)
This is a regression since last release - fixes #5076
2017-10-18 11:14:27 +05:30
Krishna Srinivas
75865efb0e fs: All parts except the last part should be of the same size (#5045)
fixes #4881
2017-10-17 12:01:28 -07:00
Harshavardhana
53e133e844 Remove NotSupported error redundant with NotImplemented (#5074) 2017-10-17 08:11:06 +05:30
Harshavardhana
b2cbade477 Support creating empty directories. (#5049)
Every so often we get requirements for creating
directories/prefixes and we end up rejecting
such requirements. This PR implements this and
allows empty directories without any new file
addition to backend.

Existing lower APIs themselves are leveraged to provide
this behavior. Only FS backend supports this for
the time being as desired.
2017-10-16 17:20:54 -07:00
Harshavardhana
0c0d1e4150 Implement backblaze-b2 gateway support (#5002)
Fixes https://github.com/minio/minio/issues/4072
2017-10-13 16:26:16 +05:30
Harshavardhana
3d0dced23c Remove go1.9 specific code for windows (#5033)
Following fix https://go-review.googlesource.com/#/c/41834/ has
been merged upstream and released with go1.9.
2017-10-13 15:31:15 +05:30
Nitish Tiwari
ad53c5d859 Remove body from POST request in webhook (#5067)
When webhook notification is configured, Minio server tries to lookup the
webhook endpoint by making a POST request with body set as releasetag.
We can remove the body from the POST request as the POST body does not
add any specific value.

This discussion on IETF group says empty POSTs are okay
http://lists.w3.org/Archives/Public/ietf-http-wg/2010JulSep/0272.html

Fixes: https://github.com/minio/minio/issues/5066
2017-10-13 13:29:01 +05:30
Harshavardhana
45463b1d6b Fix azure metadata handling for all incoming PUT requests (#5038)
s3cmd cli fails when trying to upload a file to azure gateway.
Previous fixes in azure to handle client side encryption alone
did not completely address the problem.

We need to possibilly convert all the x-amz-meta-<name>
, i.e specifically <name> should be converted into a
C# identifier as mentioned in the docs for `put-blob`.

https://docs.microsoft.com/en-us/rest/api/storageservices/put-blob

```
s3cmd put README.md s3://myanis/
upload: 'README.md' -> 's3://myanis/README.md'  [1 of 1]
 4598 of 4598   100% in    0s    47.24 kB/s  done
upload: 'README.md' -> 's3://myanis/README.md'  [1 of 1]
 4598 of 4598   100% in    0s    50.47 kB/s  done
ERROR: S3 error: 400 (InvalidArgument): Your metadata headers are not supported.
```

There is a separate issue with s3cmd after this fix is applied where
the ETag is wronly validated https://github.com/s3tools/s3cmd/issues/880
But that is an upstream s3cmd problem which wrongly interprets ETag
to be md5sum of the content that was uploaded.
2017-10-12 12:16:24 -07:00
Harshavardhana
4deefa3695 tests: Remove dependency on check.v1 (#5034)
This PR addresses a long standing dependency on
`gopkg.in/check.v1` project used for our tests.
All tests are re-written to use the go default
testing framework instead.

There was no reason for us to use an external
package where Go tools are sufficient for this.
2017-10-10 02:14:42 -07:00
Harshavardhana
099b5293a9 Move gateway unsupported functions into a common struct. (#5009)
This is done to avoid repeated declaration of not-implemented
functions for each gateway. It also avoids a possible bug in go
https://github.com/golang/go/issues/18468 which is triggered on
our multiple PRs already.
2017-10-09 16:41:35 -07:00
Aditya Manthramurthy
4a0a491ca1 Refactor update check code (#5020)
- Add release-time conversion helpers
- Split GetCurrentReleaseTime() into two simpler functions.
- Avoid appending strings when assembling user-agent string.
- Reorder release info URLs to check the newer URLs earlier.
- Remove trivial low-level functions created solely for the purpose of
  writing tests.
- Remove some unnecessary tests.
2017-10-09 16:12:13 -07:00
Harshavardhana
db6b6e9518 S3 peers should be initialized properly (#5024)
Fixes #4991
2017-10-08 20:23:42 -07:00
Bala FA
d8a11c8f4b fix build failure for go1.9 (#4872) 2017-10-06 17:00:15 -07:00
Harshavardhana
0b546ddfd4 Return errors in PutObject()/PutObjectPart() if input size is -1. (#5015)
Amazon S3 API expects all incoming stream has a content-length
set it was superflous for us to support object layer which supports
unknown sized stream as well, this PR removes such requirements
and explicitly error out if input stream is less than zero.
2017-10-06 09:38:01 -07:00
Krishna Srinivas
d1712a46a7 Enable ListMultipartUploads and ListObjectParts for FS (#4996)
* Enable ListMultipartUploads and ListObjectParts for FS.
Previously we had disabled ListMultipartUploads and ListObjectParts
to see if any clients break. Docker registry broke. This patch
enables ListMultipartUploads and ListObjectParts, however
ListMultipartUploads with prefix based listing is not
supported (which is not used by docker registry anyway).
i.e ListMultipartUploads will need exact object name.
2017-10-05 16:08:25 -07:00
Bala FA
88938340b3 remove all dead codes (#5019)
Fixes #5012
2017-10-05 12:25:45 -07:00
Krishnan Parthasarathi
13a7033505 Translate s3 gateway errors at object layer (#5006) 2017-10-05 12:24:45 -07:00
Harshavardhana
89d528a4ed Allow CopyObject() in S3 gateway to support metadata (#5000)
Fixes #4924
2017-10-03 10:38:25 -07:00
A. Elleuch
53f3d2fd65 Push max threads to little less than kernel limit (#5001)
Let Minio server use more threads than allowed by golang runtime. This
is important to better deal with high load.
2017-10-03 10:37:45 -07:00