Commit Graph

5294 Commits

Author SHA1 Message Date
kannappanr 2d84b02bc4 Check for absence of checksum field and attributes. (#6298)
Fixes #6295
2018-08-20 16:58:47 -07:00
poornas 8b2801bd46 Update documentation to show how to update minio config fields (#6301)
- with recent commit 1fb2e9ef95, config
can no longer be updated by editing config.json. This is because config
has been migrated inside the minio backend. Update documentation on
how to set/get configuration using mc admin config command.
2018-08-20 13:37:10 -07:00
Harshavardhana 7d7e21aebb Merge initConfig logic to ConfigSys (#6312) 2018-08-19 13:57:18 -07:00
Nitish Tiwari bf14e5ce1b Fix distributed doc as total storage capacity is not displayed anymore (#6309) 2018-08-18 10:01:24 -07:00
Andreas Auernhammer d531080b7e add SSE-KMS not-implemented error handling (#6234)
This commit adds error handling for SSE-KMS requests to
HEAD, GET, PUT and COPY operations. The server responds
with `not implemented` if a client sends a SSE-KMS
request.
2018-08-17 21:07:19 -07:00
Harshavardhana a6b8a5487a Fail gateway properly with an error on port conflicts (#6303) 2018-08-17 21:06:36 -07:00
Minio Trusted 6c0d53a1c5 Update yaml files to latest version RELEASE.2018-08-18T03-49-57Z 2018-08-18 03:54:06 +00:00
Harshavardhana 9f14433cbd Ensure that setConfig uses latest functionality (#6302) 2018-08-17 18:51:34 -07:00
Harshavardhana 50a817e3d3 Use new listener which implements enhanced tcp features (#6289)
This package provide customizable TCP net.Listener with various
performance-related options:

 * SO_REUSEPORT. This option allows linear scaling server performance
   on multi-CPU servers.
   See https://www.nginx.com/blog/socket-sharding-nginx-release-1-9-1/ for details.
 * TCP_DEFER_ACCEPT. This option expects the server reads from the accepted
   connection before writing to them.
 * TCP_FASTOPEN. See https://lwn.net/Articles/508865/ for details.
2018-08-17 18:44:02 -07:00
Harshavardhana 5a4a57700b Add select docs and fix return values for Select API (#6300) 2018-08-17 17:11:39 -07:00
Harshavardhana 3de5a3157f Enhance picking valid xlMeta based on quorum (#6297)
This PR borrows the idea from getFormatXLQuorum()
2018-08-17 14:42:04 -07:00
Kaan Kabalak 50dec08002 Correct link paths in Chinese documentation (#6299) 2018-08-17 13:16:17 -07:00
poornas e71ef905f9 Add support for SSE-S3 server side encryption with vault (#6192)
Add support for sse-s3 encryption with vault as KMS.

Also refactoring code to make use of headers and functions defined in
crypto package and clean up duplicated code.
2018-08-17 12:52:14 -07:00
junpeng liu 3d197c1449 Modify several translation errors (#6038) 2018-08-17 12:04:09 +05:30
Harshavardhana 65de2d68c0 Allow for proper garbage collection pooling bytes.Buffer (#6266) 2018-08-16 18:37:43 -07:00
Harshavardhana 1103ad2d08 Watch for symlinked certs in container envs (#6282)
Fixes #6278
2018-08-16 18:37:21 -07:00
Harshavardhana eab947cf42 Make sure to update modTime in erasure metadata (#6296)
This is to ensure that when we update xl.json with new
parts have the latest modtime, in-turn avoids consistency
issues when the disk is offline.
2018-08-16 17:55:01 -07:00
Harshavardhana 0fe9e95250 Validate prefixes on all sets (#6294)
This PR fixes a regression introduced in 8eb838bf91
where hashing technique was used on prefixes to get the right set
to perform the operation, this is not correct since prefixes and
their corresponding keys might hash to a different value depending
on the key length.

For prefixes/directories we should look everywhere to support proper
quorum based listing.

Fixes #6293
2018-08-16 16:49:38 -07:00
kannappanr c7946ab9ab
Remove unnecessary error log messages (#6186) 2018-08-16 12:57:49 -07:00
Harshavardhana f5df3b4795 Remove select docs (#6287)
Select API is sufficiently documented, this doc is also incomplete.

- https://aws.amazon.com/blogs/aws/s3-glacier-select/
- https://aws.amazon.com/blogs/developer/introducing-support-for-amazon-s3-select-in-the-aws-sdk-for-ruby/
- https://aws.amazon.com/blogs/developer/introducing-support-for-amazon-s3-select-in-the-aws-sdk-for-javascript/
- https://aws.amazon.com/blogs/developer/category/storage/s3-select/
2018-08-15 19:47:22 -07:00
Harshavardhana f26325c988 Support supplying custom drives per set count (#6261) 2018-08-15 16:35:21 -07:00
Arjun Mishra 7c14cdb60e S3 Select API Support for CSV (#6127)
Add support for trivial where clause cases
2018-08-15 03:30:19 -07:00
Harshavardhana 0e02328c98 Migrate config.json from config-dir to backend (#6195)
This PR is the first set of changes to move the config
to the backend, the changes use the existing `config.json`
allows it to be migrated such that we can save it in on
backend disks.

In future releases, we will slowly migrate out of the
current architecture.

Fixes #6182
2018-08-15 10:11:47 +05:30
Harshavardhana 380524ae27
Unlock read lock on uploadID upon errors (#6283) 2018-08-14 18:35:30 -07:00
kannappanr 0286e61aee Log disk not found error just once (#6059)
Modified the LogIf function to log only if the error passed
is not on the ignored errors list.

Currently, only disk not found error is added to the list.
Added a new function in logger package called LogAlwaysIf, 
which will print on any error.

Fixes #5997
2018-08-14 13:58:48 -07:00
wd256 ff29aed05d gcs: Translate S3 user-defined metadata prefix to/from GCS custom metadata prefix (#6270) 2018-08-14 11:53:39 -07:00
Harshavardhana 64f2c61813 Implement memory efficient readdir for windows (#6247)
Fixes #6164
2018-08-09 14:52:29 -07:00
Andreas Auernhammer 525c04fd07 crypto: add SSE-KMS HTTP header detection (#6228)
This commit adds support for detecting SSE-KMS headers.
The server should be able to detect SSE-KMS headers to
at least fail such S3 requests with not implemented.
2018-08-09 13:02:57 -07:00
Sanat Mouli efac90461a Fix bug in Share Object Modal (#6257)
Fixes #6249
2018-08-09 09:45:43 -07:00
Durgadas Kamath 71979376b5 Add s390x support (#6263) 2018-08-08 15:49:12 -07:00
Anis Elleuch 5a1ae862a7 Avoid sending an error after 206 HTTP code (#6264)
When a S3 client sends a GET Object with a range header, 206 http
code is returned indicating success, however the call of the object
layer's GetObject() inside the handler can return an error and will lead
to writing an XML error message, which is obviously wrong since
we already sent 206 http code. So in the case, we just stop sending
data to the S3 client, this latter can still detect if there is no
error when comparing received data with Content-Length header
in the Get Object response.
2018-08-08 15:39:47 -07:00
Anis Elleuch 6df20734f9 Avoid logging the body of the http 206 response (#6258)
When an S3 client issues a GET request with range specified, Minio
server returns some partial data with 206 http code. The latter
is sent in MINIO_HTTP_TRACE output which is incorrect. This PR
fixes the issue.
2018-08-08 12:34:42 -07:00
Sanat Mouli 98bce72295 Fix output path for go-bindata-assetfs (#6253) 2018-08-08 10:46:02 +05:30
Harshavardhana 2f1756489e Add tracing capabilities for internode rpc Servers (#6254) 2018-08-07 15:21:30 +05:30
Harshavardhana 9719640e34 Use sha256-simd instead of crypto/sha256 (#6252) 2018-08-06 18:00:10 -07:00
Krishna Srinivas ce02ab613d Simplify erasure code by separating bitrot from erasure code (#5959) 2018-08-06 15:14:08 -07:00
Oleg Kovalov 37de2dbd3b simplifying if-else chains to switches (#6208) 2018-08-06 10:26:40 -07:00
Harshavardhana a82500f162 Support dumb terminals by turning off color (#6246)
ANSI colors do not work on dumb terminals, in situations
when minio is running as a service under systemd.

This PR ensures we turn off color in those situations.
2018-08-06 18:16:49 +05:30
Harshavardhana 2dede2fdc2 Add reliable RemoveAll to handle racy situations (#6227) 2018-08-06 09:45:28 +05:30
Harshavardhana 13fbb96736 Hold locks granularly in nslockMap (#6242)
With benchmarks increases the performance for small files
by almost 4x times the previous releases.
2018-08-06 08:55:25 +05:30
Harshavardhana eabfcea34e
Add granular locking in retryTicker (#6236)
This is to avoid serializing RPC contention on ongoing
parallel operations, the blocking profile indicating
all calls were being serialized through setRetryTicker.
2018-08-03 18:57:00 -07:00
Andreas Auernhammer a078703214 catch crypto.* errors and add SSE-S3 invalid algorithm err (#6229)
This commit adds the crypto.* errors to the
`toAPIErrorCode` switch. Further this commit adds an S3
API error code returned whenever the client specifes a
SSE-S3 request with an invalid algorithm parameter.
2018-08-03 16:55:45 -07:00
Harshavardhana bd2b22572f
Increase max idle connections from 100 to 4096 (#6244)
This is to be inline with our benchmarking results
2018-08-03 16:29:28 -07:00
poornas 5f69f04909 nas gateway: fix regression in global bucket policy initialization (#6243)
Fixes #6238
globalPolicySys used to be initialized in fs/xl layer. The referenced
commit moved this logic to server/gateway initialization,but a check
to avoid double initialization prevented globalPolicySys to be loaded
from disk for NAS.

fixes regression from commit be1700f595
2018-08-03 15:12:18 -07:00
Minio Trusted a1a426e523 Update yaml files to latest version RELEASE.2018-08-02T23-11-36Z 2018-08-02 23:17:30 +00:00
Harshavardhana a091b1a3ee Fix admin API doc formatting (#6235) 2018-08-02 14:21:38 -07:00
Harshavardhana 556a51120c Deprecate ListLocks and ClearLocks (#6233)
No locks are ever left in memory, we also
have a periodic interval of clearing stale locks
anyways. The lock instrumentation was not complete
and was seldom used.

Deprecate this for now and bring it back later if
it is really needed. This also in-turn seems to improve
performance slightly.
2018-08-02 23:09:42 +05:30
poornas eb391a53c1 check for syscall errors in posix-errors helper functions (#6232) 2018-08-02 10:38:51 +05:30
Harshavardhana e17e09ea3c Handle POST object upload without filename param (#6221)
POST mime/multipart upload style can have filename value optional
which leads to implementation issues in Go releases in their
standard mime/multipart library.

When `filename` doesn't exist Go doesn't update `form.File` which
we rely on to extract the incoming file data, strangely when `filename`
is not specified this data is buffered in memory and is now part of
`form.Value` instead of `form.File` which creates an inconsistent
behavior.

This PR tries to fix this in our code for the time being, but ideal PR
would be to fix the upstream mime/multipart library to handle the
above situation consistently.
2018-08-01 14:19:11 -07:00
Andreas Auernhammer 76c423392a crypto: add `GenerateIV` from random IV generation (#6215)
This commit adds a `GenerateIV` function to simplify
the generation of random IVs.

It adds some unit tests for `GenerateIV` in key_test.go
2018-08-01 01:02:07 -07:00