Commit Graph

5164 Commits

Author SHA1 Message Date
Andreas Auernhammer 6c93c60424 crypto: add a basic KMS implementation (#6161)
This commit adds a basic KMS implementation for an
operator-specified SSE-S3 master key. The master key
is wrapped as KMS such that using SSE-S3 with master key
and SSE-S3 with KMS can use the same code.

Bindings for a remote / true KMS (like hashicorp vault)
will be added later on.
2018-07-17 22:40:34 -07:00
Krishna Srinivas 0c9f4c9092 formatMetaV1 should be "inherited" by disk format structs (#6134) 2018-07-16 20:26:42 -07:00
Andreas Auernhammer 289d6ce1d7 crypto: update SSE-S3 and SSE-C key derivation (#6152)
This commit updates the key derivation to reflect the
latest change of crypto/doc.go. This includes handling
the insecure legacy KDF.

Since #6064 is fixed, the 3. test case for object key
generation is enabled again.
2018-07-16 07:49:50 -07:00
Rob Girard 2a12e694f3 Changed command line examples (#6149)
Order for server:disk originally provided wouldn't stripe 
wide and may lead to availability issues.

Also added Short-form examples using {1...n} and a 
warning about {1..2} vs {1...3}
2018-07-14 20:48:38 +05:30
Harshavardhana db26d3c9e2 Fix handling files at leaf attempting disk.ListDir() (#6155)
Return an ignorable error upon readDir() failure on
a file.
2018-07-14 12:11:48 +05:30
Stefan Husch 914c76a801 Implement lock.Open() to fix #5642 (#6150) 2018-07-13 18:37:02 -07:00
Krishnan Parthasarathi a1ef90be52 gcs: Limit number of objects listed to max-keys (#6133) 2018-07-13 10:27:26 -07:00
Minio Trusted 7c4a41b933 Update yaml files to latest version RELEASE.2018-07-13T00-09-07Z 2018-07-13 00:43:51 +00:00
Nitish Tiwari 2aa18cafc6 Update federation target to etcd/clientv3 (#6119)
With CoreDNS now supporting etcdv3 as the DNS backend, we
can update our federation target to etcdv3. Users will now be
able to use etcdv3 server as the federation backbone.

Minio will update bucket data to etcdv3 and CoreDNS can pick
that data up and serve it as bucket style DNS path.
2018-07-12 14:12:40 -07:00
Andreas Auernhammer adf7340394 fix size computation for en/decrypted objects (#6147)
This PR fixes the size calculation for encrypted multipart
objects.
2018-07-12 11:23:32 -07:00
Harshavardhana b11a8eb3f4
Support multiple values for x-amz-meta header (#6145)
Fixes #5595
2018-07-12 09:40:14 -07:00
Andreas Auernhammer 15771ebe8d Fix decrypted object size and key derivation in CopyObjectPart (#6141)
This commit fixes the size calculation for multipart
objects. The decrypted size of an encrypted multipart
object is the sum of the decrypted part sizes.

Also fixes the key derivation in CopyObjectPart.
Instead of using the same object-encryption-key for each
part now an unique per-part key is derived.

Updates #6139
2018-07-12 21:59:56 +05:30
Praveen raj Mani 44865596db SignatureV4 validation with Metadata in the presignedUrl (#5894)
The `X-Amz-Meta-`/`X-Minio-Meta-` will now be recognized in query string also.

Fixes #5857 #5950
2018-07-10 20:27:10 -07:00
Nitish Tiwari c9bc7e47b9
Update distributed docs (#6123)
We need to clarify that distributed Minio doesn't strictly need a
fresh drive, instead it just needs a fresh directory on the drive.
2018-07-10 07:32:24 +05:30
Anis Elleuch be1700f595 Avoid startup abort when a notify target is down (#6126)
Minio server was preventing itself to start when any notification
target is down and not running. The PR changes the behavior by
avoiding startup abort in that case, so the user will still
be able to access Minio server using mc admin commands after
a restart or set config commands.
2018-07-10 07:20:31 +05:30
Minio Trusted 42c5b64e4e Update yaml files to latest version RELEASE.2018-07-10T01-42-11Z 2018-07-10 01:46:03 +00:00
Krishna Srinivas 40ed0d1f5d Support 1GB disk size (#6137)
Pivotal CF by default has 1GB disk option which causes minio to not start
2018-07-09 18:23:49 -07:00
Andreas Auernhammer b181a693fb fix object rebinding SSE-C security guarantee violation (#6121)
This commit fixes a weakness of the key-encryption-key
derivation for SSE-C encrypted objects. Before this
change the key-encryption-key was not bound to / didn't
depend on the object path. This allows an attacker to
repalce objects - encrypted with the same
client-key - with each other.

This change fixes this issue by updating the
key-encryption-key derivation to include:
 - the domain (in this case SSE-C)
 - a canonical object path representation
 - the encryption & key derivation algorithm

Changing the object path now causes the KDF to derive a
different key-encryption-key such that the object-key
unsealing fails.
Including the domain (SSE-C) and encryption & key
derivation algorithm is not directly neccessary for this
fix. However, both will be included for the SSE-S3 KDF.
So they are included here to avoid updating the KDF
again when we add SSE-S3.

The leagcy KDF 'DARE-SHA256' is only used for existing
objects and never for new objects / key rotation.
2018-07-09 17:18:28 -07:00
Anis Elleuch 4ddc222f46 fix: Propagate bucket policy update in a distributed setup (#6135)
Commit 0d52126023 caused a regression in setting
a new bucket policy in a distributed setup. The reason is that gob is not able
to encode fields declared as interfaces unless we provide GobEncode() and GobDecode()
This PR adds them by using json marshaller and unmarshaller that are already
implemented for Functions interface.
2018-07-09 02:18:48 -07:00
Mike Scarlett c310cbbe89 Update comments regarding GCS component count (#6131) 2018-07-06 17:07:11 -07:00
Harshavardhana 0ef0d7e685 pkg/certs: On windows watch for directory changes to load certs (#6128)
This PR fixes an issue when configuring Minio TLS on windows
2018-07-05 16:33:37 -07:00
wksw c62813c887 completed sample config and fix wrong json format (#6096)
missing worm configuration and wrong json format
2018-07-05 13:40:07 +05:30
poornas 1da362538b cache: allow ellipse style entries for MINIO_CACHE_DRIVES (#6088)
Fixes #5863
2018-07-03 16:54:10 -07:00
Krishna Srinivas e40a5e05e1 Do notification in background to not block S3 client REST calls (#6005) 2018-07-03 11:09:36 -07:00
ebozduman b0b0fb4c8d Validate Minio config.json file on the client side (#6067) 2018-07-03 11:07:46 -07:00
kannappanr 726e75611e
Do not log BucketNotFound errors on minio console (#6114) 2018-07-03 11:04:55 -07:00
Rafael Peria de Sene 317e648c0d Add support for ppc64le (#6116)
Signed-off-by: Rafael Peria de Sene <rpsene@br.ibm.com>
2018-07-03 09:42:19 -07:00
Harshavardhana 80b3e9cb03 use appropriate HTTP status for storage is full (#6117) 2018-07-03 09:40:14 -07:00
Harshavardhana 6c85706c24 Use GetSourceIP for source ip as request params (#6109)
Fixes #6108
2018-07-02 14:40:18 -07:00
Harshavardhana d7ced9a8b5 Upgrade gjson to bring in new performance improvements (#6106) 2018-07-02 13:31:11 -07:00
Praveen raj Mani 360f3f9335 Checking the existence of the bucket in DeleteObjectHandler (#6085)
Fixes #6077
2018-06-30 22:35:43 -07:00
wd256 25f9b0bc3b Handle ListObjectsV2 start-after parameter in ObjectLayer (#6078) 2018-07-01 09:52:45 +05:30
Harshavardhana a5453c307f Fix kernel tuning script to ignore write failures (#6107)
Certain SCSI drivers do not allow certain tuning parameters
like nr_requests, max_sectors_kb to be changed, ignore these
errors silently as this script is simply a best effort.

Fixes #6103
2018-06-30 14:55:21 -07:00
Harshavardhana 92a6676a2f Avoid unnecessary logging for policy not found errors (#6104) 2018-06-29 06:30:10 -07:00
Minio Trusted f53d511798 Update yaml files to latest version RELEASE.2018-06-29T02-11-29Z 2018-06-29 02:14:58 +00:00
Harshavardhana e5e522fc61
docs: fix all Chinese doc links for the new docs site (#6097)
Additionally fix typos, default to US locale words
2018-06-28 16:02:02 -07:00
Harshavardhana de251483d1 Avoid ticker timer to simplify disk usage (#6101)
This PR simplifies the code to avoid tracking
any running usage events. This PR also brings
in an upper threshold of upto 1 minute suspend
the usage function after which the usage would
proceed without waiting any longer.
2018-06-28 15:05:45 -07:00
Andreas Auernhammer 805186ab97 new package: cmd/crypto (#6062)
This commit introduces a new crypto package providing
AWS S3 related cryptographic building blocks to implement
SSE-S3 (master key or KMS) and SSE-C.

This change only adds some basic functionallity esp.
related to SSE-S3 and documents the general approach
for SSE-S3 and SSE-C.
2018-06-28 12:47:42 -07:00
Praveen raj Mani ea76e72054 Incorrect error message for insufficient volume fix (#6099)
Reply back with appropriate error message when the server is spawn
with volume of insufficient size (< 1GiB).

Fixes #5993.
2018-06-28 12:01:05 -07:00
Kaan Kabalak 8bd7a19d50 Refactor login page to get rid of warning on test (#6083) 2018-06-28 10:30:57 -07:00
Harshavardhana 25de775560 disable disk-usage when export is root mount path (#6091)
disk usage crawling is not needed when a tenant
is not sharing the same disk for multiple other
tenants. This PR adds an optimization when we
see a setup uses entire disk, we simply rely on
statvfs() to give us total usage.

This PR also additionally adds low priority
scheduling for usage check routine, such that
other go-routines blocked will be automatically
unblocked and prioritized before usage.
2018-06-27 18:59:38 -07:00
Harshavardhana abf209b1dd load bucket policies using object layer API (#6084)
This PR fixes an issue during gateway mode
where underlying policies were not translated
into meaningful policies.
2018-06-27 12:29:48 +05:30
Harshavardhana d9d13c898c docs: Fix admin heal API description (#6073) 2018-06-26 10:53:14 -07:00
Nitish Tiwari ad79c626c6
Throw 404 for head requests for prefixes without trailing "/" (#5966)
Minio server returns 403 (access denied) for head requests to prefixes
without trailing "/", this is different from S3 behaviour. S3 returns
404 in such cases.

Fixes #6080
2018-06-26 06:54:00 +05:30
Andreas Auernhammer cd152f404a replace os.Exit with panic for logger.CriticalIf (#6065)
This commit prevents complete server failures caused by
`logger.CriticalIf` calls. Instead of calling `os.Exit(1)`
the function now executes a panic with a special value
indicating that a critical error happend. At the top HTTP
handler layer panics are recovered and if its a critical
error the client gets an InternalServerError status code.

Further this allows unit tests to cover critical-error code
paths.
2018-06-25 13:51:49 -07:00
Praveen raj Mani 5fbdd70de9 Minio handle for list_objects_v2 with a blank ContinuationToken (#5940)
minio will now return an error for a blank continuation token in list_objects_v2,so as in s3.

Fixes #5931
2018-06-25 12:35:43 -07:00
Ashish Kumar Sinha 0bbdd02a57 Updating disk storage for FS/Erasure mode (#6081)
Updating the disk storage stats for FS/Erasure coded backend
2018-06-25 10:46:48 -07:00
Annanay Agarwal 78abe5234e Add functionality to make cache limit configurable (#5991) 2018-06-25 10:24:12 -07:00
Minio Trusted f46ee54194 Update yaml files to latest version RELEASE.2018-06-22T23-48-46Z 2018-06-22 23:52:10 +00:00
Harshavardhana 6005dbf01f Fix broken doc links (#6068) 2018-06-22 09:14:41 +05:30