Commit Graph

27 Commits

Author SHA1 Message Date
Harshavardhana
233824bf92 Configure http2 with higher maxconcurrent streams (#7363)
This value is needed for Minio's internode communication,
read the meaning of this value as per the HTTP 2.0 spec

https://http2.github.io/http2-spec/#rfc.section.5.1.2
2019-03-14 11:57:35 -07:00
Harshavardhana
5e69a107d8 Support detecting mountpoints correctly (#7288)
Currently windows support was relying on Symlink as
a way to detect a drive, this doesn't work in latest
Windows 2016, fix this to use a proper mechanism by
using win32 APIs.

Additionally also add support for detecting bind mounts
on Linux.
2019-02-26 18:04:53 -08:00
Harshavardhana
55ef51a99d Vendorize all recent changes to minio-go (#7135)
- Default support for S3 dualstack endpoints (IPv6 support)
- Support granular policy conditionals in List operations
- Support proxy cookies for stickiness
2019-01-23 19:22:09 +05:30
Harshavardhana
3265112d04 Remove gateway implementations for manta, sia and b2 (#7115) 2019-01-20 08:10:58 -08:00
Praveen raj Mani
6571641735 Persist offline mqtt events in the queueDir and replay (#7037) 2019-01-14 12:39:00 +05:30
kannappanr
c1798cc89a
Update GCS storage library to support GetObject API on gzipped objects (#6506)
Fixes #6280
2018-09-21 11:43:10 -07:00
Nitish Tiwari
2aa18cafc6 Update federation target to etcd/clientv3 (#6119)
With CoreDNS now supporting etcdv3 as the DNS backend, we
can update our federation target to etcdv3. Users will now be
able to use etcdv3 server as the federation backbone.

Minio will update bucket data to etcdv3 and CoreDNS can pick
that data up and serve it as bucket style DNS path.
2018-07-12 14:12:40 -07:00
Nitish Tiwari
36c39d04da Update vendorized minio-go to support start-after param (#6043)
Fixes #6032
2018-06-14 15:08:02 -07:00
Andreas Auernhammer
ca8520fb31 vendor x/sys/cpu and update x/crypto/blake2b (#5870)
This change updates the vendor'd x/crypto/blake2b package and
adds x/sys/cpu.

This change fixes an issue for Go1.11 affecting x/crypto/blake2b
since G1.11 removes the runtime-internal functions `suports_avx()` and
`supports_avx2()`.
2018-05-01 11:16:56 +05:30
Harshavardhana
4af89543cf Update minio-go dependencies to latest 5.0.0 release (#5640)
With following changes

- Add SSE and refactor encryption API (#942) <Andreas Auernhammer>
- add copyObject test changing metadata and preserving etag (#944) <Harshavardhana>
- Add SSE-C tests for multipart, copy, get range operations (#941) <Harshavardhana>
- Removing conditional check for notificationInfoCh in api-notication (#940) <Matthew Magaldi>
- Honor prefix parameter in ListBucketPolicies API (#929) <kannappanr>
- test for empty objects uploaded with SSE-C headers (#927) <kannappanr>
- Encryption headers should also be set during initMultipart (#930) <Harshavardhana>
- Add support for Content-Language metadata header (#928) <kannappanr>
- Fix check for duplicate notification configuration entries (#917) <kannappanr>
- allow OS to cleanup sockets in TIME_WAIT (#925) <Harshavardhana>
- Sign V2: Fix signature calculation in virtual host style (#921) <A. Elleuch>
- bucket policy: Support json string in Principal field (#919) <A. Elleuch>
- Fix copyobject failure for empty files (#918) <kannappanr>
- Add new constructor NewWithOptions to SDK (#915) <poornas>
- Support redirect headers to sign again with new Host header. (#829) <Harshavardhana>
- Fail in PutObject if invalid user metadata is passed <Harshavadhana>
- PutObjectOptions Header: Don't include invalid header <Isaac Hess>
- increase max retry count to 10 (#913) <poornas>
- Add new regions for Paris and China west. (#905) <Harshavardhana>
- fix s3signer to use req.Host header (#899) <Bartłomiej Nogaś>
2018-03-14 19:38:29 +05:30
Paul Stack
7d75d61621 Add Support for Manta Object Storage as a Gateway (#5025)
Manta is an Object Storage by [Joyent](https://www.joyent.com/)

This PR adds initial support for Manta. It is intended as non-production 
ready so that feedback can be obtained.
2017-12-20 13:37:56 +05:30
Harshavardhana
1f77708a30 Limit number of connections upto system maxlimit (#5109) 2017-12-20 13:30:14 +05:30
Harshavardhana
eb7c690ea9 Support in-place upgrades of new minio binary and releases. (#4961)
This PR allows 'minio update' to not only shows update banner
but also allows for in-place upgrades.

Updates are done safely by validating the downloaded
sha256 of the binary.

Fixes #4781
2017-12-15 12:33:42 -08:00
Harshavardhana
8d59f35523 Add GetInfo() support for solaris (#5174)
Fixes #5173
2017-11-13 12:54:38 -08:00
Andreas Auernhammer
ca6b4773ed add SSE-C support for HEAD, GET, PUT (#4894)
This change adds server-side-encryption support for HEAD, GET and PUT
operations. This PR only addresses single-part PUTs and GETs without
HTTP ranges.

Further this change adds the concept of reserved object metadata which is required
to make encrypted objects tamper-proof and provide API compatibility to AWS S3.
This PR adds the following reserved metadata entries:
- X-Minio-Internal-Server-Side-Encryption-Iv          ('guarantees' tamper-proof property)
- X-Minio-Internal-Server-Side-Encryption-Kdf         (makes Key-MAC computation negotiable in future)
- X-Minio-Internal-Server-Side-Encryption-Key-Mac     (provides AWS S3 API compatibility)

The prefix `X-Minio_Internal` specifies an internal metadata entry which must not
send to clients. All client requests containing a metadata key starting with `X-Minio-Internal`
must also rejected. This is implemented by a generic-handler.

This PR implements SSE-C separated from client-side-encryption (CSE). This cannot decrypt
server-side-encrypted objects on the client-side. However, clients can encrypted the same object
with CSE and SSE-C.

This PR does not address:
 - SSE-C Copy and Copy part
 - SSE-C GET with HTTP ranges
 - SSE-C multipart PUT
 - SSE-C Gateway

Each point must be addressed in a separate PR.

Added to vendor dir:
 - x/crypto/chacha20poly1305
 - x/crypto/poly1305
 - github.com/minio/sio
2017-11-07 15:18:59 -08:00
Harshavardhana
0c0d1e4150 Implement backblaze-b2 gateway support (#5002)
Fixes https://github.com/minio/minio/issues/4072
2017-10-13 16:26:16 +05:30
Andreas Auernhammer
b233345f19 remove bcrypt code from code-base (#4844) (#4845)
Bcrypt is not neccessary and not used properly. This change
replace the whole bcrypt hash computation through a constant time
compare and removes bcrypt from the code base.
2017-08-23 15:59:37 -07:00
Remco
ace4f9fd15 Implement gateway support Google Cloud Storage 2017-06-16 14:47:02 -07:00
splinter98
8293f546af Add support for MQTT server as a notification target (#4474)
This implementation is similar to AMQP notifications:

* Notifications are published on a single topic as a JSON feed
* Topic is configurable, as is the QoS. Uses the paho.mqtt.golang
  library for the mqtt connection, and supports connections over tcp
  and websockets, with optional secure tls support.
* Additionally the minio server configuration has been bumped up
  so mqtt configuration can be added.
* Configuration migration code is added with tests.

MQTT is an ISO standard M2M/IoT messaging protocol and was
originally designed for applications for limited bandwidth
networks. Today it's use is growing in the IoT space.
2017-06-14 17:27:49 -07:00
Peter Tribble
2b96d9f706 Enable build on solaris (#4115) 2017-04-23 11:10:18 -07:00
Aditya Manthramurthy
a2a8d54bb6 Add access format support for Elasticsearch notification target (#4006)
This change adds `access` format support for notifications to a
Elasticsearch server, and it refactors `namespace` format support.

In the case of `access` format, for each event in Minio, a JSON
document is inserted into Elasticsearch with its timestamp set to the
event's timestamp, and with the ID generated automatically by
elasticsearch. No events are modified or deleted in this mode.

In the case of `namespace` format, for each event in Minio, a JSON
document is keyed together by the bucket and object name is updated in
Elasticsearch. In the case of an object being created or over-written
in Minio, a new document or an existing document is inserted into the
Elasticsearch index. If an object is deleted in Minio, the
corresponding document is deleted from the Elasticsearch index.

Additionally, this change upgrades Elasticsearch support to the 5.x
series. This is a breaking change, and users of previous elasticsearch
versions should upgrade.

Also updates documentation on Elasticsearch notification target usage
and has a link to an elasticsearch upgrade guide.

This is the last patch that finally resolves #3928.
2017-03-31 14:11:27 -07:00
Andreas Auernhammer
f38222c0cc update the blake2b implementation (#3724)
Fixes a performance bug caused by SSE-AVX register savings on amd64.
2017-02-09 15:01:00 -08:00
Harshavardhana
d41dcb784b Move to blake2b-simd due to perf problems in golang.org/x/crypto
Ref https://github.com/golang/go/issues/18563
2017-01-24 18:07:34 -08:00
Andreas Auernhammer
6ee27daac1 fix blake2b tests on non-amd64 machines (#3496)
Fix the TestHashes Test for non-amd64 machines
2016-12-25 17:03:08 -08:00
Andreas Auernhammer
1ac36a95aa replace blake2b implementation (#3481)
* replace blake2b implementation
replace the blake2b-simd with the golang/x/crypto implementation

```
name        old time/op    new time/op     delta
Size64-8       715ns ±13%      614ns ± 3%    ~     (p=0.084 n=6+6)
Size128-8      612ns ± 5%      634ns ± 8%    ~     (p=0.084 n=6+6)
Size1K-8      2.18µs ± 5%     2.09µs ± 7%    ~     (p=0.084 n=6+6)
Size8K-8      13.1µs ± 2%     13.4µs ± 3%    ~     (p=0.084 n=6+6)
Size32K-8     48.5µs ± 1%     49.5µs ± 3%    ~     (p=0.775 n=6+6)
Size128K-8     199µs ± 0%      198µs ± 3%    ~     (p=0.468 n=6+6)

name        old speed      new speed       delta
Size64-8    92.6MB/s ±11%  104.2MB/s ± 3%    ~     (p=0.139 n=6+6)
Size128-8    208MB/s ± 6%    202MB/s ± 8%    ~     (p=0.102 n=6+6)
Size1K-8     466MB/s ± 7%    492MB/s ± 7%    ~     (p=0.139 n=6+6)
Size8K-8     621MB/s ± 2%    610MB/s ± 3%    ~     (p=0.102 n=6+6)
Size32K-8    672MB/s ± 2%    669MB/s ± 1%    ~     (p=0.818 n=6+6)
Size128K-8   657MB/s ± 1%    672MB/s ± 0%  +2.28%  (p=0.002 n=6+6)

name        old time/op   new time/op   delta
Size64-4      334ns ± 1%    243ns ± 0%  -27.14%  (p=0.029 n=4+4)
Size128-4     296ns ± 1%    242ns ± 0%  -18.21%  (p=0.029 n=4+4)
Size1K-4     1.44µs ± 0%   1.28µs ± 0%  -10.83%  (p=0.029 n=4+4)
Size8K-4     10.0µs ± 0%    9.4µs ± 0%   -6.23%  (p=0.029 n=4+4)
Size32K-4    39.8µs ± 1%   37.3µs ± 0%   -6.31%  (p=0.029 n=4+4)
Size128K-4    162µs ± 3%    149µs ± 0%   -7.72%  (p=0.029 n=4+4)

name        old speed     new speed     delta
Size64-4    192MB/s ± 1%  263MB/s ± 0%  +37.24%  (p=0.029 n=4+4)
Size128-4   431MB/s ± 0%  526MB/s ± 0%  +22.04%  (p=0.029 n=4+4)
Size1K-4    713MB/s ± 0%  800MB/s ± 0%  +12.17%  (p=0.029 n=4+4)
Size8K-4    815MB/s ± 0%  869MB/s ± 0%   +6.64%  (p=0.029 n=4+4)
Size32K-4   823MB/s ± 1%  878MB/s ± 0%   +6.72%  (p=0.029 n=4+4)
Size128K-4  810MB/s ± 3%  877MB/s ± 0%   +8.23%  (p=0.029 n=4+4)
```
See: https://go-review.googlesource.com/#/c/34319/
2016-12-21 14:20:01 -08:00
Harshavardhana
35d438e0ff vendorize: update all vendorized packages. (#2206)
Bring in new changes from upstream for all the packages.

Important ones include
   - gorilla/mux
   - logrus
   - jwt
2016-07-14 14:59:20 -07:00
Harshavardhana
0a9496462a jwt: Add JWT support for minio server.
Please read JWT.md before using this feature.
2016-01-22 17:38:05 -08:00