Andreas Auernhammer
b7ee0bbbc9
madmin: Migrate to provable secure channel construction ( #8395 )
...
This commit replaces the currently used channel construction
for en/decrypting config data with a provable secure scheme.
2019-10-15 02:36:04 +05:30
Minio Trusted
ae752ed1fa
Update yaml files to latest version RELEASE.2019-10-11T00-38-09Z
2019-10-11 00:45:13 +00:00
Harshavardhana
6a4ef2e48e
Initialize configs correctly, move notification config ( #8367 )
...
This PR also removes deprecated tests, adds checks
to avoid races reproduced on CI/CD.
2019-10-09 11:41:15 +05:30
Ashish Kumar Sinha
74008446fe
CPU hardware info ( #8187 )
2019-10-03 20:18:38 +05:30
Harshavardhana
90bfa6260a
Fix LDAP TLS support to use custom CAs ( #8352 )
2019-10-03 01:44:57 +05:30
Harshavardhana
4ec9b349d0
azure: Fix upload corruption with PutObject() on certain sizes ( #8330 )
...
On objects bigger than 100MiB can have a corrupted object
stored due to partial blockListing attempted right after
each blocks uploaded. Simplify this code to ensure that
all the blocks successfully uploaded are committed right
away.
This PR also updates the azure-sdk-go to latest release.
2019-09-30 18:42:18 -07:00
Andreas Auernhammer
8771e83545
Update klauspost/reedsolomon dependency to 1.9.3 ( #8328 )
...
This commit updates the reedsolomon dependency
since it contains an fix for an unexpected property
of the `Split` function.
See: klauspost/reedsolomon#109
2019-10-01 02:31:00 +05:30
Klaus Post
ff726969aa
Switch to Snappy -> S2 compression ( #8189 )
2019-09-25 23:08:24 -07:00
Harshavardhana
be70ef59e7
Update minio-go to latest release ( #8297 )
...
- Change LDAP XML names to follow server changes
- Fix listObjects hang when listing special characters
- set encoding type to url in listobjects API
2019-09-24 01:01:54 -07:00
Klaus Post
ddea0bdf11
Concurrent CSV parsing and reduce S3 select allocations ( #8200 )
...
```
CSV parsing, BEFORE:
BenchmarkReaderBasic-12 2842 407533 ns/op 397860 B/op 957 allocs/op
BenchmarkReaderReplace-12 2718 429914 ns/op 397844 B/op 957 allocs/op
BenchmarkReaderReplaceTwo-12 2718 435556 ns/op 397855 B/op 957 allocs/op
BenchmarkAggregateCount_100K-12 171 6798974 ns/op 16667102 B/op 308077 allocs/op
BenchmarkAggregateCount_1M-12 19 65657411 ns/op 168057743 B/op 3146610 allocs/op
BenchmarkSelectAll_10M-12 1 20882119900 ns/op 2758799896 B/op 41978762 allocs/op
CSV parsing, AFTER:
BenchmarkReaderBasic-12 3721 312549 ns/op 101920 B/op 338 allocs/op
BenchmarkReaderReplace-12 3776 318810 ns/op 101993 B/op 340 allocs/op
BenchmarkReaderReplaceTwo-12 3610 330967 ns/op 102012 B/op 341 allocs/op
BenchmarkAggregateCount_100K-12 295 4149588 ns/op 3553623 B/op 103261 allocs/op
BenchmarkAggregateCount_1M-12 30 37746503 ns/op 33827931 B/op 1049435 allocs/op
BenchmarkSelectAll_10M-12 1 17608495800 ns/op 1416504040 B/op 21007082 allocs/op
~ benchcmp old.txt new.txt
benchmark old ns/op new ns/op delta
BenchmarkReaderBasic-12 407533 312549 -23.31%
BenchmarkReaderReplace-12 429914 318810 -25.84%
BenchmarkReaderReplaceTwo-12 435556 330967 -24.01%
BenchmarkAggregateCount_100K-12 6798974 4149588 -38.97%
BenchmarkAggregateCount_1M-12 65657411 37746503 -42.51%
BenchmarkSelectAll_10M-12 20882119900 17608495800 -15.68%
benchmark old allocs new allocs delta
BenchmarkReaderBasic-12 957 338 -64.68%
BenchmarkReaderReplace-12 957 340 -64.47%
BenchmarkReaderReplaceTwo-12 957 341 -64.37%
BenchmarkAggregateCount_100K-12 308077 103261 -66.48%
BenchmarkAggregateCount_1M-12 3146610 1049435 -66.65%
BenchmarkSelectAll_10M-12 41978762 21007082 -49.96%
benchmark old bytes new bytes delta
BenchmarkReaderBasic-12 397860 101920 -74.38%
BenchmarkReaderReplace-12 397844 101993 -74.36%
BenchmarkReaderReplaceTwo-12 397855 102012 -74.36%
BenchmarkAggregateCount_100K-12 16667102 3553623 -78.68%
BenchmarkAggregateCount_1M-12 168057743 33827931 -79.87%
BenchmarkSelectAll_10M-12 2758799896 1416504040 -48.66%
```
```
BenchmarkReaderHuge/97K-12 2200 540840 ns/op 184.32 MB/s 1604450 B/op 687 allocs/op
BenchmarkReaderHuge/194K-12 1522 752257 ns/op 265.04 MB/s 2143135 B/op 1335 allocs/op
BenchmarkReaderHuge/389K-12 1190 947858 ns/op 420.69 MB/s 3221831 B/op 2630 allocs/op
BenchmarkReaderHuge/778K-12 806 1472486 ns/op 541.61 MB/s 5201856 B/op 5187 allocs/op
BenchmarkReaderHuge/1557K-12 426 2575269 ns/op 619.36 MB/s 9101330 B/op 10233 allocs/op
BenchmarkReaderHuge/3115K-12 286 4034656 ns/op 790.66 MB/s 12397968 B/op 16099 allocs/op
BenchmarkReaderHuge/6230K-12 172 6830563 ns/op 934.05 MB/s 16008416 B/op 26844 allocs/op
BenchmarkReaderHuge/12461K-12 100 11409467 ns/op 1118.39 MB/s 22655163 B/op 48107 allocs/op
BenchmarkReaderHuge/24922K-12 66 19780395 ns/op 1290.19 MB/s 35158559 B/op 90216 allocs/op
BenchmarkReaderHuge/49844K-12 34 37282559 ns/op 1369.03 MB/s 60528624 B/op 174497 allocs/op
```
2019-09-13 14:18:35 -07:00
Krishnan Parthasarathi
6ba323b009
Add ability to test drive speeds on a MinIO setup ( #7664 )
...
- Extends existing Admin API to measure disk performance
2019-09-13 03:22:30 +05:30
Harshavardhana
5512baab21
Fix go mod tidy errors and fix platform independent downloads ( #8222 )
...
Refer https://github.com/gin-gonic/gin/issues/1673
2019-09-12 04:57:59 +05:30
Minio Trusted
3d65dc8d94
Update yaml files to latest version RELEASE.2019-09-11T19-53-16Z
2019-09-11 19:59:05 +00:00
Aditya Manthramurthy
a0456ce940
LDAP STS API ( #8091 )
...
Add LDAP based users-groups system
This change adds support to integrate an LDAP server for user
authentication. This works via a custom STS API for LDAP. Each user
accessing the MinIO who can be authenticated via LDAP receives
temporary credentials to access the MinIO server.
LDAP is enabled only over TLS.
User groups are also supported via LDAP. The administrator may
configure an LDAP search query to find the group attribute of a user -
this may correspond to any attribute in the LDAP tree (that the user
has access to view). One or more groups may be returned by such a
query.
A group is mapped to an IAM policy in the usual way, and the server
enforces a policy corresponding to all the groups and the user's own
mapped policy.
When LDAP is configured, the internal MinIO users system is disabled.
2019-09-10 04:42:29 +05:30
Harshavardhana
94e5cb7576
Migrate to go1.13 to avail all new features ( #8203 )
...
Read more https://blog.golang.org/go1.13
2019-09-08 16:44:15 -07:00
Klaus Post
33aec08e8c
Clean up go. mod ( #8197 )
...
Golang proposes to clean up these and is fairly insisting about it ;)
2019-09-06 14:57:55 -07:00
Minio Trusted
5bde31d021
Update yaml files to latest version RELEASE.2019-09-05T23-24-38Z
2019-09-05 23:30:40 +00:00
Harshavardhana
b52a3e523c
Avoid using fastjson parser pool, move back to jsoniter ( #8190 )
...
It looks like from implementation point of view fastjson
parser pool doesn't behave the same way as expected
when dealing many `xl.json` from multiple disks.
The fastjson parser pool usage ends up returning incorrect
xl.json entries for checksums, with references pointing
to older entries. This led to the subtle bug where checksum
info is duplicated from a previous xl.json read of a different
file from different disk.
2019-09-06 04:21:27 +05:30
Minio Trusted
daa57e8e5a
Update yaml files to latest version RELEASE.2019-08-29T00-25-01Z
2019-08-29 00:30:50 +00:00
Minio Trusted
de736f2864
Update yaml files to latest version RELEASE.2019-08-21T19-40-07Z
2019-08-21 19:46:01 +00:00
Harshavardhana
9ca7470ccc
Avoid using jsoniter, move to fastjson ( #8063 )
...
This is to avoid using unsafe.Pointer type
code dependency for MinIO, this causes
crashes on ARM64 platforms
Refer #8005 collection of runtime crashes due
to unsafe.Pointer usage incorrectly. We have
seen issues like this before when using
jsoniter library in the past.
This PR hopes to fix this using fastjson
2019-08-19 08:35:52 -10:00
Minio Trusted
26640033b0
Update yaml files to latest version RELEASE.2019-08-14T20-37-41Z
2019-08-14 20:42:16 +00:00
Minio Trusted
55b385beee
Update yaml files to latest version RELEASE.2019-08-07T01-59-21Z
2019-08-07 02:05:41 +00:00
kannappanr
930943f058
Fix IAM users migration regression in etcd ( #8029 )
...
PR #8008 did not migrate user data stored in etcd.
This PR fixes that.
2019-08-06 17:06:31 -07:00
Minio Trusted
dfa8835720
Update yaml files to latest version RELEASE.2019-08-01T22-18-54Z
2019-08-01 22:27:41 +00:00
Minio Trusted
c301f5882d
Update yaml files to latest version RELEASE.2019-07-31T18-57-56Z
2019-07-31 19:06:20 +00:00
Harshavardhana
6f2b4675fa
Add krb5 support for HDFS gateway ( #7933 )
2019-07-24 18:05:48 -07:00
Minio Trusted
5b71c21330
Update yaml files to latest version RELEASE.2019-07-24T02-02-23Z
2019-07-24 02:09:19 +00:00
Krishnan Parthasarathi
559a59220e
Add initial support for bucket lifecycle ( #7563 )
...
This PR is based off @sinhaashish's PR for object lifecycle
management, which includes support only for,
- Expiration of object
- Filter using object prefix (_not_ object tags)
N B the code for actual expiration of objects will be included in a
subsequent PR.
2019-07-19 21:20:33 +01:00
Minio Trusted
b058e32348
Update yaml files to latest version RELEASE.2019-07-17T22-54-12Z
2019-07-17 22:59:33 +00:00
Harshavardhana
16a45e5aff
Fix dynamic help vars for sub-commands ( #7925 )
...
The fix in #7646 introduced a regression which
was left unnoticed, the fix didn't work for
sub-commands unfortunately. This fixes it
by moving v1.21.0 version of the minio/cli
package.
Fixes #7924
2019-07-12 23:32:27 -07:00
Praveen raj Mani
bba562235b
Enable persistent event store in elasticsearch ( #7564 )
2019-07-12 08:23:20 +05:30
Minio Trusted
2d96745156
Update yaml files to latest version RELEASE.2019-07-10T00-34-56Z
2019-07-10 00:39:45 +00:00
mzukowski-reef
9d49688c87
Switch to kurin/blazer from minio/blazer fork for b2 gateway ( #7879 )
2019-07-09 08:14:02 -07:00
Minio Trusted
22bc15d89b
Update yaml files to latest version RELEASE.2019-07-05T21-20-21Z
2019-07-05 21:24:43 +00:00
Krishna Srinivas
a2e904b966
Support any string as delimiter for listing ( #7882 )
2019-07-05 14:06:12 -07:00
Krishna Srinivas
338e9a9be9
Put object client disconnect ( #7824 )
...
Fail putObject and postpolicy in case client prematurely disconnects
Use request's context to cancel lock requests on client disconnects
2019-06-28 22:09:17 -07:00
Minio Trusted
5db60a6c59
Update yaml files to latest version RELEASE.2019-06-27T21-13-50Z
2019-06-27 21:18:52 +00:00
Minio Trusted
e43d3a075c
Update yaml files to latest version RELEASE.2019-06-19T18-24-42Z
2019-06-19 18:31:37 +00:00
Minio Trusted
59f7266081
Update yaml files to latest version RELEASE.2019-06-15T23-07-18Z
2019-06-15 23:17:28 +00:00
Harshavardhana
99bf4d0c42
[sec] Match ${aws:username} exactly instead of prefix match ( #7791 )
...
This PR fixes a security issue where an IAM user based
on his policy is granted more privileges than restricted
by the users IAM policy.
This is due to an issue of prefix based Matcher() function
which was incorrectly matching prefix based on resource
prefixes instead of exact match.
2019-06-15 11:27:17 -07:00
Minio Trusted
da2887f914
Update yaml files to latest version RELEASE.2019-06-13T01-41-13Z
2019-06-13 01:46:00 +00:00
Harshavardhana
c22439c82e
Update minio-go v6.0.29 ( #7778 )
...
Bring improved retry logic
2019-06-12 18:09:21 -07:00
Harshavardhana
b4ab778cb2
Fix user IAM policy regression, reload policy appropriately ( #7770 )
...
Introduce in commit 7e4c9a9e1e
Fixes #7769
2019-06-12 14:49:45 -07:00
Minio Trusted
d3a2efbf91
Update yaml files to latest version RELEASE.2019-06-11T00-44-33Z
2019-06-11 00:50:31 +00:00
Harshavardhana
cb1566c6e6
S3 Gateway: Handle restricted access credentials ( #7757 )
2019-06-07 15:49:13 -07:00
Minio Trusted
df19546cc8
Update yaml files to latest version RELEASE.2019-06-04T01-15-58Z
2019-06-04 01:22:32 +00:00
Harshavardhana
ff7799452b
Fix build failure due to nats project migration
2019-06-03 17:49:49 -07:00
Minio Trusted
48ffb7731a
Update yaml files to latest version RELEASE.2019-06-01T03-46-14Z
2019-06-01 03:52:47 +00:00
Harshavardhana
2c0b3cadfc
Update go mod with sem versions of our libraries ( #7687 )
2019-05-29 16:35:12 -07:00