520552ffa9
S3 select: flush when reaching limit ( #8279 )
...
Add missing flush when reaching select limit.
2019-09-20 11:00:17 -07:00
b823d6d7bd
remove the unused code for decrypting `io.Writer` ( #8277 )
...
This commit removes unused code for decrypting
`io.Writer` since the actual implementation only
decrypts `io.Reader`
2019-09-20 14:51:07 +05:30
a9d724120f
remove TLS 1.3 opt-in code ( #8275 )
...
This commit removes the TLS 1.3 opt-in code.
Since TLS 1.3 is opt-out for >= Go 1.13 this
code is not needed anymore.
2019-09-20 01:51:44 +05:30
dc0dce9beb
Check for connection errors during event transit in elasticsearch and webhook ( #8273 )
2019-09-19 08:23:43 -07:00
0e1408844b
fix get container id bug ( #8196 )
2019-09-19 10:19:51 +05:30
e34369c860
prepare SSE-S3 metadata parsing for K/V data key store ( #8259 )
...
This commit allows the MinIO server to parse the metadata if:
- either the `X-Minio-Internal-Server-Side-Encryption-S3-Key-Id`
and the `X-Minio-Internal-Server-Side-Encryption-S3-Kms-Sealed-Key`
entries are present.
- or *both* headers are not present.
This is in service to support a K/V data key store.
2019-09-19 04:08:09 +05:30
8dc897b5f5
Update yaml files to latest version RELEASE.2019-09-18T21-55-05Z
2019-09-18 22:02:41 +00:00
456ce4cc92
Add rootCAs support to Kafka & MQTT ( #8236 )
...
Fixes #8211
2019-09-18 23:43:04 +05:30
696f4ceee2
Add notes on configuring LDAP STS with Microsoft Active Directory ( #8260 )
2019-09-17 17:21:37 -07:00
dac1cf5a9a
S3 Select: Parsing tweaks ( #8261 )
...
* Don't output empty lines.
* Trim whitespace from byte to int/float/bool conversions.
2019-09-17 17:21:23 -07:00
cb01516a26
In HDFS gateway fix non-empty folder behavior ( #8254 )
...
To be compatible with our FS and Erasure coded
mode deployments, make sure that we do not send
200 OK for folders which have files inside.
Fixes #8143
2019-09-18 01:59:59 +05:30
dfa1b417a8
Update comments in ldap example client code ( #8257 )
2019-09-18 01:59:31 +05:30
31bee6b6ed
Remove size query parameter from PerfInfo handler ( #8258 )
2019-09-18 01:59:12 +05:30
04b92124c5
fs/xl: Log warning if cache config specified ( #8251 )
...
in non-gateway mode.
2019-09-16 19:55:52 -07:00
5392eee250
Avoid recursion and use a simple loop to merge entries ( #8239 )
...
This avoids stack overflows when there are
lot of entries to be skipped, this PR also
optimizes the code to reuse the buffers.
2019-09-17 06:08:37 +05:30
fa32c71a56
Build staticcheck as pre-built binaries are not available for s390x ( #8220 )
2019-09-17 05:59:15 +05:30
c9b8bd8de2
S3 Select: optimize output ( #8238 )
...
Queue output items and reuse them.
Remove the unneeded type system in sql and just use the Go type system.
In best case this is more than an order of magnitude speedup:
```
BenchmarkSelectAll_1M-12 1 1841049400 ns/op 274299728 B/op 4198522 allocs/op
BenchmarkSelectAll_1M-12 14 84833400 ns/op 169228346 B/op 3146541 allocs/op
```
2019-09-17 05:56:27 +05:30
017456df63
Wait clearing the close channel ( #8250 )
...
Close channel should not be nilled before goroutines have exited.
Fixes potential hang on closing.
2019-09-16 16:18:01 -07:00
14b137aa66
posix/readDir should populate name for DT_UKNOWN ( #8240 )
...
In commit a8296445ad
2019-09-17 03:04:01 +05:30
3064da7b08
return error during part listing when no quorum ( #8241 )
...
This commit fixes a subtle bug that (probably)
caused an issue affecting encrypted multipart objects.
When a cluster has no quorum this bug causes `ListObjectParts`
to return nil as error instead of a quorum error.
Thanks to @harshavardhana for detecting this.
2019-09-17 02:57:34 +05:30
76df027264
Allow caching only in gateway mode. ( #8232 )
...
This PR changes cache on PUT behavior to background fill the cache
after PutObject completes. This will avoid concurrency issues as in #8219 .
Added cleanup of partially filled cache to prevent cache corruption
- Fixes #8208
2019-09-17 02:54:04 +05:30
208efb843b
Update Kafka notification example doc ( #8212 )
2019-09-16 19:24:55 +05:30
9ac12cf898
Remove unusued Set/GetConfigKeys API ( #8235 )
2019-09-13 16:34:34 -07:00
ddea0bdf11
Concurrent CSV parsing and reduce S3 select allocations ( #8200 )
...
```
CSV parsing, BEFORE:
BenchmarkReaderBasic-12 2842 407533 ns/op 397860 B/op 957 allocs/op
BenchmarkReaderReplace-12 2718 429914 ns/op 397844 B/op 957 allocs/op
BenchmarkReaderReplaceTwo-12 2718 435556 ns/op 397855 B/op 957 allocs/op
BenchmarkAggregateCount_100K-12 171 6798974 ns/op 16667102 B/op 308077 allocs/op
BenchmarkAggregateCount_1M-12 19 65657411 ns/op 168057743 B/op 3146610 allocs/op
BenchmarkSelectAll_10M-12 1 20882119900 ns/op 2758799896 B/op 41978762 allocs/op
CSV parsing, AFTER:
BenchmarkReaderBasic-12 3721 312549 ns/op 101920 B/op 338 allocs/op
BenchmarkReaderReplace-12 3776 318810 ns/op 101993 B/op 340 allocs/op
BenchmarkReaderReplaceTwo-12 3610 330967 ns/op 102012 B/op 341 allocs/op
BenchmarkAggregateCount_100K-12 295 4149588 ns/op 3553623 B/op 103261 allocs/op
BenchmarkAggregateCount_1M-12 30 37746503 ns/op 33827931 B/op 1049435 allocs/op
BenchmarkSelectAll_10M-12 1 17608495800 ns/op 1416504040 B/op 21007082 allocs/op
~ benchcmp old.txt new.txt
benchmark old ns/op new ns/op delta
BenchmarkReaderBasic-12 407533 312549 -23.31%
BenchmarkReaderReplace-12 429914 318810 -25.84%
BenchmarkReaderReplaceTwo-12 435556 330967 -24.01%
BenchmarkAggregateCount_100K-12 6798974 4149588 -38.97%
BenchmarkAggregateCount_1M-12 65657411 37746503 -42.51%
BenchmarkSelectAll_10M-12 20882119900 17608495800 -15.68%
benchmark old allocs new allocs delta
BenchmarkReaderBasic-12 957 338 -64.68%
BenchmarkReaderReplace-12 957 340 -64.47%
BenchmarkReaderReplaceTwo-12 957 341 -64.37%
BenchmarkAggregateCount_100K-12 308077 103261 -66.48%
BenchmarkAggregateCount_1M-12 3146610 1049435 -66.65%
BenchmarkSelectAll_10M-12 41978762 21007082 -49.96%
benchmark old bytes new bytes delta
BenchmarkReaderBasic-12 397860 101920 -74.38%
BenchmarkReaderReplace-12 397844 101993 -74.36%
BenchmarkReaderReplaceTwo-12 397855 102012 -74.36%
BenchmarkAggregateCount_100K-12 16667102 3553623 -78.68%
BenchmarkAggregateCount_1M-12 168057743 33827931 -79.87%
BenchmarkSelectAll_10M-12 2758799896 1416504040 -48.66%
```
```
BenchmarkReaderHuge/97K-12 2200 540840 ns/op 184.32 MB/s 1604450 B/op 687 allocs/op
BenchmarkReaderHuge/194K-12 1522 752257 ns/op 265.04 MB/s 2143135 B/op 1335 allocs/op
BenchmarkReaderHuge/389K-12 1190 947858 ns/op 420.69 MB/s 3221831 B/op 2630 allocs/op
BenchmarkReaderHuge/778K-12 806 1472486 ns/op 541.61 MB/s 5201856 B/op 5187 allocs/op
BenchmarkReaderHuge/1557K-12 426 2575269 ns/op 619.36 MB/s 9101330 B/op 10233 allocs/op
BenchmarkReaderHuge/3115K-12 286 4034656 ns/op 790.66 MB/s 12397968 B/op 16099 allocs/op
BenchmarkReaderHuge/6230K-12 172 6830563 ns/op 934.05 MB/s 16008416 B/op 26844 allocs/op
BenchmarkReaderHuge/12461K-12 100 11409467 ns/op 1118.39 MB/s 22655163 B/op 48107 allocs/op
BenchmarkReaderHuge/24922K-12 66 19780395 ns/op 1290.19 MB/s 35158559 B/op 90216 allocs/op
BenchmarkReaderHuge/49844K-12 34 37282559 ns/op 1369.03 MB/s 60528624 B/op 174497 allocs/op
```
2019-09-13 14:18:35 -07:00
e7f491a14b
Use optimized sha256-simd whenever possible ( #8227 )
...
Avoid using `crypto/sha256` and use always
`github.com/minio/sha256-simd`
2019-09-14 00:39:39 +05:30
8700945cdf
Handle connection failures on webhook/url pings ( #8204 )
...
Properly handle connection failures while replaying events
Fixes #8194
2019-09-12 16:44:51 -07:00
ff6aabd9c0
Honor standard HTTP headers for sourceIP ( #8233 )
...
Behind load balancers we should be tracing sourceIP
preserved by load balancers.
2019-09-13 03:59:59 +05:30
6ba323b009
Add ability to test drive speeds on a MinIO setup ( #7664 )
...
- Extends existing Admin API to measure disk performance
2019-09-13 03:22:30 +05:30
e7b3f39064
xl: Fix verifying non streaming highway algo with a dist setup ( #8230 )
...
VerifyFile in the distributed setup does not work with
the non streaming highway hash. The reason is that the
internode mux router did not expect `storageRESTBitrotHash`
parameter.
2019-09-12 13:08:02 -07:00
9fa727d154
Provide a friendlier error when an update fails ( #8228 )
...
Add upgrading documentation as well
2019-09-13 01:33:42 +05:30
73e4e99942
Hosts should be skipped, when calculating local info ( #8191 )
...
endpoint.IsLocal will not have .Host entries so
using them to skip double entries will never work.
change the code such that we look for endpoint.Host
outside of endpoint.IsLocal logic to skip double
hosts appropriately.
Move these functions to their appropriate file.
2019-09-12 23:36:12 +05:30
a87fc7d09b
Use the B2 'list' endpoint to determine file ID ( #8169 )
...
- More effective deletion and checking for existence.
- Rever Dockerfile.
- Add a 'GOPROXY' to the Dockerfile to workaround Apache issues.
2019-09-12 22:48:47 +05:30
475df52a19
Fix etcd watch regression in IAM subsystem ( #8224 )
...
Fixes #8223
2019-09-12 07:24:25 +05:30
5512baab21
Fix go mod tidy errors and fix platform independent downloads ( #8222 )
...
Refer https://github.com/gin-gonic/gin/issues/1673
2019-09-12 04:57:59 +05:30
77963078a2
Remove maintainers.md use only contributing.md ( #8215 )
2019-09-12 04:57:11 +05:30
3f258062d8
bitrot: Verify file size inside storage interface ( #7932 )
2019-09-12 02:19:53 +05:30
3d65dc8d94
Update yaml files to latest version RELEASE.2019-09-11T19-53-16Z
2019-09-11 19:59:05 +00:00
53e4887e02
Simplify and cleanup metadata r/w functions ( #8146 )
2019-09-11 22:52:12 +05:30
a7be313230
Start using new errors package ( #8207 )
2019-09-11 22:51:43 +05:30
e12f52e2c6
Enhancements to daily-sweeper routine to reduce CPU load ( #8209 )
...
- ListObjectsHeal should list only objects
which need healing, not the entire namespace.
- DeleteObjects() to be used to delete 1000s of
objects in bulk instead of serially.
2019-09-11 00:38:44 +05:30
432cb38dbd
Add reference to MinIO operator in Kubernetes docs ( #8098 )
2019-09-10 11:37:31 -07:00
18fedc67d5
friendly prompt for s3select MalformedXML error ( #8171 )
...
partly fix #7911
2019-09-09 21:33:27 -07:00
a0456ce940
LDAP STS API ( #8091 )
...
Add LDAP based users-groups system
This change adds support to integrate an LDAP server for user
authentication. This works via a custom STS API for LDAP. Each user
accessing the MinIO who can be authenticated via LDAP receives
temporary credentials to access the MinIO server.
LDAP is enabled only over TLS.
User groups are also supported via LDAP. The administrator may
configure an LDAP search query to find the group attribute of a user -
this may correspond to any attribute in the LDAP tree (that the user
has access to view). One or more groups may be returned by such a
query.
A group is mapped to an IAM policy in the usual way, and the server
enforces a policy corresponding to all the groups and the user's own
mapped policy.
When LDAP is configured, the internal MinIO users system is disabled.
2019-09-10 04:42:29 +05:30
94e5cb7576
Migrate to go1.13 to avail all new features ( #8203 )
...
Read more https://blog.golang.org/go1.13
2019-09-08 16:44:15 -07:00
33aec08e8c
Clean up go. mod ( #8197 )
...
Golang proposes to clean up these and is fairly insisting about it ;)
2019-09-06 14:57:55 -07:00
5bde31d021
Update yaml files to latest version RELEASE.2019-09-05T23-24-38Z
2019-09-05 23:30:40 +00:00
b52a3e523c
Avoid using fastjson parser pool, move back to jsoniter ( #8190 )
...
It looks like from implementation point of view fastjson
parser pool doesn't behave the same way as expected
when dealing many `xl.json` from multiple disks.
The fastjson parser pool usage ends up returning incorrect
xl.json entries for checksums, with references pointing
to older entries. This led to the subtle bug where checksum
info is duplicated from a previous xl.json read of a different
file from different disk.
2019-09-06 04:21:27 +05:30
428836d4e1
Ignore "connection reset" errors while initializing the targets with queue store enabled ( #8185 )
...
Fixes #8178
2019-09-05 13:55:48 -07:00
259a5d825b
cache - fix corruption when client prematurely terminates request ( #8155 )
2019-09-05 23:33:32 +05:30
29f64355ce
Allow caching on single PutObject ( #8100 )
2019-09-05 19:50:16 +05:30
Klaus Post
Andreas Auernhammer
Praveen raj Mani
Yao Zongyou
Minio Trusted
Aditya Manthramurthy
Harshavardhana
Krishnan Parthasarathi
poornas
Vivian Kong
k-tipp
Anis Elleuch
Alex Pardoe
Nitish Tiwari