MyFavMirrors/minio
1
0
Fork 0
mirror of https://github.com/minio/minio.git synced 2024-12-25 22:55:54 -05:00
Code Issues Packages Projects Releases Wiki Activity
6,104 Commits 4 Branches 506 Tags 189 MiB
7bf093c06a
Commit Graph

1324 Commits

Author SHA1 Message Date
Ashish Kumar Sinha
18cb15559d Add network hardware info (#8358) 
peerRESTVersion changed to v6
 2019-10-17 04:09:49 -07:00
Harshavardhana
5afb1b6747
Add support for {jwt:sub} substitutions for policies (#8393) 
Fixes #8345
 2019-10-16 08:59:59 -07:00
Harshavardhana
f2cc97a44c
Improve MQTT token registration retry (#8397) 2019-10-15 19:39:14 -07:00
Harshavardhana
d48fd6fde9
Remove unusued params and functions (#8399) 2019-10-15 18:35:41 -07:00
Andreas Auernhammer
b7ee0bbbc9 madmin: Migrate to provable secure channel construction (#8395) 
This commit replaces the currently used channel construction
for en/decrypting config data with a provable secure scheme.
 2019-10-15 02:36:04 +05:30
Harshavardhana
c33bae057f Fix build issues with event target logger 2019-10-11 19:07:09 -07:00
poornas
d7060c4c32 Allow logging targets to be configured to receive minio (#8347) 
specific errors, `application` errors or `all` by default.

console logging on server by default lists all logs -
enhance admin console API to accept `type` as query parameter to
subscribe to application/minio logs.
 2019-10-11 18:50:54 -07:00
Harshavardhana
bd10640846 Fix logger and audit http endpoint config lookup 2019-10-11 18:33:25 -07:00
Harshavardhana
175b07d6e4
Fix queueStore stops working with concurrent PUT/DELETE requests (#8381) 
- This PR fixes situation to avoid underflow, this is possible
  because of disconnected operations in replay/sendEvents
- Hold right locks if Del() operation is performed in Get()
- Remove panic in the code and use loggerOnce
- Remove Timer and instead use Ticker instead for proper ticks
 2019-10-11 17:46:03 -07:00
Harshavardhana
f16df2a4e7 Avoid crash in PopulatePublicKey() by re-initializing the args (#8388) 
This is to avoid nil pointer dereference when method by pointer
reference and method by value reference are implemented.

Fixes #8387
 2019-10-12 03:37:08 +05:30
Harshavardhana
290ad0996f Move etcd, logger, crypto into their own packages (#8366) 
- Deprecates _MINIO_PROFILER, `mc admin profile` does the job
- Move ENVs to common location in cmd/config/
 2019-10-08 11:17:56 +05:30
Harshavardhana
3b8adf7528 Move storageclass config handling into cmd/config/storageclass (#8360) 
Continuation of the changes done in PR #8351 to refactor,
add tests and move global handling into a more idiomatic
style for Go as packages.
 2019-10-07 11:20:24 +05:30
Klaus Post
002ac82631 S3 Select: Add parser support for lists. (#8329) 2019-10-06 07:52:45 -07:00
Harshavardhana
589e32a4ed Refactor config and split them in packages (#8351) 
This change is related to larger config migration PR
change, this is a first stage change to move our
configs to `cmd/config/` - divided into its subsystems
 2019-10-04 23:05:33 +05:30
Ashish Kumar Sinha
74008446fe CPU hardware info (#8187) 2019-10-03 20:18:38 +05:30
Harshavardhana
fb1374f2f7 Rename iam/validator -> iam/openid and add tests (#8340) 
Refactor as part of config migration
 2019-10-02 03:37:20 +05:30
Klaus Post
c1a17c2561 S3 Select: Aggregate AVG/SUM as float (#8326) 
Force sum/average to be calculated as a float.

As noted in #8221

> run SELECT AVG(CAST (Score as int)) FROM S3Object on

```
Name,Score
alice,80
bob,81
```

> AWS S3 gives 80.5 and MinIO gives 80.

This also makes overflows much more unlikely.
 2019-09-27 16:12:03 -07:00
Klaus Post
1c5b05c130 S3 select: Fix output conversion on select * (#8303) 
Fixes #8268
 2019-09-27 12:33:14 -07:00
Bala FA
2a2ff96ee1 change ReadPerf into ReadThroughput in NetPerfInfo. (#8316) 
Previously `ReadPerf` was in time.Duration is changed to `ReadThroughput` in uint64.
 2019-09-27 00:01:18 +05:30
Harshavardhana
fd53057654 Add InfoCannedPolicy API to fetch only necessary policy (#8307) 
This PR adds
- InfoCannedPolicy() API for efficiency in fetching policies
- Send group memberships for LDAPUser if available
 2019-09-26 23:53:13 +05:30
Klaus Post
be313f1758 S3 Select: Workaround java buffer size (#8312) 
Updates #7475

The Java implementation has a 128KB buffer and a message must be emitted before that is used. #7475 therefore limits the message size to 128KB. But up to 256 bytes are written to the buffer in each call. This means we must emit a message before shorter than 128KB.

Therefore we change the limit to 128KB minus 256 bytes.
 2019-09-26 04:56:20 +05:30
poornas
2e02e1889b Cleanup ResponseWriter function for audit and trace (#8283) 2019-09-24 02:04:28 +05:30
poornas
4925bc3e80 log server startup messages to admin console api (#8264) 2019-09-22 13:54:32 +05:30
Klaus Post
520552ffa9 S3 select: flush when reaching limit (#8279) 
Add missing flush when reaching select limit.
 2019-09-20 11:00:17 -07:00
Praveen raj Mani
dc0dce9beb Check for connection errors during event transit in elasticsearch and webhook (#8273) 2019-09-19 08:23:43 -07:00
Praveen raj Mani
456ce4cc92 Add rootCAs support to Kafka & MQTT (#8236) 
Fixes #8211
 2019-09-18 23:43:04 +05:30
Klaus Post
dac1cf5a9a S3 Select: Parsing tweaks (#8261) 
* Don't output empty lines.
* Trim whitespace from byte to int/float/bool conversions.
 2019-09-17 17:21:23 -07:00
Klaus Post
c9b8bd8de2 S3 Select: optimize output (#8238) 
Queue output items and reuse them.
Remove the unneeded type system in sql and just use the Go type system.

In best case this is more than an order of magnitude speedup:

```
BenchmarkSelectAll_1M-12    	       1	1841049400 ns/op	274299728 B/op	 4198522 allocs/op
BenchmarkSelectAll_1M-12    	      14	  84833400 ns/op	169228346 B/op	 3146541 allocs/op
```
 2019-09-17 05:56:27 +05:30
Klaus Post
017456df63 Wait clearing the close channel (#8250) 
Close channel should not be nilled before goroutines have exited.

Fixes potential hang on closing.
 2019-09-16 16:18:01 -07:00
Harshavardhana
9ac12cf898
Remove unusued Set/GetConfigKeys API (#8235) 2019-09-13 16:34:34 -07:00
Klaus Post
ddea0bdf11 Concurrent CSV parsing and reduce S3 select allocations (#8200) 
```
CSV parsing, BEFORE:
BenchmarkReaderBasic-12         	    2842	    407533 ns/op	  397860 B/op	     957 allocs/op
BenchmarkReaderReplace-12       	    2718	    429914 ns/op	  397844 B/op	     957 allocs/op
BenchmarkReaderReplaceTwo-12    	    2718	    435556 ns/op	  397855 B/op	     957 allocs/op
BenchmarkAggregateCount_100K-12    	     171	   6798974 ns/op	16667102 B/op	  308077 allocs/op
BenchmarkAggregateCount_1M-12    	      19	  65657411 ns/op	168057743 B/op	 3146610 allocs/op
BenchmarkSelectAll_10M-12    	       1	20882119900 ns/op	2758799896 B/op	41978762 allocs/op

CSV parsing, AFTER:
BenchmarkReaderBasic-12         	    3721	    312549 ns/op	  101920 B/op	     338 allocs/op
BenchmarkReaderReplace-12       	    3776	    318810 ns/op	  101993 B/op	     340 allocs/op
BenchmarkReaderReplaceTwo-12    	    3610	    330967 ns/op	  102012 B/op	     341 allocs/op
BenchmarkAggregateCount_100K-12    	     295	   4149588 ns/op	 3553623 B/op	  103261 allocs/op
BenchmarkAggregateCount_1M-12    	      30	  37746503 ns/op	33827931 B/op	 1049435 allocs/op
BenchmarkSelectAll_10M-12    	       1	17608495800 ns/op	1416504040 B/op	21007082 allocs/op

~ benchcmp old.txt new.txt
benchmark                           old ns/op       new ns/op       delta
BenchmarkReaderBasic-12             407533          312549          -23.31%
BenchmarkReaderReplace-12           429914          318810          -25.84%
BenchmarkReaderReplaceTwo-12        435556          330967          -24.01%
BenchmarkAggregateCount_100K-12     6798974         4149588         -38.97%
BenchmarkAggregateCount_1M-12       65657411        37746503        -42.51%
BenchmarkSelectAll_10M-12           20882119900     17608495800     -15.68%

benchmark                           old allocs     new allocs     delta
BenchmarkReaderBasic-12             957            338            -64.68%
BenchmarkReaderReplace-12           957            340            -64.47%
BenchmarkReaderReplaceTwo-12        957            341            -64.37%
BenchmarkAggregateCount_100K-12     308077         103261         -66.48%
BenchmarkAggregateCount_1M-12       3146610        1049435        -66.65%
BenchmarkSelectAll_10M-12           41978762       21007082       -49.96%

benchmark                           old bytes      new bytes      delta
BenchmarkReaderBasic-12             397860         101920         -74.38%
BenchmarkReaderReplace-12           397844         101993         -74.36%
BenchmarkReaderReplaceTwo-12        397855         102012         -74.36%
BenchmarkAggregateCount_100K-12     16667102       3553623        -78.68%
BenchmarkAggregateCount_1M-12       168057743      33827931       -79.87%
BenchmarkSelectAll_10M-12           2758799896     1416504040     -48.66%
```

```
BenchmarkReaderHuge/97K-12         	    2200	    540840 ns/op	 184.32 MB/s	 1604450 B/op	     687 allocs/op
BenchmarkReaderHuge/194K-12        	    1522	    752257 ns/op	 265.04 MB/s	 2143135 B/op	    1335 allocs/op
BenchmarkReaderHuge/389K-12        	    1190	    947858 ns/op	 420.69 MB/s	 3221831 B/op	    2630 allocs/op
BenchmarkReaderHuge/778K-12        	     806	   1472486 ns/op	 541.61 MB/s	 5201856 B/op	    5187 allocs/op
BenchmarkReaderHuge/1557K-12       	     426	   2575269 ns/op	 619.36 MB/s	 9101330 B/op	   10233 allocs/op
BenchmarkReaderHuge/3115K-12       	     286	   4034656 ns/op	 790.66 MB/s	12397968 B/op	   16099 allocs/op
BenchmarkReaderHuge/6230K-12       	     172	   6830563 ns/op	 934.05 MB/s	16008416 B/op	   26844 allocs/op
BenchmarkReaderHuge/12461K-12      	     100	  11409467 ns/op	1118.39 MB/s	22655163 B/op	   48107 allocs/op
BenchmarkReaderHuge/24922K-12      	      66	  19780395 ns/op	1290.19 MB/s	35158559 B/op	   90216 allocs/op
BenchmarkReaderHuge/49844K-12      	      34	  37282559 ns/op	1369.03 MB/s	60528624 B/op	  174497 allocs/op
```
 2019-09-13 14:18:35 -07:00
Praveen raj Mani
8700945cdf Handle connection failures on webhook/url pings (#8204) 
Properly handle connection failures while replaying events

Fixes #8194
 2019-09-12 16:44:51 -07:00
Krishnan Parthasarathi
6ba323b009 Add ability to test drive speeds on a MinIO setup (#7664) 
- Extends existing Admin API to measure disk performance
 2019-09-13 03:22:30 +05:30
Harshavardhana
53e4887e02 Simplify and cleanup metadata r/w functions (#8146) 2019-09-11 22:52:12 +05:30
Harshavardhana
a7be313230 Start using new errors package (#8207) 2019-09-11 22:51:43 +05:30
Yao Zongyou
18fedc67d5 friendly prompt for s3select MalformedXML error (#8171) 
partly fix #7911
 2019-09-09 21:33:27 -07:00
Harshavardhana
b52a3e523c Avoid using fastjson parser pool, move back to jsoniter (#8190) 
It looks like from implementation point of view fastjson
parser pool doesn't behave the same way as expected
when dealing many `xl.json` from multiple disks.

The fastjson parser pool usage ends up returning incorrect
xl.json entries for checksums, with references pointing
to older entries. This led to the subtle bug where checksum
info is duplicated from a previous xl.json read of a different
file from different disk.
 2019-09-06 04:21:27 +05:30
Praveen raj Mani
428836d4e1 Ignore "connection reset" errors while initializing the targets with queue store enabled (#8185) 
Fixes #8178
 2019-09-05 13:55:48 -07:00
Andreas Auernhammer
810a44e951 KMS Admin-API: add route and handler for KMS key info (#7955) 
This commit adds an admin API route and handler for
requesting status information about a KMS key.

Therefore, the client specifies the KMS key ID (when
empty / not set the server takes the currently configured
default key-ID) and the server tries to perform a dummy encryption,
re-wrap and decryption operation. If all three succeed we know that
the server can access the KMS and has permissions to generate, re-wrap
and decrypt data keys (policy is set correctly).
 2019-09-05 01:49:44 +05:30
poornas
8a71b0ec5a Add admin API to send console log messages (#7784) 
Utilized by mc admin console command.
 2019-09-03 23:40:48 +05:30
Yao Zongyou
ec9bfd3aef speed up the performance of s3select on csv (#7945) 2019-08-31 00:07:40 -07:00
Bala FA
fa3546bb03 Add NetPerfInfo() API in madmin (#8112) 2019-08-31 08:27:53 +05:30
Harshavardhana
83d4c5763c
Decouple ServiceUpdate to ServerUpdate to be more native (#8138) 
The change now is to ensure that we take custom URL as
well for updating the deployment, this is required for
hotfix deliveries for certain deployments - other than
the community release.

This commit changes the previous work d65a2c6725
with newer set of requirements.

Also deprecates PeerUptime()
 2019-08-28 15:04:43 -07:00
Ashish Kumar Sinha
c6f86d35d3 Rectify Examples and README.MD (#8141) 2019-08-28 10:56:02 -07:00
Harshavardhana
d65a2c6725
Implement cluster-wide in-place updates (#8070) 
This PR is a breaking change and also deprecates
`minio update` command, from this release onwards
all users are advised to just use `mc admin update`
 2019-08-27 11:37:47 -07:00
Krishna Srinivas
c38ada1a26 write() to disk in 4MB blocks for better performance (#7888) 2019-08-23 15:36:46 -07:00
poornas
7bf1caa0fe Fix broken link to go install docs (#8090) 2019-08-15 16:00:50 -07:00
Aditya Manthramurthy
bf9b619d86 Set the policy mapping for a user or group (#8036) 
Add API to set policy mapping for a user or group

Contains a breaking Admin APIs change.

- Also enforce all applicable policies
- Removes the previous /set-user-policy API

 Bump up peerRESTVersion

Add get user info API to show groups of a user
 2019-08-13 13:41:06 -07:00
maihde
0ed6daab59 fix: #8051 so that stale DNS entries are cleaned-up (#8053) 2019-08-13 08:49:26 -07:00
Praveen raj Mani
e96f19c867 Use default ports for the schemes when dialing (#7960) 
Fixes #7946
 2019-08-13 16:43:11 +05:30