MyFavMirrors/minio
1
0
Fork 0
mirror of https://github.com/minio/minio.git synced 2024-12-26 07:05:55 -05:00
Code Issues Packages Projects Releases Wiki Activity
6,247 Commits 4 Branches 506 Tags 189 MiB
a3c8ef79a4
Commit Graph

6247 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Nitish Tiwari
3df7285c3c Add Support for Cache and S3 related metrics in Prometheus endpoint (#8591) 
This PR adds support below metrics

- Cache Hit Count
- Cache Miss Count
- Data served from Cache (in Bytes)
- Bytes received from AWS S3
- Bytes sent to AWS S3
- Number of requests sent to AWS S3

Fixes #8549
 2019-12-05 23:16:06 -08:00
Aleksandr Petruhin
d2dc964cb5 Support TLS auth for Kafka notification target (#8609) 2019-12-05 15:31:46 -08:00
Harshavardhana
d8e3de0cae Ensure comment is always a valid key (#8604) 
Also fix LDAP leaky connection
 2019-12-05 18:17:42 +05:30
Harshavardhana
c9940d8c3f Final changes to config sub-system (#8600) 
- Introduces changes such as certain types of
  errors that can be ignored or which need to 
  go into safe mode.
- Update help text as per the review
 2019-12-04 15:32:37 -08:00
Harshavardhana
794eb54da8 Export command prints turned-off sub-sys as comments (#8594) 
This PR also tries to

- Preserve the order of keys printed in export command
- Fix cache to be enabled with _STATE env to keep
  backward compatibility
 2019-12-03 10:50:20 -08:00
Harshavardhana
2ab8d5e47f Enable build verification with race (#8583) 2019-12-02 15:54:26 -08:00
Clemens Wolff
947bc8c7d3 Update Azure Gateway to azure-storage-blob SDK (#8537) 
The azure-sdk-for-go/storage package has been in maintenance-
only mode since February 2018 (see [1]) and will be deprecated in the future.
 2019-12-02 09:32:19 -08:00
Harshavardhana
5d3d57c12a
Start using error wrapping with fmt.Errorf (#8588) 
Use fatih/errwrap to fix all the code to use
error wrapping with fmt.Errorf()
 2019-12-02 09:28:01 -08:00
Harshavardhana
0bfd20a8e3
Add client_id support for OpenID (#8579) 
- One click OpenID authorization on Login page
- Add client_id help, config keys etc

Thanks to @egorkaru @ihostage for the
original work and testing.
 2019-11-29 21:37:42 -08:00
Klaus Post
db3dbcce3a Print goroutines when shutdown hangs (#8574) 2019-11-29 19:40:08 +05:30
Harshavardhana
b21835f195 Honor DurationSeconds properly for WebIdentity (#8581) 
Also cleanup code to add various constants for
verbatim strings across the code base.

Fixes #8482
 2019-11-29 18:57:54 +05:30
Klaus Post
c7844fb1fb posix: cache disk ID for a short while (#8564) 
`*posix.getDiskID()` takes up to 30% of all CPU due to the `os.Stat` call on `GET` calls.

Before:
```
Operation: GET - Concurrency: 12
* Average: 1333.97 MB/s, 1365.99 obj/s, 1365.98 ops ended/s (4m59.975s)
* First Byte: Average: 7.801487ms, Median: 7.9974ms, Best: 1.9822ms, Worst: 110.0021ms

Aggregated, split into 299 x 1s time segments:
* Fastest: 1453.50 MB/s, 1488.38 obj/s, 1492.00 ops ended/s (1s)
* 50% Median: 1360.47 MB/s, 1393.12 obj/s, 1393.00 ops ended/s (1s)
* Slowest: 978.68 MB/s, 1002.17 obj/s, 1004.00 ops ended/s (1s)
```

After:
```
Operation: GET - Concurrency: 12
* Average: 1706.07 MB/s, 1747.02 obj/s, 1747.01 ops ended/s (4m59.985s)
* First Byte: Average: 5.797886ms, Median: 5.9959ms, Best: 996.3µs, Worst: 84.0007ms

Aggregated, split into 299 x 1s time segments:
* Fastest: 1830.03 MB/s, 1873.96 obj/s, 1872.00 ops ended/s (1s)
* 50% Median: 1735.04 MB/s, 1776.68 obj/s, 1776.00 ops ended/s (1s)
* Slowest: 994.94 MB/s, 1018.82 obj/s, 1018.00 ops ended/s (1s)
```

TLDR; `os.Stat` is not free.
 2019-11-29 02:57:14 -08:00
Harshavardhana
2ff8132e2d Fix the regression introduced in #8580 2019-11-27 16:13:07 -08:00
Harshavardhana
30e80d0a86
Add ReadFrom,WriteTo helpers for server config (#8580) 2019-11-27 09:36:08 -08:00
Klaus Post
87443af49e Cache Windows mount point info (#8572) 
It seems like every upload creates a new StorageInfo object.

This makes it very sensitive to slow code paths.

A particular slow function is `IsLikelyMountPoint` on Windows since `windows.GetVolumePathName` is slow.

We cache the result of this operation.

Before:

```
Operation: PUT - Concurrency: 12
* Average: 70.46 MB/s, 70.46 obj/s, 70.46 ops ended/s (59.58s)

Aggregated, split into 59 x 1s time segments:
* Fastest: 77.67 MB/s, 77.67 obj/s, 79.00 ops ended/s (1s)
* 50% Median: 70.84 MB/s, 70.84 obj/s, 70.00 ops ended/s (1s)
* Slowest: 61.51 MB/s, 61.51 obj/s, 61.00 ops ended/s (1s)
```

After:
```
Operation: PUT - Concurrency: 12
* Average: 166.80 MB/s, 166.80 obj/s, 166.74 ops ended/s (59.8s)

Aggregated, split into 59 x 1s time segments:
* Fastest: 183.83 MB/s, 183.83 obj/s, 186.00 ops ended/s (1s)
* 50% Median: 167.47 MB/s, 167.47 obj/s, 166.00 ops ended/s (1s)
* Slowest: 139.30 MB/s, 139.30 obj/s, 141.00 ops ended/s (1s)
```
 2019-11-27 20:10:13 +05:30
Harshavardhana
5d65428b29
Handle localhost distributed setups properly (#8577) 
Fixes an issue reported by @klauspost and @vadmeste

This PR also allows users to expand their clusters
from single node XL deployment to distributed mode.
 2019-11-26 11:42:10 -08:00
Harshavardhana
78eb3b78bb
Repurpose Get/SetConfig as import/export support (#8578) 2019-11-26 10:08:25 -08:00
Harshavardhana
720442b1a2
Add lock expiry handler to expire state locks (#8562) 2019-11-25 16:39:43 -08:00
Harshavardhana
e542084c37
Add etcd path prefix for all IAM assets (#8569) 
Currently, we use the top-level prefix "config/"
for all our IAM assets, instead of to provide
tenant-level separation bring 'path_prefix'
to namespace the access properly.

Fixes #8567
 2019-11-25 16:33:34 -08:00
poornas
f931fc7bfb Fix retention enforcement in Compliance mode (#8556) 
In compliance mode, the retention date can be extended with 
governance bypass permissions
 2019-11-25 10:58:39 -08:00
Harshavardhana
0a56e33ce1 Preserve client sent config appropriately (#8566) 2019-11-22 13:46:05 -08:00
Harshavardhana
c3771df641
Add bootstrap REST handler for verifying server config (#8550) 2019-11-22 12:45:13 -08:00
Klaus Post
890b493a2e Use random file name for write check (#8563) 
Since there may be multiple writes going on concurrently
Use a random file name for the write check to avoid collisions.
 2019-11-22 09:50:17 -08:00
Bruce Wang
fb708b6b64 minimum Go version 1.13 (#8561) 2019-11-22 12:02:34 +05:30
Harshavardhana
f96e902f63 Do not rely on quorum for StorageInfo() (#8557) 
StorageInfo() call is supposed to give each
server/disk information independently, rely
on this appropriately so that `mc admin info server`
gets correct information all the time.
 2019-11-21 22:08:41 -08:00
Sergey Morgunov
06bd1e582a Log in with OIDC not work with MINIO_DOMAIN (#8558) (#8559) 2019-11-21 17:45:15 -08:00
Harshavardhana
fb43d64dc3
Fix healing on multiple zones (#8555) 
It is expected in zone healing underlying
callers should return appropriate errors
 2019-11-21 13:18:32 -08:00
Harshavardhana
fd0fa4e5c5 Add NTP retention time (#8548) 2019-11-21 18:22:35 +05:30
Harshavardhana
4e9de58675 Avoid pointer based copy, instead use Clone() (#8547) 
This PR adds functional test to test expanded
cluster syntax.
 2019-11-21 17:54:51 +05:30
Harshavardhana
9565641b9b
Enhance ListObjectsV2 API to return UserDefined metadata (#8539) 2019-11-21 01:54:49 -08:00
poornas
4da68cfcfc Handle indexes correctly in DeleteMultipleObjectsHandler (#8544) 
Regression from #8509 which changes objectsToDelete entry
from a list to map. This will cause index out of range
panic if object is not selected for delete.
 2019-11-20 17:51:10 -08:00
poornas
a9b87c0a16 Fix typo in event name (#8545) 2019-11-20 16:35:01 -08:00
Harshavardhana
5ac4b517c9
Order all keys in config (#8541) 
New changes

- return default values when sub-sys is
  not configured.
- state is hidden parameter now
- remove worm mode to be saved in config
 2019-11-20 15:10:24 -08:00
poornas
ca96560d56 Add object retention at the per object (#8528) 
level - this PR builds on #8120 which
added PutBucketObjectLockConfiguration and
GetBucketObjectLockConfiguration APIS

This PR implements PutObjectRetention,
GetObjectRetention API and enhances
PUT and GET API operations to display
governance metadata if permissions allow.
 2019-11-20 13:18:09 -08:00
Nitish Tiwari
cc1a84b62e Fix heal result item output to properly count drives and sets (#8543) 2019-11-20 10:10:26 -08:00
Harshavardhana
8392d2f510 Preserve same deploymentID on all zones (#8542) 2019-11-20 15:39:30 +05:30
Harshavardhana
347b29d059 Implement bucket expansion (#8509) 2019-11-19 17:42:27 -08:00
Harshavardhana
3a34d98db8
Initialize local nsLocker for gateway instances (#8540) 2019-11-19 16:45:35 -08:00
Harshavardhana
7cdb67680e
Add help with order of keys (#8535) 2019-11-19 13:48:13 -08:00
poornas
929951fd49 Add support for multiple admins (#8487) 
Also define IAM policies for administering
MinIO server
 2019-11-19 02:03:18 -08:00
Harshavardhana
13a3d17321
Do not add comments after migration (#8530) 
Also filter out empty comments from being
printed.
 2019-11-16 14:57:36 -08:00
Harshavardhana
a8e156d6a5
Fix cache locking to use local namespace locking (#8529) 2019-11-16 13:44:28 -08:00
svistoi
c9be601988 NATS TLS specify CA and client TLS authentication (#8389) 
- added ability to specify CA for self-signed certificates
- added option to authenticate using client certificates
- added unit tests for nats connections
 2019-11-15 09:13:23 -08:00
poornas
13e2b97ad9 Fix regression in caching on single PUT (#8526) 
Regression caused by #8120
 2019-11-15 15:46:27 +05:30
Ville Skyttä
95e5d7a9c3 Improve access and secret key validation error, sync with implementation (#8516) 2019-11-14 14:47:35 -08:00
Harshavardhana
32c200fe12 Fix console logger crash in gateway mode (#8525) 
This PR also fixes config migration only
for credentials and region which are valid
and set.

Also fix implicit `state="on"` behavior
 2019-11-14 14:19:57 -08:00
Klaus Post
1dd38750f7 Remove read-ahead for small files (#8522) 
We should only read ahead if we are reading big files. We enable it for files >= 16MB.

Benchmark on 64KB objects.

Before:

```
Operation: GET
Errors: 0
Average: 59.976s, 87.13 MB/s, 1394.07 ops ended/s.
Fastest: 1s, 90.99 MB/s, 1455.00 ops ended/s.
50% Median: 1s, 87.53 MB/s, 1401.00 ops ended/s.
Slowest: 1s, 81.39 MB/s, 1301.00 ops ended/s.
```

After:

```
Operation: GET
Errors: 0
Average: 59.992s, 207.99 MB/s, 3327.85 ops ended/s.
Fastest: 1s, 219.20 MB/s, 3507.00 ops ended/s.
50% Median: 1s, 210.54 MB/s, 3368.00 ops ended/s.
Slowest: 1s, 179.14 MB/s, 2865.00 ops ended/s.
```

The 64KB buffer is actually a small disadvantage for this case, but I believe it will be better in general than no buffer.
 2019-11-14 12:58:41 -08:00
Praveen raj Mani
e3273bc5bf Fix runtime panic in BackendDisks.Merge() (#8524) 2019-11-14 10:17:41 -08:00
Harshavardhana
26a866a202
Fix review comments and new changes in config (#8515) 
- Migrate and save only settings which are enabled
- Rename logger_http to logger_webhook and
  logger_http_audit to audit_webhook
- No more pretty printing comments, comment
  is a key=value pair now.
- Avoid quotes on values which do not have space in them
- `state="on"` is implicit for all SetConfigKV unless
  specified explicitly as `state="off"`
- Disabled IAM users should be disabled always
 2019-11-13 17:38:05 -08:00
Anis Elleuch
60690a7e1d fs: Fix setting new deployment ID in format when not present (#8517) 
The code does not properly set a new deployemnt ID when not present
in format.json: it loops twice without releasing write lock on format.json
causing an infinite locking error on the same file.

This commit fixes and simplifies a little the code.
 2019-11-13 12:18:23 -08:00