Commit Graph

6124 Commits

Author SHA1 Message Date
Klaus Post
38e6d911ea S3 Select: Detect full object (#8456)
Check if select is `SELECT s.* from S3Object s` and forward it to All

Fixes #8371 and makes this case run significantly faster.
2019-10-30 13:46:55 +05:30
Harshavardhana
47b13cdb80 Add etcd part of config support, add noColor/json support (#8439)
- Add color/json mode support for get/help commands
- Support ENV help for all sub-systems
- Add support for etcd as part of config
2019-10-30 00:04:39 -07:00
Klaus Post
51456e6adc Select: Support Square Bracket Lists (#8457)
Allows for S3 compatible `SELECT * from s3object s WHERE id IN [3,2]`

Fixes #8422
2019-10-30 11:34:40 +05:30
Harshavardhana
1f481c0967 Return appropriate error if user,group,policy doesn't exist (#8465)
Fixes https://github.com/minio/mc/issues/2944
2019-10-29 19:50:26 -07:00
Andreas Auernhammer
e31b7cdcd5 madmin: update sio version and use Algorithm constants (#8463)
This commit bumps the version of the `sio` library
from v0.2.0 => v0.3.0. Now, `madmin` can use the
`Algorithm` type constants that make the encrypt/decrypt
code simpler.
2019-10-29 15:36:47 -07:00
Anis Elleuch
4cec0501ca heal: Remove daily sweeper code (#8462)
This has no effect on the functional change of the server
2019-10-29 14:13:05 -07:00
cc
1b6de05a51 refine NextMarker comments (#8450) 2019-10-28 13:18:12 -07:00
Kanagaraj M
4e6bf136a8 fix browser blank page on gateways (#8453)
StorageInfo component will not be rendered if
Storage used is null.
2019-10-28 10:48:02 -07:00
Harshavardhana
a2825702f8
Increase maximum 1000 List keys to 10000 (#8444) 2019-10-28 10:36:15 -07:00
Anis Elleuch
a49d4a9cb2 xl: Rewrite auto-healing and implement auto new-disk healer (#8114)
The new auto healing model selects one node always responsible
for auto-healing the whole cluster, erasure set by erasure set.
If that node dies, another node will be elected as a leading
operator to perform healing.

This code also adds a goroutine which checks each 10 minutes
if there are any new unformatted disks and performs its healing
in that case, only the erasure set which has the new disk will
be healed.
2019-10-28 10:27:49 -07:00
Nitish Tiwari
10b526ed86 Fix panic during trace requests (#8448)
While Tracing requests on server, type assertion on logger.ResponseWriter
caused nil pointer exception because of recordAPIStats{} being
used as ResponseWriter. This PR avoids the type assertion and
initializes a new logger.ResponseWriter.

Fixes regression introduced in #8003
2019-10-27 08:49:16 -07:00
Krishna Srinivas
980bf78b4d Detect underlying disk mount/unmount (#8408) 2019-10-25 10:37:53 -07:00
Harshavardhana
8aaaa46be9 Fix typo in prometheus getBucketLocation metrics (#8438) 2019-10-23 18:54:22 -07:00
Kaan Kabalak
53862d4a28 Use sanitized raw link for logo.svg in README (#8436) 2019-10-23 13:02:35 -07:00
Harshavardhana
ddc908207d Fix README.md link to logo.svg 2019-10-23 10:22:06 -07:00
Harshavardhana
ee4a6a823d Migrate config to KV data format (#8392)
- adding oauth support to MinIO browser (#8400) by @kanagaraj
- supports multi-line get/set/del for all config fields
- add support for comments, allow toggle
- add extensive validation of config before saving
- support MinIO browser to support proper claims, using STS tokens
- env support for all config parameters, legacy envs are also
  supported with all documentation now pointing to latest ENVs
- preserve accessKey/secretKey from FS mode setups
- add history support implements three APIs
  - ClearHistory
  - RestoreHistory
  - ListHistory
- add help command support for each config parameters
- all the bug fixes after migration to KV, and other bug
  fixes encountered during testing.
2019-10-22 22:59:13 -07:00
Praveen raj Mani
8836d57e3c The prometheus metrics refractoring (#8003)
The measures are consolidated to the following metrics

- `disk_storage_used` : Disk space used by the disk.
- `disk_storage_available`: Available disk space left on the disk.
- `disk_storage_total`: Total disk space on the disk.
- `disks_offline`: Total number of offline disks in current MinIO instance.
- `disks_total`: Total number of disks in current MinIO instance.
- `s3_requests_total`: Total number of s3 requests in current MinIO instance.
- `s3_errors_total`: Total number of errors in s3 requests in current MinIO instance.
- `s3_requests_current`: Total number of active s3 requests in current MinIO instance.
- `internode_rx_bytes_total`: Total number of internode bytes received by current MinIO server instance.
- `internode_tx_bytes_total`: Total number of bytes sent to the other nodes by current MinIO server instance.
- `s3_rx_bytes_total`: Total number of s3 bytes received by current MinIO server instance.
- `s3_tx_bytes_total`: Total number of s3 bytes sent by current MinIO server instance.
- `minio_version_info`: Current MinIO version with commit-id.
- `s3_ttfb_seconds_bucket`: Histogram that holds the latency information of the requests.

And this PR also modifies the current StorageInfo queries

- Decouples StorageInfo from ServerInfo .
- StorageInfo is enhanced to give endpoint information.

NOTE: ADMIN API VERSION IS BUMPED UP IN THIS PR

Fixes #7873
2019-10-22 21:01:14 -07:00
poornas
f01d53b20f cache: do not evict entry on ErrClosedPipe (#8432)
Fixes: #8431. If client prematurely closes the read end of the pipe,
cache entry should not be evicted.
2019-10-22 15:04:25 -07:00
Harshavardhana
40fcd3dc48 Deprecate listDirFactory in HealObjects, rely on ListObjectsHeal (#8419) 2019-10-22 03:13:04 +05:30
poornas
1b74ce3924 Ensure actual object size is sent in notification (#8418)
Fixes: #8407
2019-10-20 23:48:19 -07:00
Anis Elleuch
7bf093c06a xl: Fix isObject() to consider not found disks (#8411)
xl.isObject() returns 'nil' for not found disks when
calculating the existance of xl.json for a given object,
which what StatFile() is also doing (setting nil) if
xl.json exists.

This commit avoids this confusion by setting errDiskNotFound
error when the storage disk is not found.
2019-10-18 23:19:11 +05:30
Kaan Kabalak
140a7eadb4 Fix browser showing compressed instead of actual object size (#8412)
Fixes #8930
2019-10-18 18:21:52 +05:30
Harshavardhana
fce2d6ddd1
Remote update should be on by default (#8413)
Fixes a regression introduced in PR #8351
2019-10-17 20:54:02 -07:00
Ashish Kumar Sinha
18cb15559d Add network hardware info (#8358)
peerRESTVersion changed to v6
2019-10-17 04:09:49 -07:00
poornas
3adc311c1c Fix regression in admin router when no route matches (#8409)
When `mc admin user add` is attempted in gateway mode without
etcd setup, NoSuchBucket error is returned instead of MethodNotAllowed.
Regression from commit - e48005ddc7
2019-10-16 20:39:23 -07:00
Anis Elleuch
ee05280721 fs: Remove stale background append temporary file (#8404)
Background append creates a temporary file which appends
uploaded parts as long as they are available, but when a
client stops the upload, the temporary file is not removed
by any way.

This commit removes the temporary file when the server does
its regular removing stale multipart uploads.
2019-10-17 00:27:52 +05:30
poornas
c4e2af8ca3 Remove cache env from server help message (#8405) 2019-10-16 23:22:57 +05:30
Harshavardhana
5afb1b6747
Add support for {jwt:sub} substitutions for policies (#8393)
Fixes #8345
2019-10-16 08:59:59 -07:00
Harshavardhana
f2cc97a44c
Improve MQTT token registration retry (#8397) 2019-10-15 19:39:14 -07:00
Harshavardhana
d48fd6fde9
Remove unusued params and functions (#8399) 2019-10-15 18:35:41 -07:00
Andreas Auernhammer
b7ee0bbbc9 madmin: Migrate to provable secure channel construction (#8395)
This commit replaces the currently used channel construction
for en/decrypting config data with a provable secure scheme.
2019-10-15 02:36:04 +05:30
Harshavardhana
68a519a468
Use errgroups instead of sync.WaitGroup as needed (#8354) 2019-10-14 09:44:51 -07:00
Harshavardhana
c33bae057f Fix build issues with event target logger 2019-10-11 19:07:09 -07:00
poornas
d7060c4c32 Allow logging targets to be configured to receive minio (#8347)
specific errors, `application` errors or `all` by default.

console logging on server by default lists all logs -
enhance admin console API to accept `type` as query parameter to
subscribe to application/minio logs.
2019-10-11 18:50:54 -07:00
Minio Trusted
8964ef821f Update yaml files to latest version RELEASE.2019-10-12T01-39-57Z 2019-10-12 01:44:42 +00:00
Harshavardhana
bd10640846 Fix logger and audit http endpoint config lookup 2019-10-11 18:33:25 -07:00
Harshavardhana
175b07d6e4
Fix queueStore stops working with concurrent PUT/DELETE requests (#8381)
- This PR fixes situation to avoid underflow, this is possible
  because of disconnected operations in replay/sendEvents
- Hold right locks if Del() operation is performed in Get()
- Remove panic in the code and use loggerOnce
- Remove Timer and instead use Ticker instead for proper ticks
2019-10-11 17:46:03 -07:00
Harshavardhana
f16df2a4e7 Avoid crash in PopulatePublicKey() by re-initializing the args (#8388)
This is to avoid nil pointer dereference when method by pointer
reference and method by value reference are implemented.

Fixes #8387
2019-10-12 03:37:08 +05:30
Ashish Kumar Sinha
1c90485b56 Remove duplicate cpu hardware info (#8384) 2019-10-12 00:15:43 +05:30
Harshavardhana
2320a877bc
Add CREDITS license from all dependent projects (#8382) 2019-10-11 11:24:06 -07:00
Minio Trusted
ae752ed1fa Update yaml files to latest version RELEASE.2019-10-11T00-38-09Z 2019-10-11 00:45:13 +00:00
Aditya Manthramurthy
c566cc6b61 Fix errors in ldap doc (#8374) 2019-10-10 12:28:39 +05:30
Harshavardhana
36e12a6038 Assume local endpoints appropriately in k8s deployments (#8375)
On Kubernetes/Docker setups DNS resolves inappropriately
sometimes where there are situations same endpoints with
multiple disks come online indicating either one of them
is local and some of them are not local. This situation
can never happen and its only a possibility in orchestrated
deployments with dynamic DNS. Following code ensures that we
treat if one of the endpoint says its local for a given host
it is true for all endpoints for the same host. Following code
ensures that this assumption is true and it works in all
scenarios and it is safe to assume for a given host.

This PR also adds validation such that we do not crash the
server if there are bugs in the endpoints list in dsync
initialization.

Thanks to Daniel Valdivia <hola@danielvaldivia.com> for
reproducing this, this fix is needed as part of the
https://github.com/minio/m3 project.
2019-10-10 10:14:17 +05:30
Kaan Kabalak
42531db37e Update browser UI assets (#8373) 2019-10-10 08:22:47 +05:30
Harshavardhana
6a4ef2e48e Initialize configs correctly, move notification config (#8367)
This PR also removes deprecated tests, adds checks
to avoid races reproduced on CI/CD.
2019-10-09 11:41:15 +05:30
Harshavardhana
d2a8be6fc2 gateway/hdfs: Fix isObjectDir to behave correctly (#8368) 2019-10-09 04:20:43 +05:30
Harshavardhana
290ad0996f Move etcd, logger, crypto into their own packages (#8366)
- Deprecates _MINIO_PROFILER, `mc admin profile` does the job
- Move ENVs to common location in cmd/config/
2019-10-08 11:17:56 +05:30
Mark Felder
bffc378a4f Secret key field is not for creating a new password (#8362) 2019-10-07 10:55:50 -07:00
Harshavardhana
3b8adf7528 Move storageclass config handling into cmd/config/storageclass (#8360)
Continuation of the changes done in PR #8351 to refactor,
add tests and move global handling into a more idiomatic
style for Go as packages.
2019-10-07 11:20:24 +05:30
Klaus Post
002ac82631 S3 Select: Add parser support for lists. (#8329) 2019-10-06 07:52:45 -07:00