Commit Graph

2009 Commits

Author SHA1 Message Date
Andreas Auernhammer
810a44e951 KMS Admin-API: add route and handler for KMS key info (#7955)
This commit adds an admin API route and handler for
requesting status information about a KMS key.

Therefore, the client specifies the KMS key ID (when
empty / not set the server takes the currently configured
default key-ID) and the server tries to perform a dummy encryption,
re-wrap and decryption operation. If all three succeed we know that
the server can access the KMS and has permissions to generate, re-wrap
and decrypt data keys (policy is set correctly).
2019-09-05 01:49:44 +05:30
Praveen raj Mani
341d61e3d8 Fix for web-uploads in federated mode (#8175)
Fixes #8173
2019-09-04 23:14:02 +05:30
poornas
8a71b0ec5a Add admin API to send console log messages (#7784)
Utilized by mc admin console command.
2019-09-03 23:40:48 +05:30
Anis Elleuch
b3c19e2d4b storage: Expect empty param in REST requests (#8167)
Empty parameter was forgotten to be added to restQueries() function,
scanning with deep parameter wasn't working properly for distributed
setup.
2019-08-31 13:51:25 +05:30
Bala FA
fa3546bb03 Add NetPerfInfo() API in madmin (#8112) 2019-08-31 08:27:53 +05:30
Harshavardhana
42e716a094
formatsToDrivesInfo should return drives with correct order (#8157)
This is a defensive change to avoid any future issues,
from this part of the code. New change also ensures
to populate UUID if present for the right disk.
2019-08-30 14:11:18 -07:00
Andreas Auernhammer
6b2ed0fc47 fix DownloadZIP for encrypted objects (#8159)
This commit fixes the web ZIP download handler for
encrypted objects. The decryption logic has moved into
`getObjectNInfo`. So trying to decrypt the (already decrypted)
content again in the ZIP handler obviously causes an error.

This commit fixes this by removing the decryption logic from the
the handler.

Fixes #7965
2019-08-30 10:46:09 -07:00
Harshavardhana
0cd0f6c255
Avoid error modification during IAM migration (#8156)
The underlying errors are important, for IAM
requirements and should wait appropriately at
the caller level, this allows for distributed
setups to run properly and not fail prematurely
during startup.

Also additionally fix the onlineDisk counting
2019-08-30 10:41:02 -07:00
Aditya Manthramurthy
847a3ea0a2 Add unit tests and refactor to improve coverage (#7617) 2019-08-29 13:53:27 -07:00
Aditya Manthramurthy
1f3d270de8 Fix delete policy routing (#8145) 2019-08-29 07:07:43 +05:30
Aditya Manthramurthy
eb18c82976 Remove policy query param from being rejected for objects (#8144) 2019-08-28 16:58:40 -07:00
Krishna Srinivas
2ab0681c0c Do not ignore Lock()'s return value (#8142) 2019-08-28 16:12:57 -07:00
Harshavardhana
83d4c5763c
Decouple ServiceUpdate to ServerUpdate to be more native (#8138)
The change now is to ensure that we take custom URL as
well for updating the deployment, this is required for
hotfix deliveries for certain deployments - other than
the community release.

This commit changes the previous work d65a2c6725
with newer set of requirements.

Also deprecates PeerUptime()
2019-08-28 15:04:43 -07:00
Harshavardhana
d65a2c6725
Implement cluster-wide in-place updates (#8070)
This PR is a breaking change and also deprecates
`minio update` command, from this release onwards
all users are advised to just use `mc admin update`
2019-08-27 11:37:47 -07:00
Harshavardhana
70136fb55b
Look for network errors appropriately for RemoteStorageAPI (#8128)
net.Error is very unreliable in providing better error
handling, we need to ensure that we always have a fallback
option in case of network failures.

This fixes an important issue in our distributed server
setups when one of the servers is down, all deployments
out there are recommended to upgrade after this fix is
merged to ensure that availability is not lost.

Fixes #8127
Fixes #8016
Fixes #7964
2019-08-25 13:32:49 -07:00
Harshavardhana
d6dd98e597
Avoid data-race in getDisksInfo call (#8126) 2019-08-23 17:03:15 -07:00
Krishna Srinivas
c38ada1a26 write() to disk in 4MB blocks for better performance (#7888) 2019-08-23 15:36:46 -07:00
poornas
48bc3f1d53 Allow cached content to be encrypted (#8001)
If MINIO_CACHE_ENCRYPTION_MASTER_KEY is set,
automatically encrypt all cached content on disk.
2019-08-23 10:13:22 -07:00
Praveen raj Mani
e211f6f52e Parallelize the DiskInfo calls in xl.StorageInfo() (#8115) 2019-08-22 20:02:40 -07:00
Harshavardhana
f13f421e84
Allow CopyObject in pathStyle across federated instances (#8064)
Fixes #7976
2019-08-21 22:02:39 -10:00
Aditya Manthramurthy
cd03bfb3cf Fix ignoring claims in list buckets call (#8118) 2019-08-21 19:20:11 -10:00
poornas
2e19619e79 browser: Avoid logging BucketNotEmpty error (#8110) 2019-08-21 10:01:46 -10:00
Harshavardhana
2fa98b1d6a Convert errAuthentication as AccessDenied appropriately (#8105)
Fixes #8062
2019-08-21 09:13:15 +05:30
kannappanr
99a4298938 Use a non-strict invalid bucket name check in Get and Delete object (#8073) 2019-08-20 17:40:52 -10:00
Harshavardhana
069badc7e9
Allow CopyObjectPart to work in federated setups (#8066)
Fixes #8065
2019-08-20 07:19:22 -10:00
Harshavardhana
c601cb2f1e
Add listBucketObjectsVersions implementation (#8093)
This API implementation simply behaves like listObjects()
but returns back single version for each object, this
implementation should be considered dummy it is only
meant for some applications which rely on this.
2019-08-19 11:02:54 -10:00
Harshavardhana
9ca7470ccc
Avoid using jsoniter, move to fastjson (#8063)
This is to avoid using unsafe.Pointer type
code dependency for MinIO, this causes
crashes on ARM64 platforms

Refer #8005 collection of runtime crashes due
to unsafe.Pointer usage incorrectly. We have
seen issues like this before when using
jsoniter library in the past.

This PR hopes to fix this using fastjson
2019-08-19 08:35:52 -10:00
Harshavardhana
b3ca304c01
Avoid excessive listing attempts in the daily sweep (#8081)
Add better dynamic timeouts for locks, also
add jitters before launching daily sweep to ensure
that not all the servers in distributed setup
are not trying to hold locks to begin the sweep
round.

Also, add enough delay for incoming requests based
on totalSetCount*totalDriveCount.

A possible fix for #8071
2019-08-19 08:22:32 -10:00
Bala FA
60f52f461f add network read performance collection support. (#8038)
ReST API on /minio/admin/v1/performance?perfType=net[?size=N] 
returns

```
{
  "PEER-1": [
             {
	       "addr": ADDR,
	       "readPerf": DURATION,
	       "error": ERROR,
	     },
	     ...
	   ],
  ...
  ...
  "PEER-N": [
             {
	       "addr": ADDR,
	       "readPerf": DURATION,
	       "error": ERROR,
	     },
	     ...
	   ]
}
```
2019-08-19 08:26:32 +05:30
Harshavardhana
a15bb19d37
Allow audit logging to work while tracing (#8077)
It is observed that when `mc admin trace` is being
used due to ResponseWriter wrapper, we loose information
about statusCode,statusText for audit logging.

This PR fixes this behavior
2019-08-15 16:17:46 -07:00
Harshavardhana
6e7962bf35
Return if paths are empty in DeleteFileBulk (#8085)
This avoids a network call, also fixes an issue
when empty paths are passed the underlying call
fails with "405 Method Not Allowed".

This is reproducible when you are deleting a
non-existent object.

Fixes #8083
2019-08-15 13:15:49 -07:00
Aditya Manthramurthy
825e29f301 Check if user or group is disabled when evaluating policy (#8078) 2019-08-14 16:59:16 -07:00
Krishnan Parthasarathi
bbb56739bd Add User-Agent header with MinIO release details in http logs (#7843)
This would allow http log target server to distinguish between log
messages across different versions of MinIO deployments.
2019-08-14 11:43:43 -07:00
Nitish Tiwari
1cd801b2e9 Fix DeleteObjects() to remove renamed objects inside (#8072) 2019-08-14 11:15:25 -07:00
Aditya Manthramurthy
bf9b619d86 Set the policy mapping for a user or group (#8036)
Add API to set policy mapping for a user or group

Contains a breaking Admin APIs change.

- Also enforce all applicable policies
- Removes the previous /set-user-policy API

 Bump up peerRESTVersion

Add get user info API to show groups of a user
2019-08-13 13:41:06 -07:00
maihde
0ed6daab59 fix: #8051 so that stale DNS entries are cleaned-up (#8053) 2019-08-13 08:49:26 -07:00
Harshavardhana
bf8ec8ad73
Cleanup ui-errors and print proper error messages (#8068)
* Cleanup ui-errors and print proper error messages

Change HELP to HINT instead, handle more error
cases when starting up MinIO. One such is related
to #8048

* Apply suggestions from code review
2019-08-12 21:25:34 -07:00
Harshavardhana
8ce424bacd Enhance audit logging to capture responseTimes (#8067)
Audit logging requires to have

- timeToFirstByte
- timeToResponse

timing information
2019-08-12 20:32:34 -07:00
Anis Elleuch
cea3e3f7a6 browser: Add user-agent header filter to gorilla mux route (#8040)
When a peer client which higher version sends a request to a peer
server with lower version, the returned status code is 200 OK instead
of 405 code. The reason is that the peer client request reaches the
browser handler, which registers itself by '/minio' route but without
any other constraints. Adding filtering by user agent header to the
browser route so internal requests to old endpoints versions return
405 error code.
2019-08-12 17:05:30 -07:00
Harshavardhana
af36c92cab
With ListBuckets() access-list only buckets the user has access (#8037)
This is a behavior change from AWS S3, but it is done with
better judgment on our end to allow the listing of buckets only
which user has access to.

The advantage is this declutters the UI for users and only
lists bucket which they have access to.

Precursor for this feature to be applicable is a policy
must have the following actions

```
s3:ListAllMyBuckets
```
and
```
s3:ListBucket
```

enabled in the policy.
2019-08-12 10:27:38 -07:00
Jakob Ackermann
1b258da108 [web-router] update the white list for favicons (#8024) 2019-08-11 22:17:02 -07:00
Andreas Auernhammer
35427a017d fix type conversion in UpdateKey for Vault (#8058)
This commit fixes a type conversion in the `UpdateKey`
implementation of Vault.
2019-08-11 22:20:25 +05:30
Harshavardhana
5a28ef0d47 Bump readiness check upto 10000 go-routines (#8057)
Most of our current workloads reach this value
regularly, it doesn't make sense to keep 1000
go-routine limit.
2019-08-10 18:13:14 +05:30
poornas
3385bf3da8 Rewrite cache implementation to cache only on GET (#7694)
Fixes #7458
Fixes #7573 
Fixes #7938 
Fixes #6934
Fixes #6265 
Fixes #6630 

This will allow the cache to consistently work for
server and gateways. Range GET requests will
be cached in the background after the request
is served from the backend.

- All cached content is automatically bitrot protected.

- Avoid ETag verification if a cache-control header
is set and the cached content is still valid.

- This PR changes the cache backend format, and all existing
content will be migrated to the new format. Until the data is
migrated completely, all content will be served from the backend.
2019-08-09 17:09:08 -07:00
Anis Elleuch
1ce8d2c476 Add bucket lifecycle expiry feature (#7834) 2019-08-09 10:02:41 -07:00
Harshavardhana
a8296445ad
Safely use unsafe.Pointer to avoid crashes on ARM (#8027)
Refactor the Dirent parsing code such that when we
calculate offsets are correct based on the platform
This PR fixes a silent potential crash on ARM
architecture.
2019-08-09 08:54:11 -07:00
Aditya Manthramurthy
5d2b5ee6a9 Refactor IAM to use new IAMStorageAPI (#7999) 2019-08-08 15:10:04 -07:00
kannappanr
930943f058
Fix IAM users migration regression in etcd (#8029)
PR #8008 did not migrate user data stored in etcd.
This PR fixes that.
2019-08-06 17:06:31 -07:00
Harshavardhana
e6d8e272ce
Use const slashSeparator instead of "/" everywhere (#8028) 2019-08-06 12:08:58 -07:00
Harshavardhana
b52b90412b Avoid data-transfer in distributed locking (#8004) 2019-08-05 11:45:30 -07:00