Commit Graph

6293 Commits

Author SHA1 Message Date
Harshavardhana
586614c73f fix: temp credentials shouldn't allow policy/group changes (#8675)
This PR fixes the issue where we might allow policy changes
for temporary credentials out of band, this situation allows
privilege escalation for those temporary credentials. We
should disallow any external actions on temporary creds
as a practice and we should clearly differentiate which
are static and which are temporary credentials.

Refer #8667
2019-12-19 14:21:21 -08:00
Harshavardhana
d140074773 fix: replica set deployment for multi tenants (#8673)
Changes in IP underneath are dynamic in replica sets
with multiple tenants, so deploying in that fashion
will not work until we wait for atleast one participatory
server to be local.

This PR also ensures that multi-tenant zone expansion also
works in replica set k8s deployments.

Introduces a new ENV `KUBERNETES_REPLICA_SET` check to call
appropriate code paths.
2019-12-19 13:45:56 -08:00
Harshavardhana
39face27cf Simplify k8s replicated set deployment (#8666)
Continuation from #8629 which basically broke
zone deployments on k8s statefulset environment
due to incorrect assumptions which made it work
on replicated set.

Fix this properly such that this container works
for both replicated set and stateful set deployment
2019-12-18 17:05:24 -08:00
Andreas Auernhammer
e047ac52b8 remove github.com/minio/kes as a dependency (#8665)
This commit removes github.com/minio/kes as
a dependency and implements the necessary
client-side functionality without relying
on the KES project.

This resolves the licensing issue since
KES is licensed under AGPL while MinIO
is licensed under Apache.
2019-12-18 15:10:57 -08:00
poornas
04de3ea4bd Change cache purge routine granularity to hours (#8660)
With this PR,cache eviction will continue until
no LRU entries older than an hour can be cache
evicted or sufficient percentage of disk space
has been reclaimed.
2019-12-18 13:49:10 -08:00
Amol Umbarkar
e6ce9da087 fix BucketForward Handler for federated setup (#8646)
fixes #8595
2019-12-18 14:06:03 +05:30
Harshavardhana
c9c0d5eec2 Allow CNAME records when specified as MINIO_PUBLIC_IPS (#8662)
This is necessary for `m3` global bucket support
2019-12-18 11:02:45 +05:30
Harshavardhana
63c3114657 fix: doc notifications formatting issues (#8661) 2019-12-17 17:34:17 -08:00
Minio Trusted
63ac1d6d18 Update yaml files to latest version RELEASE.2019-12-17T23-16-33Z 2019-12-17 23:21:28 +00:00
Harshavardhana
9bb0869b73
fix: populate buckets on etcd after config has loaded (#8658) 2019-12-17 13:50:07 -08:00
Harshavardhana
5f2318567e
Allow metadata updates on meta bucket even in WORM mode (#8657)
This ensures that we can update the

- .minio.sys is updated for accounting/data usage purposes
- .minio.sys is updated to indicate if backend is encrypted
  or not.
2019-12-17 10:13:12 -08:00
kannappanr
16ac4a3c64 PutBucketLifeCycleConfiguration: Return 200 instead of 204 (#8656) 2019-12-17 07:39:49 -08:00
Harshavardhana
c8d82588c2 Fix crash in console logger and also handle bucket DNS updates (#8654)
Also fix listenBucketNotification bugs seen by minio-js
listen bucket notification API.
2019-12-16 20:30:57 -08:00
Praveen raj Mani
842d0241ed Fix aws-sdk-php functional test cases for GCS gateway (#8613)
Fixes #8570
2019-12-16 12:41:49 +05:30
Harshavardhana
1dc5f2d0af
Remove safe mode for invalid entries in config (#8650)
The approach is that now safe mode is only invoked when
we cannot read the config or under some catastrophic
situations, but not under situations when config entries
are invalid or unreachable. This allows for maximum
availability for MinIO and not fail on our users unlike
most of our historical releases.
2019-12-14 17:27:57 -08:00
Harshavardhana
c10ecacf91 Always use SourceIP for host target filtering (#8649) 2019-12-14 11:12:59 -08:00
poornas
1cf3e3b7b5 PutBucket: Case-insensitive validation of x-amz-bucket-object-lock-enabled (#8648)
Fix: case insensitive validation of x-amz-bucket-object-lock-enabled header in PutBucket handler
2019-12-13 15:51:28 -08:00
Andreas Auernhammer
c3d4c1f584 add minio/keys KMS integration (#8631)
This commit adds support for the minio/kes KMS.
See: https://github.com/minio/kes

In particular you can configure it as KMS by:
 - `export MINIO_KMS_KES_ENDPOINT=`  // Server URL
 - `export MINIO_KMS_KES_KEY_FILE=`  // TLS client private key
 - `export MINIO_KMS_KES_CERT_FILE=` // TLS client certificate
 - `export MINIO_KMS_KES_CA_PATH=`   // Root CAs issuing server cert
 - `export MINIO_KMS_KES_KEY_NAME=`  // The name of the (default)
master key
2019-12-13 12:57:11 -08:00
Harshavardhana
471a3a650a
fix: Don't allow to set unconfigured notification ARNs (#8643)
Fixes #8642
2019-12-13 12:36:45 -08:00
Harshavardhana
cc02bf0442
Remove old ListenBucketNotification API (#8645) 2019-12-13 11:33:11 -08:00
Harshavardhana
39e8e4f4aa
Allow empty target KVS for notification targets (#8644)
This is allowed with enable=off arg value
2019-12-12 17:02:14 -08:00
poornas
80558e839d Clear cache if reverting to backend (#8637)
Clear cached entry before reverting to backend for
encrypted objects or those under retention to avoid
stale objects remaining in cache.
2019-12-12 15:11:27 -08:00
Harshavardhana
ca62ac65d4
Reject mandatory KVS if not set for any sub-sys (#8641) 2019-12-12 14:55:07 -08:00
Harshavardhana
f5abe4e1f1
Support ListenBucketNotificationV2 streaming (#8622) 2019-12-12 10:01:23 -08:00
Klaus Post
3211cb5df6 Add encryption buffer (#8626)
Quite hard to measure difference:

```
λ warp cmp put-before.csv.zst put-after2.csv.zst
Operation: PUT
Operations: 340 -> 353
* Average: +4.11% (+22.7 MB/s) throughput, +4.11% (+0.2) obj/s
* 50% Median: +1.58% (+7.3 MB/s) throughput, +1.58% (+0.1) obj/s
```

Difference is likely bigger on Intel platforms due to higher syscall costs.
2019-12-12 10:01:15 -08:00
Ashish Kumar Sinha
abc266caa1 Add bucket and object count along with total object size (#8639) 2019-12-12 09:58:59 -08:00
Harshavardhana
c364f0af6c Start using custom HTTP transport for webhook endpoints (#8630)
Use a more performant http transport for webhook
endpoints with proper connection pooling, appropriate
timeouts etc.
2019-12-12 06:53:50 -08:00
Anis Elleuch
555969ee42 Add data usage collect with its new admin API (#8553)
Admin data usage info API returns the following

(Only FS & XL, for now)

- Number of buckets
- Number of objects
- The total size of objects
- Objects histogram
- Bucket sizes
2019-12-12 06:02:37 -08:00
Ashish Kumar Sinha
e2c5d29017 Bucket,Object count & Usage removed if set to default (#8638) 2019-12-11 21:56:47 -08:00
Harshavardhana
fa00a84709
Avoid crashes on peers if IAMSys is not initialized (#8636) 2019-12-11 20:46:57 -08:00
kannappanr
d266b3a066
Admin Info: Modify Uptime to return seconds (#8635) 2019-12-11 17:56:02 -08:00
Leletir
db3f41fcb4 Doc: change url for Total Population CSV (#8633) 2019-12-11 14:37:48 -08:00
Ashish Kumar Sinha
24fb1bf258 New Admin Info (#8497) 2019-12-11 14:27:03 -08:00
Harshavardhana
8b803491af
fix: CacheOpts parsing tests (#8632) 2019-12-11 13:26:18 -08:00
Harshavardhana
10b2f15f6f Add randomize sleep times for lock checkers (#8628) 2019-12-11 10:57:05 -08:00
Harshavardhana
3e9ab5f4a9
Fix k8s replica set deployment (#8629)
In replica sets, hosts resolve to localhost
IP automatically until the deployment fully
comes up. To avoid this issue we need to
wait for such resolution.
2019-12-10 20:28:22 -08:00
brnbs
46b4dd8e20 Add command to list added lifecycle configurations (#8627) 2019-12-10 11:38:25 -08:00
Klaus Post
bf3a97d3aa S3 Select: Concurrent LINES delimited json parsing (#8610)
The speedup is ~5x on a 6 core CPU
2019-12-09 06:55:31 -08:00
Krishna Srinivas
3b67f629a4 Retry peer notification of events (#8621) 2019-12-09 05:29:37 -08:00
poornas
3c30e4503d Cache only the range requested for range GETs (#8599) 2019-12-08 13:58:04 -08:00
poornas
8390bc26db Fix cache hit metrics. (#8617) 2019-12-07 23:14:33 +05:30
Nitish Tiwari
24ad59316d
Use atomic.Uint64 for gateway metrics count instead of mutex (#8615) 2019-12-07 11:21:52 +05:30
poornas
be0c8b1ec0 Add support for missing Cache-Control directives (#8619)
no-cache, only-if-cached and no-store directives are
being enforced in this PR.
2019-12-07 07:49:36 +05:30
Harshavardhana
476111968a Update help messages with new wording (#8616)
Final update to all messages across sub-systems
after final review, the only change here is that
NATS now has TLS and TLSSkipVerify to be consistent
for all other notification targets.
2019-12-06 13:53:51 -08:00
Klaus Post
f1e2e1cc9e S3 Select: Mismatched types don't match (#8608)
When comparing for equality, if types cannot be matched, they don't match.
2019-12-06 07:24:41 -08:00
Harshavardhana
97deba2a7c GetKVS should add new keys automatically, preserve order (#8612) 2019-12-06 16:13:10 +05:30
Nitish Tiwari
3df7285c3c Add Support for Cache and S3 related metrics in Prometheus endpoint (#8591)
This PR adds support below metrics

- Cache Hit Count
- Cache Miss Count
- Data served from Cache (in Bytes)
- Bytes received from AWS S3
- Bytes sent to AWS S3
- Number of requests sent to AWS S3

Fixes #8549
2019-12-05 23:16:06 -08:00
Aleksandr Petruhin
d2dc964cb5 Support TLS auth for Kafka notification target (#8609) 2019-12-05 15:31:46 -08:00
Harshavardhana
d8e3de0cae Ensure comment is always a valid key (#8604)
Also fix LDAP leaky connection
2019-12-05 18:17:42 +05:30
Harshavardhana
c9940d8c3f Final changes to config sub-system (#8600)
- Introduces changes such as certain types of
  errors that can be ignored or which need to 
  go into safe mode.
- Update help text as per the review
2019-12-04 15:32:37 -08:00