Commit Graph

757 Commits

Author SHA1 Message Date
Anis Elleuch
3011692d93 doc: Add missing ``` in compression documentation (#8872) 2020-01-21 15:50:54 +05:30
Harshavardhana
86252ec7e1
fix: document _ENABLE for all notification targets (#8864)
Fixes #8863
2020-01-20 16:48:19 -08:00
bijialin
c260182412 zh_CN docs: add Expanding existing distributed setup (#8862) 2020-01-20 08:51:09 -08:00
Nitish Tiwari
61c17c8933 Add ObjectTagging Support (#8754)
This PR adds support for AWS S3 ObjectTagging API as explained here
https://docs.aws.amazon.com/AmazonS3/latest/dev/object-tagging.html
2020-01-20 08:45:59 -08:00
kannappanr
005ebbb9b2 Typo: Add missing '<' character (#8845)
Fixes #8833
2020-01-17 13:31:46 -08:00
poornas
60e60f68dd Add support for object locking with legal hold. (#8634) 2020-01-16 15:41:56 -08:00
Minio Trusted
ba758361b3 Update yaml files to latest version RELEASE.2020-01-16T22-40-29Z 2020-01-16 22:47:41 +00:00
Minio Trusted
080e0c2323 Update yaml files to latest version RELEASE.2020-01-16T03-05-44Z 2020-01-16 03:11:40 +00:00
ebozduman
d74818b227 Typo in MySQL notification doc (#8807) 2020-01-13 13:06:42 -08:00
Harshavardhana
3f20011862 docs: update how to query the keys and args (#8797)
remove link to missing config.json.sample
2020-01-11 15:13:13 -08:00
Etienne
5f34b5e6a5 Update README.md (#8788) 2020-01-10 21:08:09 -08:00
Kevin Humphreys
656146b699 doc: Prometheus metrics name fix (#8774)
changed docs to reflect proper Prometheus metrics
2020-01-09 18:36:58 -08:00
Harshavardhana
5e40b9a563
fix: docs for live/ready check implementation details 2020-01-09 18:29:24 -08:00
Joe Adams
89d1221217 Fix typo in prometheus monitoring docs (#8780) 2020-01-09 09:08:41 -08:00
Harshavardhana
60813bef29
Allow proper setCount SLAs across zones (#8752)
Fixes scenario where zones are appropriately
handled, along with supporting overriding set
count. The new fix also ensures that we handle
the various setup types properly.

Update documentation to properly indicate the
behavior.

Fixes #8750

Co-authored-by: Nitish Tiwari <nitish@minio.io>
2020-01-07 09:13:44 -08:00
Harshavardhana
7a06e158f1
Add docs for sysctl tuning parameters (#8749) 2020-01-06 00:30:04 -08:00
Harshavardhana
5eab3db344 Fix doc reference for prometheus (#8748) 2020-01-05 13:44:39 -08:00
Brian Candler
9f44fcd540 Clarify behaviour of erasure coding sets (#8745) 2020-01-05 13:00:11 -08:00
Minio Trusted
84e55e2e6f Update yaml files to latest version RELEASE.2020-01-03T19-12-21Z 2020-01-03 19:20:28 +00:00
Harshavardhana
dd311623df Update design doc with zone implementation details (#8738)
Fixes #8719
2020-01-02 16:46:16 -08:00
Nitish Tiwari
97d799b9f0 doc: Update Kubernetes examples to use Readiness Checks (#8727) 2020-01-01 08:25:09 -08:00
Minio Trusted
8eba97da74 Update yaml files to latest version RELEASE.2019-12-30T05-45-39Z 2019-12-30 05:56:46 +00:00
Harshavardhana
669c9da85d Disable federated buckets when etcd is namespaced (#8709)
This is to ensure that when we have multiple tenants
deployed all sharing the same etcd for global bucket
should avoid listing each others buckets, this leads
to information leak which should be avoided unless
etcd is not namespaced for IAM assets in which case
it can be assumed that its a federated setup.

Federated setup and namespaced IAM assets on etcd
is not supported since namespacing is only useful
when you wish to separate the tenants as isolated
instances of MinIO.

This PR allows a new type of behavior, primarily
driven by the usecase of m3(mkube) multi-tenant
deployments with global bucket support.
2019-12-29 08:56:45 -08:00
Minio Trusted
4364102363 Update yaml files to latest version RELEASE.2019-12-24T23-04-45Z 2019-12-24 23:15:44 +00:00
Harshavardhana
0f092c2be8 Add env values and description for config sub-systems (#8684)
The output is directly borrowed from `mc` command output
2019-12-21 22:52:16 +05:30
Minio Trusted
dc1eb57f4e Update yaml files to latest version RELEASE.2019-12-19T22-52-26Z 2019-12-19 22:58:19 +00:00
Harshavardhana
63c3114657 fix: doc notifications formatting issues (#8661) 2019-12-17 17:34:17 -08:00
Minio Trusted
63ac1d6d18 Update yaml files to latest version RELEASE.2019-12-17T23-16-33Z 2019-12-17 23:21:28 +00:00
Harshavardhana
c8d82588c2 Fix crash in console logger and also handle bucket DNS updates (#8654)
Also fix listenBucketNotification bugs seen by minio-js
listen bucket notification API.
2019-12-16 20:30:57 -08:00
Leletir
db3f41fcb4 Doc: change url for Total Population CSV (#8633) 2019-12-11 14:37:48 -08:00
brnbs
46b4dd8e20 Add command to list added lifecycle configurations (#8627) 2019-12-10 11:38:25 -08:00
Harshavardhana
476111968a Update help messages with new wording (#8616)
Final update to all messages across sub-systems
after final review, the only change here is that
NATS now has TLS and TLSSkipVerify to be consistent
for all other notification targets.
2019-12-06 13:53:51 -08:00
Nitish Tiwari
3df7285c3c Add Support for Cache and S3 related metrics in Prometheus endpoint (#8591)
This PR adds support below metrics

- Cache Hit Count
- Cache Miss Count
- Data served from Cache (in Bytes)
- Bytes received from AWS S3
- Bytes sent to AWS S3
- Number of requests sent to AWS S3

Fixes #8549
2019-12-05 23:16:06 -08:00
Aleksandr Petruhin
d2dc964cb5 Support TLS auth for Kafka notification target (#8609) 2019-12-05 15:31:46 -08:00
Harshavardhana
c9940d8c3f Final changes to config sub-system (#8600)
- Introduces changes such as certain types of
  errors that can be ignored or which need to 
  go into safe mode.
- Update help text as per the review
2019-12-04 15:32:37 -08:00
Harshavardhana
0bfd20a8e3
Add client_id support for OpenID (#8579)
- One click OpenID authorization on Login page
- Add client_id help, config keys etc

Thanks to @egorkaru @ihostage for the
original work and testing.
2019-11-29 21:37:42 -08:00
Harshavardhana
b21835f195 Honor DurationSeconds properly for WebIdentity (#8581)
Also cleanup code to add various constants for
verbatim strings across the code base.

Fixes #8482
2019-11-29 18:57:54 +05:30
poornas
f931fc7bfb Fix retention enforcement in Compliance mode (#8556)
In compliance mode, the retention date can be extended with 
governance bypass permissions
2019-11-25 10:58:39 -08:00
Harshavardhana
5ac4b517c9
Order all keys in config (#8541)
New changes

- return default values when sub-sys is
  not configured.
- state is hidden parameter now
- remove worm mode to be saved in config
2019-11-20 15:10:24 -08:00
poornas
ca96560d56 Add object retention at the per object (#8528)
level - this PR builds on #8120 which
added PutBucketObjectLockConfiguration and
GetBucketObjectLockConfiguration APIS

This PR implements PutObjectRetention,
GetObjectRetention API and enhances
PUT and GET API operations to display
governance metadata if permissions allow.
2019-11-20 13:18:09 -08:00
Harshavardhana
347b29d059 Implement bucket expansion (#8509) 2019-11-19 17:42:27 -08:00
poornas
929951fd49 Add support for multiple admins (#8487)
Also define IAM policies for administering
MinIO server
2019-11-19 02:03:18 -08:00
Harshavardhana
26a866a202
Fix review comments and new changes in config (#8515)
- Migrate and save only settings which are enabled
- Rename logger_http to logger_webhook and
  logger_http_audit to audit_webhook
- No more pretty printing comments, comment
  is a key=value pair now.
- Avoid quotes on values which do not have space in them
- `state="on"` is implicit for all SetConfigKV unless
  specified explicitly as `state="off"`
- Disabled IAM users should be disabled always
2019-11-13 17:38:05 -08:00
Harshavardhana
e9b2bf00ad Support MinIO to be deployed on more than 32 nodes (#8492)
This PR implements locking from a global entity into
a more localized set level entity, allowing for locks
to be held only on the resources which are writing
to a collection of disks rather than a global level.

In this process this PR also removes the top-level
limit of 32 nodes to an unlimited number of nodes. This
is a precursor change before bring in bucket expansion.
2019-11-13 12:17:45 -08:00
Harshavardhana
64759189f5 Update web-identity example to use minio-go SDK (#8501) 2019-11-13 16:00:54 +05:30
Bala FA
fb48ca5020 Add Get/Put Bucket Lock Configuration API support (#8120)
This feature implements [PUT Bucket object lock configuration][1] and
[GET Bucket object lock configuration][2]. After object lock
configuration is set, existing and new objects are set to WORM for
specified duration. Currently Governance mode works exactly like
Compliance mode.

Fixes #8101

[1] https://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketPUTObjectLockConfiguration.html
[2] https://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketGETObjectLockConfiguration.html
2019-11-12 14:50:18 -08:00
Harshavardhana
aa04f97f95 Config migration should handle plain-text (#8506)
This PR fixes issues found in config migration

 - StorageClass migration error when rrs is empty
 - Plain-text migration of older config
 - Do not run in safe mode with incorrect credentials
 - Update logger_http documentation for _STATE env

Refer more reported issues at #8434
2019-11-11 12:01:21 -08:00
Harshavardhana
822eb5ddc7 Bring in safe mode support (#8478)
This PR refactors object layer handling such
that upon failure in sub-system initialization
server reaches a stage of safe-mode operation
wherein only certain API operations are enabled
and available.

This allows for fixing many scenarios such as

 - incorrect configuration in vault, etcd,
   notification targets
 - missing files, incomplete config migrations
   unable to read encrypted content etc
 - any other issues related to notification,
   policies, lifecycle etc
2019-11-09 09:27:23 -08:00
Harshavardhana
d28bcb4f84 Migrate all backend at .minio.sys/config to encrypted backend (#8474)
- Supports migrating only when the credential ENVs are set,
  so any FS mode deployments which do not have ENVs set will
  continue to remain as is.
- Credential ENVs can be rotated using MINIO_ACCESS_KEY_OLD
  and MINIO_SECRET_KEY_OLD envs, in such scenarios it allowed
  to rotate the encrypted content to a new admin key.
2019-11-01 15:53:16 -07:00
Harshavardhana
47b13cdb80 Add etcd part of config support, add noColor/json support (#8439)
- Add color/json mode support for get/help commands
- Support ENV help for all sub-systems
- Add support for etcd as part of config
2019-10-30 00:04:39 -07:00
Harshavardhana
ee4a6a823d Migrate config to KV data format (#8392)
- adding oauth support to MinIO browser (#8400) by @kanagaraj
- supports multi-line get/set/del for all config fields
- add support for comments, allow toggle
- add extensive validation of config before saving
- support MinIO browser to support proper claims, using STS tokens
- env support for all config parameters, legacy envs are also
  supported with all documentation now pointing to latest ENVs
- preserve accessKey/secretKey from FS mode setups
- add history support implements three APIs
  - ClearHistory
  - RestoreHistory
  - ListHistory
- add help command support for each config parameters
- all the bug fixes after migration to KV, and other bug
  fixes encountered during testing.
2019-10-22 22:59:13 -07:00
Praveen raj Mani
8836d57e3c The prometheus metrics refractoring (#8003)
The measures are consolidated to the following metrics

- `disk_storage_used` : Disk space used by the disk.
- `disk_storage_available`: Available disk space left on the disk.
- `disk_storage_total`: Total disk space on the disk.
- `disks_offline`: Total number of offline disks in current MinIO instance.
- `disks_total`: Total number of disks in current MinIO instance.
- `s3_requests_total`: Total number of s3 requests in current MinIO instance.
- `s3_errors_total`: Total number of errors in s3 requests in current MinIO instance.
- `s3_requests_current`: Total number of active s3 requests in current MinIO instance.
- `internode_rx_bytes_total`: Total number of internode bytes received by current MinIO server instance.
- `internode_tx_bytes_total`: Total number of bytes sent to the other nodes by current MinIO server instance.
- `s3_rx_bytes_total`: Total number of s3 bytes received by current MinIO server instance.
- `s3_tx_bytes_total`: Total number of s3 bytes sent by current MinIO server instance.
- `minio_version_info`: Current MinIO version with commit-id.
- `s3_ttfb_seconds_bucket`: Histogram that holds the latency information of the requests.

And this PR also modifies the current StorageInfo queries

- Decouples StorageInfo from ServerInfo .
- StorageInfo is enhanced to give endpoint information.

NOTE: ADMIN API VERSION IS BUMPED UP IN THIS PR

Fixes #7873
2019-10-22 21:01:14 -07:00
poornas
d7060c4c32 Allow logging targets to be configured to receive minio (#8347)
specific errors, `application` errors or `all` by default.

console logging on server by default lists all logs -
enhance admin console API to accept `type` as query parameter to
subscribe to application/minio logs.
2019-10-11 18:50:54 -07:00
Minio Trusted
8964ef821f Update yaml files to latest version RELEASE.2019-10-12T01-39-57Z 2019-10-12 01:44:42 +00:00
Minio Trusted
ae752ed1fa Update yaml files to latest version RELEASE.2019-10-11T00-38-09Z 2019-10-11 00:45:13 +00:00
Aditya Manthramurthy
c566cc6b61 Fix errors in ldap doc (#8374) 2019-10-10 12:28:39 +05:30
Harshavardhana
e85df07518 Add prometheus auth-type to turn-off authentication (#8356)
Also this PR moves the original doc from cookbook to
MinIO repo under docs/metrics/prometheus/

Fixes #8323
2019-10-04 23:48:59 +05:30
Minio Trusted
83066f953c Update yaml files to latest version RELEASE.2019-10-02T21-19-38Z 2019-10-02 21:25:57 +00:00
Harshavardhana
90bfa6260a Fix LDAP TLS support to use custom CAs (#8352) 2019-10-03 01:44:57 +05:30
Ashish Kumar Sinha
42c821e164 Replace proxy with forward in coredns example (#8344) 2019-10-01 22:23:08 +05:30
Harshavardhana
20b907d8fb
Update hadoop docs to indicate new committers (#8060) 2019-09-30 21:34:06 -07:00
Ashish Kumar Sinha
4f981a0b42 Change YAML files to reference the newer APIs (#8299) 2019-09-30 00:04:55 -07:00
Aditya Manthramurthy
f3022e891d Add ToC for all STS API docs (#8318) 2019-09-27 09:57:51 -07:00
Minio Trusted
c28405a5c2 Update yaml files to latest version RELEASE.2019-09-26T19-42-35Z 2019-09-26 19:47:59 +00:00
Harshavardhana
3094615e38 Update LDAP document to reflect work user/group sub-commands (#8313)
This PR also fixes markdown formatting
2019-09-26 23:48:26 +05:30
Klaus Post
ff726969aa Switch to Snappy -> S2 compression (#8189) 2019-09-25 23:08:24 -07:00
Aditya Manthramurthy
704be85987 Fix numbering in doc (#8311) 2019-09-26 04:48:59 +05:30
Aditya Manthramurthy
c8da04ba5b Add section on user/group policy for AD/LDAP integration (#8310) 2019-09-26 04:45:09 +05:30
Minio Trusted
9ed423b13f Update yaml files to latest version RELEASE.2019-09-25T18-25-51Z 2019-09-25 18:31:05 +00:00
Harshavardhana
77dc2031a2 Fix LDAP responseXML to be named appropriately (#8285)
This PR additionally also adds support for missing

- Session policy support for AD/LDAP
- Add API request/response parameters detail
- Update example to take ldap username,
  password input from the command line
- Fixes session policy handling for
  ClientGrants and WebIdentity
2019-09-24 03:51:16 +05:30
Andreas Auernhammer
cb7d23cb17 remove SSE-S3 key rotation in CopyObject (#8278)
This commit removes the SSE-S3 key rotation functionality
from CopyObject since there will be a dedicated Admin-API
for this purpose.

Also update the security documentation to link to mc and
the admin documentation.
2019-09-24 02:05:04 +05:30
Julio Pintos
3b05e175d7 fix link to let's encrypt doc (#8294) 2019-09-23 23:53:11 +05:30
Praveen raj Mani
ad75683bde Authorize prometheus endpoint with bearer token (#7640) 2019-09-22 20:27:12 +05:30
Minio Trusted
8dc897b5f5 Update yaml files to latest version RELEASE.2019-09-18T21-55-05Z 2019-09-18 22:02:41 +00:00
Aditya Manthramurthy
696f4ceee2 Add notes on configuring LDAP STS with Microsoft Active Directory (#8260) 2019-09-17 17:21:37 -07:00
Aditya Manthramurthy
dfa1b417a8 Update comments in ldap example client code (#8257) 2019-09-18 01:59:31 +05:30
poornas
76df027264 Allow caching only in gateway mode. (#8232)
This PR changes cache on PUT behavior to background fill the cache
after PutObject completes. This will avoid concurrency issues as in #8219.

Added cleanup of partially filled cache to prevent cache corruption
- Fixes #8208
2019-09-17 02:54:04 +05:30
k-tipp
208efb843b Update Kafka notification example doc (#8212) 2019-09-16 19:24:55 +05:30
Klaus Post
ddea0bdf11 Concurrent CSV parsing and reduce S3 select allocations (#8200)
```
CSV parsing, BEFORE:
BenchmarkReaderBasic-12         	    2842	    407533 ns/op	  397860 B/op	     957 allocs/op
BenchmarkReaderReplace-12       	    2718	    429914 ns/op	  397844 B/op	     957 allocs/op
BenchmarkReaderReplaceTwo-12    	    2718	    435556 ns/op	  397855 B/op	     957 allocs/op
BenchmarkAggregateCount_100K-12    	     171	   6798974 ns/op	16667102 B/op	  308077 allocs/op
BenchmarkAggregateCount_1M-12    	      19	  65657411 ns/op	168057743 B/op	 3146610 allocs/op
BenchmarkSelectAll_10M-12    	       1	20882119900 ns/op	2758799896 B/op	41978762 allocs/op

CSV parsing, AFTER:
BenchmarkReaderBasic-12         	    3721	    312549 ns/op	  101920 B/op	     338 allocs/op
BenchmarkReaderReplace-12       	    3776	    318810 ns/op	  101993 B/op	     340 allocs/op
BenchmarkReaderReplaceTwo-12    	    3610	    330967 ns/op	  102012 B/op	     341 allocs/op
BenchmarkAggregateCount_100K-12    	     295	   4149588 ns/op	 3553623 B/op	  103261 allocs/op
BenchmarkAggregateCount_1M-12    	      30	  37746503 ns/op	33827931 B/op	 1049435 allocs/op
BenchmarkSelectAll_10M-12    	       1	17608495800 ns/op	1416504040 B/op	21007082 allocs/op

~ benchcmp old.txt new.txt
benchmark                           old ns/op       new ns/op       delta
BenchmarkReaderBasic-12             407533          312549          -23.31%
BenchmarkReaderReplace-12           429914          318810          -25.84%
BenchmarkReaderReplaceTwo-12        435556          330967          -24.01%
BenchmarkAggregateCount_100K-12     6798974         4149588         -38.97%
BenchmarkAggregateCount_1M-12       65657411        37746503        -42.51%
BenchmarkSelectAll_10M-12           20882119900     17608495800     -15.68%

benchmark                           old allocs     new allocs     delta
BenchmarkReaderBasic-12             957            338            -64.68%
BenchmarkReaderReplace-12           957            340            -64.47%
BenchmarkReaderReplaceTwo-12        957            341            -64.37%
BenchmarkAggregateCount_100K-12     308077         103261         -66.48%
BenchmarkAggregateCount_1M-12       3146610        1049435        -66.65%
BenchmarkSelectAll_10M-12           41978762       21007082       -49.96%

benchmark                           old bytes      new bytes      delta
BenchmarkReaderBasic-12             397860         101920         -74.38%
BenchmarkReaderReplace-12           397844         101993         -74.36%
BenchmarkReaderReplaceTwo-12        397855         102012         -74.36%
BenchmarkAggregateCount_100K-12     16667102       3553623        -78.68%
BenchmarkAggregateCount_1M-12       168057743      33827931       -79.87%
BenchmarkSelectAll_10M-12           2758799896     1416504040     -48.66%
```

```
BenchmarkReaderHuge/97K-12         	    2200	    540840 ns/op	 184.32 MB/s	 1604450 B/op	     687 allocs/op
BenchmarkReaderHuge/194K-12        	    1522	    752257 ns/op	 265.04 MB/s	 2143135 B/op	    1335 allocs/op
BenchmarkReaderHuge/389K-12        	    1190	    947858 ns/op	 420.69 MB/s	 3221831 B/op	    2630 allocs/op
BenchmarkReaderHuge/778K-12        	     806	   1472486 ns/op	 541.61 MB/s	 5201856 B/op	    5187 allocs/op
BenchmarkReaderHuge/1557K-12       	     426	   2575269 ns/op	 619.36 MB/s	 9101330 B/op	   10233 allocs/op
BenchmarkReaderHuge/3115K-12       	     286	   4034656 ns/op	 790.66 MB/s	12397968 B/op	   16099 allocs/op
BenchmarkReaderHuge/6230K-12       	     172	   6830563 ns/op	 934.05 MB/s	16008416 B/op	   26844 allocs/op
BenchmarkReaderHuge/12461K-12      	     100	  11409467 ns/op	1118.39 MB/s	22655163 B/op	   48107 allocs/op
BenchmarkReaderHuge/24922K-12      	      66	  19780395 ns/op	1290.19 MB/s	35158559 B/op	   90216 allocs/op
BenchmarkReaderHuge/49844K-12      	      34	  37282559 ns/op	1369.03 MB/s	60528624 B/op	  174497 allocs/op
```
2019-09-13 14:18:35 -07:00
Minio Trusted
3d65dc8d94 Update yaml files to latest version RELEASE.2019-09-11T19-53-16Z 2019-09-11 19:59:05 +00:00
Nitish Tiwari
432cb38dbd Add reference to MinIO operator in Kubernetes docs (#8098) 2019-09-10 11:37:31 -07:00
Aditya Manthramurthy
a0456ce940 LDAP STS API (#8091)
Add LDAP based users-groups system

This change adds support to integrate an LDAP server for user
authentication. This works via a custom STS API for LDAP. Each user
accessing the MinIO who can be authenticated via LDAP receives
temporary credentials to access the MinIO server.

LDAP is enabled only over TLS.

User groups are also supported via LDAP. The administrator may
configure an LDAP search query to find the group attribute of a user -
this may correspond to any attribute in the LDAP tree (that the user
has access to view). One or more groups may be returned by such a
query.

A group is mapped to an IAM policy in the usual way, and the server
enforces a policy corresponding to all the groups and the user's own
mapped policy.

When LDAP is configured, the internal MinIO users system is disabled.
2019-09-10 04:42:29 +05:30
Minio Trusted
5bde31d021 Update yaml files to latest version RELEASE.2019-09-05T23-24-38Z 2019-09-05 23:30:40 +00:00
Nitish Tiwari
496fba3e9a
Return 200 OK for liveness checks while distributed cluster starts (#8176)
With this PR, liveness check responds with 200 OK with "server-not-
initialized" header while objectLayer gets initialized. The header
is removed as objectLayer is initialized. This is to allow
MinIO distributed cluster to get started when running on an
orchestration platforms like Docker Swarm.

This PR also updates sample Swarm yaml files to use correct values
for healthcheck fields.

Fixes #8140
2019-09-05 14:50:56 +05:30
Harshavardhana
0772438125
Update orchestration docs to use ellipses syntax (#8153) 2019-09-01 10:02:15 -07:00
Minio Trusted
daa57e8e5a Update yaml files to latest version RELEASE.2019-08-29T00-25-01Z 2019-08-29 00:30:50 +00:00
poornas
48bc3f1d53 Allow cached content to be encrypted (#8001)
If MINIO_CACHE_ENCRYPTION_MASTER_KEY is set,
automatically encrypt all cached content on disk.
2019-08-23 10:13:22 -07:00
Dee Koder
4f3c9d7677
Update doc with Object Tagging not supported (#8116) 2019-08-21 16:10:59 -07:00
Minio Trusted
de736f2864 Update yaml files to latest version RELEASE.2019-08-21T19-40-07Z 2019-08-21 19:46:01 +00:00
Nitish Tiwari
e5fb6294a7 Remove healthcheck script for Docker image (#8095)
There are multiple possibilities for running MinIO within
a container e.g. configurable address, non-root user etc.
This makes it difficult to identify actual IP / Port to
use to check healthcheck status from within a container.

It is simpler to use external healthcheck mechanisms
like healthcheck command in docker-compose to check
for MinIO health status. This is similar to how checks
work in Kubernetes as well.

This PR removes the healthcheck script used inside
Docker container and ad documentation on how to
use docker-compose based healthcheck mechanism.
2019-08-17 12:44:04 -07:00
Harshavardhana
cebeca3075 Fix multi-user doc to reflect the latest change 2019-08-14 14:12:23 -07:00
Minio Trusted
26640033b0 Update yaml files to latest version RELEASE.2019-08-14T20-37-41Z 2019-08-14 20:42:16 +00:00
kannappanr
bc79b435a2 Doc: Fix PutObject API known limitations text (#7704) 2019-08-13 08:50:43 -07:00
Harshavardhana
8ce424bacd Enhance audit logging to capture responseTimes (#8067)
Audit logging requires to have

- timeToFirstByte
- timeToResponse

timing information
2019-08-12 20:32:34 -07:00
Cary FitzHugh
a0715f3b59 Adding Notes on Swarm Network Aliases (#8054) 2019-08-10 21:25:01 +05:30
Harshavardhana
5a28ef0d47 Bump readiness check upto 10000 go-routines (#8057)
Most of our current workloads reach this value
regularly, it doesn't make sense to keep 1000
go-routine limit.
2019-08-10 18:13:14 +05:30
poornas
3385bf3da8 Rewrite cache implementation to cache only on GET (#7694)
Fixes #7458
Fixes #7573 
Fixes #7938 
Fixes #6934
Fixes #6265 
Fixes #6630 

This will allow the cache to consistently work for
server and gateways. Range GET requests will
be cached in the background after the request
is served from the backend.

- All cached content is automatically bitrot protected.

- Avoid ETag verification if a cache-control header
is set and the cached content is still valid.

- This PR changes the cache backend format, and all existing
content will be migrated to the new format. Until the data is
migrated completely, all content will be served from the backend.
2019-08-09 17:09:08 -07:00
Anis Elleuch
1ce8d2c476 Add bucket lifecycle expiry feature (#7834) 2019-08-09 10:02:41 -07:00
Harshavardhana
d759a7ce99
Fix time formatting of Expiration field in STS (#8043)
Without explicit conversion to UTC() from Unix
time the zone information is lost, this leads
to XML marshallers marshaling the time into
a wrong format.

This PR fixes the compatibility issue with AWS STS
API by keeping Expiration format close to ISO8601
or RFC3339

Fixes #8041
2019-08-08 15:44:57 -07:00
Harshavardhana
9ee619ea14 Remove deprecated ENVs 2019-08-07 18:57:38 -07:00
Minio Trusted
55b385beee Update yaml files to latest version RELEASE.2019-08-07T01-59-21Z 2019-08-07 02:05:41 +00:00
Andreas Auernhammer
a6f4cf61f2 add UpdateKey method to KMS interface (#7974)
This commit adds a new method `UpdateKey` to the KMS
interface.

The purpose of `UpdateKey` is to re-wrap an encrypted
data key (the key generated & encrypted with a master key by e.g.
Vault).
For example, consider Vault with a master key ID: `master-key-1`
and an encrypted data key `E(dk)` for a particular object. The
data key `dk` has been generated randomly when the object was created.
Now, the KMS operator may "rotate" the master key `master-key-1`.
However, the KMS cannot forget the "old" value of that master key
since there is still an object that requires `dk`, and therefore,
the `D(E(dk))`.
With the `UpdateKey` method call MinIO can ask the KMS to decrypt
`E(dk)` with the old key (internally) and re-encrypted `dk` with
the new master key value: `E'(dk)`.

However, this operation only works for the same master key ID.
When rotating the data key (replacing it with a new one) then
we perform a `UnsealKey` operation with the 1st master key ID
and then a `GenerateKey` operation with the 2nd master key ID.

This commit also updates the KMS documentation and removes
the `encrypt` policy entry (we don't use `encrypt`) and
add a policy entry for `rewarp`.
2019-08-01 15:47:47 -07:00
Minio Trusted
dfa8835720 Update yaml files to latest version RELEASE.2019-08-01T22-18-54Z 2019-08-01 22:27:41 +00:00
Minio Trusted
c301f5882d Update yaml files to latest version RELEASE.2019-07-31T18-57-56Z 2019-07-31 19:06:20 +00:00
Praveen raj Mani
b0cea1c0f3 Enable event persistence in AMQP (#7565) 2019-07-25 11:20:24 -07:00
Harshavardhana
6f2b4675fa
Add krb5 support for HDFS gateway (#7933) 2019-07-24 18:05:48 -07:00
Harshavardhana
a4ce1daf99 docs: Use --user to start container in non-root (#7966) 2019-07-24 17:35:52 -07:00
Praveen raj Mani
55d4eee6f1 Enable event persistence in MySQL and PostgreSQL (#7629) 2019-07-24 10:18:29 -07:00
Minio Trusted
5b71c21330 Update yaml files to latest version RELEASE.2019-07-24T02-02-23Z 2019-07-24 02:09:19 +00:00
Praveen raj Mani
c9349747ca Enable event-persistence in NATS and NATS-Streaming (#7612) 2019-07-23 10:37:25 -07:00
Praveen raj Mani
2b9b907f9c Enable event persistence in Redis (#7601) 2019-07-23 10:22:08 -07:00
Harshavardhana
87e6533cf3 Add some design docs for distributed setup (#7950) 2019-07-23 07:48:10 +05:30
Minio Trusted
b058e32348 Update yaml files to latest version RELEASE.2019-07-17T22-54-12Z 2019-07-17 22:59:33 +00:00
Lucas
ea66a52ed1 Add KMS master key from Docker secret (#7825) 2019-07-17 20:55:26 +01:00
Harshavardhana
55dd017e62 Deprecate auto detection of container user (#7930)
There is no reliable way to handle fallbacks for
MinIO deployments, due to various command line
options and multiple locations which require
access inside container.

Parsing command line options is tricky to figure
out which is the backend disk etc, we did try
to fix this in implementations of check-user.go
but it wasn't complete and introduced more bugs.

This PR simplifies the entire approach to rather
than running Docker container as non-root by default
always, it allows users to opt-in. Such that they
are aware that that is what they are planning to do.

In-fact there are other ways docker containers can
be run as regular users, without modifying our
internal behavior and adding more complexities.
2019-07-17 19:20:55 +01:00
Harshavardhana
a57c747667 Document vault in prod mode instead of dev mode (#7928) 2019-07-16 01:32:15 +01:00
Praveen raj Mani
bf278ca36f Enable event persistence in NSQ (#7579) 2019-07-12 10:41:57 +01:00
Praveen raj Mani
bba562235b Enable persistent event store in elasticsearch (#7564) 2019-07-12 08:23:20 +05:30
Minio Trusted
2d96745156 Update yaml files to latest version RELEASE.2019-07-10T00-34-56Z 2019-07-10 00:39:45 +00:00
Minio Trusted
22bc15d89b Update yaml files to latest version RELEASE.2019-07-05T21-20-21Z 2019-07-05 21:24:43 +00:00
Praveen raj Mani
bb871a7c31 Enable event persistence in webhook (#7614) 2019-07-05 15:21:41 +05:30
iliul
a39e810965 docs: Fix dead link of HighwayHash (#7847)
Signed-off-by: Lei Liu <liul.stone@gmail.com>
2019-07-03 14:32:58 -07:00
mizuno-keyence
09103991ea [Bugfix] duplicating flag registration (#7853) 2019-07-03 14:31:19 -07:00
Matthew Wegner
0bcd8abc5c doc: "admin user policy" command typo (#7865)
Under "change user policy", the `mc admin set-policy` command is wrong.  It should be `mc admin user set-policy`.
2019-07-02 11:48:26 -07:00
Minio Trusted
5db60a6c59 Update yaml files to latest version RELEASE.2019-06-27T21-13-50Z 2019-06-27 21:18:52 +00:00
Klaus Post
61229b38f7 Update compression README.md (#7823)
- Snappy is not and RLE compressor, it is LZ77 based.
- Add `xz` as a common file type.
- Add most common media container types.
- Never heard of `application/x-spoon`. Google turns up a blank as well.
- Change link to minio blog post on compression & encryption.
2019-06-23 13:52:58 +05:30
Andreas Auernhammer
219d841496 remove encryption key section for certtool docs (#7820)
This commit removes the encryption key section from
the certool.exe docs because:
 - MinIO does not support any TLS cipher that encrypts
   something with the private key. We only support PFS
   ciphers.
 - The doc comment is not really accurate anyway.
2019-06-21 10:39:02 -07:00
poornas
680fdf6f90 Update doc link (#7814) 2019-06-20 16:49:21 -07:00
Harshavardhana
1af6e8cb72
Add support for session policies in STS APIs (#7747)
This PR adds support for adding session policies
for further restrictions on STS credentials, useful
in situations when applications want to generate
creds for multiple interested parties with different
set of policy restrictions.

This session policy is not mandatory, but optional.

Fixes #7732
2019-06-20 15:28:33 -07:00
Harshavardhana
35c38e4bd8 Add docs regarding trace (#7808) 2019-06-19 14:28:03 -07:00
Minio Trusted
e43d3a075c Update yaml files to latest version RELEASE.2019-06-19T18-24-42Z 2019-06-19 18:31:37 +00:00
Harshavardhana
43e0ef4248 Update docs with new settings (#7803) 2019-06-18 17:52:27 -07:00
Minio Trusted
59f7266081 Update yaml files to latest version RELEASE.2019-06-15T23-07-18Z 2019-06-15 23:17:28 +00:00
Harshavardhana
4a4048fe27 Migrate minio etcd config to backend config (#7751)
etcd when used in federated setups, currently
mandates that all clusters should have same
config.json, which is too restrictive and makes
federation a restrictive environment.

This change makes it apparent that each cluster
needs to be independently managed if necessary
from `mc admin info` command line.

Each cluster with in federation can have their
own root credentials and as well as separate
regions. This way buckets get further restrictions
and allows for root creds to be not common
across clusters/data centers.

Existing data in etcd gets migrated to backend
on each clusters, upon start. Once done
users can change their config entries
independently.
2019-06-15 03:07:54 -07:00
Minio Trusted
da2887f914 Update yaml files to latest version RELEASE.2019-06-13T01-41-13Z 2019-06-13 01:46:00 +00:00
Harshavardhana
a075015293 doc: Merge large bucket with distributed docs (#7761) 2019-06-11 13:44:33 -07:00
Minio Trusted
d3a2efbf91 Update yaml files to latest version RELEASE.2019-06-11T00-44-33Z 2019-06-11 00:50:31 +00:00
Harshavardhana
002a205c9c Fix OPA result response handling (#7763)
Also update the document with updated rego policy
and updated OPA agent REST API.

This PR is to fix a regression caused by PR #7637
2019-06-10 17:06:32 -07:00
Harshavardhana
91ceae23d0 Add support for customizable user (#7569) 2019-06-10 20:27:42 +05:30
Harshavardhana
cb1566c6e6 S3 Gateway: Handle restricted access credentials (#7757) 2019-06-07 15:49:13 -07:00
Waldemar Quevedo
8b0c86298d Update import for NATS clients (#7743)
Signed-off-by: Waldemar Quevedo <wally@synadia.com>
2019-06-04 11:16:07 +05:30
Minio Trusted
df19546cc8 Update yaml files to latest version RELEASE.2019-06-04T01-15-58Z 2019-06-04 01:22:32 +00:00
Minio Trusted
48ffb7731a Update yaml files to latest version RELEASE.2019-06-01T03-46-14Z 2019-06-01 03:52:47 +00:00
Harshavardhana
0c16b1c9a7 Fix docs.min.io linking issues (#7710) 2019-05-30 17:19:25 -07:00
Harshavardhana
2c0b3cadfc Update go mod with sem versions of our libraries (#7687) 2019-05-29 16:35:12 -07:00
Praveen raj Mani
763fce909b Enable event persistence in kafka (#7633) 2019-05-29 13:19:48 -07:00
Harshavardhana
cb7f9ba286 Add disaggregated architecture document (#7691)
This document describes how to configure Spark,
Hive to use MinIO as replacement for HDFS.
2019-05-29 08:04:51 +05:30
Scott Edlund
f6fd407e47 Add limitation to b2.md (#7680)
The B2 gateway does not implement reporting md5sum as an etag response from PutObject.  Add to Known Limitations.
2019-05-27 21:04:43 -07:00
Minio Trusted
7a02faab72 Update yaml files to latest version RELEASE.2019-05-23T00-29-34Z 2019-05-23 00:35:13 +00:00
Dee Koder
e252114f06 Revert "cache: Rewrite to cache only on download (#7575)" (#7684)
This reverts commit a13b58f630.
2019-05-22 14:54:15 -07:00
poornas
a13b58f630 cache: Rewrite to cache only on download (#7575)
This will allow cache to consistently work for
server and gateways. Range GET requests will
be cached in the background after the request
is served from the backend.

Fixes: #7458, #7573, #6265, #6630
2019-05-22 08:30:27 +05:30
ebozduman
78be3f8947 Removes the incorrect coverage badge from the docs (#7651) 2019-05-16 12:11:49 +05:30
Praveen raj Mani
998f01fadc Support IPv6 hosts for postgres connections (#7168)
Fixes #7145
2019-05-03 17:31:33 +05:30
Eco
3517f64d20 Clarify server maximums via Federation (#7608)
To eliminate confusion about 32 server limit, clarify that the limit is only per cluster, but that unlimited clusters can be Federated
2019-05-01 16:20:25 -07:00
parnigot
ba76cd3268 Minor TLS documentation improvements (#7592)
Added the explicit name for private.key and public.crt
2019-04-29 12:56:34 -07:00
Minio Trusted
ab711fe1a2 Update yaml files to latest version RELEASE.2019-04-23T23-50-36Z 2019-04-23 23:55:31 +00:00
Minio Trusted
d42496cc74 Update yaml files to latest version RELEASE.2019-04-18T21-44-59Z 2019-04-18 21:52:22 +00:00
Minio Trusted
7686340621 Update yaml files to latest version RELEASE.2019-04-18T01-15-57Z 2019-04-18 01:22:00 +00:00
Harshavardhana
620e462413 Implement S3-HDFS gateway (#7440)
- [x] Support bucket and regular object operations
- [x] Supports Select API on HDFS
- [x] Implement multipart API support
- [x] Completion of ListObjects support
2019-04-17 09:52:08 -07:00
kannappanr
5ecac91a55
Replace Minio refs in docs with MinIO and links (#7494) 2019-04-09 11:39:42 -07:00
Minio Trusted
ea052a2a00 Update yaml files to latest version RELEASE.2019-04-09T01-22-30Z 2019-04-09 01:30:27 +00:00
Minio Trusted
082bd5f31f Update yaml files to latest version RELEASE.2019-04-04T18-31-46Z 2019-04-04 18:39:37 +00:00
Harshavardhana
619611933a
Remove policy nesting errors (#7449)
Policy nesting has been supported for a while
now, we should remove references of code and
docs indicating nesting is not allowed anymore.
2019-03-31 08:42:43 -07:00
Minio Trusted
8cf707779c Update yaml files to latest version RELEASE.2019-03-27T22-35-21Z 2019-03-27 22:41:57 +00:00
Minio Trusted
8af45f0135 Update yaml files to latest version RELEASE.2019-03-20T22-38-47Z 2019-03-20 22:49:06 +00:00
poornas
a0ee7be050 update link on README (#7361) 2019-03-14 15:18:12 +05:30
Minio Trusted
8377d00574 Update yaml files to latest version RELEASE.2019-03-13T21-59-47Z 2019-03-13 22:06:26 +00:00
Minio Trusted
6387407e79 Update yaml files to latest version RELEASE.2019-03-06T22-47-10Z 2019-03-06 22:56:46 +00:00
Harshavardhana
c3ca954684 Implement AssumeRole API for Minio users (#7267)
For actual API reference read here

https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html

Documentation is added and updated as well at docs/sts/assume-role.md

Fixes #6381
2019-02-27 17:46:55 -08:00
Minio Trusted
d3125857a8 Update yaml files to latest version RELEASE.2019-02-26T19-51-46Z 2019-02-26 19:58:09 +00:00
Praveen raj Mani
78d116c487 Event persistence for MQTT (#7268)
- The events will be persisted in queueStore if `queueDir` is set.
- Else, if queueDir is not set events persist in memory.

The events are replayed back when the mqtt broker is back online.
2019-02-25 18:01:13 -08:00
dcharbonnier
2fc341394d fix json syntax (#7285) 2019-02-25 13:02:59 +05:30
Harshavardhana
7923b83953 Support multiple-domains in MINIO_DOMAIN (#7274)
Fixes #7173
2019-02-23 08:48:01 +05:30
Minio Trusted
3aabe45fd9 Update yaml files to latest version RELEASE.2019-02-20T22-44-29Z 2019-02-20 22:49:40 +00:00
Minio Trusted
6d778a883f Update yaml files to latest version RELEASE.2019-02-14T00-21-45Z 2019-02-14 00:27:52 +00:00
Minio Trusted
9f9e0fe085 Update yaml files to latest version RELEASE.2019-02-12T21-58-47Z 2019-02-12 22:07:04 +00:00
Harshavardhana
9f87283cd5 Revert and bring back B2 gateway implementation (#7224)
This PR is simply a revert of 3265112d04
just for B2 gateway.
2019-02-12 12:44:22 +05:30
Harshavardhana
a8cd70f3e5 Remove GPL go-lzo dependency for parquet-go (#7220)
Also remove any other unused dependencies
2019-02-11 14:57:24 +05:30
Prashant Shahi
f75f707ff4 Added documentation for MINIO_HTTP_TRACE (#7207) 2019-02-10 11:57:43 +05:30
Adam
18c4ecbbef Fix mc event add event flag (#7210) 2019-02-08 16:18:34 -08:00
Minio Trusted
5fb813a5cc Update yaml files to latest version RELEASE.2019-02-06T21-16-36Z 2019-02-06 21:24:08 +00:00
Aditya Manthramurthy
f04f8bbc78 Add support for Timestamp data type in SQL Select (#7185)
This change adds support for casting strings to Timestamp via CAST:
`CAST('2010T' AS TIMESTAMP)`

It also implements the following date-time functions:
  - UTCNOW()
  - DATE_ADD()
  - DATE_DIFF()
  - EXTRACT()

For values passed to these functions, date-types are automatically
inferred.
2019-02-04 20:54:45 -08:00
Harshavardhana
e005910051 Add more information in our select docs (#7177) 2019-02-01 11:34:56 -08:00
Minio Trusted
9dc9f03c02 Update yaml files to latest version RELEASE.2019-01-31T00-31-19Z 2019-01-31 00:37:43 +00:00
poornas
d0015b4d66 update kms docs example to set a longer period for token renewal (#7149) 2019-01-29 08:04:07 -08:00
Praveen raj Mani
fad59da29d clientID removed in the MQTT config (#7157)
More than one client can't use the same clientID for MQTT connection. 
This causes problem in distributed deployments where config is shared 
across nodes, as each Minio instance tries to connect to MQTT using the
same clientID.

This commit removes the clientID field in config, and allows
MQTT client to create random clientID for each node.
2019-01-29 15:00:15 +05:30
Aditya Manthramurthy
2786055df4 Add new SQL parser to support S3 Select syntax (#7102)
- New parser written from scratch, allows easier and complete parsing
  of the full S3 Select SQL syntax. Parser definition is directly
  provided by the AST defined for the SQL grammar.

- Bring support to parse and interpret SQL involving JSON path
  expressions; evaluation of JSON path expressions will be
  subsequently added.

- Bring automatic type inference and conversion for untyped
  values (e.g. CSV data).
2019-01-28 17:59:48 -08:00
Minio Trusted
e8c18bc145 Update yaml files to latest version RELEASE.2019-01-23T23-18-58Z 2019-01-23 23:25:02 +00:00
Harshavardhana
bd25f31100 Use IAM creds only if endpoint is S3 (#7111)
Requirements like being able to run minio gateway in ec2
pointing to a Minio deployment wouldn't work properly
because IAM creds take precendence on ec2.

Add checks such that we only enable AWS specific features
if our backend URL points to actual AWS S3 not S3 compatible
endpoints.
2019-01-23 11:12:33 -08:00
Harshavardhana
55ef51a99d Vendorize all recent changes to minio-go (#7135)
- Default support for S3 dualstack endpoints (IPv6 support)
- Support granular policy conditionals in List operations
- Support proxy cookies for stickiness
2019-01-23 19:22:09 +05:30
Harshavardhana
3265112d04 Remove gateway implementations for manta, sia and b2 (#7115) 2019-01-20 08:10:58 -08:00
Alex Simenduev
6dd8a83c5a change credential chain order in s3 gateway to mimic official docs (#7091) 2019-01-17 10:31:51 -08:00
Minio Trusted
94c52e3816 Update yaml files to latest version RELEASE.2019-01-16T21-44-08Z 2019-01-16 21:51:40 +00:00
Harshavardhana
a2f66abbe8
Update STS API docs with Version query param (#7071) 2019-01-16 09:38:32 +05:30
Andreas Auernhammer
b28661b673 doc: add security documentation to provide some background info (#7028)
This commit adds some documentation about the design of the
SSE-C and SSE-S3 implementation. It describes how the Minio server
encrypt objects and manages keys.
2019-01-15 14:27:57 -08:00
Praveen raj Mani
6571641735 Persist offline mqtt events in the queueDir and replay (#7037) 2019-01-14 12:39:00 +05:30
Minio Trusted
de1d39e436 Update yaml files to latest version RELEASE.2019-01-10T00-21-20Z 2019-01-10 00:28:50 +00:00
poornas
5a80cbec2a Add double encryption at S3 gateway. (#6423)
This PR adds pass-through, single encryption at gateway and double
encryption support (gateway encryption with pass through of SSE
headers to backend).

If KMS is set up (either with Vault as KMS or using
MINIO_SSE_MASTER_KEY),gateway will automatically perform
single encryption. If MINIO_GATEWAY_SSE is set up in addition to
Vault KMS, double encryption is performed.When neither KMS nor
MINIO_GATEWAY_SSE is set, do a pass through to backend.

When double encryption is specified, MINIO_GATEWAY_SSE can be set to
"C" for SSE-C encryption at gateway and backend, "S3" for SSE-S3
encryption at gateway/backend or both to support more than one option.

Fixes #6323, #6696
2019-01-05 14:16:42 -08:00
Harshavardhana
2d19011a1d Add support for AssumeRoleWithWebIdentity (#6985) 2019-01-04 13:48:12 -08:00
Harshavardhana
e82dcd195c Deprecate config-dir bring in certs-dir for TLS configuration (#7033)
This PR is to provide indication that config-dir will be removed
in future and all users should migrate to new --certs-dir option

Fixes #7016
Fixes #7032
2019-01-02 10:05:16 -08:00