MyFavMirrors/minio
1
0
Fork 0
mirror of https://github.com/minio/minio.git synced 2025-11-10 14:09:48 -05:00
Code Issues Packages Projects Releases Wiki Activity

Migrate config to KV data format (#8392)

Browse Source 
- adding oauth support to MinIO browser (#8400) by @kanagaraj
- supports multi-line get/set/del for all config fields
- add support for comments, allow toggle
- add extensive validation of config before saving
- support MinIO browser to support proper claims, using STS tokens
- env support for all config parameters, legacy envs are also
  supported with all documentation now pointing to latest ENVs
- preserve accessKey/secretKey from FS mode setups
- add history support implements three APIs
  - ClearHistory
  - RestoreHistory
  - ListHistory
- add help command support for each config parameters
- all the bug fixes after migration to KV, and other bug
  fixes encountered during testing.
This commit is contained in:
Harshavardhana
2019-10-22 22:59:13 -07:00
committed by kannappanr
parent 8836d57e3c
commit ee4a6a823d
185 changed files with 8228 additions and 3597 deletions
Download Patch File Download Diff File Expand all files Collapse all files

480
docs/bucket/notifications/README.md
View File

@@ -32,59 +32,40 @@ Install RabbitMQ from [here](https://www.rabbitmq.com/).
### Step 1: Add AMQP endpoint to MinIO
The MinIO server configuration file is stored on the backend in json format. The AMQP configuration is located in the `amqp` key under the `notify` top-level key. Create a configuration key-value pair here for your AMQP instance. The key is a name for your AMQP endpoint, and the value is a collection of key-value parameters described in the table below.
The AMQP configuration is located under the sub-system `notify_amqp` top-level key. Create a configuration key-value pair here for your AMQP instance. The key is a name for your AMQP endpoint, and the value is a collection of key-value parameters described in the table below.
| Parameter | Type | Description |
| :------------- | :------- | :------------------------------------------------------------------------------- |
| `enable` | _bool_ | (Required) Is this server endpoint configuration active/enabled? |
| `url` | _string_ | (Required) AMQP server endpoint, e.g. `amqp://myuser:mypassword@localhost:5672` |
| `exchange` | _string_ | Name of the exchange. |
| `routingKey` | _string_ | Routing key for publishing. |
| `exchangeType` | _string_ | Kind of exchange. |
| `deliveryMode` | _uint8_ | Delivery mode for publishing. 0 or 1 - transient; 2 - persistent. |
| `mandatory` | _bool_ | Publishing related bool. |
| `immediate` | _bool_ | Publishing related bool. |
| `durable` | _bool_ | Exchange declaration related bool. |
| `internal` | _bool_ | Exchange declaration related bool. |
| `noWait` | _bool_ | Exchange declaration related bool. |
| `autoDeleted` | _bool_ | Exchange declaration related bool. |
| `queueDir` | _string_ | Persistent store for events when AMQP broker is offline |
| `queueLimit` | _int_ | Set the maximum event limit for the persistent store. The default limit is 10000 |
| Parameter | Description |
| :------------- | :------------------------------------------------------------------------------- |
| `state` | (Required) Is this server endpoint configuration active/enabled? |
| `url` | (Required) AMQP server endpoint, e.g. `amqp://myuser:mypassword@localhost:5672` |
| `exchange` | Name of the exchange. |
| `routing_key` | Routing key for publishing. |
| `exchange_type` | Kind of exchange. |
| `delivery_mode` | Delivery mode for publishing. 0 or 1 - transient; 2 - persistent. |
| `mandatory` | Publishing related bool. |
| `immediate` | Publishing related bool. |
| `durable` | Exchange declaration related bool. |
| `internal` | Exchange declaration related bool. |
| `no_wait` | Exchange declaration related bool. |
| `auto_deleted` | Exchange declaration related bool. |
| `queue_dir` | Persistent store for events when AMQP broker is offline |
| `queue_limit` | Set the maximum event limit for the persistent store. The default limit is 10000 |
MinIO supports persistent event store. The persistent store will backup events when the AMQP broker goes offline and replays it when the broker comes back online. The event store can be configured by setting the directory path in `queue_dir` field and the maximum limit of events in the queue_dir in `queue_limit` field. For eg, the `queue_dir` can be `/home/events` and `queue_limit` can be `1000`. By default, the `queue_limit` is set to 10000.
To update the configuration, use `mc admin config get notify_amqp` command to get the current configuration for `notify_amqp`.
```sh
$ mc admin config get myminio/ notify_amqp
notify_amqp:1 delivery_mode="0" exchange_type="" no_wait="off" queue_dir="" queue_limit="0" state="off" url="" auto_deleted="off" durable="off" exchange="" internal="off" mandatory="off" routing_key=""
```
Use `mc admin config set` command to update the configuration for the deployment.Restart the MinIO server to put the changes into effect. The server will print a line like `SQS ARNs: arn:minio:sqs::1:amqp` at start-up if there were no errors.
An example configuration for RabbitMQ is shown below:
```json
"amqp": {
"1": {
"enable": true,
"url": "amqp://myuser:mypassword@localhost:5672",
"exchange": "bucketevents",
"routingKey": "bucketlogs",
"exchangeType": "fanout",
"deliveryMode": 0,
"mandatory": false,
"immediate": false,
"durable": false,
"internal": false,
"noWait": false,
"autoDeleted": false,
"queueDir": "",
"queueLimit": 0
}
}
```
MinIO supports persistent event store. The persistent store will backup events when the AMQP broker goes offline and replays it when the broker comes back online. The event store can be configured by setting the directory path in `queueDir` field and the maximum limit of events in the queueDir in `queueLimit` field. For eg, the `queueDir` can be `/home/events` and `queueLimit` can be `1000`. By default, the `queueLimit` is set to 10000.
To update the configuration, use `mc admin config get` command to get the current configuration file for the minio deployment in json format, and save it locally.
```sh
$ mc admin config get myminio/ > /tmp/myconfig
```
After updating the AMQP configuration in /tmp/myconfig , use `mc admin config set` command to update the configuration for the deployment.Restart the MinIO server to put the changes into effect. The server will print a line like `SQS ARNs: arn:minio:sqs::1:amqp` at start-up if there were no errors.
```sh
$ mc admin config set myminio < /tmp/myconfig
$ mc admin config set myminio/ notify_amqp:1 exchange="bucketevents" exchange_type="fanout" mandatory="false" no_wait="false" state="on" url="amqp://myuser:mypassword@localhost:5672" auto_deleted="false" delivery_mode="0" durable="false" internal="false" routing_key="bucketlogs"
```
MinIO supports all the exchanges available in [RabbitMQ](https://www.rabbitmq.com/). For this setup, we are using `fanout` exchange.
@@ -97,7 +78,7 @@ We will enable bucket event notification to trigger whenever a JPEG image is upl
```
mc mb myminio/images
mc event add myminio/images arn:minio:sqs::1:amqp --suffix .jpg
mc event add myminio/images arn:minio:sqs::1:amqp --suffix .jpg
mc event list myminio/images
arn:minio:sqs::1:amqp s3:ObjectCreated:*,s3:ObjectRemoved:* Filter: suffix=”.jpg”
```
@@ -162,48 +143,33 @@ Install an MQTT Broker from [here](https://mosquitto.org/).
### Step 1: Add MQTT endpoint to MinIO
The MinIO server configuration file is stored on the backend in json format. The MQTT configuration is located in the `mqtt` key under the `notify` top-level key. Create a configuration key-value pair here for your MQTT instance. The key is a name for your MQTT endpoint, and the value is a collection of key-value parameters described in the table below.
The MQTT configuration is located as `notify_mqtt` key. Create a configuration key-value pair here for your MQTT instance. The key is a name for your MQTT endpoint, and the value is a collection of key-value parameters described in the table below.
| Parameter | Type | Description |
| :----------- | :------- | :------------------------------------------------------------------------------- |
| `enable` | _bool_ | (Required) Is this server endpoint configuration active/enabled? |
| `broker` | _string_ | (Required) MQTT server endpoint, e.g. `tcp://localhost:1883` |
| `topic` | _string_ | (Required) Name of the MQTT topic to publish on, e.g. `minio` |
| `qos` | _int_ | Set the Quality of Service Level |
| `username` | _string_ | Username to connect to the MQTT server (if required) |
| `password` | _string_ | Password to connect to the MQTT server (if required) |
| `queueDir` | _string_ | Persistent store for events when MQTT broker is offline |
| `queueLimit` | _int_ | Set the maximum event limit for the persistent store. The default limit is 10000 |
| Parameter | Description |
| :----------- | :------------------------------------------------------------------------------- |
| `state` | (Required) Is this server endpoint configuration active/enabled? |
| `broker` | (Required) MQTT server endpoint, e.g. `tcp://localhost:1883` |
| `topic` | (Required) Name of the MQTT topic to publish on, e.g. `minio` |
| `qos` | Set the Quality of Service Level |
| `username` | Username to connect to the MQTT server (if required) |
| `password` | Password to connect to the MQTT server (if required) |
| `queue_dir` | Persistent store for events when MQTT broker is offline |
| `queue_limit` | Set the maximum event limit for the persistent store. The default limit is 10000 |
An example configuration for MQTT is shown below:
```json
"mqtt": {
"1": {
"enable": true,
"broker": "tcp://localhost:1883",
"topic": "minio",
"qos": 1,
"username": "",
"password": "",
"queueDir": "",
"queueLimit": 0
}
}
```
MinIO supports persistent event store. The persistent store will backup events when the MQTT broker goes offline and replays it when the broker comes back online. The event store can be configured by setting the directory path in `queue_dir` field and the maximum limit of events in the queue_dir in `queue_limit` field. For eg, the `queue_dir` can be `/home/events` and `queue_limit` can be `1000`. By default, the `queue_limit` is set to 10000.
MinIO supports persistent event store. The persistent store will backup events when the MQTT broker goes offline and replays it when the broker comes back online. The event store can be configured by setting the directory path in `queueDir` field and the maximum limit of events in the queueDir in `queueLimit` field. For eg, the `queueDir` can be `/home/events` and `queueLimit` can be `1000`. By default, the `queueLimit` is set to 10000.
To update the configuration, use `mc admin config get` command to get the current configuration file for the minio deployment in json format, and save it locally.
To update the configuration, use `mc admin config get` command to get the current configuration.
```sh
$ mc admin config get myminio/ > /tmp/myconfig
$ mc admin config get myminio/ notify_mqtt
notify_mqtt:1 broker="" password="" queue_dir="" queue_limit="0" reconnect_interval="0s" state="off" keep_alive_interval="0s" qos="0" topic="" username=""
```
After updating the MQTT configuration in /tmp/myconfig , use `mc admin config set` command to update the configuration for the deployment.Restart the MinIO server to put the changes into effect. The server will print a line like `SQS ARNs: arn:minio:sqs::1:mqtt` at start-up if there were no errors.
Use `mc admin config set` command to update the configuration for the deployment. Restart the MinIO server to put the changes into effect. The server will print a line like `SQS ARNs: arn:minio:sqs::1:mqtt` at start-up if there were no errors.
```sh
$ mc admin config set myminio < /tmp/myconfig
$ mc admin config set myminio notify_mqtt:1 broker="tcp://localhost:1883" password="" queue_dir="" queue_limit="0" reconnect_interval="0s" state="on" keep_alive_interval="0s" qos="1" topic="minio" username=""
```
MinIO supports any MQTT server that supports MQTT 3.1 or 3.1.1 and can connect to them over TCP, TLS, or a Websocket connection using `tcp://`, `tls://`, or `ws://` respectively as the scheme for the broker url. See the [Go Client](http://www.eclipse.org/paho/clients/golang/) documentation for more information.
@@ -289,46 +255,34 @@ MinIO requires a 5.x series version of Elasticsearch. This is the latest major r
### Step 2: Add Elasticsearch endpoint to MinIO
The MinIO server configuration file is stored on the backend in json format. The Elasticsearch configuration is located in the `elasticsearch` key under the `notify` top-level key. Create a configuration key-value pair here for your Elasticsearch instance. The key is a name for your Elasticsearch endpoint, and the value is a collection of key-value parameters described in the table below.
The Elasticsearch configuration is located in the `notify_elasticsearch` key. Create a configuration key-value pair here for your Elasticsearch instance. The key is a name for your Elasticsearch endpoint, and the value is a collection of key-value parameters described in the table below.
| Parameter | Type | Description |
| :----------- | :------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `enable` | _bool_ | (Required) Is this server endpoint configuration active/enabled? |
| `format` | _string_ | (Required) Either `namespace` or `access`. |
| `url` | _string_ | (Required) The Elasticsearch server's address, with optional authentication info. For example: `http://localhost:9200` or with authentication info `http://elastic:MagicWord@127.0.0.1:9200`. |
| `index` | _string_ | (Required) The name of an Elasticsearch index in which MinIO will store documents. |
| `queueDir` | _string_ | Persistent store for events when Elasticsearch broker is offline |
| `queueLimit` | _int_ | Set the maximum event limit for the persistent store. The default limit is 10000 |
| Parameter | Description |
| :----------- | :------------------------------------------------------------------------------- |
| `state` | (Required) Is this server endpoint configuration active/enabled? |
| `format` | (Required) Either `namespace` or `access`. |
| `url` | (Required) The Elasticsearch server's address, with optional authentication info. |
| `index` | (Required) The name of an Elasticsearch index in which MinIO will store documents. |
| `queue_dir` | Persistent store for events when Elasticsearch broker is offline |
| `queue_limit` | Set the maximum event limit for the persistent store. The default limit is 10000 |
An example of Elasticsearch configuration is as follows:
For example: `http://localhost:9200` or with authentication info `http://elastic:MagicWord@127.0.0.1:9200`.
```json
"elasticsearch": {
"1": {
"enable": true,
"format": "namespace",
"url": "http://127.0.0.1:9200",
"index": "minio_events",
"queueDir": "",
"queueLimit": 0
}
},
```
MinIO supports persistent event store. The persistent store will backup events when the Elasticsearch broker goes offline and replays it when the broker comes back online. The event store can be configured by setting the directory path in `queueDir` field and the maximum limit of events in the queueDir in `queueLimit` field. For eg, the `queueDir` can be `/home/events` and `queueLimit` can be `1000`. By default, the `queueLimit` is set to 10000.
MinIO supports persistent event store. The persistent store will backup events when the Elasticsearch broker goes offline and replays it when the broker comes back online. The event store can be configured by setting the directory path in `queue_dir` field and the maximum limit of events in the queue_dir in `queue_limit` field. For eg, the `queue_dir` can be `/home/events` and `queue_limit` can be `1000`. By default, the `queue_limit` is set to 10000.
If Elasticsearch has authentication enabled, the credentials can be supplied to MinIO via the `url` parameter formatted as `PROTO://USERNAME:PASSWORD@ELASTICSEARCH_HOST:PORT`.
To update the configuration, use `mc admin config get` command to get the current configuration file for the minio deployment in json format, and save it locally.
To update the configuration, use `mc admin config get` command to get the current configuration.
```sh
$ mc admin config get myminio/ > /tmp/myconfig
$ mc admin config get myminio/ notify_elasticsearch
notify_elasticsearch:1 queue_limit="0" state="off" url="" format="namespace" index="" queue_dir=""
```
After updating the Elasticsearch configuration in /tmp/myconfig , use `mc admin config set` command to update the configuration for the deployment.Restart the MinIO server to put the changes into effect. The server will print a line like `SQS ARNs: arn:minio:sqs::1:elasticsearch` at start-up if there were no errors.
Use `mc admin config set` command to update the configuration for the deployment. Restart the MinIO server to put the changes into effect. The server will print a line like `SQS ARNs: arn:minio:sqs::1:elasticsearch` at start-up if there were no errors.
```sh
$ mc admin config set myminio < /tmp/myconfig
$ mc admin config set myminio notify_elasticsearch:1 queue_limit="0" state="on" url="http://127.0.0.1:9200" format="namespace" index="minio_events" queue_dir=""
```
Note that, you can add as many Elasticsearch server endpoint configurations as needed by providing an identifier (like "1" in the example above) for the Elasticsearch instance and an object of per-server configuration parameters.
@@ -448,42 +402,28 @@ The steps below show how to use this notification target in `namespace` and `acc
The MinIO server configuration file is stored on the backend in json format.The Redis configuration is located in the `redis` key under the `notify` top-level key. Create a configuration key-value pair here for your Redis instance. The key is a name for your Redis endpoint, and the value is a collection of key-value parameters described in the table below.
| Parameter | Type | Description |
| :--------- | :------- | :------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `enable` | _bool_ | (Required) Is this server endpoint configuration active/enabled? |
| `format` | _string_ | (Required) Either `namespace` or `access`. |
| `address` | _string_ | (Required) The Redis server's address. For example: `localhost:6379`. |
| `password` | _string_ | (Optional) The Redis server's password. |
| `key` | _string_ | (Required) The name of the redis key under which events are stored. A hash is used in case of `namespace` format and a list in case of `access` format. |
| Parameter | Description |
| :--------- | :------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `state` | (Required) Is this server endpoint configuration active/enabled? |
| `format` | (Required) Either `namespace` or `access`. |
| `address` | (Required) The Redis server's address. For example: `localhost:6379`. |
| `password` | (Optional) The Redis server's password. |
| `key` | (Required) The name of the redis key under which events are stored. A hash is used in case of `namespace` format and a list in case of `access` format. |
| | |
An example of Redis configuration is as follows:
MinIO supports persistent event store. The persistent store will backup events when the Redis broker goes offline and replays it when the broker comes back online. The event store can be configured by setting the directory path in `queue_dir` field and the maximum limit of events in the queue_dir in `queue_limit` field. For eg, the `queue_dir` can be `/home/events` and `queue_limit` can be `1000`. By default, the `queue_limit` is set to 10000.
```json
"redis": {
"1": {
"enable": true,
"format": "namespace",
"address": "127.0.0.1:6379",
"password": "yoursecret",
"key": "bucketevents",
"queueDir": "",
"queueLimit": 0
}
}
```
MinIO supports persistent event store. The persistent store will backup events when the Redis broker goes offline and replays it when the broker comes back online. The event store can be configured by setting the directory path in `queueDir` field and the maximum limit of events in the queueDir in `queueLimit` field. For eg, the `queueDir` can be `/home/events` and `queueLimit` can be `1000`. By default, the `queueLimit` is set to 10000.
To update the configuration, use `mc admin config get` command to get the current configuration file for the minio deployment in json format, and save it locally.
To update the configuration, use `mc admin config get` command to get the current configuration.
```sh
$ mc admin config get myminio/ > /tmp/myconfig
$ mc admin config get myminio/ notify_redis
notify_redis:1 address="" format="namespace" key="" password="" queue_dir="" queue_limit="0" state="off"
```
After updating the Redis configuration in /tmp/myconfig , use `mc admin config set` command to update the configuration for the deployment.Restart the MinIO server to put the changes into effect. The server will print a line like `SQS ARNs: arn:minio:sqs::1:redis` at start-up if there were no errors.
Use `mc admin config set` command to update the configuration for the deployment.Restart the MinIO server to put the changes into effect. The server will print a line like `SQS ARNs: arn:minio:sqs::1:redis` at start-up if there were no errors.
```sh
$ mc admin config set myminio < /tmp/myconfig
$ mc admin config set myminio/ notify_redis:1 address="127.0.0.1:6379" format="namespace" key="bucketevents" password="yoursecret" queue_dir="" queue_limit="0" state="on"
```
Note that, you can add as many Redis server endpoint configurations as needed by providing an identifier (like "1" in the example above) for the Redis instance and an object of per-server configuration parameters.
@@ -498,7 +438,7 @@ With the `mc` tool, the configuration is very simple to add. Let us say that the
```
mc mb myminio/images
mc event add myminio/images arn:minio:sqs::1:redis --suffix .jpg
mc event add myminio/images arn:minio:sqs::1:redis --suffix .jpg
mc event list myminio/images
arn:minio:sqs::1:redis s3:ObjectCreated:*,s3:ObjectRemoved:* Filter: suffix=”.jpg”
```
@@ -540,71 +480,24 @@ Install NATS from [here](http://nats.io/).
### Step 1: Add NATS endpoint to MinIO
The NATS configuration block in `config.json` is as follows:
MinIO supports persistent event store. The persistent store will backup events when the NATS broker goes offline and replays it when the broker comes back online. The event store can be configured by setting the directory path in `queue_dir` field and the maximum limit of events in the queue_dir in `queue_limit` field. For eg, the `queue_dir` can be `/home/events` and `queue_limit` can be `1000`. By default, the `queue_limit` is set to 10000.
```
"nats": {
"1": {
"enable": true,
"address": "0.0.0.0:4222",
"subject": "bucketevents",
"username": "yourusername",
"password": "yoursecret",
"token": "",
"secure": false,
"pingInterval": 0,
"queueDir": "",
"queueLimit": 0,
"streaming": {
"enable": false,
"clusterID": "",
"async": false,
"maxPubAcksInflight": 0
}
}
},
```
MinIO supports persistent event store. The persistent store will backup events when the NATS broker goes offline and replays it when the broker comes back online. The event store can be configured by setting the directory path in `queueDir` field and the maximum limit of events in the queueDir in `queueLimit` field. For eg, the `queueDir` can be `/home/events` and `queueLimit` can be `1000`. By default, the `queueLimit` is set to 10000.
To update the configuration, use `mc admin config get` command to get the current configuration file for the minio deployment in json format, and save it locally.
To update the configuration, use `mc admin config get` command to get the current configuration file for the minio deployment.
```sh
$ mc admin config get myminio/ > /tmp/myconfig
$ mc admin config get myminio/ notify_nats
notify_nats:1 password="yoursecret" streaming_max_pub_acks_in_flight="10" subject="" address="0.0.0.0:4222" state="on" token="" username="yourusername" ping_interval="0" queue_limit="0" secure="off" streaming_async="on" queue_dir="" streaming_cluster_id="test-cluster" streaming_enable="on"
```
After updating the NATS configuration in /tmp/myconfig , use `mc admin config set` command to update the configuration for the deployment.Restart MinIO server to reflect config changes. `bucketevents` is the subject used by NATS in this example.
Use `mc admin config set` command to update the configuration for the deployment.Restart MinIO server to reflect config changes. `bucketevents` is the subject used by NATS in this example.
```sh
$ mc admin config set myminio < /tmp/myconfig
$ mc admin config set myminio notify_nats:1 password="yoursecret" streaming_max_pub_acks_in_flight="10" subject="" address="0.0.0.0:4222" state="on" token="" username="yourusername" ping_interval="0" queue_limit="0" secure="off" streaming_async="on" queue_dir="" streaming_cluster_id="test-cluster" streaming_enable="on"
```
MinIO server also supports [NATS Streaming mode](http://nats.io/documentation/streaming/nats-streaming-intro/) that offers additional functionality like `At-least-once-delivery`, and `Publisher rate limiting`. To configure MinIO server to send notifications to NATS Streaming server, update the MinIO server configuration file as follows:
```
"nats": {
"1": {
"enable": true,
"address": "0.0.0.0:4222",
"subject": "bucketevents",
"username": "yourusername",
"password": "yoursecret",
"token": "",
"secure": false,
"pingInterval": 0,
"queueDir": "",
"queueLimit": 0,
"streaming": {
"enable": true,
"clusterID": "test-cluster",
"async": true,
"maxPubAcksInflight": 10
}
}
},
```
Read more about sections `clusterID`, `clientID` on [NATS documentation](https://github.com/nats-io/nats-streaming-server/blob/master/README.md). Section `maxPubAcksInflight` is explained [here](https://github.com/nats-io/stan.go#publisher-rate-limiting).
Read more about sections `cluster_id`, `client_id` on [NATS documentation](https://github.com/nats-io/nats-streaming-server/blob/master/README.md). Section `maxPubAcksInflight` is explained [here](https://github.com/nats-io/stan.go#publisher-rate-limiting).
### Step 2: Enable bucket notification using MinIO client
@@ -612,7 +505,7 @@ We will enable bucket event notification to trigger whenever a JPEG image is upl
```
mc mb myminio/images
mc event add myminio/images arn:minio:sqs::1:nats --suffix .jpg
mc event add myminio/images arn:minio:sqs::1:nats --suffix .jpg
mc event list myminio/images
arn:minio:sqs::1:nats s3:ObjectCreated:*,s3:ObjectRemoved:* Filter: suffix=”.jpg”
```
@@ -767,53 +660,35 @@ MinIO requires PostgreSQL version 9.5 or above. MinIO uses the [`INSERT ON CONFL
### Step 2: Add PostgreSQL endpoint to MinIO
The MinIO server configuration file is stored on the backend in json format. The PostgreSQL configuration is located in the `postgresql` key under the `notify` top-level key. Create a configuration key-value pair here for your PostgreSQL instance. The key is a name for your PostgreSQL endpoint, and the value is a collection of key-value parameters described in the table below.
The PostgreSQL configuration is located in the `notify_postgresql` key. Create a configuration key-value pair here for your PostgreSQL instance. The key is a name for your PostgreSQL endpoint, and the value is a collection of key-value parameters described in the table below.
| Parameter | Type | Description |
| :----------------- | :------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `enable` | _bool_ | (Required) Is this server endpoint configuration active/enabled? |
| `format` | _string_ | (Required) Either `namespace` or `access`. |
| `connectionString` | _string_ | (Optional) [Connection string parameters](https://godoc.org/github.com/lib/pq#hdr-Connection_String_Parameters) for the PostgreSQL server. Can be used to set `sslmode` for example. |
| `table` | _string_ | (Required) Table name in which events will be stored/updated. If the table does not exist, the MinIO server creates it at start-up. |
| `host` | _string_ | (Optional) Host name of the PostgreSQL server. Defaults to `localhost`. IPv6 host should be enclosed with `[` and `]` |
| `port` | _string_ | (Optional) Port on which to connect to PostgreSQL server. Defaults to `5432`. |
| `user` | _string_ | (Optional) Database user name. Defaults to user running the server process. |
| `password` | _string_ | (Optional) Database password. |
| `database` | _string_ | (Optional) Database name. |
| Parameter | Description |
| :----------------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `state` | (Required) Is this server endpoint configuration active/enabled? |
| `format` | (Required) Either `namespace` or `access`. |
| `connection_string` | (Optional) [Connection string parameters](https://godoc.org/github.com/lib/pq#hdr-Connection_String_Parameters) for the PostgreSQL server. Can be used to set `sslmode` for example. |
| `table` | (Required) Table name in which events will be stored/updated. If the table does not exist, the MinIO server creates it at start-up. |
| `host` | (Optional) Host name of the PostgreSQL server. Defaults to `localhost`. IPv6 host should be enclosed with `[` and `]` |
| `port` | (Optional) Port on which to connect to PostgreSQL server. Defaults to `5432`. |
| `user` | (Optional) Database user name. Defaults to user running the server process. |
| `password` | (Optional) Database password. |
| `database` | (Optional) Database name. |
| | |
An example of PostgreSQL configuration is as follows:
```
"postgresql": {
"1": {
"enable": true,
"format": "namespace",
"connectionString": "sslmode=disable",
"table": "bucketevents",
"host": "127.0.0.1",
"port": "5432",
"user": "postgres",
"password": "password",
"database": "minio_events",
"queueDir": "",
"queueLimit": 0
}
}
```
MinIO supports persistent event store. The persistent store will backup events when the PostgreSQL connection goes offline and replays it when the broker comes back online. The event store can be configured by setting the directory path in `queueDir` field and the maximum limit of events in the queueDir in `queueLimit` field. For eg, the `queueDir` can be `/home/events` and `queueLimit` can be `1000`. By default, the `queueLimit` is set to 10000.
MinIO supports persistent event store. The persistent store will backup events when the PostgreSQL connection goes offline and replays it when the broker comes back online. The event store can be configured by setting the directory path in `queue_dir` field and the maximum limit of events in the queue_dir in `queue_limit` field. For eg, the `queue_dir` can be `/home/events` and `queue_limit` can be `1000`. By default, the `queue_limit` is set to 10000.
Note that for illustration here, we have disabled SSL. In the interest of security, for production this is not recommended.
To update the configuration, use `mc admin config get` command to get the current configuration file for the minio deployment in json format, and save it locally.
To update the configuration, use `mc admin config get` command to get the current configuration.
```sh
$ mc admin config get myminio/ > /tmp/myconfig
$ mc admin config get myminio notify_postgres
notify_postgres:1 password="" port="" queue_dir="" connection_string="" host="" queue_limit="0" state="off" table="" username="" database="" format="namespace"
```
After updating the Postgres configuration in /tmp/myconfig , use `mc admin config set` command to update the configuration for the deployment.Restart the MinIO server to put the changes into effect. The server will print a line like `SQS ARNs: arn:minio:sqs::1:postgresql` at start-up if there were no errors.
Use `mc admin config set` command to update the configuration for the deployment. Restart the MinIO server to put the changes into effect. The server will print a line like `SQS ARNs: arn:minio:sqs::1:postgresql` at start-up if there were no errors.
```sh
$ mc admin config set myminio < /tmp/myconfig
$ mc admin config set myminio notify_postgres:1 password="password" port="5432" queue_dir="" connection_string="sslmode=disable" host="127.0.0.1" queue_limit="0" state="on" table="bucketevents" username="postgres" database="minio_events" format="namespace"
```
Note that, you can add as many PostgreSQL server endpoint configurations as needed by providing an identifier (like "1" in the example above) for the PostgreSQL instance and an object of per-server configuration parameters.
@@ -877,53 +752,32 @@ MinIO requires MySQL version 5.7.8 or above. MinIO uses the [JSON](https://dev.m
### Step 2: Add MySQL server endpoint configuration to MinIO
The MinIO server configuration file is stored on the backend in json format. The MySQL configuration is located in the `mysql` key under the `notify` top-level key. Create a configuration key-value pair here for your MySQL instance. The key is a name for your MySQL endpoint, and the value is a collection of key-value parameters described in the table below.
The MySQL configuration is located in the `notify_mysql` key. Create a configuration key-value pair here for your MySQL instance. The key is a name for your MySQL endpoint, and the value is a collection of key-value parameters described in the table below.
| Parameter | Type | Description |
| :---------- | :------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `enable` | _bool_ | (Required) Is this server endpoint configuration active/enabled? |
| `format` | _string_ | (Required) Either `namespace` or `access`. |
| `dsnString` | _string_ | (Optional) [Data-Source-Name connection string](https://github.com/go-sql-driver/mysql#dsn-data-source-name) for the MySQL server. If not specified, the connection information specified by the `host`, `port`, `user`, `password` and `database` parameters are used. |
| `table` | _string_ | (Required) Table name in which events will be stored/updated. If the table does not exist, the MinIO server creates it at start-up. |
| `host` | _string_ | Host name of the MySQL server (used only if `dsnString` is empty). |
| `port` | _string_ | Port on which to connect to the MySQL server (used only if `dsnString` is empty). |
| `user` | _string_ | Database user-name (used only if `dsnString` is empty). |
| `password` | _string_ | Database password (used only if `dsnString` is empty). |
| `database` | _string_ | Database name (used only if `dsnString` is empty). |
| Parameter | Description | | :---------- | :-------------- | | `state` | (Required) Is this server endpoint configuration active/enabled? |
| `format` | (Required) Either `namespace` or `access`. | | `dsn_string` | (Optional) [Data-Source-Name connection string](https://github.com/go-sql-driver/mysql#dsn-data-source-name) for the MySQL server. |
| `table` | (Required) Table name in which events will be stored/updated. If the table does not exist, the MinIO server creates it at start-up. |
| `host` | Host name of the MySQL server (used only if `dsnString` is empty). |
| `port` | Port on which to connect to the MySQL server (used only if `dsn_string` is empty). |
| `user` | Database user-name (used only if `dsnString` is empty). |
| `password` | Database password (used only if `dsnString` is empty). |
| `database` | Database name (used only if `dsnString` is empty). |
An example of MySQL configuration is as follows:
`dns_string` is optional, if not specified, the connection information specified by the `host`, `port`, `user`, `password` and `database` parameters are used.
```json
"mysql": {
"1": {
"enable": true,
"dsnString": "",
"format": "namespace",
"table": "minio_images",
"host": "172.17.0.1",
"port": "3306",
"user": "root",
"password": "password",
"database": "miniodb",
"queueDir": "",
"queueLimit": 0
}
}
```
MinIO supports persistent event store. The persistent store will backup events when the MySQL connection goes offline and replays it when the broker comes back online. The event store can be configured by setting the directory path in `queue_dir` field and the maximum limit of events in the queue_dir in `queue_limit` field. For eg, the `queue_dir` can be `/home/events` and `queue_limit` can be `1000`. By default, the `queue_limit` is set to 10000.
MinIO supports persistent event store. The persistent store will backup events when the MySQL connection goes offline and replays it when the broker comes back online. The event store can be configured by setting the directory path in `queueDir` field and the maximum limit of events in the queueDir in `queueLimit` field. For eg, the `queueDir` can be `/home/events` and `queueLimit` can be `1000`. By default, the `queueLimit` is set to 10000.
To update the configuration, use `mc admin config get` command to get the current configuration file for the minio deployment in json format, and save it locally.
To update the configuration, use `mc admin config get` command to get the current configuration.
```sh
$ mc admin config get myminio/ > /tmp/myconfig
$ mc admin config get myminio/ notify_mysql
notify_mysql:1 table="" database="" format="namespace" password="" port="" queue_dir="" queue_limit="0" state="off" username="" dsn_string="" host=""
```
After updating the MySQL configuration in /tmp/myconfig , use `mc admin config set` command to update the configuration for the deployment.Restart the MinIO server to put the changes into effect. The server will print a line like `SQS ARNs: arn:minio:sqs::1:mysql` at start-up if there were no errors.
Use `mc admin config set` command to update the configuration for the deployment. Restart the MinIO server to put the changes into effect. The server will print a line like `SQS ARNs: arn:minio:sqs::1:mysql` at start-up if there were no errors.
```sh
$ mc admin config set myminio < /tmp/myconfig
$ mc admin config set myminio notify_mysql:1 table="minio_images" database="miniodb" format="namespace" password="" port="3306" queue_dir="" queue_limit="0" state="on" username="root" dsn_string="" host="172.17.0.1"
```
Note that, you can add as many MySQL server endpoint configurations as needed by providing an identifier (like "1" in the example above) for the MySQL instance and an object of per-server configuration parameters.
@@ -980,42 +834,19 @@ MinIO requires Kafka version 0.10 or 0.9. Internally MinIO uses the [Shopify/sar
### Step 2: Add Kafka endpoint to MinIO
The MinIO server configuration file is stored on the backend in json format. Update the kafka configuration block in `config.json` as follows:
MinIO supports persistent event store. The persistent store will backup events when the kafka broker goes offline and replays it when the broker comes back online. The event store can be configured by setting the directory path in `queue_dir` field and the maximum limit of events in the queue_dir in `queue_limit` field. For eg, the `queue_dir` can be `/home/events` and `queue_limit` can be `1000`. By default, the `queue_limit` is set to 10000.
```
"kafka": {
"1": {
"enable": true,
"brokers": ["localhost:9092"],
"topic": "bucketevents",
"queueDir": "",
"queueLimit": 0,
"tls": {
"enable": false,
"skipVerify": false,
"clientAuth": 0
},
"sasl": {
"enable": false,
"username": "",
"password": ""
}
}
}
```
MinIO supports persistent event store. The persistent store will backup events when the kafka broker goes offline and replays it when the broker comes back online. The event store can be configured by setting the directory path in `queueDir` field and the maximum limit of events in the queueDir in `queueLimit` field. For eg, the `queueDir` can be `/home/events` and `queueLimit` can be `1000`. By default, the `queueLimit` is set to 10000.
To update the configuration, use `mc admin config get` command to get the current configuration file for the minio deployment in json format, and save it locally.
To update the configuration, use `mc admin config get` command to get the current configuration.
```sh
$ mc admin config get myminio/ > /tmp/myconfig
$ mc admin config get myminio/ notify_kafka
notify_kafka:1 tls_skip_verify="off" state="off" queue_dir="" queue_limit="0" sasl_enable="off" sasl_password="" sasl_username="" tls_client_auth="0" tls_enable="off" brokers="" topic=""
```
After updating the Kafka configuration in /tmp/myconfig , use `mc admin config set` command to update the configuration for the deployment.Restart the MinIO server to put the changes into effect. The server will print a line like `SQS ARNs: arn:minio:sqs::1:kafka` at start-up if there were no errors.`bucketevents` is the topic used by kafka in this example.
Use `mc admin config set` command to update the configuration for the deployment. Restart the MinIO server to put the changes into effect. The server will print a line like `SQS ARNs: arn:minio:sqs::1:kafka` at start-up if there were no errors.`bucketevents` is the topic used by kafka in this example.
```sh
$ mc admin config set myminio < /tmp/myconfig
$ mc admin config set myminio notify_kafka:1 tls_skip_verify="off" state="on" queue_dir="" queue_limit="0" sasl_enable="off" sasl_password="" sasl_username="" tls_client_auth="0" tls_enable="off" brokers="localhost:9092,localhost:9093" topic="bucketevents"
```
### Step 3: Enable bucket notification using MinIO client
@@ -1110,30 +941,19 @@ kafkacat -b localhost:9092 -t bucketevents
### Step 1: Add Webhook endpoint to MinIO
The MinIO server configuration file is stored on the backend in json format. Update the Webhook configuration block in `config.json` as follows
MinIO supports persistent event store. The persistent store will backup events when the webhook goes offline and replays it when the broker comes back online. The event store can be configured by setting the directory path in `queue_dir` field and the maximum limit of events in the queue_dir in `queue_limit` field. For eg, the `queue_dir` can be `/home/events` and `queue_limit` can be `1000`. By default, the `queue_limit` is set to 10000.
```
"webhook": {
"1": {
"enable": true,
"endpoint": "http://localhost:3000/",
"queueDir": "",
"queueLimit": 0
}
```
MinIO supports persistent event store. The persistent store will backup events when the webhook goes offline and replays it when the broker comes back online. The event store can be configured by setting the directory path in `queueDir` field and the maximum limit of events in the queueDir in `queueLimit` field. For eg, the `queueDir` can be `/home/events` and `queueLimit` can be `1000`. By default, the `queueLimit` is set to 10000.
To update the configuration, use `mc admin config get` command to get the current configuration file for the minio deployment in json format, and save it locally.
To update the configuration, use `mc admin config get` command to get the current configuration.
```sh
$ mc admin config get myminio/ > /tmp/myconfig
$ mc admin config get myminio/ notify_webhook
notify_webhook:1 queue_limit="0" state="off" endpoint="" queue_dir=""
```
After updating the webhook configuration in /tmp/myconfig , use `mc admin config set` command to update the configuration for the deployment.Here the endpoint is the server listening for webhook notifications. Save the file and restart the MinIO server for changes to take effect. Note that the endpoint needs to be live and reachable when you restart your MinIO server.
Use `mc admin config set` command to update the configuration for the deployment. Here the endpoint is the server listening for webhook notifications. Save the settings and restart the MinIO server for changes to take effect. Note that the endpoint needs to be live and reachable when you restart your MinIO server.
```sh
$ mc admin config set myminio < /tmp/myconfig
$ mc admin config set myminio notify_webhook:1 queue_limit="0" state="on" endpoint="http://localhost:3000" queue_dir=""
```
### Step 2: Enable bucket notification using MinIO client
@@ -1200,39 +1020,19 @@ docker run --rm -p 4150-4151:4150-4151 nsqio/nsq /nsqd
### Step 1: Add NSQ endpoint to MinIO
The MinIO server configuration file is stored on the backend in json format. The NSQ configuration is located in the `nsq` key under the `notify` top-level key. Create a configuration key-value pair here for your NSQ instance. The key is a name for your NSQ endpoint, and the value is a collection of key-value parameters.
MinIO supports persistent event store. The persistent store will backup events when the NSQ broker goes offline and replays it when the broker comes back online. The event store can be configured by setting the directory path in `queue_dir` field and the maximum limit of events in the queue_dir in `queue_limit` field. For eg, the `queue_dir` can be `/home/events` and `queue_limit` can be `1000`. By default, the `queue_limit` is set to 10000.
An example configuration for NSQ is shown below:
```json
"nsq": {
"1": {
"enable": true,
"nsqdAddress": "127.0.0.1:4150",
"topic": "minio",
"tls": {
"enable": false,
"skipVerify": true
},
"queueDir": "",
"queueLimit": 0
}
}
MinIO supports persistent event store. The persistent store will backup events when the NSQ broker goes offline and replays it when the broker comes back online. The event store can be configured by setting the directory path in `queueDir` field and the maximum limit of events in the queueDir in `queueLimit` field. For eg, the `queueDir` can be `/home/events` and `queueLimit` can be `1000`. By default, the `queueLimit` is set to 10000.
```
To update the configuration, use `mc admin config get` command to get the current configuration file for the MinIO deployment in json format, and save it locally.
To update the configuration, use `mc admin config get` command to get the current configuration for `notify_nsq`.
```sh
$ mc admin config get myminio/ > /tmp/myconfig
$ mc admin config get myminio/ notify_nsq
notify_nsq:1 nsqd_address="" queue_dir="" queue_limit="0" state="off" tls_enable="off" tls_skip_verify="off" topic=""
```
After updating the NSQ configuration in /tmp/myconfig , use `mc admin config set` command to update the configuration for the deployment.Restart the MinIO server to put the changes into effect. The server will print a line like `SQS ARNs: arn:minio:sqs::1:nsq` at start-up if there were no errors.
Use `mc admin config set` command to update the configuration for the deployment. Restart the MinIO server to put the changes into effect. The server will print a line like `SQS ARNs: arn:minio:sqs::1:nsq` at start-up if there were no errors.
```sh
$ mc admin config set myminio < /tmp/myconfig
$ mc admin config set myminio notify_nsq:1 nsqd_address="127.0.0.1:4150" queue_dir="" queue_limit="0" state="on" tls_enable="off" tls_skip_verify="on" topic="minio"
```
Note that, you can add as many NSQ daemon endpoint configurations as needed by providing an identifier (like "1" in the example above) for the NSQ instance and an object of per-server configuration parameters.
@@ -1267,5 +1067,3 @@ You should receive the following event notification via NSQ once the upload comp
```
{"EventName":"s3:ObjectCreated:Put","Key":"images/gopher.jpg","Records":[{"eventVersion":"2.0","eventSource":"minio:s3","awsRegion":"","eventTime":"2018-10-31T09:31:11Z","eventName":"s3:ObjectCreated:Put","userIdentity":{"principalId":"21EJ9HYV110O8NVX2VMS"},"requestParameters":{"sourceIPAddress":"10.1.1.1"},"responseElements":{"x-amz-request-id":"1562A792DAA53426","x-minio-origin-endpoint":"http://10.0.3.1:9000"},"s3":{"s3SchemaVersion":"1.0","configurationId":"Config","bucket":{"name":"images","ownerIdentity":{"principalId":"21EJ9HYV110O8NVX2VMS"},"arn":"arn:aws:s3:::images"},"object":{"key":"gopher.jpg","size":162023,"eTag":"5337769ffa594e742408ad3f30713cd7","contentType":"image/jpeg","userMetadata":{"content-type":"image/jpeg"},"versionId":"1","sequencer":"1562A792DAA53426"}},"source":{"host":"","port":"","userAgent":"MinIO (linux; amd64) minio-go/v6.0.8 mc/DEVELOPMENT.GOGET"}}]}
```
_NOTE_ If you are running [distributed MinIO](https://docs.min.io/docs/distributed-minio-quickstart-guide), modify `~/.minio/config.json` on all the nodes with your bucket event notification backend configuration.

44
docs/compression/README.md
View File

@@ -16,47 +16,27 @@ Install MinIO - [MinIO Quickstart Guide](https://docs.min.io/docs/minio-quicksta
Compression can be enabled by updating the `compress` config settings for MinIO server config. Config `compress` settings take extensions and mime-types to be compressed.
```json
"compress": {
"enabled": true,
"extensions": [".txt",".log",".csv", ".json", ".tar"],
"mime-types": ["text/*","application/json","application/xml"]
}
```
$ mc admin config get myminio compression
compression extensions=".txt,.log,.csv,.json,.tar,.xml,.bin" mime_types="text/*,application/json,application/xml" state="off"```
Default config includes most common highly compressible content extensions and mime-types.
```
$ mc admin config set myminio compression extensions=".pdf" mime_types="application/pdf" state="on"
```
Since text, log, csv, json files are highly compressible, These extensions/mime-types are included by default for compression.
Having compression enabled and no extensions or mime types will attempt to compress anything that isn't explicitly known to be already compressed content.
Settings for enabling compression on all content, except for types listed below:
```json
"compress": {
"enabled": true,
"extensions": [],
"mime-types": []
}
To enable compression for all content, except for types listed below:
```
Incompressible content will be skipped with quite low CPU usage and storage overhead, typically at several GB/s.
To update the configuration, use `mc admin config get` command to get the current configuration file for the minio cluster in json format, and save it locally.
```sh
$ mc admin config get myminio/ > /tmp/myconfig
```
After updating the compression configuration in /tmp/myconfig , use `mc admin config set` command to update the configuration for the cluster. Restart the MinIO server to put the changes into effect.
```sh
$ mc admin config set myminio < /tmp/myconfig
~ mc admin config set myminio compression extensions="" mime_types="" state="on"
```
The compression settings may also be set through environment variables. When set, environment variables override the defined `compress` config settings in the server config.
```bash
export MINIO_COMPRESS="true"
export MINIO_COMPRESS="on"
export MINIO_COMPRESS_EXTENSIONS=".pdf,.doc"
export MINIO_COMPRESS_MIMETYPES="application/pdf"
export MINIO_COMPRESS_MIME_TYPES="application/pdf"
```
### 3. Note

212
docs/config/README.md
View File

@@ -1,4 +1,4 @@
# MinIO Server Config Guide [![Slack](https://slack.min.io/slack?type=svg)](https://slack.min.io) [![Go Report Card](https://goreportcard.com/badge/minio/minio)](https://goreportcard.com/report/minio/minio) [![Docker Pulls](https://img.shields.io/docker/pulls/minio/minio.svg?maxAge=604800)](https://hub.docker.com/r/minio/minio/)
# MinIO Server Config Guide [![Slack](https://slack.min.io/slack?type=svg)](https://slack.min.io) [![Docker Pulls](https://img.shields.io/docker/pulls/minio/minio.svg?maxAge=604800)](https://hub.docker.com/r/minio/minio/)
## Configuration Directory
@@ -19,119 +19,76 @@ TLS certificates by default are stored under ``${HOME}/.minio/certs`` directory.
Following is the directory structure for MinIO server with TLS certificates.
```sh
$ tree ~/.minio
$ mc tree --files ~/.minio
/home/user1/.minio
├── certs
│   ├── CAs
│   ├── private.key
│   └── public.crt
└─ certs
├─ CAs
├─ private.key
└─ public.crt
```
You can provide a custom certs directory using `--certs-dir` command line option.
### Accessing configuration file
All configuration changes can be made using [`mc admin config` get/set commands](https://github.com/minio/mc/blob/master/docs/minio-admin-complete-guide.md). Following sections provide brief explanation of fields and how to customize them. A complete example of `config.json` is available [here](https://raw.githubusercontent.com/minio/minio/master/docs/config/config.sample.json)
#### Editing configuration file fields
##### Get current configuration for MinIO deployment
```sh
$ mc admin config get myminio/ > /tmp/myconfig
```
##### Set current configuration for MinIO deployment
```sh
$ mc admin config set myminio < /tmp/myconfig
```
The `mc admin` config API will evolve soon to be able to configure specific fields using get/set commands.
#### Version
|Field|Type|Description|
|:---|:---|:---|
|``version``|_string_| `version` determines the configuration file format. Any older version will automatically be migrated to the latest version upon startup. [DO NOT EDIT THIS FIELD MANUALLY]|
#### Credential
|Field|Type|Description|
|:---|:---|:---|
|``credential``| | Auth credential for object storage and web access.|
|``credential.accessKey`` | _string_ | Access key of minimum 3 characters in length. You may override this field with `MINIO_ACCESS_KEY` environment variable.|
|``credential.secretKey`` | _string_ | Secret key of minimum 8 characters in length. You may override this field with `MINIO_SECRET_KEY` environment variable.|
> NOTE: In distributed setup it is mandatory to use environment variables `MINIO_ACCESS_KEY` and `MINIO_SECRET_KEY` for credentials.
Example:
```sh
export MINIO_ACCESS_KEY=admin
export MINIO_SECRET_KEY=password
minio server /data
```
#### Region
|Field|Type|Description|
|:---|:---|:---|
|``region``| _string_ | `region` describes the physical location of the server. By default it is blank. You may override this field with `MINIO_REGION` environment variable. If you are unsure leave it unset.|
| Field | Type | Description |
|:--------------------------|:---------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| ``region name=my_region`` | _string_ | `region` describes the physical location of the server. By default it is blank. You may override this field with `MINIO_REGION_NAME` environment variable. If you are unsure leave it unset. |
Example:
```sh
export MINIO_REGION="my_region"
export MINIO_REGION_NAME="my_region"
minio server /data
```
#### Worm
|Field|Type|Description|
|:---|:---|:---|
|``worm``| _string_ | Enable this to turn on Write-Once-Read-Many. By default it is set to `off`. You may override this field with ``MINIO_WORM`` environment variable.|
| Field | Type | Description |
|:-----------------------|:---------|:--------------------------------------------------------------------------------------------------------------------------------------------------|
| ``worm state=on`` | _string_ | Enable this to turn on Write-Once-Read-Many. By default it is set to `off`. You may override this field with ``MINIO_WORM`` environment variable. |
Example:
```sh
export MINIO_WORM=on
export MINIO_WORM_STATE=on
minio server /data
```
### Storage Class
|Field|Type|Description|
|:---|:---|:---|
|``storageclass``| | Set storage class for configurable data and parity, as per object basis.|
|``storageclass.standard`` | _string_ | Value for standard storage class. It should be in the format `EC:Parity`, for example to set 4 disk parity for standard storage class objects, set this field to `EC:4`.|
|``storageclass.rrs`` | _string_ | Value for reduced redundancy storage class. It should be in the format `EC:Parity`, for example to set 3 disk parity for reduced redundancy storage class objects, set this field to `EC:3`.|
| Field | Type | Description |
|:-------------------------------|:---------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| ``storageclass`` | | Set storage class for configurable data and parity, as per object basis. |
| ``storageclass standard=EC:4`` | _string_ | Value for standard storage class. It should be in the format `EC:Parity`, for example to set 4 disk parity for standard storage class objects, set this field to `EC:4`. |
| ``storageclass rrs=EC:2`` | _string_ | Value for reduced redundancy storage class. It should be in the format `EC:Parity`, for example to set 3 disk parity for reduced redundancy storage class objects, set this field to `EC:3`. |
By default, parity for objects with standard storage class is set to `N/2`, and parity for objects with reduced redundancy storage class objects is set to `2`. Read more about storage class support in MinIO server [here](https://github.com/minio/minio/blob/master/docs/erasure/storage-class/README.md).
### Cache
|Field|Type|Description|
|:---|:---|:---|
|``drives``| _[]string_ | List of mounted file system drives with [`atime`](http://kerolasa.github.io/filetimes.html) support enabled|
|``exclude`` | _[]string_ | List of wildcard patterns for prefixes to exclude from cache |
|``expiry`` | _int_ | Days to cache expiry |
|``maxuse`` | _int_ | Percentage of disk available to cache |
| Field | Type | Description |
|:-------------------------------------------------------------|:-----------|:------------------------------------------------------------------------------------------------------------|
| ``cache drives="/mnt/drive1;/mnt/drive2;/mnt/cache{1...3}"`` | _[]string_ | List of mounted file system drives with [`atime`](http://kerolasa.github.io/filetimes.html) support enabled |
| ``cache exclude="*.pdf;mybucket/*"`` | _[]string_ | List of wildcard patterns for prefixes to exclude from cache |
| ``cache expiry=90`` | _int_ | Days to cache expiry |
| ``cache quota=70`` | _int_ | Percentage of disk available to cache |
| | | |
#### Notify
|Field|Type|Description|
|:---|:---|:---|
|``notify``| |Notify enables bucket notification events for lambda computing via the following targets.|
|``notify.amqp``| |[Configure to publish MinIO events via AMQP target.](https://docs.min.io/docs/minio-bucket-notification-guide#AMQP)|
|``notify.nats``| |[Configure to publish MinIO events via NATS target.](https://docs.min.io/docs/minio-bucket-notification-guide#NATS)|
|``notify.elasticsearch``| |[Configure to publish MinIO events via Elasticsearch target.](https://docs.min.io/docs/minio-bucket-notification-guide#Elasticsearch)|
|``notify.redis``| |[Configure to publish MinIO events via Redis target.](https://docs.min.io/docs/minio-bucket-notification-guide#Redis)|
|``notify.postgresql``| |[Configure to publish MinIO events via PostgreSQL target.](https://docs.min.io/docs/minio-bucket-notification-guide#PostgreSQL)|
|``notify.kafka``| |[Configure to publish MinIO events via Apache Kafka target.](https://docs.min.io/docs/minio-bucket-notification-guide#apache-kafka)|
|``notify.webhook``| |[Configure to publish MinIO events via Webhooks target.](https://docs.min.io/docs/minio-bucket-notification-guide#webhooks)|
|``notify.mysql``| |[Configure to publish MinIO events via MySql target.](https://docs.min.io/docs/minio-bucket-notification-guide#MySQL)|
|``notify.mqtt``| |[Configure to publish MinIO events via MQTT target.](https://docs.min.io/docs/minio-bucket-notification-guide#MQTT)|
| Field | Type | Description |
|:-------------------------|:-----|:--------------------------------------------------------------------------------------------------------------------------------------|
| ``notify_amqp`` | | [Configure to publish MinIO events via AMQP target.](https://docs.min.io/docs/minio-bucket-notification-guide#AMQP) |
| ``notify_nats`` | | [Configure to publish MinIO events via NATS target.](https://docs.min.io/docs/minio-bucket-notification-guide#NATS) |
| ``notify_elasticsearch`` | | [Configure to publish MinIO events via Elasticsearch target.](https://docs.min.io/docs/minio-bucket-notification-guide#Elasticsearch) |
| ``notify_redis`` | | [Configure to publish MinIO events via Redis target.](https://docs.min.io/docs/minio-bucket-notification-guide#Redis) |
| ``notify_postgresql`` | | [Configure to publish MinIO events via PostgreSQL target.](https://docs.min.io/docs/minio-bucket-notification-guide#PostgreSQL) |
| ``notify_kafka`` | | [Configure to publish MinIO events via Apache Kafka target.](https://docs.min.io/docs/minio-bucket-notification-guide#apache-kafka) |
| ``notify_webhook`` | | [Configure to publish MinIO events via Webhooks target.](https://docs.min.io/docs/minio-bucket-notification-guide#webhooks) |
| ``notify_mysql`` | | [Configure to publish MinIO events via MySql target.](https://docs.min.io/docs/minio-bucket-notification-guide#MySQL) |
| ``notify_mqtt`` | | [Configure to publish MinIO events via MQTT target.](https://docs.min.io/docs/minio-bucket-notification-guide#MQTT) |
### Accessing configuration file
All configuration changes can be made using [`mc admin config` get/set commands](https://github.com/minio/mc/blob/master/docs/minio-admin-complete-guide.md). Following sections provide brief explanation of fields and how to customize them. A complete example of `config.json` is available [here](https://raw.githubusercontent.com/minio/minio/master/docs/config/config.sample.json)
## Environment only settings
@@ -162,95 +119,6 @@ export MINIO_DOMAIN=sub1.mydomain.com,sub2.mydomain.com
minio server /data
```
### HTTP Trace
HTTP tracing can be enabled by using [`mc admin trace`](https://github.com/minio/mc/blob/master/docs/minio-admin-complete-guide.md#command-trace---display-minio-server-http-trace) command.
Example:
```sh
minio server /data
```
Default trace is succinct only to indicate the API operations being called and the HTTP response status.
```sh
mc admin trace myminio
17:21:45.729309964 objectAPIHandlers.GetBucketLocation localhost:9000/vk-photos/?location= 200 OK
17:21:45.738167329 objectAPIHandlers.HeadBucket localhost:9000/vk-photos/ 200 OK
17:21:45.747676811 objectAPIHandlers.ListObjectsV1 localhost:9000/vk-photos/?delimiter=%2F&max-keys=1000&prefix= 200 OK
```
To trace entire HTTP request
```sh
mc admin trace --verbose myminio
127.0.0.1 [REQUEST objectAPIHandlers.GetBucketLocation] [17:23:21.404025835]
127.0.0.1 GET /yyy/?location=
127.0.0.1 Host: localhost:9000
127.0.0.1 Content-Length: 0
127.0.0.1 User-Agent: MinIO (linux; amd64) minio-go/v6.0.29 mc/2019-06-15T10:29:41Z
127.0.0.1 X-Amz-Content-Sha256: UNSIGNED-PAYLOAD
127.0.0.1 X-Amz-Date: 20190619T172321Z
127.0.0.1 Authorization: AWS4-HMAC-SHA256 Credential=Q3AM3UQ867SPQQA43P2F/20190619/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=8e53d8574db3424aa00dd73637575512b250c923edcad3cbf58a727305205709
127.0.0.1
127.0.0.1 [RESPONSE] [17:23:21.404780651]
127.0.0.1 200 OK
127.0.0.1 X-Amz-Request-Id: 15A9A965FF7A7546
127.0.0.1 X-Minio-Deployment-Id: 41e39f4a-3b66-415b-9ddf-025d76a58668
127.0.0.1 X-Xss-Protection: 1; mode=block
127.0.0.1 Accept-Ranges: bytes
127.0.0.1 Server: MinIO/DEVELOPMENT.2019-06-18T17-17-02Z
127.0.0.1 Content-Type: application/xml
127.0.0.1 Vary: Origin
127.0.0.1 X-Amz-Bucket-Region: us-east-1
127.0.0.1 Content-Length: 137
127.0.0.1 Content-Security-Policy: block-all-mixed-content
127.0.0.1 <?xml version="1.0" encoding="UTF-8"?>
<LocationConstraint xmlns="http://s3.amazonaws.com/doc/2006-03-01/">us-east-1</LocationConstraint>127.0.0.1
127.0.0.1 [REQUEST objectAPIHandlers.HeadBucket] [17:23:21.412985428]
127.0.0.1 HEAD /yyy/
127.0.0.1 Host: localhost:9000
127.0.0.1 User-Agent: MinIO (linux; amd64) minio-go/v6.0.29 mc/2019-06-15T10:29:41Z
127.0.0.1 X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
127.0.0.1 X-Amz-Date: 20190619T172321Z
127.0.0.1 Authorization: AWS4-HMAC-SHA256 Credential=Q3AM3UQ867SPQQA43P2F/20190619/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=e0a02a62d39994d0206586f01dd2ab3a4aea74e60da9ff4d427629f705c62c02
127.0.0.1 Content-Length: 0
127.0.0.1
127.0.0.1 [RESPONSE] [17:23:21.413457159]
127.0.0.1 200 OK
127.0.0.1 Vary: Origin
127.0.0.1 Accept-Ranges: bytes
127.0.0.1 Content-Length: 0
127.0.0.1 X-Amz-Bucket-Region: us-east-1
127.0.0.1 X-Amz-Request-Id: 15A9A9660005982D
127.0.0.1 X-Minio-Deployment-Id: 41e39f4a-3b66-415b-9ddf-025d76a58668
127.0.0.1 X-Xss-Protection: 1; mode=block
127.0.0.1 Content-Security-Policy: block-all-mixed-content
127.0.0.1 Server: MinIO/DEVELOPMENT.2019-06-18T17-17-02Z
127.0.0.1
127.0.0.1 [REQUEST objectAPIHandlers.ListObjectsV1] [17:23:21.423153668]
127.0.0.1 GET /yyy/?delimiter=%2F&max-keys=1000&prefix=
127.0.0.1 Host: localhost:9000
127.0.0.1 Content-Length: 0
127.0.0.1 User-Agent: MinIO (linux; amd64) minio-go/v6.0.29 mc/2019-06-15T10:29:41Z
127.0.0.1 X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
127.0.0.1 X-Amz-Date: 20190619T172321Z
127.0.0.1 Authorization: AWS4-HMAC-SHA256 Credential=Q3AM3UQ867SPQQA43P2F/20190619/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=46ee3d2fc5085432b94bc3205076abd8166ffa3e35c639f84e9684c7c6a181c9
127.0.0.1
127.0.0.1 [RESPONSE] [17:23:21.424260967]
127.0.0.1 200 OK
127.0.0.1 Content-Security-Policy: block-all-mixed-content
127.0.0.1 Content-Type: application/xml
127.0.0.1 Server: MinIO/DEVELOPMENT.2019-06-18T17-17-02Z
127.0.0.1 X-Amz-Bucket-Region: us-east-1
127.0.0.1 X-Minio-Deployment-Id: 41e39f4a-3b66-415b-9ddf-025d76a58668
127.0.0.1 Accept-Ranges: bytes
127.0.0.1 Content-Length: 253
127.0.0.1 Vary: Origin
127.0.0.1 X-Amz-Request-Id: 15A9A966009F94A6
127.0.0.1 X-Xss-Protection: 1; mode=block
127.0.0.1 <?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>yyy</Name><Prefix></Prefix><Marker></Marker><MaxKeys>1000</MaxKeys><Delimiter>/</Delimiter><IsTruncated>false</IsTruncated></ListBucketResult>
```
## Explore Further
* [MinIO Quickstart Guide](https://docs.min.io/docs/minio-quickstart-guide)
* [Configure MinIO Server with TLS](https://docs.min.io/docs/how-to-secure-access-to-minio-server-with-tls)

527
docs/config/config.sample.json
View File

@@ -1,219 +1,340 @@
{
"version": "33",
"credential": {
"accessKey": "36J9X8EZI4KEV1G7EHXA",
"secretKey": "ECk2uqOoNqvtJIMQ3WYugvmNPL_-zm3WcRqP5vUM",
"expiration": "1970-01-01T00:00:00Z",
"status": "enabled"
},
"region": "",
"worm": "off",
"storageclass": {
"standard": "",
"rrs": ""
},
"cache": {
"drives": [],
"expiry": 90,
"maxuse": 80,
"exclude": []
"_": {
"drives": "",
"exclude": "",
"expiry": "90",
"quota": "80",
"state": "off"
}
},
"kms": {
"vault": {
"compression": {
"_": {
"extensions": ".txt,.log,.csv,.json,.tar,.xml,.bin",
"mime_types": "text/*,application/json,application/xml",
"state": "off"
}
},
"identity_ldap": {
"_": {
"group_name_attribute": "",
"group_search_base_dn": "",
"group_search_filter": "",
"server_addr": "",
"state": "off",
"sts_expiry": "",
"username_format": ""
}
},
"identity_openid": {
"_": {
"config_url": "http://localhost:8080/auth/realms/demo/.well-known/openid-configuration",
"state": "on"
}
},
"kms_vault": {
"_": {
"auth_approle_id": "",
"auth_approle_secret": "",
"auth_type": "",
"capath": "",
"endpoint": "",
"auth": {
"type": "",
"approle": {
"id": "",
"secret": ""
}
},
"key-id": {
"name": "",
"version": 0
}
"key_name": "",
"key_version": "0",
"namespace": "",
"state": "off"
}
},
"notify": {
"amqp": {
"1": {
"enable": false,
"url": "",
"exchange": "",
"routingKey": "",
"exchangeType": "",
"deliveryMode": 0,
"mandatory": false,
"immediate": false,
"durable": false,
"internal": false,
"noWait": false,
"autoDeleted": false,
"queueDir": "",
"queueLimit": 0
}
"logger_http": {
"1": {
"auth_token": "",
"endpoint": "",
"state": "off"
},
"elasticsearch": {
"1": {
"enable": false,
"format": "",
"url": "",
"index": "",
"queueDir": "",
"queueLimit": 0
}
},
"kafka": {
"1": {
"enable": false,
"brokers": null,
"topic": "",
"queueDir": "",
"queueLimit": 0,
"tls": {
"enable": false,
"skipVerify": false,
"clientAuth": 0
},
"sasl": {
"enable": false,
"username": "",
"password": ""
}
}
},
"mqtt": {
"1": {
"enable": false,
"broker": "",
"topic": "",
"qos": 0,
"username": "",
"password": "",
"reconnectInterval": 0,
"keepAliveInterval": 0,
"queueDir": "",
"queueLimit": 0
}
},
"mysql": {
"1": {
"enable": false,
"format": "",
"dsnString": "",
"table": "",
"host": "",
"port": "",
"user": "",
"password": "",
"database": "",
"queueDir": "",
"queueLimit": 0
}
},
"nats": {
"1": {
"enable": false,
"address": "",
"subject": "",
"username": "",
"password": "",
"token": "",
"secure": false,
"pingInterval": 0,
"queueDir": "",
"queueLimit": 0,
"streaming": {
"enable": false,
"clusterID": "",
"async": false,
"maxPubAcksInflight": 0
}
}
},
"nsq": {
"1": {
"enable": false,
"nsqdAddress": "",
"topic": "",
"tls": {
"enable": false,
"skipVerify": true
},
"queueDir": "",
"queueLimit": 0
}
},
"postgresql": {
"1": {
"enable": false,
"format": "",
"connectionString": "",
"table": "",
"host": "",
"port": "",
"user": "",
"password": "",
"database": "",
"queueDir": "",
"queueLimit": 0
}
},
"redis": {
"1": {
"enable": false,
"format": "",
"address": "",
"password": "",
"key": "",
"queueDir": "",
"queueLimit": 0
}
},
"webhook": {
"1": {
"enable": false,
"endpoint": "",
"queueDir": "",
"queueLimit": 0
}
"_": {
"auth_token": "",
"endpoint": "",
"state": "off"
}
},
"logger": {
"console": {
"enabled": true
"logger_http_audit": {
"_": {
"auth_token": "",
"endpoint": "",
"state": "off"
}
},
"notify_amqp": {
"1": {
"auto_deleted": "off",
"delivery_mode": "0",
"durable": "off",
"exchange": "",
"exchange_type": "",
"internal": "off",
"mandatory": "off",
"no_wait": "off",
"queue_dir": "",
"queue_limit": "0",
"routing_key": "",
"state": "off",
"url": ""
},
"http": {
"target1": {
"enabled": false,
"type": "minio",
"endpoint": "https://username:password@example.com/api"
}
"_": {
"auto_deleted": "off",
"delivery_mode": "0",
"durable": "off",
"exchange": "",
"exchange_type": "",
"internal": "off",
"mandatory": "off",
"no_wait": "off",
"queue_dir": "",
"queue_limit": "0",
"routing_key": "",
"state": "off",
"url": ""
}
},
"compress": {
"enabled": false,
"extensions": [
".txt",
".log",
".csv",
".json"
],
"mime-types": [
"text/csv",
"text/plain",
"application/json"
]
},
"openid": {
"jwks": {
"url": null
"notify_elasticsearch": {
"1": {
"format": "namespace",
"index": "",
"queue_dir": "",
"queue_limit": "0",
"state": "off",
"url": ""
},
"_": {
"format": "namespace",
"index": "",
"queue_dir": "",
"queue_limit": "0",
"state": "off",
"url": ""
}
},
"policy": {
"opa": {
"url": null,
"authToken": ""
"notify_kafka": {
"1": {
"brokers": "",
"queue_dir": "",
"queue_limit": "0",
"sasl_enable": "off",
"sasl_password": "",
"sasl_username": "",
"state": "off",
"tls_client_auth": "0",
"tls_enable": "off",
"tls_skip_verify": "off",
"topic": ""
},
"_": {
"brokers": "",
"queue_dir": "",
"queue_limit": "0",
"sasl_enable": "off",
"sasl_password": "",
"sasl_username": "",
"state": "off",
"tls_client_auth": "0",
"tls_enable": "off",
"tls_skip_verify": "off",
"topic": ""
}
},
"notify_mqtt": {
"1": {
"broker": "",
"keep_alive_interval": "0s",
"password": "",
"qos": "0",
"queue_dir": "",
"queue_limit": "0",
"reconnect_interval": "0s",
"state": "off",
"topic": "",
"username": ""
},
"_": {
"broker": "",
"keep_alive_interval": "0s",
"password": "",
"qos": "0",
"queue_dir": "",
"queue_limit": "0",
"reconnect_interval": "0s",
"state": "off",
"topic": "",
"username": ""
}
},
"notify_mysql": {
"1": {
"database": "",
"dsn_string": "",
"format": "namespace",
"host": "",
"password": "",
"port": "",
"queue_dir": "",
"queue_limit": "0",
"state": "off",
"table": "",
"username": ""
},
"_": {
"database": "",
"dsn_string": "",
"format": "namespace",
"host": "",
"password": "",
"port": "",
"queue_dir": "",
"queue_limit": "0",
"state": "off",
"table": "",
"username": ""
}
},
"notify_nats": {
"1": {
"address": "",
"password": "",
"ping_interval": "0",
"queue_dir": "",
"queue_limit": "0",
"secure": "off",
"state": "off",
"streaming_async": "off",
"streaming_cluster_id": "",
"streaming_enable": "off",
"streaming_max_pub_acks_in_flight": "0",
"subject": "",
"token": "",
"username": ""
},
"_": {
"address": "",
"password": "",
"ping_interval": "0",
"queue_dir": "",
"queue_limit": "0",
"secure": "off",
"state": "off",
"streaming_async": "off",
"streaming_cluster_id": "",
"streaming_enable": "off",
"streaming_max_pub_acks_in_flight": "0",
"subject": "",
"token": "",
"username": ""
}
},
"notify_nsq": {
"1": {
"nsqd_address": "",
"queue_dir": "",
"queue_limit": "0",
"state": "off",
"tls_enable": "off",
"tls_skip_verify": "off",
"topic": ""
},
"_": {
"nsqd_address": "",
"queue_dir": "",
"queue_limit": "0",
"state": "off",
"tls_enable": "off",
"tls_skip_verify": "off",
"topic": ""
}
},
"notify_postgres": {
"1": {
"connection_string": "",
"database": "",
"format": "namespace",
"host": "",
"password": "",
"port": "",
"queue_dir": "",
"queue_limit": "0",
"state": "off",
"table": "",
"username": ""
},
"_": {
"connection_string": "",
"database": "",
"format": "namespace",
"host": "",
"password": "",
"port": "",
"queue_dir": "",
"queue_limit": "0",
"state": "off",
"table": "",
"username": ""
}
},
"notify_redis": {
"1": {
"address": "",
"format": "namespace",
"key": "",
"password": "",
"queue_dir": "",
"queue_limit": "0",
"state": "off"
},
"_": {
"address": "",
"format": "namespace",
"key": "",
"password": "",
"queue_dir": "",
"queue_limit": "0",
"state": "off"
}
},
"notify_webhook": {
"1": {
"endpoint": "",
"queue_dir": "",
"queue_limit": "0",
"state": "off"
},
"_": {
"auth_token": "",
"endpoint": "",
"queue_dir": "",
"queue_limit": "0",
"state": "off"
}
},
"policy_opa": {
"_": {
"auth_token": "",
"state": "off",
"url": ""
}
},
"region": {
"_": {
"name": "us-east-1"
}
},
"storageclass": {
"_": {
"rrs": "",
"standard": "",
"state": "off"
}
},
"worm": {
"_": {
"state": "off"
}
}
}
}

25
docs/debugging/README.md Normal file
View File

@@ -0,0 +1,25 @@
# MinIO Server Debugging Guide [![Slack](https://slack.min.io/slack?type=svg)](https://slack.min.io) [![Docker Pulls](https://img.shields.io/docker/pulls/minio/minio.svg?maxAge=604800)](https://hub.docker.com/r/minio/minio/)
### HTTP Trace
HTTP tracing can be enabled by using [`mc admin trace`](https://github.com/minio/mc/blob/master/docs/minio-admin-complete-guide.md#command-trace---display-minio-server-http-trace) command.
Example:
```sh
minio server /data
```
Default trace is succinct only to indicate the API operations being called and the HTTP response status.
```sh
mc admin trace myminio
```
To trace entire HTTP request
```sh
mc admin trace --verbose myminio
```
To trace entire HTTP request and also internode communication
```sh
mc admin trace --all --verbose myminio
```

4
docs/disk-caching/DESIGN.md
View File

@@ -12,7 +12,7 @@ minio gateway <name> -h
MINIO_CACHE_DRIVES: List of mounted cache drives or directories delimited by ";"
MINIO_CACHE_EXCLUDE: List of cache exclusion patterns delimited by ";"
MINIO_CACHE_EXPIRY: Cache expiry duration in days
MINIO_CACHE_MAXUSE: Maximum permitted usage of the cache in percentage (0-100).
MINIO_CACHE_QUOTA: Maximum permitted usage of the cache in percentage (0-100).
...
...
@@ -22,7 +22,7 @@ minio gateway <name> -h
$ export MINIO_CACHE_DRIVES="/mnt/drive1;/mnt/drive2;/mnt/export{1..24}"
$ export MINIO_CACHE_EXCLUDE="mybucket/*;*.pdf"
$ export MINIO_CACHE_EXPIRY=40
$ export MINIO_CACHE_MAXUSE=80
$ export MINIO_CACHE_QUOTA=80
$ minio gateway s3
```

3
docs/disk-caching/README.md
View File

@@ -18,10 +18,11 @@ Disk caching can be enabled by setting the `cache` environment variables for Min
Following example uses `/mnt/drive1`, `/mnt/drive2` ,`/mnt/cache1` ... `/mnt/cache3` for caching, with expiry up to 90 days while excluding all objects under bucket `mybucket` and all objects with '.pdf' as extension while starting a s3 gateway setup. Cache max usage is restricted to 80% of disk capacity in this example.
```bash
export MINIO_CACHE="on"
export MINIO_CACHE_DRIVES="/mnt/drive1;/mnt/drive2;/mnt/cache{1...3}"
export MINIO_CACHE_EXPIRY=90
export MINIO_CACHE_EXCLUDE="*.pdf;mybucket/*"
export MINIO_CACHE_MAXUSE=80
export MINIO_CACHE_QUOTA=80
minio gateway s3
```

6
docs/erasure/storage-class/README.md
View File

@@ -62,12 +62,14 @@ Default value for `REDUCED_REDUNDANCY` storage class is `2`.
The format to set storage class environment variables is as follows
`MINIO_STORAGE_CLASS_STATE=on`
`MINIO_STORAGE_CLASS_STANDARD=EC:parity`
`MINIO_STORAGE_CLASS_RRS=EC:parity`
For example, set `MINIO_STORAGE_CLASS_RRS` parity 2 and `MINIO_STORAGE_CLASS_STANDARD` parity 3
```sh
export MINIO_STORAGE_CLASS_STATE=on
export MINIO_STORAGE_CLASS_STANDARD=EC:3
export MINIO_STORAGE_CLASS_RRS=EC:2
```
@@ -77,7 +79,7 @@ more details.
*Note*
- If `STANDARD` storage class is set via environment variables or `mc admin config` get/set commands, and `x-amz-storage-class` is not present in request metadata, MinIO server will
- If `STANDARD` storage class is set via environment variables or `mc admin config` get/set commands, and `x-amz-storage-class` is not present in request metadata, MinIO server will
apply `STANDARD` storage class to the object. This means the data and parity disks will be used as set in `STANDARD` storage class.
- If storage class is not defined before starting MinIO server, and subsequent PutObject metadata field has `x-amz-storage-class` present
@@ -108,4 +110,4 @@ if err != nil {
log.Fatalln(err)
}
log.Println("Uploaded", "my-objectname", " of size: ", n, "Successfully.")
```
```

45
docs/kms/README.md
View File

@@ -12,7 +12,7 @@ MinIO supports two different KMS concepts:
by enabling or disabling the corresponding master keys on demand.
- Direct KMS master keys:
MinIO can also be configured to directly use a master key specified by the environment variable `MINIO_SSE_MASTER_KEY` or with a docker secret key.
MinIO can also be configured to directly use a master key specified by the environment variable `MINIO_KMS_MASTER_KEY` or with a docker secret key.
Direct master keys are useful if the storage backend is not on the same machine as the MinIO server, e.g.,
if network drives or MinIO gateway is used and an external KMS would cause too much management overhead.
@@ -174,27 +174,28 @@ The AppRole ID, AppRole Secret Id, Vault endpoint and Vault key name can now be
You'll need the Vault endpoint, AppRole ID, AppRole SecretID and encryption key-ring name defined in step 2.1.2
```
export MINIO_SSE_VAULT_APPROLE_ID=8c03926c-6c51-7a1d-cf7d-62e48ab8d6d7
export MINIO_SSE_VAULT_APPROLE_SECRET=edd8738c-6efe-c226-74f9-ef5b66e119d7
export MINIO_SSE_VAULT_ENDPOINT=http://vault-endpoint-ip:8200
export MINIO_SSE_VAULT_KEY_NAME=my-minio-key
export MINIO_SSE_VAULT_AUTH_TYPE=approle
export MINIO_KMS_VAULT_STATE=on
export MINIO_KMS_VAULT_APPROLE_ID=8c03926c-6c51-7a1d-cf7d-62e48ab8d6d7
export MINIO_KMS_VAULT_APPROLE_SECRET=edd8738c-6efe-c226-74f9-ef5b66e119d7
export MINIO_KMS_VAULT_ENDPOINT=http://vault-endpoint-ip:8200
export MINIO_KMS_VAULT_KEY_NAME=my-minio-key
export MINIO_KMS_VAULT_AUTH_TYPE=approle
minio server ~/export
```
Optionally, set `MINIO_SSE_VAULT_CAPATH` to a directory of PEM-encoded CA cert files to use mTLS for client-server authentication.
Optionally, set `MINIO_KMS_VAULT_CAPATH` to a directory of PEM-encoded CA cert files to use mTLS for client-server authentication.
```
export MINIO_SSE_VAULT_CAPATH=/home/user/custom-certs
export MINIO_KMS_VAULT_CAPATH=/home/user/custom-certs
```
An additional option is to set `MINIO_SSE_VAULT_NAMESPACE` if AppRole and Transit Secrets engine have been scoped to Vault Namespace
An additional option is to set `MINIO_KMS_VAULT_NAMESPACE` if AppRole and Transit Secrets engine have been scoped to Vault Namespace
```
export MINIO_SSE_VAULT_NAMESPACE=ns1
export MINIO_KMS_VAULT_NAMESPACE=ns1
```
Note: If [Vault Namespaces](https://learn.hashicorp.com/vault/operations/namespaces) are in use, MINIO_SSE_VAULT_NAMESPACE variable needs to be set before setting approle and transit secrets engine.
Note: If [Vault Namespaces](https://learn.hashicorp.com/vault/operations/namespaces) are in use, MINIO_KMS_VAULT_VAULT_NAMESPACE variable needs to be set before setting approle and transit secrets engine.
MinIO gateway to S3 supports encryption. Three encryption modes are possible - encryption can be set to ``pass-through`` to backend, ``single encryption`` (at the gateway) or ``double encryption`` (single encryption at gateway and pass through to backend). This can be specified by setting MINIO_GATEWAY_SSE and KMS environment variables set in Step 2.1.2.
@@ -205,11 +206,12 @@ and "c" for sse-c encryption. More than one encryption option can be set, delimi
```sh
export MINIO_GATEWAY_SSE="s3;c"
export MINIO_SSE_VAULT_APPROLE_ID=9b56cc08-8258-45d5-24a3-679876769126
export MINIO_SSE_VAULT_APPROLE_SECRET=4e30c52f-13e4-a6f5-0763-d50e8cb4321f
export MINIO_SSE_VAULT_ENDPOINT=https://vault-endpoint-ip:8200
export MINIO_SSE_VAULT_KEY_NAME=my-minio-key
export MINIO_SSE_VAULT_AUTH_TYPE=approle
export MINIO_KMS_VAULT_STATE=on
export MINIO_KMS_VAULT_APPROLE_ID=9b56cc08-8258-45d5-24a3-679876769126
export MINIO_KMS_VAULT_APPROLE_SECRET=4e30c52f-13e4-a6f5-0763-d50e8cb4321f
export MINIO_KMS_VAULT_ENDPOINT=https://vault-endpoint-ip:8200
export MINIO_KMS_VAULT_KEY_NAME=my-minio-key
export MINIO_KMS_VAULT_AUTH_TYPE=approle
minio gateway s3
```
@@ -221,7 +223,7 @@ A KMS master key consists of a master-key ID (CMK) and the 256 bit master key en
A KMS master key can be specified directly using:
```
export MINIO_SSE_MASTER_KEY=my-minio-key:6368616e676520746869732070617373776f726420746f206120736563726574
export MINIO_KMS_MASTER_KEY=my-minio-key:6368616e676520746869732070617373776f726420746f206120736563726574
```
Please use your own master key. A random master key can be generated using e.g. this command on Linux/Mac/BSD* systems:
@@ -235,16 +237,15 @@ head -c 32 /dev/urandom | xxd -c 32 -ps
Alternatively, you may pass a master key as a [Docker secret](https://docs.docker.com/engine/swarm/secrets/).
```bash
echo "my-minio-key:6368616e676520746869732070617373776f726420746f206120736563726574" | docker secret create sse_master_key
echo "my-minio-key:6368616e676520746869732070617373776f726420746f206120736563726574" | docker secret create kms_master_key
```
Obviously, do not use this demo key for anything real!
To use another secret name, follow the instructions above and replace sse_master_key with your custom names (e.g. my_sse_master_key).
Then, set the MINIO_SSE_MASTER_KEY_FILE environment variable to your secret name:
To use another secret name, follow the instructions above and replace kms_master_key with your custom names (e.g. my_kms_master_key). Then, set the MINIO_KMS_MASTER_KEY_FILE environment variable to your secret name:
```bash
export MINIO_SSE_MASTER_KEY_FILE=my_sse_master_key
export MINIO_KMS_MASTER_KEY_FILE=my_kms_master_key
```
### 3. Test your setup
@@ -262,7 +263,7 @@ plaintext - for example if sensitive data is stored on public cloud storage.
To enable auto-encryption set the environment variable to `on`:
```
export MINIO_SSE_AUTO_ENCRYPTION=on
export MINIO_KMS_AUTO_ENCRYPTION=on
```
To verify auto-encryption, use the `mc` command:

72
docs/logging/README.md
View File

@@ -8,68 +8,50 @@ MinIO supports currently two target types
- http
### Console Target
Console target logs to `/dev/stderr` and is enabled by default. To turn-off console logging you would have to update your MinIO server configuration using `mc admin config set` command.
Assuming `mc` is already [configured](https://docs.min.io/docs/minio-client-quickstart-guide.html)
```
mc admin config get myminio/ > /tmp/config
```
Edit the `/tmp/config` and toggle `console` field `enabled` from `true` to `false`.
```json
"logger": {
"console": {
"enabled": false
}
},
```
Once changed, now you may set the changed config to server through following commands.
```
mc admin config set myminio/ < /tmp/config
mc admin restart myminio/
```
Console target is on always and cannot be disabled.
### HTTP Target
HTTP target logs to a generic HTTP endpoint in JSON format and is not enabled by default. To enable HTTP target logging you would have to update your MinIO server configuration using `mc admin config set` command.
Assuming `mc` is already [configured](https://docs.min.io/docs/minio-client-quickstart-guide.html)
```
mc admin config get myminio/ > /tmp/config
mc admin config get myminio/ logger_http
logger_http:target1 auth_token="" endpoint="" state="off"
```
Edit the `/tmp/config` and toggle `http` field `enabled` from `false` to `true`.
```json
"logger": {
"console": {
"enabled": false
},
"http": {
"1": {
"enabled": true,
"endpoint": "http://endpoint:port/path"
}
}
},
```
mc admin config set myminio logger_http:target1 auth_token="" endpoint="http://endpoint:port/path" state="on"
mc admin service restart myminio
```
NOTE: `http://endpoint:port/path` is a placeholder value to indicate the URL format, please change this accordingly as per your configuration.
Once changed, now you may set the changed config to server through following commands.
```
mc admin config set myminio/ < /tmp/config
mc admin restart myminio/
```
MinIO also honors environment variable for HTTP target logging as shown below, this setting will override the endpoint settings in the MinIO server config.
```
MINIO_LOGGER_HTTP_ENDPOINT=http://localhost:8080/minio/logs minio server /mnt/data
export MINIO_LOGGER_HTTP_AUTH_TOKEN_target1="token"
export MINIO_LOGGER_HTTP_ENDPOINT_target1=http://localhost:8080/minio/logs
minio server /mnt/data
```
## Audit Targets
For audit logging MinIO supports only HTTP target type for now. Audit logging is currently only available through environment variable.
Assuming `mc` is already [configured](https://docs.min.io/docs/minio-client-quickstart-guide.html)
```
MINIO_AUDIT_LOGGER_HTTP_ENDPOINT=http://localhost:8080/minio/logs/audit minio server /mnt/data
mc admin config get myminio/ logger_http_audit
logger_http_audit:target1 auth_token="" endpoint="" state="off"
```
```
mc admin config set myminio logger_http_audit:target1 auth_token="" endpoint="http://endpoint:port/path" state="on"
mc admin service restart myminio
```
NOTE: `http://endpoint:port/path` is a placeholder value to indicate the URL format, please change this accordingly as per your configuration.
MinIO also honors environment variable for HTTP target Audit logging as shown below, this setting will override the endpoint settings in the MinIO server config.
```
export MINIO_LOGGER_HTTP_AUDIT_AUTH_TOKEN_target1="token"
export MINIO_LOGGER_HTTP_AUDIT_ENDPOINT_target1=http://localhost:8080/minio/logs
minio server /mnt/data
```
Setting this environment variable automatically enables audit logging to the HTTP target. The audit logging is in JSON format as described below.

4
docs/sts/README.md
View File

@@ -34,7 +34,7 @@ Make sure we have followed the previous step and configured each software indepe
```
export MINIO_ACCESS_KEY=minio
export MINIO_SECRET_KEY=minio123
export MINIO_IAM_JWKS_URL=https://localhost:9443/oauth2/jwks
export MINIO_IDENTITY_OPENID_CONFIG_URL=https://localhost:9443/oauth2/oidcdiscovery/.well-known/openid-configuration
minio server /mnt/data
```
@@ -46,7 +46,7 @@ Make sure we have followed the previous step and configured each software indepe
```
export MINIO_ACCESS_KEY=aws_access_key
export MINIO_SECRET_KEY=aws_secret_key
export MINIO_IAM_JWKS_URL=https://localhost:9443/oauth2/jwks
export MINIO_IDENTITY_OPENID_CONFIG_URL=https://localhost:9443/oauth2/oidcdiscovery/.well-known/openid-configuration
export MINIO_ETCD_ENDPOINTS=http://localhost:2379
minio gateway s3
```

24
docs/sts/client-grants.md
View File

@@ -93,25 +93,15 @@ http://minio.cluster:9000?Action=AssumeRoleWithClientGrants&DurationSeconds=3600
```
$ export MINIO_ACCESS_KEY=minio
$ export MINIO_SECRET_KEY=minio123
$ export MINIO_IAM_JWKS_URL=https://localhost:9443/oauth2/jwks
$ export MINIO_IAM_OPA_URL=http://localhost:8181/v1/data/httpapi/authz
$ export MINIO_IDENTITY_OPENID_CONFIG_URL=https://localhost:9443/oauth2/oidcdiscovery/.well-known/openid-configuration
$ export MINIO_POLICY_OPA_URL=http://localhost:8181/v1/data/httpapi/authz
$ minio server /mnt/export
$ mc admin config get myminio
...
{
"openid": {
"jwks": {
"url": "https://localhost:9443/oauth2/jwks"
}
}
"policy": {
"opa": {
"url": "http://localhost:8181/v1/data/httpapi/authz",
"authToken": ""
}
}
}
$ mc admin config get myminio identity_openid
identity_openid config_url="https://localhost:9443/oauth2/oidcdiscovery/.well-known/openid-configuration" state="on"
$ mc admin config get myminio policy_opa
policy_opa auth_token="" state="on" url="http://localhost:8181/v1/data/httpapi/authz"
```
Testing with an example

69
docs/sts/keycloak.md Normal file
View File

@@ -0,0 +1,69 @@
# Keycloak Quickstart Guide [![Slack](https://slack.min.io/slack?type=svg)](https://slack.min.io)
Keycloak is an open source Identity and Access Management solution aimed at modern applications and services, this document covers configuring Keycloak to be used as an identity provider for MinIO server STS API.
## 1. Prerequisites
- JAVA 1.8 and above installed
- Download and start Keycloak server by following the [installation guide](https://www.keycloak.org/docs/latest/getting_started/index.html) (finish upto section 3.4)
## 2. Configure Keycloak
- Go to Users -> Click on the user -> Attribute, add a new attribute `Key` is `policy`, `Value` is name of the policy in minio (ex: `readwrite`). Click Add and then Save.
- Go to Clients -> Click on `account` -> Settings, set `Valid Redirect URIs` to `*`, expand `Advanced Settings` and set `Access Token Lifespan` to `1 Hours`, then Save.
- Go to Clients -> Client on `account` -> Mappers -> Create, `Name` can be any text, `Mapper Type` is `User Attribute`, `User Attribute` is `policy`, `Token Claim Name` is `policy`, `Claim JSON Type` is `string`, then Save.
- Open http://localhost:8080/auth/realms/demo/.well-known/openid-configuration and see if it has `authorization_endpoint` and `jwks_uri`
## 3. Configure MinIO
```
$ export MINIO_ACCESS_KEY=minio
$ export MINIO_SECRET_KEY=minio123
$ minio server /mnt/export
```
Set `identity_openid` config and restart MinIO
```
~ mc admin config set myminio identity_openid config_url="http://localhost:8080/auth/realms/demo/.well-known/openid-configuration" state="on"
```
Once successfully set restart the MinIO instance.
```
mc admin service restart myminio
```
## 4. Using WebIdentiy API
Client ID and Client Secret can be found by clicking any of the clients listed [here](http://localhost:8080/auth/admin/master/console/#/realms/demo/clients). If you have followed the above steps docs, the default Client ID will be `account` and Client Secret can be found under `Credentials` tab.
```
$ go run web-identity.go -cid account -csec e61cb282-745b-4113-bece-29b921c735f0 -auth-ep http://localhost:8080/auth/realms/demo/protocol/openid-connect/auth -token-ep http://localhost:8080/auth/realms/demo/protocol/openid-connect/token -port 8888
2018/12/26 17:49:36 listening on http://localhost:8888/
```
This will open the login page of keycloak, upon successful login, STS credentials will be printed on the screen, for example
```
##### Credentials
{
"accessKey": "6N2BALX7ELO827DXS3GK",
"secretKey": "23JKqAD+um8ObHqzfIh+bfqwG9V8qs9tFY6MqeFR",
"expiration": "2019-10-01T07:22:34Z",
"sessionToken": "eyJhbGciOiJIUzUxMiIsInR5cCI6IkpXVCJ9.eyJhY2Nlc3NLZXkiOiI2TjJCQUxYN0VMTzgyN0RYUzNHSyIsImFjciI6IjAiLCJhdWQiOiJhY2NvdW50IiwiYXV0aF90aW1lIjoxNTY5OTEwNTUyLCJhenAiOiJhY2NvdW50IiwiZW1haWxfdmVyaWZpZWQiOmZhbHNlLCJleHAiOjE1Njk5MTQ1NTQsImlhdCI6MTU2OTkxMDk1NCwiaXNzIjoiaHR0cDovL2xvY2FsaG9zdDo4MDgxL2F1dGgvcmVhbG1zL2RlbW8iLCJqdGkiOiJkOTk4YTBlZS01NDk2LTQ4OWYtYWJlMi00ZWE5MjJiZDlhYWYiLCJuYmYiOjAsInBvbGljeSI6InJlYWR3cml0ZSIsInByZWZlcnJlZF91c2VybmFtZSI6Im5ld3VzZXIxIiwic2Vzc2lvbl9zdGF0ZSI6IjJiYTAyYTI2LWE5MTUtNDUxNC04M2M1LWE0YjgwYjc4ZTgxNyIsInN1YiI6IjY4ZmMzODVhLTA5MjItNGQyMS04N2U5LTZkZTdhYjA3Njc2NSIsInR5cCI6IklEIn0._UG_-ZHgwdRnsp0gFdwChb7VlbPs-Gr_RNUz9EV7TggCD59qjCFAKjNrVHfOSVkKvYEMe0PvwfRKjnJl3A_mBA"
}
```
These credentials can now be used to perform MinIO API operations.
## 5. Using MinIO Browser
- Open MinIO url on the browser, for example `http://localhost:9000`
- Click on `Log in with OpenID`
- Provide `Client ID` and press ENTER
- Now the user will be redirected to the Keycloak login page, upon successful login the user will be redirected to MinIO page and logged in automatically
## Explore Further
- [MinIO STS Quickstart Guide](https://docs.min.io/docs/minio-sts-quickstart-guide)
- [The MinIO documentation website](https://docs.min.io)

43
docs/sts/ldap.md
View File

@@ -40,26 +40,31 @@ MinIO can be configured to find the groups of a user from AD/LDAP by specifying
LDAP is configured via the following environment variables:
| Variable | Required? | Purpose |
|----------------------------------------------|---------------------------|--------------------------------------------------------|
| **MINIO_IDENTITY_LDAP_SERVER_ADDR** | **YES** | AD/LDAP server address |
| **MINIO_IDENTITY_LDAP_USERNAME_FORMAT** | **YES** | Format of full username DN |
| **MINIO_IDENTITY_LDAP_GROUP_SEARCH_BASE_DN** | **NO** | Base DN in AD/LDAP hierarchy to use in search requests |
| **MINIO_IDENTITY_LDAP_GROUP_SEARCH_FILTER** | **NO** | Search filter to find groups of a user |
| **MINIO_IDENTITY_LDAP_GROUP_NAME_ATTRIBUTE** | **NO** | Attribute of search results to use as group name |
| **MINIO_IDENTITY_LDAP_STS_EXPIRY** | **NO** (default: "1h") | STS credentials validity duration |
| Variable | Required? | Purpose |
|----------------------------------------------|-------------------------|-------------------------------------------------------------------------|
| **MINIO_IDENTITY_LDAP_STATE** | **YES** | Enable or disable ldap identity |
| **MINIO_IDENTITY_LDAP_SERVER_ADDR** | **YES** | AD/LDAP server address |
| **MINIO_IDENTITY_LDAP_USERNAME_FORMAT** | **YES** | Format of full username DN |
| **MINIO_IDENTITY_LDAP_GROUP_SEARCH_BASE_DN** | **NO** | Base DN in AD/LDAP hierarchy to use in search requests |
| **MINIO_IDENTITY_LDAP_GROUP_SEARCH_FILTER** | **NO** | Search filter to find groups of a user |
| **MINIO_IDENTITY_LDAP_GROUP_NAME_ATTRIBUTE** | **NO** | Attribute of search results to use as group name |
| **MINIO_IDENTITY_LDAP_STS_EXPIRY** | **NO** (default: "1h") | STS credentials validity duration |
| **MINIO_IDENTITY_LDAP_TLS_SKIP_VERIFY** | **NO** (default: "off") | Set this to 'on', to disable client verification of server certificates |
Please note that MinIO will only access the AD/LDAP server over TLS. If a self-signed certificate is being used, the certificate can be added to MinIO's certificates directory, so it can be trusted by the server.
An example setup for development or experimentation:
``` shell
export MINIO_IDENTITY_LDAP_STATE="on"
export MINIO_IDENTITY_LDAP_SERVER_ADDR=myldapserver.com:636
export MINIO_IDENTITY_LDAP_USERNAME_FORMAT="uid=${username},cn=accounts,dc=myldapserver,dc=com"
export MINIO_IDENTITY_LDAP_USERNAME_FORMAT="uid={username},cn=accounts,dc=myldapserver,dc=com"
export MINIO_IDENTITY_LDAP_GROUP_SEARCH_BASE_DN="dc=myldapserver,dc=com"
export MINIO_IDENTITY_LDAP_GROUP_SEARCH_FILTER="(&(objectclass=groupOfNames)(member=${usernamedn}))"
export MINIO_IDENTITY_LDAP_GROUP_SEARCH_FILTER="(&(objectclass=groupOfNames)(member={usernamedn})$)"
export MINIO_IDENTITY_LDAP_GROUP_NAME_ATTRIBUTE="cn"
export MINIO_IDENTITY_LDAP_STS_EXPIRY=60
export MINIO_IDENTITY_LDAP_STS_EXPIRY=60h
export MINIO_IDENTITY_LDAP_TLS_SKIP_VERIFY="on"
```
### Variable substitution in AD/LDAP configuration strings
@@ -99,18 +104,20 @@ member: CN=John,CN=Users,DC=minioad,DC=local
...
```
The lines with "..." represent skipped content not shown here from brevity. Based on the output above, we see that the username format variable looks like `cn=${username},cn=users,dc=minioad,dc=local`.
The lines with "..." represent skipped content not shown here from brevity. Based on the output above, we see that the username format variable looks like `cn={username},cn=users,dc=minioad,dc=local`.
The group search filter looks like `(&(objectclass=group)(member=${usernamedn}))` and the group name attribute is clearly `cn`.
The group search filter looks like `(&(objectclass=group)(member={usernamedn}))` and the group name attribute is clearly `cn`.
Thus the key configuration parameters look like:
```
MINIO_IDENTITY_LDAP_STATE="on"
MINIO_IDENTITY_LDAP_SERVER_ADDR='my.ldap-active-dir-server.com:636'
MINIO_IDENTITY_LDAP_USERNAME_FORMAT='cn=${username},cn=users,dc=minioad,dc=local'
MINIO_IDENTITY_LDAP_USERNAME_FORMAT='cn={username},cn=users,dc=minioad,dc=local'
MINIO_IDENTITY_LDAP_GROUP_SEARCH_BASE_DN='dc=minioad,dc=local'
MINIO_IDENTITY_LDAP_GROUP_SEARCH_FILTER='(&(objectclass=group)(member=${usernamedn}))'
MINIO_IDENTITY_LDAP_GROUP_SEARCH_FILTER='(&(objectclass=group)(member={usernamedn}))'
MINIO_IDENTITY_LDAP_GROUP_NAME_ATTRIBUTE='cn'
MINIO_IDENTITY_LDAP_TLS_SKIP_VERIFY="on"
```
## Managing User/Group Access Policy
@@ -211,10 +218,11 @@ http://minio.cluster:9000?Action=AssumeRoleWithLDAPIdentity&LDAPUsername=foouser
```
$ export MINIO_ACCESS_KEY=minio
$ export MINIO_SECRET_KEY=minio123
$ export MINIO_IDENTITY_LDAP_STATE="on"
$ export MINIO_IDENTITY_LDAP_SERVER_ADDR='my.ldap-active-dir-server.com:636'
$ export MINIO_IDENTITY_LDAP_USERNAME_FORMAT='cn=${username},cn=users,dc=minioad,dc=local'
$ export MINIO_IDENTITY_LDAP_USERNAME_FORMAT='cn={username},cn=users,dc=minioad,dc=local'
$ export MINIO_IDENTITY_LDAP_GROUP_SEARCH_BASE_DN='dc=minioad,dc=local'
$ export MINIO_IDENTITY_LDAP_GROUP_SEARCH_FILTER='(&(objectclass=group)(member=${usernamedn}))'
$ export MINIO_IDENTITY_LDAP_GROUP_SEARCH_FILTER='(&(objectclass=group)(member={usernamedn}))'
$ export MINIO_IDENTITY_LDAP_GROUP_NAME_ATTRIBUTE='cn'
$ minio server ~/test
```
@@ -230,4 +238,3 @@ $ go run ldap.go -u foouser -p foopassword
"sessionToken": "eyJhbGciOiJIUzUxMiIsInR5cCI6IkpXVCJ9.eyJhY2Nlc3NLZXkiOiJOVUlCT1JaWVRWMkhHMkJNUlNYUiIsImF1ZCI6IlBvRWdYUDZ1Vk80NUlzRU5SbmdEWGo1QXU1WWEiLCJhenAiOiJQb0VnWFA2dVZPNDVJc0VOUm5nRFhqNUF1NVlhIiwiZXhwIjoxNTM0ODk2NjI5LCJpYXQiOjE1MzQ4OTMwMjksImlzcyI6Imh0dHBzOi8vbG9jYWxob3N0Ojk0NDMvb2F1dGgyL3Rva2VuIiwianRpIjoiNjY2OTZjZTctN2U1Ny00ZjU5LWI0MWQtM2E1YTMzZGZiNjA4In0.eJONnVaSVHypiXKEARSMnSKgr-2mlC2Sr4fEGJitLcJF_at3LeNdTHv0_oHsv6ZZA3zueVGgFlVXMlREgr9LXA"
}
```

4
docs/sts/opa.md
View File

@@ -61,9 +61,9 @@ curl -X PUT --data-binary @putobject.rego \
```
### 4. Setup MinIO with OPA
MinIO server expects environment variable for OPA http API url as `MINIO_IAM_OPA_URL`, this environment variable takes a single entry.
MinIO server expects environment variable for OPA http API url as `MINIO_POLICY_OPA_URL`, this environment variable takes a single entry.
```
export MINIO_IAM_OPA_URL=http://localhost:8181/v1/data/httpapi/authz/allow
export MINIO_POLICY_OPA_URL=http://localhost:8181/v1/data/httpapi/authz/allow
minio server /mnt/data
```

4
docs/sts/sts.env
View File

@@ -1,5 +1,5 @@
export MINIO_ACCESS_KEY=minio
export MINIO_SECRET_KEY=minio123
export MINIO_IAM_JWKS_URL=http://localhost:9763/oauth2/jwks
export MINIO_IAM_OPA_URL=http://localhost:8181/v1/data/httpapi/authz
export MINIO_IDENTITY_OPENID_CONFIG_URL=https://localhost:9443/oauth2/oidcdiscovery/.well-known/openid-configuration
export MINIO_POLICY_OPA_URL=http://localhost:8181/v1/data/httpapi/authz
export MINIO_ETCD_ENDPOINTS=http://localhost:2379

11
docs/sts/web-identity.go
View File

@@ -78,6 +78,7 @@ var (
tokenEndpoint string
clientID string
clientSecret string
port int
)
func init() {
@@ -86,6 +87,7 @@ func init() {
flag.StringVar(&tokenEndpoint, "token-ep", googleOAuth2.Endpoint.TokenURL, "Token endpoint")
flag.StringVar(&clientID, "cid", "", "Client ID")
flag.StringVar(&clientSecret, "csec", "", "Client secret")
flag.IntVar(&port, "port", 8080, "Port")
}
func main() {
@@ -104,7 +106,7 @@ func main() {
AuthURL: authEndpoint,
TokenURL: tokenEndpoint,
},
RedirectURL: "http://localhost:8080/oauth2/callback",
RedirectURL: fmt.Sprintf("http://localhost:%v/oauth2/callback", port),
Scopes: []string{"openid", "profile", "email"},
}
@@ -140,7 +142,7 @@ func main() {
}
u.RawQuery = v.Encode()
req, err := http.NewRequest("POST", u.String(), nil)
req, err := http.NewRequest(http.MethodPost, u.String(), nil)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
@@ -173,6 +175,7 @@ func main() {
}
})
log.Printf("listening on http://%s/", "localhost:8080")
log.Fatal(http.ListenAndServe("localhost:8080", nil))
address := fmt.Sprintf("localhost:%v", port)
log.Printf("listening on http://%s/", address)
log.Fatal(http.ListenAndServe(address, nil))
}

36
docs/sts/web-identity.md
View File

@@ -14,6 +14,7 @@
- [Sample Response](#sample-response)
- [Testing](#testing)
- [Authorization Flow](#authorization-flow)
- [MinIO Browser](#minio-browser)
## Introduction
@@ -92,18 +93,11 @@ http://minio.cluster:9000?Action=AssumeRoleWithWebIdentity&DurationSeconds=3600&
```
$ export MINIO_ACCESS_KEY=minio
$ export MINIO_SECRET_KEY=minio123
$ export MINIO_IAM_JWKS_URL=https://www.googleapis.com/oauth2/v3/certs
$ export MINIO_IDENTITY_OPENID_CONFIG_URL=https://accounts.google.com/.well-known/openid-configuration
$ minio server /mnt/export
$ mc admin config get myminio
...
{
"openid": {
"jwks": {
"url": "https://www.googleapis.com/oauth2/v3/certs"
}
}
}
$ mc admin config get myminio identity_openid
identity_openid config_url="https://accounts.google.com/.well-known/openid-configuration" state="on"
```
Testing with an example
@@ -121,3 +115,25 @@ $ go run web-identity.go -cid 204367807228-ok7601k6gj1pgge7m09h7d79co8p35xx.apps
- Using the access token the callback handler further talks to Google OAuth2 Token URL to obtain an JWT id_token.
- Once obtained the JWT id_token is further sent to STS endpoint i.e MinIO to retrive temporary credentials.
- Temporary credentials are displayed on the browser upon successful retrieval.
## MinIO Browser
To support WebIdentity login on MinIO Browser
1. Set openid configuration and restart MinIO
```
mc admin config set myminio identity_openid jwks_url="<JWKS_URL>" config_url="<CONFIG_URL>"
mc admin service restart myminio
```
Sample URLs for Keycloak are
`config_url` - `http://localhost:8080/auth/realms/demo/.well-known/openid-configuration`,
`jwks_url` - `http://localhost:8080/auth/realms/demo/protocol/openid-connect/certs`
JWT token returned by the Identity Provider should include a custom claim for the policy, this is required to create a STS user in MinIO. The name of the custom claim could be either `policy` or `<NAMESPACE_PREFIX>policy`.
If there is no namespace then `policy_claim_prefix` can be ingored. For example if the custom claim name is `https://min.io/policy` then, `policy_claim_prefix` should be set as `https://min.io/`
2. Open MinIO Browser and click `Log in with OpenID`
3. Enter the `Client ID` obtained from Identity Provider and press ENTER
4. The user will be redirected to the Identity Provider login page
5. Upon successful login on Identity Provider page the user will be automatically logged into MinIO Browser

6
docs/sts/wso2.md
View File

@@ -66,10 +66,10 @@ Using the above `access_token` we can perform an STS request to MinIO to get tem
**We recommend setting `policy` as a custom claim for the JWT service provider follow [here](https://docs.wso2.com/display/IS550/Configuring+Claims+for+a+Service+Provider) and [here](https://docs.wso2.com/display/IS550/Handling+Custom+Claims+with+the+JWT+Bearer+Grant+Type) for relevant docs on how to configure claims for a service provider.**
### 5. Setup MinIO with JWKS URL
MinIO server expects environment variable for JWKS url as `MINIO_IAM_JWKS_URL`, this environment variable takes a single entry.
### 5. Setup MinIO with OpenID configuration URL
MinIO server expects environment variable for OpenID configuration url as `MINIO_IDENTITY_OPENID_CONFIG_URL`, this environment variable takes a single entry.
```
export MINIO_IAM_JWKS_URL=https://localhost:9443/oauth2/jwks
export MINIO_IDENTITY_OPENID_CONFIG_URL=https://localhost:9443/oauth2/oidcdiscovery/.well-known/openid-configuration
minio server /mnt/data
```

4
docs/zh_CN/config/README.md
View File

@@ -48,12 +48,12 @@ minio server /data
#### 区域（Region）
|参数|类型|描述|
|:---|:---|:---|
|``region``| _string_ | `region`描述的是服务器的物理位置，默认是`us-east-1`（美国东区1）,这也是亚马逊S3的默认区域。你可以通过`MINIO_REGION` 环境变量进行修改。如果不了解这块，建议不要随意修改|
|``region``| _string_ | `region`描述的是服务器的物理位置，默认是`us-east-1`（美国东区1）,这也是亚马逊S3的默认区域。你可以通过`MINIO_REGION_NAME` 环境变量进行修改。如果不了解这块，建议不要随意修改|
示例:
```sh
export MINIO_REGION="中国华北一区"
export MINIO_REGION_NAME="中国华北一区"
minio server /data
```