mirror of
https://github.com/minio/minio.git
synced 2025-11-07 12:52:58 -05:00
cleanup markdown docs across multiple files (#14296)
enable markdown-linter
This commit is contained in:
Harshavardhana
GitHub
@@ -6,11 +6,13 @@ Transition tiers can be added to MinIO using `mc admin tier add` command to asso
|
||||
Lifecycle transition rules can be applied to buckets (both versioned and un-versioned) by specifying the tier name defined above as the transition storage class for the lifecycle rule.
|
||||
|
||||
## Implementation
|
||||
|
||||
ILM tiering takes place when a object placed in the bucket meets lifecycle transition rules and becomes eligible for tiering. MinIO scanner (which runs at one minute intervals, each time scanning one sixteenth of the namespace), picks up the object for tiering. The data is moved to the remote tier in entirety, leaving only the object metadata on MinIO.
|
||||
|
||||
The data on the backend is stored under the `bucket/prefix` specified in the tier configuration with a custom name derived from a randomly generated uuid - e.g. `0b/c4/0bc4fab7-2daf-4d2f-8e39-5c6c6fb7e2d3`. The first two prefixes are characters 1-2,3-4 from the uuid. This format allows tiering to any cloud irrespective of whether the cloud in question supports versioning. The reference to the transitioned object name and transitioned tier is stored as part of the internal metadata for the object (or its version) on MinIO.
|
||||
|
||||
Extra metadata maintained internally in `xl.meta` for a transitioned object
|
||||
|
||||
```
|
||||
...
|
||||
"MetaSys": {
|
||||
@@ -21,6 +23,7 @@ Extra metadata maintained internally in `xl.meta` for a transitioned object
|
||||
```
|
||||
|
||||
When a transitioned object is restored temporarily to local MinIO instance via PostRestoreObject API, the object data is copied back from the remote tier, and additional metadata for the restored object is maintained as referenced below. Once the restore period expires, the local copy of the object is removed by the scanner during its periodic runs.
|
||||
|
||||
```
|
||||
...
|
||||
"MetaUsr": {
|
||||
@@ -29,16 +32,24 @@ When a transitioned object is restored temporarily to local MinIO instance via P
|
||||
"x-amz-restore": "ongoing-request=false, expiry-date=Sat, 27 Feb 2021 00:00:00 GMT",
|
||||
...
|
||||
```
|
||||
|
||||
### Encrypted/Object locked objects
|
||||
|
||||
For objects under SSE-S3 or SSE-C encryption, the encrypted content from MinIO cluster is copied as is to the remote tier without any decryption. The content is decrypted as it is streamed from remote tier on `GET/HEAD`. Objects under retention are protected because the metadata present on MinIO server ensures that the object (version) is not deleted until retention period is over. Administrators need to ensure that the remote tier bucket is under proper access control.
|
||||
|
||||
### Transition Status
|
||||
|
||||
MinIO specific extension header `X-Minio-Transition` is displayed on `HEAD/GET` to predict expected transition date on the object. Once object is transitioned to the remote tier,`x-amz-storage-class` shows the tier name to which object transitioned. Additional headers such as "X-Amz-Restore-Expiry-Days", "x-amz-restore", and "X-Amz-Restore-Request-Date" are displayed when a object is under restore/has been restored to local MinIO cluster.
|
||||
|
||||
### Expiry or removal events
|
||||
|
||||
An object that is in transition tier will be deleted once the object hits expiry date or if removed via `mc rm` (`mc rm --vid` in the case of delete of a specific object version). Other rules specific to legal hold and object locking precede any lifecycle rules.
|
||||
|
||||
### Additional notes
|
||||
|
||||
Tiering and lifecycle transition are applicable only to erasure/distributed MinIO.
|
||||
|
||||
## Explore Further
|
||||
|
||||
- [MinIO | Golang Client API Reference](https://docs.min.io/docs/golang-client-api-reference.html#SetBucketLifecycle)
|
||||
- [Object Lifecycle Management](https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lifecycle-mgmt.html)
|
||||
|
||||
@@ -3,6 +3,7 @@
|
||||
Enable object lifecycle configuration on buckets to setup automatic deletion of objects after a specified number of days or a specified date.
|
||||
|
||||
## 1. Prerequisites
|
||||
|
||||
- Install MinIO - [MinIO Quickstart Guide](https://docs.min.io/docs/minio-quickstart-guide).
|
||||
- Install `mc` - [mc Quickstart Guide](https://docs.minio.io/docs/minio-client-quickstart-guide.html)
|
||||
|
||||
@@ -45,6 +46,7 @@ Lifecycle configuration imported successfully to `play/testbucket`.
|
||||
```
|
||||
|
||||
- List the current settings
|
||||
|
||||
```
|
||||
$ mc ilm ls play/testbucket
|
||||
ID | Prefix | Enabled | Expiry | Date/Days | Transition | Date/Days | Storage-Class | Tags
|
||||
@@ -64,6 +66,7 @@ This will only work with a versioned bucket, take a look at [Bucket Versioning G
|
||||
A non-current object version is a version which is not the latest for a given object. It is possible to set up an automatic removal of non-current versions when a version becomes older than a given number of days.
|
||||
|
||||
e.g., To scan objects stored under `user-uploads/` prefix and remove versions older than one year.
|
||||
|
||||
```
|
||||
{
|
||||
"Rules": [
|
||||
@@ -81,12 +84,12 @@ e.g., To scan objects stored under `user-uploads/` prefix and remove versions ol
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
### 3.2 Automatic removal of noncurrent versions keeping only most recent ones after noncurrent days
|
||||
|
||||
It is possible to configure automatic removal of older noncurrent versions keeping only the most recent `N` using `NewerNoncurrentVersions`.
|
||||
|
||||
e.g, To remove noncurrent versions of all objects keeping the most recent 5 noncurrent versions under the prefix `user-uploads/` 30 days after they become noncurrent ,
|
||||
|
||||
```
|
||||
{
|
||||
"Rules": [
|
||||
@@ -126,6 +129,7 @@ of objects under the prefix `user-uploads/` as soon as there are more than `N` n
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Note: This rule has an implicit zero NoncurrentDays, which makes the expiry of those 'extra' noncurrent versions immediate.
|
||||
|
||||
### 3.3 Automatic removal of delete markers with no other versions
|
||||
@@ -145,6 +149,7 @@ When an object has only one version as a delete marker, the latter can be automa
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## 4. Enable ILM transition feature
|
||||
|
||||
In Erasure mode, MinIO supports tiering to public cloud providers such as GCS, AWS and Azure as well as to other MinIO clusters via the ILM transition feature. This will allow transitioning of older objects to a different cluster or the public cloud by setting up transition rules in the bucket lifecycle configuration. This feature enables applications to optimize storage costs by moving less frequently accessed data to a cheaper storage without compromising accessibility of data.
|
||||
@@ -156,14 +161,17 @@ To transition objects in a bucket to a destination bucket on a different cluster
|
||||
```
|
||||
mc admin tier add azure source AZURETIER --endpoint https://blob.core.windows.net --access-key AZURE_ACCOUNT_NAME --secret-key AZURE_ACCOUNT_KEY --bucket azurebucket --prefix testprefix1/
|
||||
```
|
||||
|
||||
> The admin user running this command needs the "admin:SetTier" and "admin:ListTier" permissions if not running as root.
|
||||
|
||||
Using above tier, set up a lifecycle rule with transition:
|
||||
|
||||
```
|
||||
mc ilm add --expiry-days 365 --transition-days 45 --storage-class "AZURETIER" myminio/srcbucket
|
||||
```
|
||||
|
||||
Note: In the case of S3, it is possible to create a tier from MinIO running in EC2 to S3 using AWS role attached to EC2 as credentials instead of accesskey/secretkey:
|
||||
|
||||
```
|
||||
mc admin tier add s3 source S3TIER --bucket s3bucket --prefix testprefix/ --use-aws-role
|
||||
```
|
||||
@@ -177,10 +185,12 @@ aws s3api restore-object --bucket srcbucket \
|
||||
```
|
||||
|
||||
### 4.1 Monitoring transition events
|
||||
|
||||
`s3:ObjectTransition:Complete` and `s3:ObjectTransition:Failed` events can be used to monitor transition events between the source cluster and transition tier. To watch lifecycle events, you can enable bucket notification on the source bucket with `mc event add` and specify `--event ilm` flag.
|
||||
|
||||
Note that transition event notification is a MinIO extension.
|
||||
|
||||
## Explore Further
|
||||
|
||||
- [MinIO | Golang Client API Reference](https://docs.min.io/docs/golang-client-api-reference.html#SetBucketLifecycle)
|
||||
- [Object Lifecycle Management](https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lifecycle-mgmt.html)
|
||||
|
||||
@@ -32,7 +32,6 @@ Various event types supported by MinIO server are
|
||||
| `s3:BucketCreated` |
|
||||
| `s3:BucketRemoved` |
|
||||
|
||||
|
||||
Use client tools like `mc` to set and listen for event notifications using the [`event` sub-command](https://docs.min.io/docs/minio-client-complete-guide#events). MinIO SDK's [`BucketNotification` APIs](https://docs.min.io/docs/golang-client-api-reference#SetBucketNotification) can also be used. The notification message MinIO sends to publish an event is a JSON message with the following [structure](https://docs.aws.amazon.com/AmazonS3/latest/dev/notification-content-structure.html).
|
||||
|
||||
Bucket events can be published to the following targets:
|
||||
@@ -64,12 +63,11 @@ notify_redis publish bucket notifications to Redis datastores
|
||||
```
|
||||
|
||||
> NOTE:
|
||||
>
|
||||
> - '\*' at the end of arg means its mandatory.
|
||||
> - '\*' at the end of the values, means its the default value for the arg.
|
||||
> - When configured using environment variables, the `:name` can be specified using this format `MINIO_NOTIFY_WEBHOOK_ENABLE_<name>`.
|
||||
|
||||
<a name="AMQP"></a>
|
||||
|
||||
## Publish MinIO events via AMQP
|
||||
|
||||
Install RabbitMQ from [here](https://www.rabbitmq.com/).
|
||||
@@ -135,7 +133,7 @@ Use `mc admin config set` command to update the configuration for the deployment
|
||||
An example configuration for RabbitMQ is shown below:
|
||||
|
||||
```sh
|
||||
$ mc admin config set myminio/ notify_amqp:1 exchange="bucketevents" exchange_type="fanout" mandatory="false" no_wait="false" url="amqp://myuser:mypassword@localhost:5672" auto_deleted="false" delivery_mode="0" durable="false" internal="false" routing_key="bucketlogs"
|
||||
mc admin config set myminio/ notify_amqp:1 exchange="bucketevents" exchange_type="fanout" mandatory="false" no_wait="false" url="amqp://myuser:mypassword@localhost:5672" auto_deleted="false" delivery_mode="0" durable="false" internal="false" routing_key="bucketlogs"
|
||||
```
|
||||
|
||||
MinIO supports all the exchanges available in [RabbitMQ](https://www.rabbitmq.com/). For this setup, we are using `fanout` exchange.
|
||||
@@ -144,7 +142,7 @@ MinIO also sends with the notifications two headers: `minio-bucket` and `minio-e
|
||||
|
||||
Note that, you can add as many AMQP server endpoint configurations as needed by providing an identifier (like "1" in the example above) for the AMQP instance and an object of per-server configuration parameters.
|
||||
|
||||
### Step 2: Enable bucket notification using MinIO client
|
||||
### Step 2: Enable RabbitMQ bucket notification using MinIO client
|
||||
|
||||
We will enable bucket event notification to trigger whenever a JPEG image is uploaded or deleted `images` bucket on `myminio` server. Here ARN value is `arn:minio:sqs::1:amqp`. To understand more about ARN please follow [AWS ARN](http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html) documentation.
|
||||
|
||||
@@ -207,8 +205,6 @@ python rabbit.py
|
||||
'{"Records":[{"eventVersion":"2.0","eventSource":"aws:s3","awsRegion":"","eventTime":"2016–09–08T22:34:38.226Z","eventName":"s3:ObjectCreated:Put","userIdentity":{"principalId":"minio"},"requestParameters":{"sourceIPAddress":"10.1.10.150:44576"},"responseElements":{},"s3":{"s3SchemaVersion":"1.0","configurationId":"Config","bucket":{"name":"images","ownerIdentity":{"principalId":"minio"},"arn":"arn:aws:s3:::images"},"object":{"key":"myphoto.jpg","size":200436,"sequencer":"147279EAF9F40933"}}}],"level":"info","msg":"","time":"2016–09–08T15:34:38–07:00"}'
|
||||
```
|
||||
|
||||
<a name="MQTT"></a>
|
||||
|
||||
## Publish MinIO events MQTT
|
||||
|
||||
Install an MQTT Broker from [here](https://mosquitto.org/).
|
||||
@@ -266,14 +262,14 @@ notify_mqtt:1 broker="" password="" queue_dir="" queue_limit="0" reconnect_inter
|
||||
Use `mc admin config set` command to update the configuration for the deployment. Restart the MinIO server to put the changes into effect. The server will print a line like `SQS ARNs: arn:minio:sqs::1:mqtt` at start-up if there were no errors.
|
||||
|
||||
```sh
|
||||
$ mc admin config set myminio notify_mqtt:1 broker="tcp://localhost:1883" password="" queue_dir="" queue_limit="0" reconnect_interval="0s" keep_alive_interval="0s" qos="1" topic="minio" username=""
|
||||
mc admin config set myminio notify_mqtt:1 broker="tcp://localhost:1883" password="" queue_dir="" queue_limit="0" reconnect_interval="0s" keep_alive_interval="0s" qos="1" topic="minio" username=""
|
||||
```
|
||||
|
||||
MinIO supports any MQTT server that supports MQTT 3.1 or 3.1.1 and can connect to them over TCP, TLS, or a Websocket connection using `tcp://`, `tls://`, or `ws://` respectively as the scheme for the broker url. See the [Go Client](http://www.eclipse.org/paho/clients/golang/) documentation for more information.
|
||||
|
||||
Note that, you can add as many MQTT server endpoint configurations as needed by providing an identifier (like "1" in the example above) for the MQTT instance and an object of per-server configuration parameters.
|
||||
|
||||
### Step 2: Enable bucket notification using MinIO client
|
||||
### Step 2: Enable MQTT bucket notification using MinIO client
|
||||
|
||||
We will enable bucket event notification to trigger whenever a JPEG image is uploaded or deleted `images` bucket on `myminio` server. Here ARN value is `arn:minio:sqs::1:mqtt`.
|
||||
|
||||
@@ -332,8 +328,6 @@ python mqtt.py
|
||||
{“Records”:[{“eventVersion”:”2.0",”eventSource”:”aws:s3",”awsRegion”:”",”eventTime”:”2016–09–08T22:34:38.226Z”,”eventName”:”s3:ObjectCreated:Put”,”userIdentity”:{“principalId”:”minio”},”requestParameters”:{“sourceIPAddress”:”10.1.10.150:44576"},”responseElements”:{},”s3":{“s3SchemaVersion”:”1.0",”configurationId”:”Config”,”bucket”:{“name”:”images”,”ownerIdentity”:{“principalId”:”minio”},”arn”:”arn:aws:s3:::images”},”object”:{“key”:”myphoto.jpg”,”size”:200436,”sequencer”:”147279EAF9F40933"}}}],”level”:”info”,”msg”:””,”time”:”2016–09–08T15:34:38–07:00"}
|
||||
```
|
||||
|
||||
<a name="Elasticsearch"></a>
|
||||
|
||||
## Publish MinIO events via Elasticsearch
|
||||
|
||||
Install [Elasticsearch](https://www.elastic.co/downloads/elasticsearch) server.
|
||||
@@ -346,7 +340,7 @@ When the _access_ format is used, MinIO appends events as documents in an Elasti
|
||||
|
||||
The steps below show how to use this notification target in `namespace` format. The other format is very similar and is omitted for brevity.
|
||||
|
||||
### Step 1: Ensure minimum requirements are met
|
||||
### Step 1: Ensure Elasticsearch minimum requirements are met
|
||||
|
||||
MinIO requires a 5.x series version of Elasticsearch. This is the latest major release series. Elasticsearch provides version upgrade migration guidelines [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-upgrade.html).
|
||||
|
||||
@@ -403,12 +397,12 @@ notify_elasticsearch:1 queue_limit="0" url="" format="namespace" index="" queue
|
||||
Use `mc admin config set` command to update the configuration for the deployment. Restart the MinIO server to put the changes into effect. The server will print a line like `SQS ARNs: arn:minio:sqs::1:elasticsearch` at start-up if there were no errors.
|
||||
|
||||
```sh
|
||||
$ mc admin config set myminio notify_elasticsearch:1 queue_limit="0" url="http://127.0.0.1:9200" format="namespace" index="minio_events" queue_dir="" username="" password=""
|
||||
mc admin config set myminio notify_elasticsearch:1 queue_limit="0" url="http://127.0.0.1:9200" format="namespace" index="minio_events" queue_dir="" username="" password=""
|
||||
```
|
||||
|
||||
Note that, you can add as many Elasticsearch server endpoint configurations as needed by providing an identifier (like "1" in the example above) for the Elasticsearch instance and an object of per-server configuration parameters.
|
||||
|
||||
### Step 3: Enable bucket notification using MinIO client
|
||||
### Step 3: Enable Elastichsearch bucket notification using MinIO client
|
||||
|
||||
We will now enable bucket event notifications on a bucket named `images`. Whenever a JPEG image is created/overwritten, a new document is added or an existing document is updated in the Elasticsearch index configured above. When an existing object is deleted, the corresponding document is deleted from the index. Thus, the rows in the Elasticsearch index, reflect the `.jpg` objects in the `images` bucket.
|
||||
|
||||
@@ -505,8 +499,6 @@ This output shows that a document has been created for the event in Elasticsearc
|
||||
|
||||
Here we see that the document ID is the bucket and object name. In case `access` format was used, the document ID would be automatically generated by Elasticsearch.
|
||||
|
||||
<a name="Redis"></a>
|
||||
|
||||
## Publish MinIO events via Redis
|
||||
|
||||
Install [Redis](http://redis.io/download) server. For illustrative purposes, we have set the database password as "yoursecret".
|
||||
@@ -565,12 +557,12 @@ notify_redis:1 address="" format="namespace" key="" password="" queue_dir="" que
|
||||
Use `mc admin config set` command to update the configuration for the deployment.Restart the MinIO server to put the changes into effect. The server will print a line like `SQS ARNs: arn:minio:sqs::1:redis` at start-up if there were no errors.
|
||||
|
||||
```sh
|
||||
$ mc admin config set myminio/ notify_redis:1 address="127.0.0.1:6379" format="namespace" key="bucketevents" password="yoursecret" queue_dir="" queue_limit="0"
|
||||
mc admin config set myminio/ notify_redis:1 address="127.0.0.1:6379" format="namespace" key="bucketevents" password="yoursecret" queue_dir="" queue_limit="0"
|
||||
```
|
||||
|
||||
Note that, you can add as many Redis server endpoint configurations as needed by providing an identifier (like "1" in the example above) for the Redis instance and an object of per-server configuration parameters.
|
||||
|
||||
### Step 2: Enable bucket notification using MinIO client
|
||||
### Step 2: Enable Redis bucket notification using MinIO client
|
||||
|
||||
We will now enable bucket event notifications on a bucket named `images`. Whenever a JPEG image is created/overwritten, a new key is added or an existing key is updated in the Redis hash configured above. When an existing object is deleted, the corresponding key is deleted from the Redis hash. Thus, the rows in the Redis hash, reflect the `.jpg` objects in the `images` bucket.
|
||||
|
||||
@@ -614,8 +606,6 @@ Here we see that MinIO performed `HSET` on `minio_events` key.
|
||||
|
||||
In case, `access` format was used, then `minio_events` would be a list, and the MinIO server would have performed an `RPUSH` to append to the list. A consumer of this list would ideally use `BLPOP` to remove list items from the left-end of the list.
|
||||
|
||||
<a name="NATS"></a>
|
||||
|
||||
## Publish MinIO events via NATS
|
||||
|
||||
Install NATS from [here](http://nats.io/).
|
||||
@@ -650,6 +640,7 @@ comment (sentence) optionally add a comment to this s
|
||||
```
|
||||
|
||||
or environment variables
|
||||
|
||||
```
|
||||
KEY:
|
||||
notify_nats[:name] publish bucket notifications to NATS endpoints
|
||||
@@ -686,14 +677,14 @@ notify_nats:1 password="yoursecret" streaming_max_pub_acks_in_flight="10" subjec
|
||||
Use `mc admin config set` command to update the configuration for the deployment.Restart MinIO server to reflect config changes. `bucketevents` is the subject used by NATS in this example.
|
||||
|
||||
```sh
|
||||
$ mc admin config set myminio notify_nats:1 password="yoursecret" streaming_max_pub_acks_in_flight="10" subject="" address="0.0.0.0:4222" token="" username="yourusername" ping_interval="0" queue_limit="0" tls="off" streaming_async="on" queue_dir="" streaming_cluster_id="test-cluster" streaming_enable="on"
|
||||
mc admin config set myminio notify_nats:1 password="yoursecret" streaming_max_pub_acks_in_flight="10" subject="" address="0.0.0.0:4222" token="" username="yourusername" ping_interval="0" queue_limit="0" tls="off" streaming_async="on" queue_dir="" streaming_cluster_id="test-cluster" streaming_enable="on"
|
||||
```
|
||||
|
||||
MinIO server also supports [NATS Streaming mode](http://nats.io/documentation/streaming/nats-streaming-intro/) that offers additional functionality like `At-least-once-delivery`, and `Publisher rate limiting`. To configure MinIO server to send notifications to NATS Streaming server, update the MinIO server configuration file as follows:
|
||||
|
||||
Read more about sections `cluster_id`, `client_id` on [NATS documentation](https://github.com/nats-io/nats-streaming-server/blob/master/README.md). Section `maxPubAcksInflight` is explained [here](https://github.com/nats-io/stan.go#publisher-rate-limiting).
|
||||
|
||||
### Step 2: Enable bucket notification using MinIO client
|
||||
### Step 2: Enable NATS bucket notification using MinIO client
|
||||
|
||||
We will enable bucket event notification to trigger whenever a JPEG image is uploaded or deleted from `images` bucket on `myminio` server. Here ARN value is `arn:minio:sqs::1:nats`. To understand more about ARN please follow [AWS ARN](http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html) documentation.
|
||||
|
||||
@@ -713,28 +704,28 @@ package main
|
||||
|
||||
// Import Go and NATS packages
|
||||
import (
|
||||
"log"
|
||||
"runtime"
|
||||
"log"
|
||||
"runtime"
|
||||
|
||||
"github.com/nats-io/nats.go"
|
||||
"github.com/nats-io/nats.go"
|
||||
)
|
||||
|
||||
func main() {
|
||||
|
||||
// Create server connection
|
||||
natsConnection, _ := nats.Connect("nats://yourusername:yoursecret@localhost:4222")
|
||||
log.Println("Connected")
|
||||
// Create server connection
|
||||
natsConnection, _ := nats.Connect("nats://yourusername:yoursecret@localhost:4222")
|
||||
log.Println("Connected")
|
||||
|
||||
// Subscribe to subject
|
||||
log.Printf("Subscribing to subject 'bucketevents'\n")
|
||||
natsConnection.Subscribe("bucketevents", func(msg *nats.Msg) {
|
||||
// Subscribe to subject
|
||||
log.Printf("Subscribing to subject 'bucketevents'\n")
|
||||
natsConnection.Subscribe("bucketevents", func(msg *nats.Msg) {
|
||||
|
||||
// Handle the message
|
||||
log.Printf("Received message '%s\n", string(msg.Data)+"'")
|
||||
})
|
||||
// Handle the message
|
||||
log.Printf("Received message '%s\n", string(msg.Data)+"'")
|
||||
})
|
||||
|
||||
// Keep the connection alive
|
||||
runtime.Goexit()
|
||||
// Keep the connection alive
|
||||
runtime.Goexit()
|
||||
}
|
||||
```
|
||||
|
||||
@@ -766,52 +757,52 @@ package main
|
||||
|
||||
// Import Go and NATS packages
|
||||
import (
|
||||
"fmt"
|
||||
"runtime"
|
||||
"fmt"
|
||||
"runtime"
|
||||
|
||||
"github.com/nats-io/stan.go"
|
||||
"github.com/nats-io/stan.go"
|
||||
)
|
||||
|
||||
func main() {
|
||||
|
||||
var stanConnection stan.Conn
|
||||
var stanConnection stan.Conn
|
||||
|
||||
subscribe := func() {
|
||||
fmt.Printf("Subscribing to subject 'bucketevents'\n")
|
||||
stanConnection.Subscribe("bucketevents", func(m *stan.Msg) {
|
||||
subscribe := func() {
|
||||
fmt.Printf("Subscribing to subject 'bucketevents'\n")
|
||||
stanConnection.Subscribe("bucketevents", func(m *stan.Msg) {
|
||||
|
||||
// Handle the message
|
||||
fmt.Printf("Received a message: %s\n", string(m.Data))
|
||||
})
|
||||
}
|
||||
// Handle the message
|
||||
fmt.Printf("Received a message: %s\n", string(m.Data))
|
||||
})
|
||||
}
|
||||
|
||||
|
||||
stanConnection, _ = stan.Connect("test-cluster", "test-client", stan.NatsURL("nats://yourusername:yoursecret@0.0.0.0:4222"), stan.SetConnectionLostHandler(func(c stan.Conn, _ error) {
|
||||
go func() {
|
||||
for {
|
||||
// Reconnect if the connection is lost.
|
||||
if stanConnection == nil || stanConnection.NatsConn() == nil || !stanConnection.NatsConn().IsConnected() {
|
||||
stanConnection, _ = stan.Connect("test-cluster", "test-client", stan.NatsURL("nats://yourusername:yoursecret@0.0.0.0:4222"), stan.SetConnectionLostHandler(func(c stan.Conn, _ error) {
|
||||
if c.NatsConn() != nil {
|
||||
c.NatsConn().Close()
|
||||
}
|
||||
_ = c.Close()
|
||||
}))
|
||||
if stanConnection != nil {
|
||||
subscribe()
|
||||
}
|
||||
stanConnection, _ = stan.Connect("test-cluster", "test-client", stan.NatsURL("nats://yourusername:yoursecret@0.0.0.0:4222"), stan.SetConnectionLostHandler(func(c stan.Conn, _ error) {
|
||||
go func() {
|
||||
for {
|
||||
// Reconnect if the connection is lost.
|
||||
if stanConnection == nil || stanConnection.NatsConn() == nil || !stanConnection.NatsConn().IsConnected() {
|
||||
stanConnection, _ = stan.Connect("test-cluster", "test-client", stan.NatsURL("nats://yourusername:yoursecret@0.0.0.0:4222"), stan.SetConnectionLostHandler(func(c stan.Conn, _ error) {
|
||||
if c.NatsConn() != nil {
|
||||
c.NatsConn().Close()
|
||||
}
|
||||
_ = c.Close()
|
||||
}))
|
||||
if stanConnection != nil {
|
||||
subscribe()
|
||||
}
|
||||
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
}()
|
||||
}))
|
||||
}()
|
||||
}))
|
||||
|
||||
// Subscribe to subject
|
||||
subscribe()
|
||||
// Subscribe to subject
|
||||
subscribe()
|
||||
|
||||
// Keep the connection alive
|
||||
runtime.Goexit()
|
||||
// Keep the connection alive
|
||||
runtime.Goexit()
|
||||
}
|
||||
|
||||
```
|
||||
@@ -834,8 +825,6 @@ The example `nats.go` program prints event notification to console.
|
||||
Received a message: {"EventType":"s3:ObjectCreated:Put","Key":"images/myphoto.jpg","Records":[{"eventVersion":"2.0","eventSource":"minio:s3","awsRegion":"","eventTime":"2017-07-07T18:46:37Z","eventName":"s3:ObjectCreated:Put","userIdentity":{"principalId":"minio"},"requestParameters":{"sourceIPAddress":"192.168.1.80:55328"},"responseElements":{"x-amz-request-id":"14CF20BD1EFD5B93","x-minio-origin-endpoint":"http://127.0.0.1:9000"},"s3":{"s3SchemaVersion":"1.0","configurationId":"Config","bucket":{"name":"images","ownerIdentity":{"principalId":"minio"},"arn":"arn:aws:s3:::images"},"object":{"key":"myphoto.jpg","size":248682,"eTag":"f1671feacb8bbf7b0397c6e9364e8c92","contentType":"image/jpeg","userDefined":{"content-type":"image/jpeg"},"versionId":"1","sequencer":"14CF20BD1EFD5B93"}},"source":{"host":"192.168.1.80","port":"55328","userAgent":"MinIO (linux; amd64) minio-go/2.0.4 mc/DEVELOPMENT.GOGET"}}],"level":"info","msg":"","time":"2017-07-07T11:46:37-07:00"}
|
||||
```
|
||||
|
||||
<a name="PostgreSQL"></a>
|
||||
|
||||
## Publish MinIO events via PostgreSQL
|
||||
|
||||
> NOTE: Until release RELEASE.2020-04-10T03-34-42Z PostgreSQL notification used to support following options:
|
||||
@@ -848,8 +837,8 @@ Received a message: {"EventType":"s3:ObjectCreated:Put","Key":"images/myphoto.jp
|
||||
> database (string) database name (used only if `connection_string` is empty)
|
||||
> ```
|
||||
>
|
||||
> These are now deprecated, if you plan to upgrade to any releases after *RELEASE.2020-04-10T03-34-42Z* make sure
|
||||
> to migrate to only using *connection_string* option. To migrate, once you have upgraded all the servers use the
|
||||
> These are now deprecated, if you plan to upgrade to any releases after _RELEASE.2020-04-10T03-34-42Z_ make sure
|
||||
> to migrate to only using _connection_string_ option. To migrate, once you have upgraded all the servers use the
|
||||
> following command to update the existing notification targets.
|
||||
>
|
||||
> ```
|
||||
@@ -858,7 +847,7 @@ Received a message: {"EventType":"s3:ObjectCreated:Put","Key":"images/myphoto.jp
|
||||
>
|
||||
> Please make sure this step is carried out, without this step PostgreSQL notification targets will not work,
|
||||
> an error message will be shown on the console upon server upgrade/restart, make sure to follow the above
|
||||
> instructions appropriately. For further questions please join our https://slack.min.io
|
||||
> instructions appropriately. For further questions please join our <https://slack.min.io>
|
||||
|
||||
Install [PostgreSQL](https://www.postgresql.org/) database server. For illustrative purposes, we have set the "postgres" user password as `password` and created a database called `minio_events` to store the events.
|
||||
|
||||
@@ -870,7 +859,7 @@ When the _access_ format is used, MinIO appends events to a table. It creates ro
|
||||
|
||||
The steps below show how to use this notification target in `namespace` format. The other format is very similar and is omitted for brevity.
|
||||
|
||||
### Step 1: Ensure minimum requirements are met
|
||||
### Step 1: Ensure PostgresSQL minimum requirements are met
|
||||
|
||||
MinIO requires PostgreSQL version 9.5 or above. MinIO uses the [`INSERT ON CONFLICT`](https://www.postgresql.org/docs/9.5/static/sql-insert.html#SQL-ON-CONFLICT) (aka UPSERT) feature, introduced in version 9.5 and the [JSONB](https://www.postgresql.org/docs/9.4/static/datatype-json.html) data-type introduced in version 9.4.
|
||||
|
||||
@@ -893,6 +882,7 @@ comment (sentence) optionally add a comment to this setti
|
||||
```
|
||||
|
||||
or environment variables
|
||||
|
||||
```
|
||||
KEY:
|
||||
notify_postgres[:name] publish bucket notifications to Postgres databases
|
||||
@@ -924,12 +914,12 @@ notify_postgres:1 queue_dir="" connection_string="" queue_limit="0" table="" fo
|
||||
Use `mc admin config set` command to update the configuration for the deployment. Restart the MinIO server to put the changes into effect. The server will print a line like `SQS ARNs: arn:minio:sqs::1:postgresql` at start-up if there were no errors.
|
||||
|
||||
```sh
|
||||
$ mc admin config set myminio notify_postgres:1 connection_string="host=localhost port=5432 dbname=minio_events user=postgres password=password sslmode=disable" table="bucketevents" format="namespace"
|
||||
mc admin config set myminio notify_postgres:1 connection_string="host=localhost port=5432 dbname=minio_events user=postgres password=password sslmode=disable" table="bucketevents" format="namespace"
|
||||
```
|
||||
|
||||
Note that, you can add as many PostgreSQL server endpoint configurations as needed by providing an identifier (like "1" in the example above) for the PostgreSQL instance and an object of per-server configuration parameters.
|
||||
|
||||
### Step 3: Enable bucket notification using MinIO client
|
||||
### Step 3: Enable PostgreSQL bucket notification using MinIO client
|
||||
|
||||
We will now enable bucket event notifications on a bucket named `images`. Whenever a JPEG image is created/overwritten, a new row is added or an existing row is updated in the PostgreSQL configured above. When an existing object is deleted, the corresponding row is deleted from the PostgreSQL table. Thus, the rows in the PostgreSQL table, reflect the `.jpg` objects in the `images` bucket.
|
||||
|
||||
@@ -968,8 +958,6 @@ key | value
|
||||
(1 row)
|
||||
```
|
||||
|
||||
<a name="MySQL"></a>
|
||||
|
||||
## Publish MinIO events via MySQL
|
||||
|
||||
> NOTE: Until release RELEASE.2020-04-10T03-34-42Z MySQL notification used to support following options:
|
||||
@@ -982,8 +970,8 @@ key | value
|
||||
> database (string) database name (used only if `dsn_string` is empty)
|
||||
> ```
|
||||
>
|
||||
> These are now deprecated, if you plan to upgrade to any releases after *RELEASE.2020-04-10T03-34-42Z* make sure
|
||||
> to migrate to only using *dsn_string* option. To migrate, once you have upgraded all the servers use the
|
||||
> These are now deprecated, if you plan to upgrade to any releases after _RELEASE.2020-04-10T03-34-42Z_ make sure
|
||||
> to migrate to only using _dsn_string_ option. To migrate, once you have upgraded all the servers use the
|
||||
> following command to update the existing notification targets.
|
||||
>
|
||||
> ```
|
||||
@@ -992,7 +980,7 @@ key | value
|
||||
>
|
||||
> Please make sure this step is carried out, without this step MySQL notification targets will not work,
|
||||
> an error message will be shown on the console upon server upgrade/restart, make sure to follow the above
|
||||
> instructions appropriately. For further questions please join our https://slack.min.io
|
||||
> instructions appropriately. For further questions please join our <https://slack.min.io>
|
||||
|
||||
Install MySQL from [here](https://dev.mysql.com/downloads/mysql/). For illustrative purposes, we have set the root password as `password` and created a database called `miniodb` to store the events.
|
||||
|
||||
@@ -1004,7 +992,7 @@ When the _access_ format is used, MinIO appends events to a table. It creates ro
|
||||
|
||||
The steps below show how to use this notification target in `namespace` format. The other format is very similar and is omitted for brevity.
|
||||
|
||||
### Step 1: Ensure minimum requirements are met
|
||||
### Step 1: Ensure MySQL minimum requirements are met
|
||||
|
||||
MinIO requires MySQL version 5.7.8 or above. MinIO uses the [JSON](https://dev.mysql.com/doc/refman/5.7/en/json.html) data-type introduced in version 5.7.8. We tested this setup on MySQL 5.7.17.
|
||||
|
||||
@@ -1027,6 +1015,7 @@ comment (sentence) optionally add a comment to this setti
|
||||
```
|
||||
|
||||
or environment variables
|
||||
|
||||
```
|
||||
KEY:
|
||||
notify_mysql[:name] publish bucket notifications to MySQL databases
|
||||
@@ -1059,14 +1048,14 @@ notify_mysql:myinstance enable=off format=namespace host= port= username= passwo
|
||||
Use `mc admin config set` command to update MySQL notification configuration for the deployment with `dsn_string` parameter:
|
||||
|
||||
```sh
|
||||
$ mc admin config set myminio notify_mysql:myinstance table="minio_images" dsn_string="root:xxxx@tcp(172.17.0.1:3306)/miniodb"
|
||||
mc admin config set myminio notify_mysql:myinstance table="minio_images" dsn_string="root:xxxx@tcp(172.17.0.1:3306)/miniodb"
|
||||
```
|
||||
|
||||
Note that, you can add as many MySQL server endpoint configurations as needed by providing an identifier (like "myinstance" in the example above) for each MySQL instance desired.
|
||||
|
||||
Restart the MinIO server to put the changes into effect. The server will print a line like `SQS ARNs: arn:minio:sqs::myinstance:mysql` at start-up, if there are no errors.
|
||||
|
||||
### Step 3: Enable bucket notification using MinIO client
|
||||
### Step 3: Enable MySQL bucket notification using MinIO client
|
||||
|
||||
We will now setup bucket notifications on a bucket named `images`. Whenever a JPEG image object is created/overwritten, a new row is added or an existing row is updated in the MySQL table configured above. When an existing object is deleted, the corresponding row is deleted from the MySQL table. Thus, the rows in the MySQL table, reflect the `.jpg` objects in the `images` bucket.
|
||||
|
||||
@@ -1106,8 +1095,6 @@ mysql> select * from minio_images;
|
||||
|
||||
```
|
||||
|
||||
<a name="apache-kafka"></a>
|
||||
|
||||
## Publish MinIO events via Kafka
|
||||
|
||||
Install Apache Kafka from [here](http://kafka.apache.org/).
|
||||
@@ -1143,6 +1130,7 @@ comment (sentence) optionally add a comment to this setting
|
||||
```
|
||||
|
||||
or environment variables
|
||||
|
||||
```
|
||||
KEY:
|
||||
notify_kafka[:name] publish bucket notifications to Kafka endpoints
|
||||
@@ -1176,10 +1164,10 @@ notify_kafka:1 tls_skip_verify="off" queue_dir="" queue_limit="0" sasl="off" sa
|
||||
Use `mc admin config set` command to update the configuration for the deployment. Restart the MinIO server to put the changes into effect. The server will print a line like `SQS ARNs: arn:minio:sqs::1:kafka` at start-up if there were no errors.`bucketevents` is the topic used by kafka in this example.
|
||||
|
||||
```sh
|
||||
$ mc admin config set myminio notify_kafka:1 tls_skip_verify="off" queue_dir="" queue_limit="0" sasl="off" sasl_password="" sasl_username="" tls_client_auth="0" tls="off" client_tls_cert="" client_tls_key="" brokers="localhost:9092,localhost:9093" topic="bucketevents" version=""
|
||||
mc admin config set myminio notify_kafka:1 tls_skip_verify="off" queue_dir="" queue_limit="0" sasl="off" sasl_password="" sasl_username="" tls_client_auth="0" tls="off" client_tls_cert="" client_tls_key="" brokers="localhost:9092,localhost:9093" topic="bucketevents" version=""
|
||||
```
|
||||
|
||||
### Step 3: Enable bucket notification using MinIO client
|
||||
### Step 3: Enable Kafka bucket notification using MinIO client
|
||||
|
||||
We will enable bucket event notification to trigger whenever a JPEG image is uploaded or deleted from `images` bucket on `myminio` server. Here ARN value is `arn:minio:sqs::1:kafka`. To understand more about ARN please follow [AWS ARN](http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html) documentation.
|
||||
|
||||
@@ -1263,8 +1251,6 @@ kafkacat -b localhost:9092 -t bucketevents
|
||||
}
|
||||
```
|
||||
|
||||
<a name="webhooks"></a>
|
||||
|
||||
## Publish MinIO events via Webhooks
|
||||
|
||||
[Webhooks](https://en.wikipedia.org/wiki/Webhook) are a way to receive information when it happens, rather than continually polling for that data.
|
||||
@@ -1288,6 +1274,7 @@ comment (sentence) optionally add a comment to this setting
|
||||
```
|
||||
|
||||
or environment variables
|
||||
|
||||
```
|
||||
KEY:
|
||||
notify_webhook[:name] publish bucket notifications to webhook endpoints
|
||||
@@ -1311,10 +1298,10 @@ notify_webhook:1 endpoint="" auth_token="" queue_limit="0" queue_dir="" client_c
|
||||
Use `mc admin config set` command to update the configuration for the deployment. Here the endpoint is the server listening for webhook notifications. Save the settings and restart the MinIO server for changes to take effect. Note that the endpoint needs to be live and reachable when you restart your MinIO server.
|
||||
|
||||
```sh
|
||||
$ mc admin config set myminio notify_webhook:1 queue_limit="0" endpoint="http://localhost:3000" queue_dir=""
|
||||
mc admin config set myminio notify_webhook:1 queue_limit="0" endpoint="http://localhost:3000" queue_dir=""
|
||||
```
|
||||
|
||||
### Step 2: Enable bucket notification using MinIO client
|
||||
### Step 2: Enable Webhook bucket notification using MinIO client
|
||||
|
||||
We will enable bucket event notification to trigger whenever a JPEG image is uploaded to `images` bucket on `myminio` server. Here ARN value is `arn:minio:sqs::1:webhook`. To learn more about ARN please follow [AWS ARN](http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html) documentation.
|
||||
|
||||
@@ -1365,8 +1352,6 @@ mc ls myminio/images-thumbnail
|
||||
[2017-02-08 11:39:40 IST] 992B images-thumbnail.jpg
|
||||
```
|
||||
|
||||
<a name="NSQ"></a>
|
||||
|
||||
## Publish MinIO events to NSQ
|
||||
|
||||
Install an NSQ Daemon from [here](https://nsq.io/). Or use the following Docker
|
||||
@@ -1397,6 +1382,7 @@ comment (sentence) optionally add a comment to this setting
|
||||
```
|
||||
|
||||
or environment variables
|
||||
|
||||
```
|
||||
KEY:
|
||||
notify_nsq[:name] publish bucket notifications to NSQ endpoints
|
||||
@@ -1420,12 +1406,12 @@ notify_nsq:1 nsqd_address="" queue_dir="" queue_limit="0" tls="off" tls_skip_ve
|
||||
Use `mc admin config set` command to update the configuration for the deployment. Restart the MinIO server to put the changes into effect. The server will print a line like `SQS ARNs: arn:minio:sqs::1:nsq` at start-up if there were no errors.
|
||||
|
||||
```sh
|
||||
$ mc admin config set myminio notify_nsq:1 nsqd_address="127.0.0.1:4150" queue_dir="" queue_limit="0" tls="off" tls_skip_verify="on" topic="minio"
|
||||
mc admin config set myminio notify_nsq:1 nsqd_address="127.0.0.1:4150" queue_dir="" queue_limit="0" tls="off" tls_skip_verify="on" topic="minio"
|
||||
```
|
||||
|
||||
Note that, you can add as many NSQ daemon endpoint configurations as needed by providing an identifier (like "1" in the example above) for the NSQ instance and an object of per-server configuration parameters.
|
||||
|
||||
### Step 2: Enable bucket notification using MinIO client
|
||||
### Step 2: Enable NSQ bucket notification using MinIO client
|
||||
|
||||
We will enable bucket event notification to trigger whenever a JPEG image is uploaded or deleted `images` bucket on `myminio` server. Here ARN value is `arn:minio:sqs::1:nsq`.
|
||||
|
||||
|
||||
@@ -7,25 +7,26 @@ Buckets can be configured to have `Hard` quota - it disallows writes to the buck
|
||||
> NOTE: Bucket quotas are not supported under gateway or standalone single disk deployments.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Install MinIO - [MinIO Quickstart Guide](https://docs.min.io/docs/minio-quickstart-guide).
|
||||
- [Use `mc` with MinIO Server](https://docs.min.io/docs/minio-client-quickstart-guide)
|
||||
|
||||
## Set bucket quota configuration
|
||||
|
||||
### Set a hard quota of 1GB for a bucket `mybucket` on MinIO object storage:
|
||||
### Set a hard quota of 1GB for a bucket `mybucket` on MinIO object storage
|
||||
|
||||
```sh
|
||||
$ mc admin bucket quota myminio/mybucket --hard 1gb
|
||||
mc admin bucket quota myminio/mybucket --hard 1gb
|
||||
```
|
||||
|
||||
### Verify the quota configured on `mybucket` on MinIO
|
||||
|
||||
```sh
|
||||
$ mc admin bucket quota myminio/mybucket
|
||||
mc admin bucket quota myminio/mybucket
|
||||
```
|
||||
|
||||
### Clear bucket quota configuration for `mybucket` on MinIO
|
||||
|
||||
```sh
|
||||
$ mc admin bucket quota myminio/mybucket --clear
|
||||
mc admin bucket quota myminio/mybucket --clear
|
||||
```
|
||||
|
||||
@@ -3,9 +3,11 @@
|
||||
This document explains the design approach of server side bucket replication. If you're looking to get started with replication, we suggest you go through the [Bucket replication guide](https://github.com/minio/minio/blob/master/docs/bucket/replication/README.md) first.
|
||||
|
||||
## Overview
|
||||
|
||||
Replication relies on immutability provided by versioning to sync objects between the configured source and replication target. Replication results in the object data, metadata, last modification time and version ID all being identical between the source and target. Thus version ordering is automatically guaranteed on the source and target clusters.
|
||||
|
||||
### Replication of object version and metadata
|
||||
|
||||
If an object meets replication rules as set in the replication configuration, `X-Amz-Replication-Status` is first set to `PENDING` as the PUT operation completes and replication is queued (unless synchronous replication is in place). After replication is performed, the metadata on the source object version changes to `COMPLETED` or `FAILED` depending on whether replication succeeded. The object version on the target shows `X-Amz-Replication-Status` of `REPLICA`
|
||||
|
||||
All replication failures are picked up by the scanner which runs at a one minute frequency, each time scanning upto a sixteenth of the namespace. Object versions marked `PENDING` or `FAILED` are re-queued for replication.
|
||||
@@ -36,6 +38,7 @@ An additional header `X-Minio-Replication-Delete-Status` is returned which would
|
||||
Note that synchronous replication, i.e. when remote target is configured with --sync mode in `mc admin bucket remote add` does not apply to `DELETE` operations. The version being deleted on the source cluster needs to maintain state and ensure that the operation is mirrored to the target cluster prior to completing on the source object version. Since this needs to account for the target cluster availability and the need to serialize concurrent DELETE operations on different versions of the same object during multi DELETE operations, the current implementation queues the `DELETE` operations in both sync and async modes.
|
||||
|
||||
### Existing object replication
|
||||
|
||||
Existing object replication works similar to regular replication. Objects qualifying for existing object replication are detected when scanner runs, and will be replicated if existing object replication is enabled and applicable replication rules are satisfied. Because replication depends on the immutability of versions, only pre-existing objects created while versioning was enabled can be replicated. Even if replication rules are disabled and re-enabled later, the objects created during the interim will be synced as the scanner queues them. For saving iops, objects qualifying for
|
||||
existing object replication are not marked as `PENDING` prior to replication.
|
||||
|
||||
@@ -46,6 +49,7 @@ If the remote site is fully lost and objects previously replicated need to be re
|
||||
This is a slower operation that does not use replication queues and is designed to walk the namespace and replicate objects one at a time so as not to impede server load. Ideally, resync should not be initiated for multiple buckets simultaneously - progress of the syncing can be monitored by looking at `mc replicate resync status alias/bucket --remote-bucket <arn>`. In the event that resync operation failed to replicate some versions, they would be picked up by the healing mechanism in-built as part of the scanner. If the resync operation reports a failed status or in the event of a cluster restart while resync is in progress, a fresh `resync start` can be issued - this will replicate previously unsynced content at the cost of additional overhead in additional metadata updates.
|
||||
|
||||
### Multi destination replication
|
||||
|
||||
The replication design for multiple sites works in a similar manner as described above for two site scenario. However there are some
|
||||
important exceptions.
|
||||
|
||||
@@ -94,6 +98,7 @@ If 3 or more targets are participating in active-active replication, the replica
|
||||
```
|
||||
|
||||
### Additional replication metadata for DeleteMarker
|
||||
|
||||
```
|
||||
...
|
||||
{
|
||||
@@ -150,5 +155,6 @@ If 3 or more targets are participating in active-active replication, the replica
|
||||
```
|
||||
|
||||
## Explore Further
|
||||
|
||||
- [MinIO Bucket Versioning Implementation](https://docs.minio.io/docs/minio-bucket-versioning-guide.html)
|
||||
- [MinIO Client Quickstart Guide](https://docs.minio.io/docs/minio-client-quickstart-guide.html)
|
||||
|
||||
@@ -7,6 +7,7 @@ The contents of this page have been migrated to the new [MinIO Baremetal Documen
|
||||
To replicate objects in a bucket to a destination bucket on a target site either in the same cluster or a different cluster, start by enabling [versioning](https://docs.minio.io/docs/minio-bucket-versioning-guide.html) for both source and destination buckets. Finally, the target site and the destination bucket need to be configured on the source MinIO server.
|
||||
|
||||
## Highlights
|
||||
|
||||
- Supports source and destination buckets to have the same name unlike AWS S3, addresses variety of usecases such as *Splunk*, *Veeam* site to site DR.
|
||||
- Supports object locking/retention across source and destination buckets natively out of the box, unlike AWS S3.
|
||||
- Simpler implementation than [AWS S3 Bucket Replication Config](https://docs.aws.amazon.com/AmazonS3/latest/dev/replication-add-config.html) with requirements such as IAM Role, AccessControlTranslation, Metrics and SourceSelectionCriteria are not needed with MinIO.
|
||||
@@ -14,6 +15,7 @@ To replicate objects in a bucket to a destination bucket on a target site either
|
||||
- Multi destination replication
|
||||
|
||||
## How to use?
|
||||
|
||||
Ensure that versioning is enabled on the source and target buckets with `mc version` command. If object locking is required, the buckets should have been created with `mc mb --with-lock`
|
||||
|
||||
Create a replication target on the source cluster as shown below:
|
||||
@@ -23,9 +25,10 @@ mc admin bucket remote add myminio/srcbucket https://accessKey:secretKey@replica
|
||||
Remote ARN = 'arn:minio:replication:us-east-1:c5be6b16-769d-432a-9ef1-4567081f3566:destbucket'
|
||||
```
|
||||
|
||||
> The user running the above command needs *s3:GetReplicationConfiguration* and *s3:GetBucketVersioning* permission on the source cluster. We do not recommend running root credentials/super admin with replication, instead create a dedicated user. The access credentials used at the destination requires *s3:ReplicateObject* permission.
|
||||
> The user running the above command needs *s3:GetReplicationConfiguration* and *s3:GetBucketVersioning* permission on the source cluster. We do not recommend running root credentials/super admin with replication, instead create a dedicated user. The access credentials used at the destination requires *s3:ReplicateObject* permission.
|
||||
|
||||
The following minimal permission policy is needed by admin user setting up replication on the `source`:
|
||||
|
||||
```
|
||||
{
|
||||
"Version": "2012-10-17",
|
||||
@@ -57,6 +60,7 @@ The following minimal permission policy is needed by admin user setting up repli
|
||||
```
|
||||
|
||||
The access key provided for the replication *target* cluster should have these minimal permissions:
|
||||
|
||||
```
|
||||
{
|
||||
"Version": "2012-10-17",
|
||||
@@ -96,9 +100,11 @@ The access key provided for the replication *target* cluster should have these m
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Please note that the permissions required by the admin user on the target cluster can be more fine grained to exclude permissions like "s3:ReplicateDelete", "s3:GetBucketObjectLockConfiguration" etc depending on whether delete replication rules are set up or if object locking is disabled on `destbucket`. The above policies assume that replication of objects, tags and delete marker replication are all enabled on object lock enabled buckets. A sample script to setup replication is provided [here](https://github.com/minio/minio/blob/master/docs/bucket/replication/setup_replication.sh)
|
||||
|
||||
Once successfully created and authorized, the `mc admin bucket remote add` command generates a replication target ARN. This command lists all the currently authorized replication targets:
|
||||
|
||||
```
|
||||
mc admin bucket remote ls myminio/srcbucket --service "replication"
|
||||
Remote ARN = 'arn:minio:replication:us-east-1:c5be6b16-769d-432a-9ef1-4567081f3566:destbucket'
|
||||
@@ -157,19 +163,25 @@ To perform bi-directional replication, repeat the above process on the target si
|
||||
|
||||
|
||||
## Replica Modification sync
|
||||
|
||||
If bi-directional replication is set up between two clusters, any metadata update on the REPLICA object is by default reflected back in the source object when `ReplicaModifications` status in the `SourceSelectionCriteria` is `Enabled`. In MinIO, this is enabled by default. If a metadata update is performed on the "REPLICA" object, its `X-Amz-Replication-Status` will change from `PENDING` to `COMPLETE` or `FAILED`, and the source object version will show `X-Amz-Replication-Status` of `REPLICA` once the replication operation is complete.
|
||||
|
||||
The replication configuration in use on a bucket can be viewed using the `mc replicate export alias/bucket` command.
|
||||
|
||||
To disable replica metadata modification syncing, use `mc replicate edit` with the --replicate flag.
|
||||
|
||||
```
|
||||
$ mc replicate edit alias/bucket --id xyz.id --replicate "delete,delete-marker"
|
||||
mc replicate edit alias/bucket --id xyz.id --replicate "delete,delete-marker"
|
||||
```
|
||||
|
||||
To re-enable replica metadata modification syncing,
|
||||
|
||||
```
|
||||
$ mc replicate edit alias/bucket --id xyz.id --replicate "delete,delete-marker,replica-metadata-sync"
|
||||
mc replicate edit alias/bucket --id xyz.id --replicate "delete,delete-marker,replica-metadata-sync"
|
||||
```
|
||||
|
||||
## MinIO Extension
|
||||
|
||||
### Replicating Deletes
|
||||
|
||||
Delete marker replication is allowed in [AWS V1 Configuration](https://aws.amazon.com/blogs/storage/managing-delete-marker-replication-in-amazon-s3/) but not in V2 configuration. The MinIO implementation above is based on V2 configuration, however it has been extended to allow both DeleteMarker replication and replication of versioned deletes with the `DeleteMarkerReplication` and `DeleteReplication` fields in the replication configuration above. By default, this is set to `Disabled` unless the user specifies it while adding a replication rule.
|
||||
@@ -183,16 +195,19 @@ Note that just like with [AWS](https://docs.aws.amazon.com/AmazonS3/latest/userg
|
||||
To add a replication rule allowing both delete marker replication, versioned delete replication or both specify the --replicate flag with comma separated values as in the example below.
|
||||
|
||||
Additional permission of "s3:ReplicateDelete" action would need to be specified on the access key configured for the target cluster if Delete Marker replication or versioned delete replication is enabled.
|
||||
|
||||
```
|
||||
mc replicate add myminio/srcbucket/Tax --priority 1 --remote-bucket "arn:minio:replication:us-east-1:c5be6b16-769d-432a-9ef1-4567081f3566:destbucket" --tags "Year=2019&Company=AcmeCorp" --storage-class "STANDARD" --replicate "delete,delete-marker"
|
||||
Replication configuration applied successfully to myminio/srcbucket.
|
||||
```
|
||||
|
||||
> NOTE: In mc versions RELEASE.2021-09-02T09-21-27Z and older, the remote target ARN needs to be passed in the --arn flag and actual remote bucket name in --remote-bucket flag of `mc replicate add`. For example, with the ARN above the replication configuration used to be added with
|
||||
|
||||
```
|
||||
mc replicate add myminio/srcbucket/Tax --priority 1 --remote-bucket destbucket --arn "arn:minio:replication:us-east-1:c5be6b16-769d-432a-9ef1-4567081f3566:destbucket" --tags "Year=2019&Company=AcmeCorp" --storage-class "STANDARD" --replicate "delete,delete-marker"
|
||||
Replication configuration applied successfully to myminio/srcbucket.
|
||||
```
|
||||
```
|
||||
|
||||
Also note that for `mc` version `RELEASE.2021-09-02T09-21-27Z` or older supports only a single remote target per bucket. To take advantage of multiple destination replication, use the latest version of `mc`
|
||||
|
||||
Status of delete marker replication can be viewed by doing a GET/HEAD on the object version - it will return a `X-Minio-Replication-DeleteMarker-Status` header and http response code of `405`. In the case of permanent deletes, if the delete replication is pending or failed to propagate to the target cluster, GET/HEAD will return additional `X-Minio-Replication-Delete-Status` header and a http response code of `405`.
|
||||
@@ -201,16 +216,19 @@ Status of delete marker replication can be viewed by doing a GET/HEAD on the obj
|
||||
|
||||
The status of replication can be monitored by configuring event notifications on the source and target buckets using `mc event add`.On the source side, the `s3:PutObject`, `s3:Replication:OperationCompletedReplication` and `s3:Replication:OperationFailedReplication` events show the status of replication in the `X-Amz-Replication-Status` metadata.
|
||||
|
||||
On the target bucket, `s3:PutObject` event shows `X-Amz-Replication-Status` status of `REPLICA` in the metadata. Additional metrics to monitor backlog state for the purpose of bandwidth management and resource allocation are exposed via Prometheus - see https://github.com/minio/minio/blob/master/docs/metrics/prometheus/list.md for more details.
|
||||
On the target bucket, `s3:PutObject` event shows `X-Amz-Replication-Status` status of `REPLICA` in the metadata. Additional metrics to monitor backlog state for the purpose of bandwidth management and resource allocation are exposed via Prometheus - see <https://github.com/minio/minio/blob/master/docs/metrics/prometheus/list.md> for more details.
|
||||
|
||||
### Sync/Async Replication
|
||||
|
||||
By default, replication is completed asynchronously. If synchronous replication is desired, set the --sync flag while adding a
|
||||
remote replication target using the `mc admin bucket remote add` command
|
||||
|
||||
```
|
||||
mc admin bucket remote add myminio/srcbucket https://accessKey:secretKey@replica-endpoint:9000/destbucket --service replication --region us-east-1 --sync --healthcheck-seconds 100
|
||||
```
|
||||
|
||||
### Existing object replication
|
||||
|
||||
Existing object replication as detailed [here](https://aws.amazon.com/blogs/storage/replicating-existing-objects-between-s3-buckets/) can be enabled by passing `existing-objects` as a value to `--replicate` flag while adding or editing a replication rule.
|
||||
|
||||
Once existing object replication is enabled, all objects or object prefixes that satisfy the replication rules and were created prior to adding replication configuration OR while replication rules were disabled will be synced to the target cluster. Depending on the number of previously existing objects, the existing objects that are now eligible to be replicated will eventually be synced to the target cluster as the scanner schedules them. This may be slower depending on the load on the cluster, latency and size of the namespace.
|
||||
@@ -226,6 +244,7 @@ Replication from a source bucket to multiple destination buckets is supported. F
|
||||
Note that on the source side, the `X-Amz-Replication-Status` changes from `PENDING` to `COMPLETED` after replication succeeds to each of the targets. On the destination side, a `X-Amz-Replication-Status` status of `REPLICA` indicates that the object was replicated successfully. Any replication failures are automatically re-attempted during a periodic disk scanner cycle.
|
||||
|
||||
## Explore Further
|
||||
|
||||
- [MinIO Bucket Replication Design](https://github.com/minio/minio/blob/master/docs/bucket/replication/DESIGN.md)
|
||||
- [MinIO Bucket Versioning Implementation](https://docs.minio.io/docs/minio-bucket-versioning-guide.html)
|
||||
- [MinIO Client Quickstart Guide](https://docs.minio.io/docs/minio-client-quickstart-guide.html)
|
||||
|
||||
@@ -18,7 +18,7 @@ Independent of retention, an object can also be under legal hold. This effective
|
||||
WORM on a bucket is enabled by setting object lock configuration. This configuration is applied to existing and new objects in the bucket. Below is an example sets `Governance` mode and one day retention time from object creation time of all objects in `mybucket`.
|
||||
|
||||
```sh
|
||||
$ awscli s3api put-object-lock-configuration --bucket mybucket --object-lock-configuration 'ObjectLockEnabled=\"Enabled\",Rule={DefaultRetention={Mode=\"GOVERNANCE\",Days=1}}'
|
||||
awscli s3api put-object-lock-configuration --bucket mybucket --object-lock-configuration 'ObjectLockEnabled=\"Enabled\",Rule={DefaultRetention={Mode=\"GOVERNANCE\",Days=1}}'
|
||||
```
|
||||
|
||||
### Set object lock
|
||||
@@ -29,7 +29,7 @@ PutObject API allows setting per object retention mode and retention duration us
|
||||
aws s3api put-object --bucket testbucket --key lockme --object-lock-mode GOVERNANCE --object-lock-retain-until-date "2019-11-20" --body /etc/issue
|
||||
```
|
||||
|
||||
See https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lock-overview.html for AWS S3 spec on object locking and permissions required for object retention and governance bypass overrides.
|
||||
See <https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lock-overview.html> for AWS S3 spec on object locking and permissions required for object retention and governance bypass overrides.
|
||||
|
||||
### Set legal hold on an object
|
||||
|
||||
@@ -39,9 +39,10 @@ PutObject API allows setting legal hold using `x-amz-object-lock-legal-hold` hea
|
||||
aws s3api put-object --bucket testbucket --key legalhold --object-lock-legal-hold-status ON --body /etc/issue
|
||||
```
|
||||
|
||||
See https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lock-overview.html for AWS S3 spec on object locking and permissions required for specifying legal hold.
|
||||
See <https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lock-overview.html> for AWS S3 spec on object locking and permissions required for specifying legal hold.
|
||||
|
||||
## Concepts
|
||||
|
||||
- If an object is under legal hold, it cannot be deleted unless the legal hold is explicitly removed for the respective version id. DeleteObjectVersion() would fail otherwise.
|
||||
- In `Compliance` mode, objects cannot be deleted by anyone until retention period is expired for the respective version id. If user has requisite governance bypass permissions, an object's retention date can be extended in `Compliance` mode.
|
||||
- Once object lock configuration is set to a bucket
|
||||
|
||||
@@ -3,10 +3,10 @@
|
||||
## Description of `xl.meta`
|
||||
|
||||
`xl.meta` is a new self describing backend format used by MinIO to support AWS S3 compatible versioning.
|
||||
This file is the source of truth for each `version` at rest. `xl.meta` is a msgpack file serialized from a
|
||||
This file is the source of truth for each `version` at rest. `xl.meta` is a msgpack file serialized from a
|
||||
well defined data structure. To understand `xl.meta` here are the few things to start with
|
||||
|
||||
`xl.meta` carries first 8 bytes an XL header which describes the current format and the format version,
|
||||
`xl.meta` carries first 8 bytes an XL header which describes the current format and the format version,
|
||||
allowing the unmarshaller's to automatically use the right data structures to parse the subsequent content in the stream.
|
||||
|
||||
### v1.0
|
||||
@@ -43,6 +43,7 @@ Therefore, the metadata is wrapped as a binary array for easy skipping.
|
||||
- DeleteMarker (a versionId to capture the DELETE sequences implemented primarily for AWS spec compatibility)
|
||||
|
||||
A sample msgpack-JSON `xl.meta`, you can debug the content inside `xl.meta` using [xl-meta.go](https://github.com/minio/minio/tree/master/docs/debugging#decoding-metadata) program.
|
||||
|
||||
```json
|
||||
{
|
||||
"Versions": [
|
||||
@@ -92,7 +93,7 @@ Version 1.3 introduces changes to help with [faster metadata reads and updates](
|
||||
| versions | msgp int | Number of versions following
|
||||
| header_1 | msgp bin array | Header of version 1
|
||||
| metadata_1 | msgp bin array | Metadata of version 1
|
||||
| ...header_n | msgp bin array | Header of last version
|
||||
| ...header_n | msgp bin array | Header of last version
|
||||
| ...metadata_n | msgp bin array | Metadata of last version
|
||||
|
||||
Each header contains a mspg array (tuple) encoded object:
|
||||
@@ -102,11 +103,11 @@ xlHeaderVersion version == 1:
|
||||
```
|
||||
//msgp:tuple xlMetaV2VersionHeader
|
||||
type xlMetaV2VersionHeader struct {
|
||||
VersionID [16]byte // Version UUID, raw.
|
||||
ModTime int64 // Unix nanoseconds.
|
||||
Signature [4]byte // Signature of metadata.
|
||||
Type uint8 // Type if the version
|
||||
Flags uint8
|
||||
VersionID [16]byte // Version UUID, raw.
|
||||
ModTime int64 // Unix nanoseconds.
|
||||
Signature [4]byte // Signature of metadata.
|
||||
Type uint8 // Type if the version
|
||||
Flags uint8
|
||||
}
|
||||
```
|
||||
|
||||
@@ -114,9 +115,9 @@ The following flags are defined:
|
||||
|
||||
```
|
||||
const (
|
||||
FreeVersion = 1 << 0
|
||||
UsesDataDir = 1 << 1
|
||||
InlineData = 1 << 2
|
||||
FreeVersion = 1 << 0
|
||||
UsesDataDir = 1 << 1
|
||||
InlineData = 1 << 2
|
||||
)
|
||||
```
|
||||
|
||||
|
||||
@@ -31,6 +31,7 @@ To permanently delete an object you need to specify the version you want to dele
|
||||
|
||||
|
||||
## Concepts
|
||||
|
||||
- All Buckets on MinIO are always in one of the following states: unversioned (the default) and all other existing deployments, versioning-enabled, or versioning-suspended.
|
||||
- Versioning state applies to all of the objects in the versioning enabled bucket. The first time you enable a bucket for versioning, objects in the bucket are thereafter always versioned and given a unique version ID.
|
||||
- Existing or newer buckets can be created with versioning enabled and eventually can be suspended as well. Existing versions of objects stay as is and can still be accessed using the version ID.
|
||||
@@ -38,13 +39,16 @@ To permanently delete an object you need to specify the version you want to dele
|
||||
- **Versioning feature is only available in erasure coded and distributed erasure coded setups**.
|
||||
|
||||
## How to configure versioning on a bucket
|
||||
|
||||
Each bucket created has a versioning configuration associated with it. By default bucket is unversioned as shown below
|
||||
|
||||
```
|
||||
<VersioningConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
|
||||
</VersioningConfiguration>
|
||||
```
|
||||
|
||||
To enable versioning, you send a request to MinIO with a versioning configuration with Status set to `Enabled`.
|
||||
|
||||
```
|
||||
<VersioningConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
|
||||
<Status>Enabled</Status>
|
||||
@@ -52,6 +56,7 @@ To enable versioning, you send a request to MinIO with a versioning configuratio
|
||||
```
|
||||
|
||||
Similarly to suspend versioning set the configuration with Status set to `Suspended`.
|
||||
|
||||
```
|
||||
<VersioningConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
|
||||
<Status>Suspended</Status>
|
||||
@@ -160,6 +165,7 @@ public class IsVersioningEnabled {
|
||||
```
|
||||
|
||||
## Explore Further
|
||||
|
||||
- [Use `minio-java` SDK with MinIO Server](https://docs.minio.io/docs/java-client-quickstart-guide.html)
|
||||
- [Object Lock and Immutablity Guide](https://docs.minio.io/docs/minio-bucket-object-lock-guide.html)
|
||||
- [MinIO Admin Complete Guide](https://docs.min.io/docs/minio-admin-complete-guide.html)
|
||||
|
||||
Reference in New Issue
Block a user