mirror of
https://github.com/minio/minio.git
synced 2025-11-07 21:02:58 -05:00
Add support for multi site replication (#12880)
This commit is contained in:
Poorna Krishnamoorthy
committed by
GitHub
GitHub
parent
0b8c5a6872
commit
c4373ef290
@@ -20,6 +20,8 @@ The description above details one way replication from source to target w.r.t in
|
||||
|
||||
For active-active replication, automatic failover occurs on `GET/HEAD` operations if object or object version requested qualifies for replication and is missing on one site, but present on the other. This allows the applications to take full advantage of two-way replication even before the two sites get fully synced.
|
||||
|
||||
In the case of multi destination replication, the replication status shows `COMPLETED` only after the replication operation succeeds on each of the targets specified in the replication configuration. If multiple targets are configured to use active-active replication and multi destination replication, the administrator should ensure that the replication features enabled (such as replica metadata sync, delete marker replication etc) are identical to avoid asymmetric state. This is because all replication activity is inherently a one-way operation from source to target, irrespective of the number of targets.
|
||||
|
||||
### Replication of DeleteMarker and versioned Delete
|
||||
|
||||
MinIO allows DeleteMarker replication and versioned delete replication by setting `--replicate delete,delete-marker` while setting up replication configuration using `mc replicate add`. The MinIO implementation is based on V2 configuration, however it has been extended to allow both DeleteMarker replication and replication of versioned deletes with the `DeleteMarkerReplication` and `DeleteReplication` fields in the replication configuration. By default, this is set to `Disabled` unless the user specifies it while adding a replication rule.
|
||||
@@ -37,56 +39,87 @@ Note that synchronous replication, i.e. when remote target is configured with --
|
||||
Existing object replication works similar to regular replication. Objects qualifying for existing object replication are detected when scanner runs, and will be replicated if existing object replication is enabled and applicable replication rules are satisfied. Because replication depends on the immutability of versions, only pre-existing objects created while versioning was enabled can be replicated. Even if replication rules are disabled and re-enabled later, the objects created during the interim will be synced as the scanner queues them. For saving iops, objects qualifying for
|
||||
existing object replication are not marked as `PENDING` prior to replication.
|
||||
|
||||
Note that objects with `null` versions, i.e. objects created prior to enabling versioning cannot be replicated as this would break the immutability guarantees provided by versioning. For replicating such objects, `mc cp alias/bucket/object alias/bucket/object` can be performed to create a server side copy of the object as a versioned object - this versioned object will replicate if replication is enabled and the previously present `null` version can then be deleted.
|
||||
|
||||
If the remote site is fully lost and objects previously replicated need to be re-synced, the `mc replicate resync` command with optional flag of `--older-than` needs to be used to trigger re-syncing of previously replicated objects. This command generates a ResetID which is a unique UUID saved to the remote target config along with the applicable date(defaults to time of initiating the reset). All objects created prior to this date are eligible for re-replication if existing object replication is enabled for the replication rule the object satisfies. At the time of completion of replication, `X-Minio-Replication-Reset-Status` is set in the metadata with the timestamp of replication and ResetID. For saving iops, the objects which are re-replicated are not first set to `PENDING` state.
|
||||
|
||||
### Multi destination replication
|
||||
The replication design for multiple sites works in a similar manner as described above for two site scenario. However there are some
|
||||
important exceptions.
|
||||
|
||||
Replication status on the source cluster will be marked as `COMPLETED` only after replication is completed on all targets. If one or more targets failed replication, the replication status is reflected as `PENDING`.
|
||||
|
||||
If 3 or more targets are participating in active-active replication, the replication configuration for replica metadata sync, delete marker replication and delete replication should match to avoid inconsistent picture between the clusters. It is not recommended to turn on asymmetric replication - for e.g. if three sites A,B,C are participating in replication, it would be better to avoid replication setups like A -> [B, C], B -> A. In this particular example, an object uploaded to A will be replicated to B,C. If replica metadata sync is turned on in site B, any metadata updates on a replica version made in B would reflect in A, but not in C.
|
||||
|
||||
### Internal metadata for replication
|
||||
|
||||
`xl.meta` that is in use for [versioning](https://github.com/minio/minio/blob/master/docs/bucket/versioning/DESIGN.md) has additional metadata for replication of objects,delete markers and versioned deletes.
|
||||
|
||||
### Metadata for object replication
|
||||
### Metadata for object replication - on source
|
||||
|
||||
```
|
||||
...
|
||||
"MetaSys": {},
|
||||
"MetaUsr": {
|
||||
"X-Amz-Replication-Status": "COMPLETED",
|
||||
"content-type": "application/octet-stream",
|
||||
"etag": "8315e643ed6a5d7c9962fc0a8ef9c11f"
|
||||
},
|
||||
"PartASizes": [
|
||||
26
|
||||
],
|
||||
"MetaSys": {
|
||||
"x-minio-internal-inline-data": "dHJ1ZQ==",
|
||||
"x-minio-internal-replication-status": "YXJuOm1pbmlvOnJlcGxpY2F0aW9uOjo2YjdmYzFlMS0wNmU4LTQxMTUtYjYxNy00YTgzZGIyODhmNTM6YnVja2V0PUNPTVBMRVRFRDthcm46bWluaW86cmVwbGljYXRpb246OmI5MGYxZWEzLWMzYWQtNDEyMy1iYWE2LWZjMDZhYmEyMjA2MjpidWNrZXQ9Q09NUExFVEVEOw==",
|
||||
"x-minio-internal-replication-timestamp": "MjAyMS0wOS0xN1QwMTo0MzozOC40MDQwMDA0ODNa",
|
||||
"x-minio-internal-tier-free-versionID": "OWZlZjk5N2QtMjMzZi00N2U3LTlkZmMtNWYxNzc3NzdlZTM2"
|
||||
},
|
||||
"MetaUsr": {
|
||||
"X-Amz-Replication-Status": "COMPLETED",
|
||||
"content-type": "application/octet-stream",
|
||||
"etag": "8315e643ed6a5d7c9962fc0a8ef9c11f"
|
||||
},
|
||||
...
|
||||
```
|
||||
|
||||
### Metadata for object replication - on target
|
||||
|
||||
```
|
||||
...
|
||||
"MetaSys": {
|
||||
"x-minio-internal-inline-data": "dHJ1ZQ==",
|
||||
"x-minio-internal-replica-status": "UkVQTElDQQ==",
|
||||
"x-minio-internal-replica-timestamp": "MjAyMS0wOS0xN1QwMTo0MzozOC4zODg5ODU4ODRa"
|
||||
},
|
||||
"MetaUsr": {
|
||||
"X-Amz-Replication-Status": "REPLICA",
|
||||
"content-type": "application/octet-stream",
|
||||
"etag": "8315e643ed6a5d7c9962fc0a8ef9c11f",
|
||||
"x-amz-storage-class": "STANDARD"
|
||||
},
|
||||
...
|
||||
```
|
||||
### Additional replication metadata for DeleteMarker
|
||||
|
||||
```
|
||||
{
|
||||
{
|
||||
"DelObj": {
|
||||
"ID": "8+jguy20TOuzUCN2PTrESA==",
|
||||
"MTime": 1613601949645331516,
|
||||
"MetaSys": {
|
||||
"X-Amz-Replication-Status": "Q09NUExFVEVE"
|
||||
}
|
||||
},
|
||||
"Type": 2
|
||||
}
|
||||
"ID": "u8H5pYQFRMKgkIgkpSKIkQ==",
|
||||
"MTime": 1631843124147668389,
|
||||
"MetaSys": {
|
||||
"x-minio-internal-replication-status": "YXJuOm1pbmlvOnJlcGxpY2F0aW9uOjpiOTBmMWVhMy1jM2FkLTQxMjMtYmFhNi1mYzA2YWJhMjIwNjI6YnVja2V0PUNPTVBMRVRFRDthcm46bWluaW86cmVwbGljYXRpb246OjZiN2ZjMWUxLTA2ZTgtNDExNS1iNjE3LTRhODNkYjI4OGY1MzpidWNrZXQ9Q09NUExFVEVEOw==",
|
||||
"x-minio-internal-replication-timestamp": "U3VuLCAzMSBEZWMgMDAwMCAxOTowMzo1OCBHTVQ="
|
||||
}
|
||||
},
|
||||
"Type": 2
|
||||
}
|
||||
```
|
||||
|
||||
### Additional replication metadata for versioned delete
|
||||
|
||||
```
|
||||
{
|
||||
"DelObj": {
|
||||
"ID": "8+jguy20TOuzUCN2PTrESA==",
|
||||
"MTime": 1613601949645331516,
|
||||
"MetaSys": {
|
||||
"purgestatus": "RkFJTEVE"
|
||||
}
|
||||
},
|
||||
"Type": 2
|
||||
}
|
||||
{
|
||||
"DelObj": {
|
||||
"ID": "u8H5pYQFRMKgkIgkpSKIkQ==",
|
||||
"MTime": 1631843124147668389,
|
||||
"MetaSys": {
|
||||
"purgestatus": "YXJuOm1pbmlvOnJlcGxpY2F0aW9uOjpiOTBmMWVhMy1jM2FkLTQxMjMtYmFhNi1mYzA2YWJhMjIwNjI6YnVja2V0PUNPTVBMRVRFO2FybjptaW5pbzpyZXBsaWNhdGlvbjo6NmI3ZmMxZTEtMDZlOC00MTE1LWI2MTctNGE4M2RiMjg4ZjUzOmJ1Y2tldD1GQUlMRUQ7",
|
||||
"x-minio-internal-replication-status": "YXJuOm1pbmlvOnJlcGxpY2F0aW9uOjpiOTBmMWVhMy1jM2FkLTQxMjMtYmFhNi1mYzA2YWJhMjIwNjI6YnVja2V0PTthcm46bWluaW86cmVwbGljYXRpb246OjZiN2ZjMWUxLTA2ZTgtNDExNS1iNjE3LTRhODNkYjI4OGY1MzpidWNrZXQ9Ow==",
|
||||
"x-minio-internal-replication-timestamp": "U3VuLCAzMSBEZWMgMDAwMCAxOTowMzo1OCBHTVQ="
|
||||
}
|
||||
},
|
||||
"Type": 2
|
||||
}
|
||||
```
|
||||
|
||||
## Explore Further
|
||||
|
||||
@@ -11,6 +11,7 @@ To replicate objects in a bucket to a destination bucket on a target site either
|
||||
- Supports object locking/retention across source and destination buckets natively out of the box, unlike AWS S3.
|
||||
- Simpler implementation than [AWS S3 Bucket Replication Config](https://docs.aws.amazon.com/AmazonS3/latest/dev/replication-add-config.html) with requirements such as IAM Role, AccessControlTranslation, Metrics and SourceSelectionCriteria are not needed with MinIO.
|
||||
- Active-Active replication
|
||||
- Multi destination replication
|
||||
|
||||
## How to use?
|
||||
Ensure that versioning is enabled on the source and target buckets with `mc version` command. If object locking is required, the buckets should have been created with `mc mb --with-lock`
|
||||
@@ -19,7 +20,7 @@ Create a replication target on the source cluster as shown below:
|
||||
|
||||
```
|
||||
mc admin bucket remote add myminio/srcbucket https://accessKey:secretKey@replica-endpoint:9000/destbucket --service replication --region us-east-1
|
||||
Role ARN = 'arn:minio:replication:us-east-1:c5be6b16-769d-432a-9ef1-4567081f3566:destbucket'
|
||||
Remote ARN = 'arn:minio:replication:us-east-1:c5be6b16-769d-432a-9ef1-4567081f3566:destbucket'
|
||||
```
|
||||
|
||||
> The user running the above command needs *s3:GetReplicationConfiguration* and *s3:GetBucketVersioning* permission on the source cluster. We do not recommend running root credentials/super admin with replication, instead create a dedicated user. The access credentials used at the destination requires *s3:ReplicateObject* permission.
|
||||
@@ -100,14 +101,14 @@ Please note that the permissions required by the admin user on the target cluste
|
||||
Once successfully created and authorized, the `mc admin bucket remote add` command generates a replication target ARN. This command lists all the currently authorized replication targets:
|
||||
```
|
||||
mc admin bucket remote ls myminio/srcbucket --service "replication"
|
||||
Role ARN = 'arn:minio:replication:us-east-1:c5be6b16-769d-432a-9ef1-4567081f3566:destbucket'
|
||||
Remote ARN = 'arn:minio:replication:us-east-1:c5be6b16-769d-432a-9ef1-4567081f3566:destbucket'
|
||||
```
|
||||
|
||||
The replication configuration can now be added to the source bucket by applying the json file with replication configuration. The Role ARN above is passed in as a json element in the configuration.
|
||||
The replication configuration can now be added to the source bucket by applying the json file with replication configuration. The Remote ARN above is passed in as a json element in the configuration.
|
||||
|
||||
```json
|
||||
{
|
||||
"Role" :"arn:minio:replication:us-east-1:c5be6b16-769d-432a-9ef1-4567081f3566:destbucket",
|
||||
"Role" :"",
|
||||
"Rules": [
|
||||
{
|
||||
"Status": "Enabled",
|
||||
@@ -130,7 +131,7 @@ The replication configuration can now be added to the source bucket by applying
|
||||
}
|
||||
},
|
||||
"Destination": {
|
||||
"Bucket": "arn:aws:s3:::destbucket",
|
||||
"Bucket": "arn:minio:replication:us-east-1:c5be6b16-769d-432a-9ef1-4567081f3566:destbucket",
|
||||
"StorageClass": "STANDARD"
|
||||
},
|
||||
"SourceSelectionCriteria": {
|
||||
@@ -181,7 +182,7 @@ To add a replication rule allowing both delete marker replication, versioned del
|
||||
|
||||
Additional permission of "s3:ReplicateDelete" action would need to be specified on the access key configured for the target cluster if Delete Marker replication or versioned delete replication is enabled.
|
||||
```
|
||||
mc replicate add myminio/srcbucket/Tax --priority 1 --arn "arn:minio:replication:us-east-1:c5be6b16-769d-432a-9ef1-4567081f3566:destbucket" --tags "Year=2019&Company=AcmeCorp" --storage-class "STANDARD" --remote-bucket "destbucket" --replicate "delete,delete-marker"
|
||||
mc replicate add myminio/srcbucket/Tax --priority 1 --remote-bucket "arn:minio:replication:us-east-1:c5be6b16-769d-432a-9ef1-4567081f3566:destbucket" --tags "Year=2019&Company=AcmeCorp" --storage-class "STANDARD" --replicate "delete,delete-marker"
|
||||
Replication configuration applied successfully to myminio/srcbucket.
|
||||
```
|
||||
|
||||
@@ -215,7 +216,11 @@ This is an expensive operation and should be initiated only once - progress of t
|
||||
|
||||
Note that ExistingObjectReplication needs to be enabled in the config via `mc replicate [add|edit]` by passing `existing-objects` as one of the values to `--replicate` flag. Only those objects meeting replication rules and having existing object replication enabled will be re-synced.
|
||||
|
||||
Multi site replication is currently not supported.
|
||||
### Multi destination replication
|
||||
|
||||
Replication from a source bucket to multiple destination buckets is supported. For each of the targets,repeat the steps to configure a remote target ARN and add replication rules to the source bucket's replication config.
|
||||
|
||||
Note that on the source side, the `X-Amz-Replication-Status` changes from `PENDING` to `COMPLETED` after replication succeeds to each of the targets. On the destination side, a `X-Amz-Replication-Status` status of `REPLICA` indicates that the object was replicated successfully. Any replication failures are automatically re-attempted during a periodic disk scanner cycle.
|
||||
|
||||
## Explore Further
|
||||
- [MinIO Bucket Replication Design](https://github.com/minio/minio/blob/master/docs/bucket/replication/DESIGN.md)
|
||||
|
||||
Reference in New Issue
Block a user