mirror of
https://github.com/minio/minio.git
synced 2024-12-24 22:25:54 -05:00
Update storage class related documents (#5399)
- Add storage usage info in storage class doc - Update distributed & erasure code doc with info on storage class
This commit is contained in:
parent
17301fe45d
commit
8a1dc10c60
@ -22,6 +22,8 @@ As with Minio in stand-alone mode, distributed Minio has a per tenant limit of m
|
|||||||
|
|
||||||
Note that with distributed Minio you can play around with the number of nodes and drives as long as the limits are adhered to. For example, you can have 2 nodes with 4 drives each, 4 nodes with 4 drives each, 8 nodes with 2 drives each, and so on.
|
Note that with distributed Minio you can play around with the number of nodes and drives as long as the limits are adhered to. For example, you can have 2 nodes with 4 drives each, 4 nodes with 4 drives each, 8 nodes with 2 drives each, and so on.
|
||||||
|
|
||||||
|
You can also use [storage classes](https://github.com/minio/minio/tree/master/docs/erasure/storage-class) to set custom data and parity distribution across total disks.
|
||||||
|
|
||||||
### Consistency Guarantees
|
### Consistency Guarantees
|
||||||
|
|
||||||
Minio follows strict **read-after-write** consistency model for all i/o operations both in distributed and standalone modes.
|
Minio follows strict **read-after-write** consistency model for all i/o operations both in distributed and standalone modes.
|
||||||
|
@ -1,30 +1,34 @@
|
|||||||
# Minio Erasure Code Quickstart Guide [![Slack](https://slack.minio.io/slack?type=svg)](https://slack.minio.io)
|
# Minio Erasure Code Quickstart Guide [![Slack](https://slack.minio.io/slack?type=svg)](https://slack.minio.io)
|
||||||
|
|
||||||
Minio protects data against hardware failures and silent data corruption using erasure code and checksums. You may lose half the number (N/2) of drives and still be able to recover the data.
|
Minio protects data against hardware failures and silent data corruption using erasure code and checksums. With the highest level of redundancy, you may lose up to half (N/2) of the total drives and still be able to recover the data.
|
||||||
|
|
||||||
## What is Erasure Code?
|
## What is Erasure Code?
|
||||||
|
|
||||||
Erasure code is a mathematical algorithm to reconstruct missing or corrupted data. Minio uses Reed-Solomon code to shard objects into N/2 data and N/2 parity blocks. This means that in a 12 drive setup, an object is sharded across as 6 data and 6 parity blocks. You can lose as many as 6 drives (be it parity or data) and still reconstruct the data reliably from the remaining drives.
|
Erasure code is a mathematical algorithm to reconstruct missing or corrupted data. Minio uses Reed-Solomon code to shard objects into variable data and parity blocks. For example, in a 12 drive setup, an object can be sharded to a variable number of data and parity blocks across all the drives - ranging from six data and six parity blocks to ten data and two parity blocks.
|
||||||
|
|
||||||
|
By default, Minio shards the objects across N/2 data and N/2 parity drives. Though, you can use [storage classes](https://github.com/minio/minio/tree/master/docs/erasure/storage-class) to use a custom configuration. We recommend N/2 data and parity blocks, as it ensures the best protection from drive failures.
|
||||||
|
|
||||||
|
In 12 drive example above, with Minio server running in the default configuration, you can lose any of the six drives and still reconstruct the data reliably from the remaining drives.
|
||||||
|
|
||||||
## Why is Erasure Code useful?
|
## Why is Erasure Code useful?
|
||||||
|
|
||||||
Erasure code protects data from multiple drives failure unlike RAID or replication. For eg RAID6 can protect against 2 drive failure whereas in Minio erasure code you can lose as many as half number of drives and still the data remains safe. Further Minio's erasure code is at object level and can heal one object at a time. For RAID, healing can only be performed at volume level which translates into huge down time. As Minio encodes each object individually with a high parity count. Storage servers once deployed should not require drive replacement or healing for the lifetime of the server. Minio's erasure coded backend is designed for operational efficiency and takes full advantage of hardware acceleration whenever available.
|
Erasure code protects data from multiple drives failure, unlike RAID or replication. For example, RAID6 can protect against two drive failure whereas in Minio erasure code you can lose as many as half of drives and still the data remains safe. Further, Minio's erasure code is at the object level and can heal one object at a time. For RAID, healing can be done only at the volume level which translates into high downtime. As Minio encodes each object individually, it can heal objects incrementally. Storage servers once deployed should not require drive replacement or healing for the lifetime of the server. Minio's erasure coded backend is designed for operational efficiency and takes full advantage of hardware acceleration whenever available.
|
||||||
|
|
||||||
![Erasure](https://github.com/minio/minio/blob/master/docs/screenshots/erasure-code.jpg?raw=true)
|
![Erasure](https://github.com/minio/minio/blob/master/docs/screenshots/erasure-code.jpg?raw=true)
|
||||||
|
|
||||||
## What is Bit Rot protection?
|
## What is Bit Rot protection?
|
||||||
|
|
||||||
Bit Rot also known as Data Rot or Silent Data Corruption is a serious data loss issue faced by disk drives today. Data on the drive may silently get corrupted without signalling an error has occurred. This makes Bit Rot more dangerous than permanent hard drive failure.
|
Bit Rot, also known as data rot or silent data corruption is a data loss issue faced by disk drives today. Data on the drive may silently get corrupted without signalling an error has occurred, making bit rot more dangerous than a permanent hard drive failure.
|
||||||
|
|
||||||
Minio's erasure coded backend uses high speed [BLAKE2](https://blog.minio.io/accelerating-blake2b-by-4x-using-simd-in-go-assembly-33ef16c8a56b#.jrp1fdwer) hash based checksums to protect against Bit Rot.
|
Minio's erasure coded backend uses high speed [BLAKE2](https://blog.minio.io/accelerating-blake2b-by-4x-using-simd-in-go-assembly-33ef16c8a56b#.jrp1fdwer) hash based checksums to protect against Bit Rot.
|
||||||
|
|
||||||
## Get Started with Minio in Erasure Code
|
## Get Started with Minio in Erasure Code
|
||||||
|
|
||||||
### 1. Prerequisites:
|
### 1. Prerequisites
|
||||||
|
|
||||||
Install Minio - [Minio Quickstart Guide](https://docs.minio.io/docs/minio-quickstart-guide)
|
Install Minio - [Minio Quickstart Guide](https://docs.minio.io/docs/minio-quickstart-guide)
|
||||||
|
|
||||||
### 2. Run Minio Server with Erasure Code.
|
### 2. Run Minio Server with Erasure Code
|
||||||
|
|
||||||
Example: Start Minio server in a 12 drives setup, using Minio binary.
|
Example: Start Minio server in a 12 drives setup, using Minio binary.
|
||||||
|
|
||||||
|
@ -9,7 +9,32 @@ set before starting Minio server. After the data and parity disks for each stora
|
|||||||
you can set the storage class of an object via request metadata field `x-amz-storage-class`. Minio server then honors the storage class by
|
you can set the storage class of an object via request metadata field `x-amz-storage-class`. Minio server then honors the storage class by
|
||||||
saving the object in specific number of data and parity disks.
|
saving the object in specific number of data and parity disks.
|
||||||
|
|
||||||
### Values for standard storage class (STANDARD)
|
## Storage usage
|
||||||
|
|
||||||
|
The selection of varying data and parity drives has a direct impact on the drive space usage. With storage class, you can optimise for high
|
||||||
|
redundancy or better drive space utilisation.
|
||||||
|
|
||||||
|
To get an idea of how various combinations of data and parity drives affect the storage usage, let’s take an example of a 100 MiB file stored
|
||||||
|
on 16 drive Minio deployment. If you use eight data and eight parity drives, the file space usage will be approximately twice, i.e. 100 MiB
|
||||||
|
file will take 200 MiB space. But, if you use ten data and six parity drives, same 100 MiB file takes around 160 MiB. If you use 14 data and
|
||||||
|
two parity drives, 100 MiB file takes only approximately 114 MiB.
|
||||||
|
|
||||||
|
Below is a list of data/parity drives and corresponding _approximate_ storage space usage on a 16 drive Minio deployment. The field _storage
|
||||||
|
usage ratio_ is simply the drive space used by the file after erasure-encoding, divided by actual file size.
|
||||||
|
|
||||||
|
| Total Drives (N) | Data Drives (D) | Parity Drives (P) | Storage Usage Ratio |
|
||||||
|
|------------------|-----------------|-------------------|---------------------|
|
||||||
|
| 16 | 8 | 8 | 2.00 |
|
||||||
|
| 16 | 9 | 7 | 1.79 |
|
||||||
|
| 16 | 10 | 6 | 1.60 |
|
||||||
|
| 16 | 11 | 5 | 1.45 |
|
||||||
|
| 16 | 12 | 4 | 1.34 |
|
||||||
|
| 16 | 13 | 3 | 1.23 |
|
||||||
|
| 16 | 14 | 2 | 1.14 |
|
||||||
|
|
||||||
|
You can calculate _approximate_ storage usage ratio using the formula - total drives (N) / data drives (D).
|
||||||
|
|
||||||
|
### Allowed values for STANDARD storage class
|
||||||
|
|
||||||
`STANDARD` storage class implies more parity than `REDUCED_REDUNDANCY` class. So, `STANDARD` parity disks should be
|
`STANDARD` storage class implies more parity than `REDUCED_REDUNDANCY` class. So, `STANDARD` parity disks should be
|
||||||
|
|
||||||
@ -20,7 +45,7 @@ Parity blocks can not be higher than data blocks, so `STANDARD` storage class pa
|
|||||||
|
|
||||||
Default value for `STANDARD` storage class is `N/2` (N is the total number of drives).
|
Default value for `STANDARD` storage class is `N/2` (N is the total number of drives).
|
||||||
|
|
||||||
### Values for reduced redundancy storage class (REDUCED_REDUNDANCY)
|
### Allowed values for REDUCED_REDUNDANCY storage class
|
||||||
|
|
||||||
`REDUCED_REDUNDANCY` implies lesser parity than `STANDARD` class. So,`REDUCED_REDUNDANCY` parity disks should be
|
`REDUCED_REDUNDANCY` implies lesser parity than `STANDARD` class. So,`REDUCED_REDUNDANCY` parity disks should be
|
||||||
|
|
||||||
|
Loading…
Reference in New Issue
Block a user