MyFavMirrors
/
minio
mirror of https://github.com/minio/minio.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
minio/docs/large-bucket
History
Harshavardhana fb96779a8a Add large bucket support for erasure coded backend (#5160) 
This PR implements an object layer which
combines input erasure sets of XL layers
into a unified namespace.

This object layer extends the existing
erasure coded implementation, it is assumed
in this design that providing > 16 disks is
a static configuration as well i.e if you started
the setup with 32 disks with 4 sets 8 disks per
pack then you would need to provide 4 sets always.

Some design details and restrictions:

- Objects are distributed using consistent ordering
  to a unique erasure coded layer.
- Each pack has its own dsync so locks are synchronized
  properly at pack (erasure layer).
- Each pack still has a maximum of 16 disks
  requirement, you can start with multiple
  such sets statically.
- Static sets set of disks and cannot be
  changed, there is no elastic expansion allowed.
- Static sets set of disks and cannot be
  changed, there is no elastic removal allowed.
- ListObjects() across sets can be noticeably
  slower since List happens on all servers,
  and is merged at this sets layer.

Fixes #5465
Fixes #5464
Fixes #5461
Fixes #5460
Fixes #5459
Fixes #5458
Fixes #5460
Fixes #5488
Fixes #5489
Fixes #5497
Fixes #5496
 		2018-02-15 17:45:57 -08:00
..
DESIGN.md Add large bucket support for erasure coded backend (#5160) 2018-02-15 17:45:57 -08:00
README.md Add large bucket support for erasure coded backend (#5160) 2018-02-15 17:45:57 -08:00

README.md

Large Bucket Support Quickstart Guide Slack Go Report Card Docker Pulls codecov

Minio large bucket support lets you use more than 16 disks by creating a number of smaller sets of erasure coded units, these units are further combined into a single namespace. Minio large bucket support is developed to solve for several real world use cases, without any special configuration changes. Some of these are

  • You already have racks with many disks.
  • You are looking for large capacity up-front for your object storage needs.

Get started

If you're aware of distributed Minio setup, the installation and running remains the same. Newer syntax to use a ... convention to abbreviate the directory arguments. Remote directories in a distributed setup are encoded as HTTP(s) URIs which can be similarly abbreviated as well.

1. Prerequisites

Install Minio - Minio Quickstart Guide.

2. Run Minio on many disks

To run Minio large bucket instances, you need to start multiple Minio servers pointing to the same disks. We'll see examples on how to do this in the following sections.

Note

  • All the nodes running distributed Minio need to have same access key and secret key. To achieve this, we export access key and secret key as environment variables on all the nodes before executing Minio server command.
  • The drive paths below are for demonstration purposes only, you need to replace these with the actual drive paths/folders.

Minio large bucket on Ubuntu 16.04 LTS standalone

You'll need the path to the disks e.g. /export1, /export2 .... /export24. Then run the following commands on all the nodes you'd like to launch Minio.

export MINIO_ACCESS_KEY=<ACCESS_KEY>
export MINIO_SECRET_KEY=<SECRET_KEY>
minio server /export{1...24}

Minio large bucket on Ubuntu 16.04 LTS servers

You'll need the path to the disks e.g. /export1, /export2 .... /export16. Then run the following commands on all the nodes you'd like to launch Minio.

export MINIO_ACCESS_KEY=<ACCESS_KEY>
export MINIO_SECRET_KEY=<SECRET_KEY>
minio server http://host{1...4}/export{1...16}

3. Test your setup

To test this setup, access the Minio server via browser or mc. You’ll see the uploaded files are accessible from the all the Minio endpoints.

Explore Further