minio/pkg
Harshavardhana fb96779a8a Add large bucket support for erasure coded backend (#5160)
This PR implements an object layer which
combines input erasure sets of XL layers
into a unified namespace.

This object layer extends the existing
erasure coded implementation, it is assumed
in this design that providing > 16 disks is
a static configuration as well i.e if you started
the setup with 32 disks with 4 sets 8 disks per
pack then you would need to provide 4 sets always.

Some design details and restrictions:

- Objects are distributed using consistent ordering
  to a unique erasure coded layer.
- Each pack has its own dsync so locks are synchronized
  properly at pack (erasure layer).
- Each pack still has a maximum of 16 disks
  requirement, you can start with multiple
  such sets statically.
- Static sets set of disks and cannot be
  changed, there is no elastic expansion allowed.
- Static sets set of disks and cannot be
  changed, there is no elastic removal allowed.
- ListObjects() across sets can be noticeably
  slower since List happens on all servers,
  and is merged at this sets layer.

Fixes #5465
Fixes #5464
Fixes #5461
Fixes #5460
Fixes #5459
Fixes #5458
Fixes #5460
Fixes #5488
Fixes #5489
Fixes #5497
Fixes #5496
2018-02-15 17:45:57 -08:00
..
auth Reduce Minio access key minimum length to 3 (#5478) 2018-02-02 09:13:30 +05:30
bpool Add large bucket support for erasure coded backend (#5160) 2018-02-15 17:45:57 -08:00
cgroup sys/stats: return cgroup mem limit, fall back to sysinfo() (#4002) 2017-04-02 10:46:16 -07:00
disk Add GetInfo() support for solaris (#5174) 2017-11-13 12:54:38 -08:00
ellipses Add large bucket support for erasure coded backend (#5160) 2018-02-15 17:45:57 -08:00
errors Convert errors tracer into a separate package (#5221) 2017-11-25 11:58:29 -08:00
handlers log: Store http request/responses in a log file (#4804) 2017-10-25 10:59:53 -07:00
hash replace all "crypto/sha256" with "github.com/minio/sha256-simd" (#5391) 2018-01-17 10:54:31 -08:00
http restrict TLS cipher suites of the server (#5245) 2018-01-13 09:12:11 +05:30
ioutil Lock free multipart backend implementation for FS (#5401) 2018-01-31 13:17:24 -08:00
lock Lock free multipart backend implementation for FS (#5401) 2018-01-31 13:17:24 -08:00
madmin Add large bucket support for erasure coded backend (#5160) 2018-02-15 17:45:57 -08:00
mimedb cleanup: All conditionals simplified under pkg. (#3875) 2017-03-09 10:13:30 -08:00
mountinfo Fail to start server if detected cross-device mounts. (#4807) 2017-08-15 15:10:50 -07:00
quick Remove go1.9 specific code for windows (#5033) 2017-10-13 15:31:15 +05:30
safe Remove go1.9 specific code for windows (#5033) 2017-10-13 15:31:15 +05:30
sync/errgroup Add large bucket support for erasure coded backend (#5160) 2018-02-15 17:45:57 -08:00
sys Push max threads to little less than kernel limit (#5001) 2017-10-03 10:37:45 -07:00
trie trie: new package (#3729) 2017-02-10 11:51:41 -08:00
wildcard pkg/wildcard: Simplify the wildcard logic further. (#2555) 2016-08-27 00:27:17 -07:00
words words: new package Damerau Levenshtein distance function. (#3929) 2017-03-19 14:23:05 -07:00