76e2713ffe
Use separate sync.Pool for writes/reads
Avoid passing buffers for io.CopyBuffer()
if the writer or reader implement io.WriteTo or io.ReadFrom
respectively then its useless for sync.Pool to allocate
buffers on its own since that will be completely ignored
by the io.CopyBuffer Go implementation.
Improve this wherever we see this to be optimal.
This allows us to be more efficient on memory usage.
```
385 // copyBuffer is the actual implementation of Copy and CopyBuffer.
386 // if buf is nil, one is allocated.
387 func copyBuffer(dst Writer, src Reader, buf []byte) (written int64, err error) {
388 // If the reader has a WriteTo method, use it to do the copy.
389 // Avoids an allocation and a copy.
390 if wt, ok := src.(WriterTo); ok {
391 return wt.WriteTo(dst)
392 }
393 // Similarly, if the writer has a ReadFrom method, use it to do the copy.
394 if rt, ok := dst.(ReaderFrom); ok {
395 return rt.ReadFrom(src)
396 }
```
From readahead package
```
// WriteTo writes data to w until there's no more data to write or when an error occurs.
// The return value n is the number of bytes written.
// Any error encountered during the write is also returned.
func (a *reader) WriteTo(w io.Writer) (n int64, err error) {
if a.err != nil {
return 0, a.err
}
n = 0
for {
err = a.fill()
if err != nil {
return n, err
}
n2, err := w.Write(a.cur.buffer())
a.cur.inc(n2)
n += int64(n2)
if err != nil {
return n, err
}
```
|
||
|---|---|---|
| .. | ||
| DESIGN.md | ||
| README.md | ||
Shared Backend MinIO Quickstart Guide 
MinIO shared mode lets you use single NAS (like NFS, GlusterFS, and other distributed filesystems) as the storage backend for multiple MinIO servers. Synchronization among MinIO servers is taken care by design. Read more about the MinIO shared mode design here.
MinIO shared mode is developed to solve several real world use cases, without any special configuration changes. Some of these are
- You have already invested in NAS and would like to use MinIO to add S3 compatibility to your storage tier.
- You need to use NAS with an S3 interface due to your application architecture requirements.
- You expect huge traffic and need a load balanced S3 compatible server, serving files from a single NAS backend.
With a proxy running in front of multiple, shared mode MinIO servers, it is very easy to create a Highly Available, load balanced, AWS S3 compatible storage system.
Get started
If you're aware of stand-alone MinIO set up, the installation and running remains the same.
1. Prerequisites
Install MinIO - MinIO Quickstart Guide.
2. Run MinIO on Shared Backend
To run MinIO shared backend instances, you need to start multiple MinIO servers pointing to the same backend storage. We'll see examples on how to do this in the following sections.
Note
- All the nodes running shared MinIO need to have same access key and secret key. To achieve this, we export access key and secret key as environment variables on all the nodes before executing MinIO server command.
- The drive paths below are for demonstration purposes only, you need to replace these with the actual drive paths/folders.
MinIO shared mode on Ubuntu 16.04 LTS.
You'll need the path to the shared volume, e.g. /path/to/nfs-volume. Then run the following commands on all the nodes you'd like to launch MinIO.
export MINIO_ROOT_USER=<ACCESS_KEY>
export MINIO_ROOT_PASSWORD=<SECRET_KEY>
minio gateway nas /path/to/nfs-volume
MinIO shared mode on Windows 2012 Server
You'll need the path to the shared volume, e.g. \\remote-server\smb. Then run the following commands on all the nodes you'd like to launch MinIO.
set MINIO_ROOT_USER=my-username
set MINIO_ROOT_PASSWORD=my-password
minio.exe gateway nas \\remote-server\smb\export
Windows Tip
If a remote volume, e.g. \\remote-server\smb is mounted as a drive, e.g. M:\. You can use net use command to map the drive to a folder.
set MINIO_ROOT_USER=my-username
set MINIO_ROOT_PASSWORD=my-password
net use m: \\remote-server\smb\export /P:Yes
minio.exe gateway nas M:\export
3. Test your setup
To test this setup, access the MinIO server via browser or mc. You’ll see the uploaded files are accessible from the all the MinIO shared backend endpoints.