Implement S3 Gateway to third party cloud storage providers. (#3756)

Currently supported backend is Azure Blob Storage.

```
export MINIO_ACCESS_KEY=azureaccountname
export MINIO_SECRET_KEY=azureaccountkey
minio gateway azure
```
This commit is contained in:
Krishna Srinivas
2017-03-16 12:21:58 -07:00
committed by Harshavardhana
parent 8426cf9aec
commit cea4cfa3a8
41 changed files with 6983 additions and 65 deletions

48
docs/gateway/README.md Normal file
View File

@@ -0,0 +1,48 @@
# Minio Gateway [![Slack](https://slack.minio.io/slack?type=svg)](https://slack.minio.io)
Minio gateway adds Amazon S3 compatibility to third party cloud storage providers. Supported providers are:
- Azure Blob Storage
## Run Minio Gateway for Azure Blob Storage
### Using Docker
```
docker run -p 9000:9000 --name azure-s3 \
-e "MINIO_ACCESS_KEY=azureaccountname" \
-e "MINIO_SECRET_KEY=azureaccountkey" \
minio/minio gateway azure
```
### Using Binary
```
export MINIO_ACCESS_KEY=azureaccountname
export MINIO_SECRET_KEY=azureaccountkey
minio gateway azure
```
## Test using Minio Client `mc`
`mc` provides a modern alternative to UNIX commands such as ls, cat, cp, mirror, diff etc. It supports filesystems and Amazon S3 compatible cloud storage services.
### Configure `mc`
```
mc config host add myazure http://gateway-ip:9000 azureaccountname azureaccountkey
```
### List containers on Azure
```
mc ls myazure
[2017-02-22 01:50:43 PST] 0B ferenginar/
[2017-02-26 21:43:51 PST] 0B my-container/
[2017-02-26 22:10:11 PST] 0B test-container1/
```
## Explore Further
- [`mc` command-line interface](https://docs.minio.io/docs/minio-client-quickstart-guide)
- [`aws` command-line interface](https://docs.minio.io/docs/aws-cli-with-minio)
- [`minfs` filesystem interface](http://docs.minio.io/docs/minfs-quickstart-guide)
- [`minio-go` Go SDK](https://docs.minio.io/docs/golang-client-quickstart-guide)

View File

@@ -0,0 +1,19 @@
## Minio Azure Gateway Limitations
Gateway inherits the following Azure limitations:
- Maximum Multipart part size is 100MB.
- Maximum Multipart object size is 10000*100 MB = 1TB
- No support for prefix based bucket policies. Only top level bucket policy is supported.
- Gateway restart implies all the ongoing multipart uploads must be restarted.
i.e clients must again start with NewMultipartUpload
This is because S3 clients send metadata in NewMultipartUpload but Azure expects metadata to
be set during CompleteMultipartUpload (PutBlockList in Azure terminology). We store the metadata
sent by the client during NewMultipartUpload in memory so that it can be set on Azure later during
CompleteMultipartUpload. When the gateway is restarted this information is lost.
- Bucket names with "." in the bucket name is not supported.
- Non-empty buckets get removed on a DeleteBucket() call.
Other limitations:
- Current implementation of ListMultipartUploads is incomplete. Right now it returns if the object with name "prefix" has any uploaded parts.
- Bucket notification not supported.