High Performance Object Storage for AI
Go to file
Harshavardhana 141a44bfbf xl: Fix ReadFile to keep the order always for reading the data back. (#1339)
Also fixes a stackoverflow bug in namespace locking.
2016-04-25 12:47:31 -07:00
buildscripts release: Add freebsd/amd64 build and remove zip, tgz. (#1316) 2016-04-13 23:34:55 -07:00
docker add environment var to explicitly indicate containerized and allow running as root (#1327) 2016-04-18 16:30:11 -07:00
pkg Implement XL layer - preliminary work. 2016-04-25 12:47:31 -07:00
vendor Implement XL layer - preliminary work. 2016-04-25 12:47:31 -07:00
.gitignore cleanup: Remove old donut/xl code and erasure implementation. 2016-04-02 17:30:35 -07:00
.mailmap Update AB's duplicate email address 2015-08-20 16:10:01 -07:00
.travis.yml cleanup: Remove old donut/xl code and erasure implementation. 2016-04-02 17:30:35 -07:00
access-key.go minio: Simplify for gosimple tool complaints. 2016-04-02 17:28:54 -07:00
accesslog-handler.go xl: Moved to minio/minio - fixes #1112 2016-02-11 15:43:36 -08:00
api-datatypes.go api: verify Location constraint for make bucket. (#1342) 2016-04-20 17:35:38 -07:00
api-errors.go object: handle Error responses and handle errDiskFull. (#1331) 2016-04-19 02:42:10 -07:00
api-headers_test.go handlers: Cleanup time handlers helpers. 2016-03-07 10:47:45 -08:00
api-headers.go refactor: refactor code to separate fs into object-layer and fs layer. (#1305) 2016-04-16 16:25:53 -07:00
api-resources.go listObjects: Cleanup and naming conventions. 2016-04-04 19:55:07 -07:00
api-response.go refactor: refactor code to separate fs into object-layer and fs layer. (#1305) 2016-04-16 16:25:53 -07:00
api-router.go list/xl: Fix the way marker is handled in leafDirectory verification. 2016-04-25 12:47:31 -07:00
appveyor.yml cleanup: Remove old donut/xl code and erasure implementation. 2016-04-02 17:30:35 -07:00
auth-handler.go signature: Handle presigned payload if set. 2016-04-07 03:04:18 -07:00
AWS-SDK-GO.md doc: Cleanup aws-sdk-go.md 2015-11-23 13:53:55 -08:00
Browser.md api: Add diskInfo as part of StatVol and ListVols. (#1349) 2016-04-21 20:07:47 -07:00
bucket-handlers.go Adding return statement after error response in the lastest commit to verify location constraint (#1348) 2016-04-21 20:08:08 -07:00
bucket-policy-handlers.go storage/server/client: Enable storage server, enable client storage. 2016-04-16 16:25:53 -07:00
bucket-policy-parser_test.go bucket-policy parset tests, and bug fixes (#1317) 2016-04-15 18:23:19 -07:00
bucket-policy-parser.go bucket-policy parset tests, and bug fixes (#1317) 2016-04-15 18:23:19 -07:00
bucket-policy.go fs: Break fs package to top-level and introduce ObjectAPI interface. 2016-04-01 15:58:39 -07:00
Bucket-Policy.md bucketpolicy: Improve bucket policy validation, avoid nested rules. 2016-03-15 17:50:23 -07:00
build-constants.go build: Add release builds, now generated with 'make release' 2016-02-23 15:14:02 -08:00
Caddy.md Update Caddy.md 2016-03-14 20:51:16 -07:00
certs.go config/main: Re-write config files - add to new config v3 2016-03-23 19:16:09 -07:00
commands.go minio: Server upon start displays a message if update is available. 2016-03-24 20:03:51 -07:00
config-migrate.go config: Migrate to the new version. Remove backend details. 2016-04-02 17:29:31 -07:00
config-old.go config: Migrate to the new version. Remove backend details. 2016-04-02 17:29:31 -07:00
config-v4.go config: console logging should be enabled by default. (#1307) 2016-04-09 14:19:59 -07:00
config.go config/main: Re-write config files - add to new config v3 2016-03-23 19:16:09 -07:00
CONTRIBUTING.md Migrate this project to minio micro services code 2015-10-16 11:26:08 -07:00
damerau-levenshtein.go xl: Moved to minio/minio - fixes #1112 2016-02-11 15:43:36 -08:00
Docker.md docker: Update docker document 2016-04-15 17:40:56 -07:00
fs-dir-common.go fs: Filter out valid paths and volnames. (#1321) 2016-04-16 19:33:29 -07:00
fs-dir-nix.go fs/object: Fix issues from review comments. 2016-04-16 17:57:14 -07:00
fs-dir-others.go fs/object: Fix issues from review comments. 2016-04-16 17:57:14 -07:00
fs-dirent-fileno.go listObjects: Cleanup and naming conventions. 2016-04-04 19:55:07 -07:00
fs-dirent-ino.go listObjects: Cleanup and naming conventions. 2016-04-04 19:55:07 -07:00
fs-utils.go fs: Filter out valid paths and volnames. (#1321) 2016-04-16 19:33:29 -07:00
fs.go Implement XL layer - preliminary work. 2016-04-25 12:47:31 -07:00
generic-handlers.go storage/server/client: Enable storage server, enable client storage. 2016-04-16 16:25:53 -07:00
globals.go config: Migrate to the new version. Remove backend details. 2016-04-02 17:29:31 -07:00
httprange.go object: handle Error responses and handle errDiskFull. (#1331) 2016-04-19 02:42:10 -07:00
INSTALLGO.md build/vet: Fix all the shadowing reports with go1.6 2016-02-23 14:34:39 -08:00
LICENSE Initial commit 2014-10-30 21:51:52 -07:00
logger_test.go xl: Moved to minio/minio - fixes #1112 2016-02-11 15:43:36 -08:00
logger-console-hook.go config/main: Re-write config files - add to new config v3 2016-03-23 19:16:09 -07:00
logger-file-hook.go logging: Enable logging across storage fs layer. (#1367) 2016-04-24 00:36:00 -07:00
logger-syslog-hook_nix.go config/main: Re-write config files - add to new config v3 2016-03-23 19:16:09 -07:00
logger-syslog-hook_windows.go config/main: Re-write config files - add to new config v3 2016-03-23 19:16:09 -07:00
logger.go logging: Enable logging across storage fs layer. (#1367) 2016-04-24 00:36:00 -07:00
Logging.md logging: Enable logging across storage fs layer. (#1367) 2016-04-24 00:36:00 -07:00
main_test.go Migrate this project to minio micro services code 2015-10-16 11:26:08 -07:00
main.go logging: Enable logging across storage fs layer. (#1367) 2016-04-24 00:36:00 -07:00
MAINTAINERS.md build: Add release builds, now generated with 'make release' 2016-02-23 15:14:02 -08:00
Makefile Implement XL layer - preliminary work. 2016-04-25 12:47:31 -07:00
minio.md config: Migrate to the new version. Remove backend details. 2016-04-02 17:29:31 -07:00
Multipart.md doc: add multipart documentation about staging files 2016-04-05 19:56:19 -07:00
network-fs.go logging: Enable logging across storage fs layer. (#1367) 2016-04-24 00:36:00 -07:00
NOTICE Fix typo 2016-03-09 13:55:58 +01:00
notifier.go minio: Server upon start displays a message if update is available. 2016-03-24 20:03:51 -07:00
object_api_suite_test.go Tests: object api multipart tests and bug fixes. 2016-04-25 10:39:28 -07:00
object-api_test.go tests: Add API suite tests back for object api. (#1352) 2016-04-21 23:40:01 -07:00
object-api-getobjectinfo_test.go tests: Add API suite tests back for object api. (#1352) 2016-04-21 23:40:01 -07:00
object-api-listobjects_test.go refactor: refactor code to separate fs into object-layer and fs layer. (#1305) 2016-04-16 16:25:53 -07:00
object-api-multipart_test.go tests: Fix a bug in TestObjectAPIIsUploadIDExists. (#1375) 2016-04-25 12:47:08 -07:00
object-api-multipart.go xl: Fix ReadFile to keep the order always for reading the data back. (#1339) 2016-04-25 12:47:31 -07:00
object-api.go list/xl: Fix the way marker is handled in leafDirectory verification. 2016-04-25 12:47:31 -07:00
object-datatypes.go api: Add diskInfo as part of StatVol and ListVols. (#1349) 2016-04-21 20:07:47 -07:00
object-errors.go Tests: object api multipart tests and bug fixes. 2016-04-25 10:39:28 -07:00
object-handlers.go Implement XL layer - preliminary work. 2016-04-25 12:47:31 -07:00
object-utils_test.go api: verify Location constraint for make bucket. (#1342) 2016-04-20 17:35:38 -07:00
object-utils.go api: Fix verification of checkLeafDirectory. (#1347) 2016-04-21 18:05:26 -07:00
README.md Update README.md 2016-04-19 10:22:40 -07:00
routers.go list/xl: Fix the way marker is handled in leafDirectory verification. 2016-04-25 12:47:31 -07:00
runtime-checks.go isContainerized should look for /.dockerenv not /.dockerinit 2016-04-18 19:26:56 -07:00
server_test.go fs: Add proper volume and path validation. 2016-04-16 16:25:53 -07:00
server-main.go Implement XL layer - preliminary work. 2016-04-25 12:47:31 -07:00
server-rlimit-nix.go server: Attempt to increase max open files. (#1328) 2016-04-18 22:05:32 -07:00
server-rlimit-win.go server: Attempt to increase max open files. (#1328) 2016-04-18 22:05:32 -07:00
signature-jwt.go signature: Use a layered approach for signature verification. 2016-03-26 15:21:05 -07:00
signature-v4-parser.go error: Signature errors should be returned with APIErrorCode. 2016-03-31 23:28:40 -07:00
signature-v4-postpolicyform.go error: Signature errors should be returned with APIErrorCode. 2016-03-31 23:28:40 -07:00
signature-v4-utils.go signature: Handle presigned payload if set. 2016-04-07 03:04:18 -07:00
signature-v4.go signature: Handle presigned payload if set. 2016-04-07 03:04:18 -07:00
storage-api-interface.go refactor: refactor code to separate fs into object-layer and fs layer. (#1305) 2016-04-16 16:25:53 -07:00
storage-datatypes.go Implement XL layer - preliminary work. 2016-04-25 12:47:31 -07:00
storage-errors.go xl: Add more fixes and cleanup. 2016-04-25 12:47:31 -07:00
storage-rpc-datatypes.go storage/server/client: Enable storage server, enable client storage. 2016-04-16 16:25:53 -07:00
storage-rpc-server.go list/xl: Fix the way marker is handled in leafDirectory verification. 2016-04-25 12:47:31 -07:00
trie.go Add trie to verify wrong inputs, and provide meaningful messages 2015-09-17 16:49:08 -07:00
typed-errors.go UI-handler: remove minio-go dependancy. 2016-04-01 13:56:32 +05:30
update-main.go release: Add freebsd/amd64 build and remove zip, tgz. (#1316) 2016-04-13 23:34:55 -07:00
utils.go api: verify Location constraint for make bucket. (#1342) 2016-04-20 17:35:38 -07:00
version_test.go Move all server and controller packages into top-level 2015-09-19 01:07:42 -07:00
version-main.go console: Fix console color printing on windows. 2016-03-04 10:07:19 -08:00
web-handlers.go api: Add diskInfo as part of StatVol and ListVols. (#1349) 2016-04-21 20:07:47 -07:00
web-router.go list/xl: Fix the way marker is handled in leafDirectory verification. 2016-04-25 12:47:31 -07:00
xl-v1-createfile.go xl: Fix ReadFile to keep the order always for reading the data back. (#1339) 2016-04-25 12:47:31 -07:00
xl-v1-healfile.go list/xl: Fix the way marker is handled in leafDirectory verification. 2016-04-25 12:47:31 -07:00
xl-v1-namespace.go xl: Fix ReadFile to keep the order always for reading the data back. (#1339) 2016-04-25 12:47:31 -07:00
xl-v1-readfile.go xl: Fix ReadFile to keep the order always for reading the data back. (#1339) 2016-04-25 12:47:31 -07:00
xl-v1.go list/xl: Fix the way marker is handled in leafDirectory verification. 2016-04-25 12:47:31 -07:00

Minio Gitter

Minio is an object storage server compatible with Amazon S3 and licensed under Apache license 2.0.

Description

Minio is an open source object storage server released under Apache License V2. It is compatible with Amazon S3 cloud storage service. Minio follows a minimalist design philosophy. Minio is light enough to be bundled with the application stack. It sits on the side of NodeJS, Redis, MySQL and the likes. Unlike databases, Minio stores objects such as photos, videos, log files, backups, container / VM images and so on. Minio is best suited for storing blobs of information ranging from KBs to 5 TBs each. In a simplistic sense, it is like a FTP server with a simple get / put API over HTTP.

Minio currently implements two backends

  • Filesystem (FS) - is available and ready for general purpose use. This version of the Minio binary is built using Filesystem storage backend for magnetic and solid state disks.
  • ErasureCoded (XL) - is work in progress and not ready for general purpose use.

Minio Client

Minio Client (mc) provides a modern alternative to Unix commands like ls, cat, cp, sync, and diff. It supports POSIX compatible filesystems and Amazon S3 compatible cloud storage systems. It is entirely written in Golang.

Amazon S3 Compatible Client Libraries

Install Build StatusBuild status

GNU/Linux

Download minio for:

$ chmod +x minio
$ ./minio --help

OS X

Download minio from https://dl.minio.io/server/minio/release/darwin-amd64/minio

$ chmod 755 minio
$ ./minio --help

Microsoft Windows

Download minio for:

C:\Users\Username\Downloads> minio.exe --help

FreeBSD

Download minio from https://dl.minio.io/server/minio/release/freebsd-amd64/minio

$ chmod 755 minio
$ ./minio --help

Docker container

Download minio for docker.

$ docker pull minio/minio

Read more here on How to configure data volume containers for Minio?

Source

NOTE: Source installation is intended for only developers and advanced users. For general use, please download official releases from https://minio.io/download.

If you do not have a working Golang environment, please follow Install Golang.

$ go get -d github.com/minio/minio
$ cd $GOPATH/src/github.com/minio/minio
$ make

How to use Minio?

Start minio server.

$ minio server ~/Photos

AccessKey: WLGDGYAQYIGI833EV05A  SecretKey: BYvgJM101sHngl2uzjXS/OBF/aMxAN06JrJ3qJlF  Region: us-east-1

Minio Object Storage:
    http://127.0.0.1:9000
    http://10.1.10.177:9000

Minio Browser:
    http://127.0.0.1:9000
    http://10.1.10.177:9000

To configure Minio Client:
    $ wget https://dl.minio.io/client/mc/release/darwin-amd64/mc
    $ chmod 755 mc
    $ ./mc config host add myminio http://localhost:9000 WLGDGYAQYIGI833EV05A BYvgJM101sHngl2uzjXS/OBF/aMxAN06JrJ3qJlF

How to use AWS CLI with Minio?

This section assumes that you have already installed aws-cli, if not please visit https://aws.amazon.com/cli/

To configure aws-cli, type aws configure and follow below steps.

$ aws configure
AWS Access Key ID [None]: YOUR_ACCESS_KEY_HERE
AWS Secret Access Key [None]: YOUR_SECRET_KEY_HERE
Default region name [None]: us-east-1
Default output format [None]: ENTER

Additionally enable aws-cli to use AWS Signature Version '4' for Minio server.

$ aws configure set default.s3.signature_version s3v4

To list your buckets.

$ aws --endpoint-url http://localhost:9000 s3 ls
2016-01-07 16:38:23 testbucket

To list contents inside bucket.

$ aws --endpoint-url http://localhost:9000 s3 ls s3://testbucket
                           PRE test/
2015-12-17 08:46:41   12232928 vim
2016-01-07 16:38:23   32232928 emacs
2015-12-09 08:05:24     138504 s3cmd

How to use AWS SDK with Minio?

Please follow the documentation here - Using aws-sdk-go with Minio server

How to use s3cmd with Minio?

This section assumes that you have already installed s3cmd, if not please visit http://s3tools.org/s3cmd

Edit the following fields in your s3cmd configuration file ~/.s3cfg .

host_base = localhost:9000
host_bucket = localhost:9000
access_key = YOUR_ACCESS_KEY_HERE
secret_key = YOUR_SECRET_KEY_HERE
signature_v2 = False
bucket_location = us-east-1

To make a bucket

$ s3cmd mb s3://mybucket
Bucket 's3://mybucket/' created

To copy an object to bucket

$ s3cmd put newfile.txt s3://testbucket
upload: 'newfile' -> 's3://testbucket/newfile'  

To copy an object to local system

$ s3cmd get s3://testbucket/newfile
download: 's3://testbucket/newfile' -> './newfile'

To sync local file/directory to a bucket

$ s3cmd sync newdemo s3://testbucket
upload: 'newdemo/newdemofile.txt' -> 's3://testbucket/newdemo/newdemofile.txt'

To sync bucket or object with local filesystem

$ s3cmd sync  s3://otherbucket otherlocalbucket
download: 's3://otherbucket/cat.jpg' -> 'otherlocalbucket/cat.jpg' 

To list buckets.

$ s3cmd ls s3://
2015-12-09 16:12  s3://testbbucket

To list contents inside bucket.

$ s3cmd ls s3://testbucket/
                       DIR   s3://testbucket/test/
2015-12-09 16:05    138504   s3://testbucket/newfile

Delete an object from bucket

$ s3cmd del s3://testbucket/newfile
delete: 's3://testbucket/newfile'

Delete a bucket

$ s3cmd rb s3://testbucket
Bucket 's3://testbucket/' removed

Contribute to Minio Project

Please follow Minio Contributor's Guide

Jobs

If you think in Lisp or Haskell and hack in go, you would blend right in. Send your github link to callhome@minio.io.