High Performance Object Storage for AI
Go to file
Harshavardhana e8990e42c2 XL: Make allocations simpler avoid redundant allocs. (#1961)
- Reduce 10MiB buffers for loopy calls to use 128KiB.
- start using 128KiB buffer where needed.
2016-06-24 02:06:23 -07:00
.github Created ISSUE_TEMPLATE with basic information 2016-06-03 10:31:08 +05:30
buildscripts filter GOPATH from stack trace (#1755) 2016-05-25 02:32:35 -07:00
docs bitrot: Start using blake2b algorithm and remove sha512 usage. (#1957) 2016-06-22 17:13:26 -07:00
pkg bitrot: Start using blake2b algorithm and remove sha512 usage. (#1957) 2016-06-22 17:13:26 -07:00
vendor bitrot: Start using blake2b algorithm and remove sha512 usage. (#1957) 2016-06-22 17:13:26 -07:00
.gitignore Remove binary files from previous commit. 2016-05-10 15:49:17 -07:00
.mailmap Update AB's duplicate email address 2015-08-20 16:10:01 -07:00
.mention-bot Add mention-bot config 2016-05-20 13:53:15 -07:00
.travis.yml cleanup: Remove old donut/xl code and erasure implementation. 2016-04-02 17:30:35 -07:00
access-key.go logging: Log only for unhandled errors, remove all the debug logging. (#1652) 2016-05-16 14:31:28 -07:00
accesslog-handler.go logging: Log only for unhandled errors, remove all the debug logging. (#1652) 2016-05-16 14:31:28 -07:00
api-datatypes.go api: verify Location constraint for make bucket. (#1342) 2016-04-20 17:35:38 -07:00
api-errors.go TestServer introduction and revamp of functional tests. (#1940) 2016-06-21 12:10:18 -07:00
api-headers_test.go handlers: Cleanup time handlers helpers. 2016-03-07 10:47:45 -08:00
api-headers.go refactor: refactor code to separate fs into object-layer and fs layer. (#1305) 2016-04-16 16:25:53 -07:00
api-resources.go List Objects version 2. (#1815) 2016-05-31 22:10:55 -07:00
api-response.go api: MakeBucket should set proper bucket location. (#1948) 2016-06-20 23:25:18 -07:00
api-router.go xl/fs: Split object layer into interface. (#1415) 2016-04-29 14:24:10 -07:00
appveyor.yml cleanup: Remove old donut/xl code and erasure implementation. 2016-04-02 17:30:35 -07:00
auth-handler.go logging: Log only for unhandled errors, remove all the debug logging. (#1652) 2016-05-16 14:31:28 -07:00
AWS-SDK-GO.md doc: Cleanup aws-sdk-go.md 2015-11-23 13:53:55 -08:00
bucket-handlers.go List Objects version 2. (#1815) 2016-05-31 22:10:55 -07:00
bucket-policy-handlers_test.go Resource matching fix to overcome issues with regular expression based match (#1476) 2016-05-04 16:56:57 -07:00
bucket-policy-handlers.go logging: Log only for unhandled errors, remove all the debug logging. (#1652) 2016-05-16 14:31:28 -07:00
bucket-policy-parser_test.go api: Add new bucket policy nesting error (#1883) 2016-06-09 01:53:56 -07:00
bucket-policy-parser.go api: Add new bucket policy nesting error (#1883) 2016-06-09 01:53:56 -07:00
bucket-policy.go server: Deadcode removal. (#1517) 2016-05-07 21:47:33 -07:00
build-constants.go filter GOPATH from stack trace (#1755) 2016-05-25 02:32:35 -07:00
Caddy.md Update Caddy.md 2016-03-14 20:51:16 -07:00
certs.go logging: Log only for unhandled errors, remove all the debug logging. (#1652) 2016-05-16 14:31:28 -07:00
commands.go logging: Log only for unhandled errors, remove all the debug logging. (#1652) 2016-05-16 14:31:28 -07:00
config-migrate.go posix: Support UNC paths on windows. (#1887) 2016-06-13 02:53:09 -07:00
config-old.go quick/config: No need to use Data() with type assertion. (#1480) 2016-05-04 20:22:15 -07:00
config-v4.go quick/config: No need to use Data() with type assertion. (#1480) 2016-05-04 20:22:15 -07:00
config.go logging: Log only for unhandled errors, remove all the debug logging. (#1652) 2016-05-16 14:31:28 -07:00
CONTRIBUTING.md Minor changes to CONTIRBUTING.md instructions (#1403) 2016-04-27 16:57:16 -07:00
damerau-levenshtein.go xl: Moved to minio/minio - fixes #1112 2016-02-11 15:43:36 -08:00
Docker.md docker: Update docker document 2016-04-15 17:40:56 -07:00
Dockerfile New Dockerfile for building & running minio inside Docker (inc. autobuild support) (#1473) (#1485) 2016-05-04 17:07:19 -07:00
erasure-createfile.go bitrot: Start using blake2b algorithm and remove sha512 usage. (#1957) 2016-06-22 17:13:26 -07:00
erasure-readfile.go XL: Make allocations simpler avoid redundant allocs. (#1961) 2016-06-24 02:06:23 -07:00
erasure-utils.go XL: Make allocations simpler avoid redundant allocs. (#1961) 2016-06-24 02:06:23 -07:00
erasure.go XL: Cleanup, comments and all the updated functions. (#1830) 2016-06-01 16:43:31 -07:00
format-config-v1_test.go XL/format: Fix a bug in checkDisksConsistency. (#1894) 2016-06-14 01:12:15 -07:00
format-config-v1.go XL: Make allocations simpler avoid redundant allocs. (#1961) 2016-06-24 02:06:23 -07:00
FreeBSD.md docs: FreeBSD minio source intructions (#1421) 2016-04-29 13:35:20 -07:00
fs-v1-metadata.go XL: Make allocations simpler avoid redundant allocs. (#1961) 2016-06-24 02:06:23 -07:00
fs-v1-multipart-common.go FS: Cleanup and Fix all multipart related operations. (#1836) 2016-06-02 12:18:56 -07:00
fs-v1-multipart.go XL: Make allocations simpler avoid redundant allocs. (#1961) 2016-06-24 02:06:23 -07:00
fs-v1.go XL: Make allocations simpler avoid redundant allocs. (#1961) 2016-06-24 02:06:23 -07:00
generic-handlers.go Fix formatting for adding ETag for exposed headers (#1687) 2016-05-18 19:17:17 -07:00
globals.go logging: Log only for unhandled errors, remove all the debug logging. (#1652) 2016-05-16 14:31:28 -07:00
handler-utils_test.go Cleanup: Moving IsValidLocationContraint to handler utils 2016-04-28 20:01:11 -07:00
handler-utils.go Cleanup: Moving IsValidLocationContraint to handler utils 2016-04-28 20:01:11 -07:00
httprange.go XL: Rename, cleanup and add more comments. (#1769) 2016-05-28 15:15:53 -07:00
INSTALLGO.md INSTALLGO.md mentions Go 1.5+ for OS X (#1936) 2016-06-20 06:17:57 -07:00
LICENSE Initial commit 2014-10-30 21:51:52 -07:00
logger_test.go logging: Log only for unhandled errors, remove all the debug logging. (#1652) 2016-05-16 14:31:28 -07:00
logger-console-hook.go logging: Log only for unhandled errors, remove all the debug logging. (#1652) 2016-05-16 14:31:28 -07:00
logger-file-hook.go posix: Mkdir() and OpenFile() should honor umask. (#1972) 2016-06-23 20:19:27 -07:00
logger-syslog-hook_nix.go logging: Log only for unhandled errors, remove all the debug logging. (#1652) 2016-05-16 14:31:28 -07:00
logger-syslog-hook_windows.go logging: Log only for unhandled errors, remove all the debug logging. (#1652) 2016-05-16 14:31:28 -07:00
logger.go filter GOPATH from stack trace (#1755) 2016-05-25 02:32:35 -07:00
main_test.go Migrate this project to minio micro services code 2015-10-16 11:26:08 -07:00
main.go XL: Re-align the code again. 2016-06-02 01:54:06 -07:00
MAINTAINERS.md build: Add release builds, now generated with 'make release' 2016-02-23 15:14:02 -08:00
Makefile docker: Remove unneeded docker files and makefile tags. 2016-06-21 15:31:30 -07:00
namespace-lock_test.go lock: Add unit tests for namespace lock (#1922) 2016-06-22 12:27:47 -07:00
namespace-lock.go XL: Cleanup, comments and all the updated functions. (#1830) 2016-06-01 16:43:31 -07:00
NOTICE Fix typo 2016-03-09 13:55:58 +01:00
notifier.go server: terminal width should fallback to 80x25. (#1895) 2016-06-12 19:35:28 -07:00
object_api_suite_test.go XL: Remove usage of reduceErr and make it isQuorum verification. (#1909) 2016-06-18 00:27:51 +05:30
object-api_test.go posix: Support UNC paths on windows. (#1887) 2016-06-13 02:53:09 -07:00
object-api-disk-not-found_test.go XL/multipart: statPart should ignore errDiskNotFound. (#1862) 2016-06-07 18:15:04 -07:00
object-api-getobjectinfo_test.go posix: Support UNC paths on windows. (#1887) 2016-06-13 02:53:09 -07:00
object-api-listobjects_test.go posix: Support UNC paths on windows. (#1887) 2016-06-13 02:53:09 -07:00
object-api-multipart_test.go Test for Complete Multipart Upload. (#1888) 2016-06-10 18:43:16 +05:30
object-common.go FS: remove .minio directory if .minio/multipart is empty. (#1899) 2016-06-16 08:50:38 +05:30
object-datatypes.go docs: Add comments for each data types. (#1881) 2016-06-09 06:24:11 -07:00
object-errors.go XL: Bring in some modularity into format verification and healing. (#1832) 2016-06-02 16:34:15 -07:00
object-handlers.go object layer: Send 200 OK and whitespace chars (#1897) 2016-06-16 09:01:06 +05:30
object-interface.go XL: bring in new storage API. (#1780) 2016-05-28 16:12:51 -07:00
object-utils_test.go Validation: Reject object names with trailing "/". (#1619) 2016-05-13 11:43:06 -07:00
object-utils.go XL: bring in new storage API. (#1780) 2016-05-28 16:12:51 -07:00
posix-dirent-fileno.go xl/fs: Split object layer into interface. (#1415) 2016-04-29 14:24:10 -07:00
posix-dirent-ino.go xl/fs: Split object layer into interface. (#1415) 2016-04-29 14:24:10 -07:00
posix-list-dir-nix.go logging: Log only for unhandled errors, remove all the debug logging. (#1652) 2016-05-16 14:31:28 -07:00
posix-list-dir-others.go posix: cleanup usage of fmt.Println() (#1934) 2016-06-19 18:52:19 -07:00
posix-prepare-path.go posix: Support UNC paths on windows. (#1887) 2016-06-13 02:53:09 -07:00
posix-utils_common.go posix: Support UNC paths on windows. (#1887) 2016-06-13 02:53:09 -07:00
posix-utils_nix_test.go posix: Mkdir() and OpenFile() should honor umask. (#1972) 2016-06-23 20:19:27 -07:00
posix-utils_nix.go posix-utils: remove unused isValidPath() (#1937) 2016-06-20 06:17:36 -07:00
posix-utils_test.go posix: Mkdir() and OpenFile() should honor umask. (#1972) 2016-06-23 20:19:27 -07:00
posix-utils_windows_test.go posix: Mkdir() and OpenFile() should honor umask. (#1972) 2016-06-23 20:19:27 -07:00
posix-utils_windows.go posix-utils: remove unused isValidPath() (#1937) 2016-06-20 06:17:36 -07:00
posix.go XL: Make allocations simpler avoid redundant allocs. (#1961) 2016-06-24 02:06:23 -07:00
README.md Add erasure code. 2016-06-03 12:50:36 -07:00
routers.go XL: Bring in some modularity into format verification and healing. (#1832) 2016-06-02 16:34:15 -07:00
rpc-client.go XL: Change AppendFile() to return only error (#1932) 2016-06-19 15:31:13 -07:00
rpc-server-datatypes.go docs: Add comments for each data types. (#1881) 2016-06-09 06:24:11 -07:00
rpc-server.go XL: Change AppendFile() to return only error (#1932) 2016-06-19 15:31:13 -07:00
runtime-checks.go isContainerized should look for /.dockerenv not /.dockerinit 2016-04-18 19:26:56 -07:00
server_test.go TestServer introduction and revamp of functional tests. (#1940) 2016-06-21 12:10:18 -07:00
server_utils_test.go xl/tests: Enable server handler tests over XL. (#1512) 2016-05-06 16:47:23 -07:00
server_xl_test.go XL: Add mis-aligned GetObject() test. (#1960) 2016-06-22 21:42:24 -07:00
server-main.go XL: Re-align the code again. 2016-06-02 01:54:06 -07:00
server-rlimit-nix.go server: Set rLimit properly to the max. (#1676) 2016-05-18 11:34:24 -07:00
server-rlimit-win.go server: Attempt to increase max open files. (#1328) 2016-04-18 22:05:32 -07:00
signals.go FS: remove .minio directory if .minio/multipart is empty. (#1899) 2016-06-16 08:50:38 +05:30
signature-jwt.go xl/fs: Split object layer into interface. (#1415) 2016-04-29 14:24:10 -07:00
signature-v4-parser.go error: Signature errors should be returned with APIErrorCode. 2016-03-31 23:28:40 -07:00
signature-v4-postpolicyform.go fs: Handle cases of PutObject for an existing prefix. (#1478) 2016-05-04 12:18:40 -07:00
signature-v4-utils.go Replacing fastsha256 with crypto/sha256 package from golang standard package (#1584) 2016-05-10 14:20:11 -07:00
signature-v4.go Replacing fastsha256 with crypto/sha256 package from golang standard package (#1584) 2016-05-10 14:20:11 -07:00
storage-datatypes.go docs: Add comments for each data types. (#1881) 2016-06-09 06:24:11 -07:00
storage-errors.go XL: Bring in some modularity into format verification and healing. (#1832) 2016-06-02 16:34:15 -07:00
storage-interface.go XL: Change AppendFile() to return only error (#1932) 2016-06-19 15:31:13 -07:00
test-utils_test.go TestServer introduction and revamp of functional tests. (#1940) 2016-06-21 12:10:18 -07:00
tree-walk-fs.go FS/ListMultipart: Fix FS list-multipart to work for unit test cases. 2016-05-28 15:15:53 -07:00
tree-walk-pool.go XL/tree-walk: Added comments, changed variable names and structure fields to improve code readability. (#1856) 2016-06-05 11:55:45 -07:00
tree-walk-xl.go posix: return errFaultyDisk on I/O errors. (#1885) 2016-06-08 22:02:10 -07:00
trie.go Add trie to verify wrong inputs, and provide meaningful messages 2015-09-17 16:49:08 -07:00
typed-errors.go UI-handler: remove minio-go dependancy. 2016-04-01 13:56:32 +05:30
update-main.go server: terminal width should fallback to 80x25. (#1895) 2016-06-12 19:35:28 -07:00
utils.go Part ID check (#1730) 2016-05-24 01:52:47 -07:00
version_test.go Move all server and controller packages into top-level 2015-09-19 01:07:42 -07:00
version-main.go console: Fix console color printing on windows. 2016-03-04 10:07:19 -08:00
web-handlers.go XL: bring in new storage API. (#1780) 2016-05-28 16:12:51 -07:00
web-router.go xl/fs: Split object layer into interface. (#1415) 2016-04-29 14:24:10 -07:00
xl-v1-bucket.go XL: Remove usage of reduceErr and make it isQuorum verification. (#1909) 2016-06-18 00:27:51 +05:30
xl-v1-common.go posix: return errFaultyDisk on I/O errors. (#1885) 2016-06-08 22:02:10 -07:00
xl-v1-healing.go XL: Make allocations simpler avoid redundant allocs. (#1961) 2016-06-24 02:06:23 -07:00
xl-v1-list-objects.go XL/tree-walk: Added comments, changed variable names and structure fields to improve code readability. (#1856) 2016-06-05 11:55:45 -07:00
xl-v1-metadata_test.go XL/Unittest: Add testcase for xlMetaV1{} and its methods. (#1938) 2016-06-20 07:35:41 -07:00
xl-v1-metadata.go XL: Make allocations simpler avoid redundant allocs. (#1961) 2016-06-24 02:06:23 -07:00
xl-v1-multipart-common.go XL: Make allocations simpler avoid redundant allocs. (#1961) 2016-06-24 02:06:23 -07:00
xl-v1-multipart.go Renaming a part from tmp namespace needs to be handled different from… (#1944) 2016-06-20 19:11:55 -07:00
xl-v1-object_test.go Added couple of unit-tests to xlObjects (#1950) 2016-06-21 15:48:27 -07:00
xl-v1-object.go XL/erasure-read: Add Comments and enable bitrot detection. 2016-06-23 00:34:47 +05:30
xl-v1-utils.go XL: Make allocations simpler avoid redundant allocs. (#1961) 2016-06-24 02:06:23 -07:00
xl-v1.go XL: Format heal should re-allocate new UUIDs not reuse. (#1953) 2016-06-22 17:18:31 -07:00

Minio Gitter

Minio is an object storage server compatible with Amazon S3 and licensed under Apache license 2.0.

Description

Minio is an open source object storage server released under Apache License V2. It is compatible with Amazon S3 cloud storage service. Minio follows a minimalist design philosophy. Minio is light enough to be bundled with the application stack. It sits on the side of NodeJS, Redis, MySQL and the likes. Unlike databases, Minio stores objects such as photos, videos, log files, backups, container / VM images and so on. Minio is best suited for storing blobs of information ranging from KBs to 5 TBs each. In a simplistic sense, it is like a FTP server with a simple get / put API over HTTP.

Minio currently implements two backends

  • Filesystem (FS) - is available and ready for general purpose use.
  • ErasureCoded (XL) - is available, but it is not ready for general purpose use.

Minio Client

Minio Client (mc) provides a modern alternative to Unix commands like ls, cat, cp, sync, and diff. It supports POSIX compatible filesystems and Amazon S3 compatible cloud storage systems. It is entirely written in Golang.

Amazon S3 Compatible Client Libraries

Install Build StatusBuild status

GNU/Linux

Download minio for:

$ chmod +x minio
$ ./minio --help

OS X

Download minio from https://dl.minio.io/server/minio/release/darwin-amd64/minio

$ chmod 755 minio
$ ./minio --help

Microsoft Windows

Download minio for:

C:\Users\Username\Downloads> minio.exe --help

FreeBSD

Download minio from https://dl.minio.io/server/minio/release/freebsd-amd64/minio

$ chmod 755 minio
$ ./minio --help

Read more here on How to configure Minio on FreeBSD with ZFS backend.

Docker container

Download minio for docker.

$ docker pull minio/minio

Read more here on How to configure data volume containers for Minio?

Source

NOTE: Source installation is intended for only developers and advanced users. For general use, please download official releases from https://minio.io/downloads.

If you do not have a working Golang environment, please follow Install Golang.

$ go get -d github.com/minio/minio
$ cd $GOPATH/src/github.com/minio/minio
$ make

How to use Minio?

Start minio server.

$ minio server ~/Photos

AccessKey: WLGDGYAQYIGI833EV05A  SecretKey: BYvgJM101sHngl2uzjXS/OBF/aMxAN06JrJ3qJlF  Region: us-east-1

Minio Object Storage:
    http://127.0.0.1:9000
    http://10.1.10.177:9000

Minio Browser:
    http://127.0.0.1:9000
    http://10.1.10.177:9000

To configure Minio Client:
    $ wget https://dl.minio.io/client/mc/release/darwin-amd64/mc
    $ chmod 755 mc
    $ ./mc config host add myminio http://localhost:9000 WLGDGYAQYIGI833EV05A BYvgJM101sHngl2uzjXS/OBF/aMxAN06JrJ3qJlF

How to use AWS CLI with Minio?

This section assumes that you have already installed aws-cli, if not please visit https://aws.amazon.com/cli/

To configure aws-cli, type aws configure and follow below steps.

$ aws configure
AWS Access Key ID [None]: YOUR_ACCESS_KEY_HERE
AWS Secret Access Key [None]: YOUR_SECRET_KEY_HERE
Default region name [None]: us-east-1
Default output format [None]: ENTER

Additionally enable aws-cli to use AWS Signature Version '4' for Minio server.

$ aws configure set default.s3.signature_version s3v4

To list your buckets.

$ aws --endpoint-url http://localhost:9000 s3 ls
2016-03-27 02:06:30 deebucket
2016-03-28 21:53:49 guestbucket
2016-03-29 13:34:34 mbtest
2016-03-26 22:01:36 mybucket
2016-03-26 15:37:02 testbucket

To list contents inside bucket.

$ aws --endpoint-url http://localhost:9000 s3 ls s3://mybucket
2016-03-30 00:26:53      69297 argparse-1.2.1.tar.gz
2016-03-30 00:35:37      67250 simplejson-3.3.0.tar.gz

To make a bucket.

$ aws --endpoint-url http://localhost:9000 s3 mb s3://mybucket
make_bucket: s3://mybucket/

To add an object to a bucket.

$ aws --endpoint-url http://localhost:9000 s3 cp simplejson-3.3.0.tar.gz s3://mybucket
upload: ./simplejson-3.3.0.tar.gz to s3://mybucket/simplejson-3.3.0.tar.gz

Delete an object from a bucket.

$ aws --endpoint-url http://localhost:9000 s3 rm s3://mybucket/argparse-1.2.1.tar.gzdelete: s3://mybucket/argparse-1.2.1.tar.gz

Remove a bucket.

$ aws --endpoint-url http://localhost:9000 s3 rb s3://mybucket
remove_bucket: s3://mybucket/

How to use AWS SDK with Minio?

Please follow the documentation here - Using aws-sdk-go with Minio server

How to use s3cmd with Minio?

This section assumes that you have already installed s3cmd, if not please visit http://s3tools.org/s3cmd

Edit the following fields in your s3cmd configuration file ~/.s3cfg .

host_base = localhost:9000
host_bucket = localhost:9000
access_key = YOUR_ACCESS_KEY_HERE
secret_key = YOUR_SECRET_KEY_HERE
signature_v2 = False
bucket_location = us-east-1

To make a bucket.

$ s3cmd mb s3://mybucket
Bucket 's3://mybucket/' created

To copy an object to bucket.

$ s3cmd put newfile s3://testbucket
upload: 'newfile' -> 's3://testbucket/newfile'  

To copy an object to local system.

$ s3cmd get s3://testbucket/newfile
download: 's3://testbucket/newfile' -> './newfile'

To sync local file/directory to a bucket.

$ s3cmd sync newdemo s3://testbucket
upload: 'newdemo/newdemofile.txt' -> 's3://testbucket/newdemo/newdemofile.txt'

To sync bucket or object with local filesystem.

$ s3cmd sync  s3://otherbucket otherlocalbucket
download: 's3://otherbucket/cat.jpg' -> 'otherlocalbucket/cat.jpg'

To list buckets.

$ s3cmd ls s3://
2015-12-09 16:12  s3://testbbucket

To list contents inside bucket.

$ s3cmd ls s3://testbucket/
                       DIR   s3://testbucket/test/
2015-12-09 16:05    138504   s3://testbucket/newfile

Delete an object from bucket.

$ s3cmd del s3://testbucket/newfile
delete: 's3://testbucket/newfile'

Delete a bucket.

$ s3cmd rb s3://testbucket
Bucket 's3://testbucket/' removed

Contribute to Minio Project

Please follow Minio Contributor's Guide

Jobs

If you think in Lisp or Haskell and hack in go, you would blend right in. Send your github link to callhome@minio.io.