MyFavMirrors/minio
1
0
Fork 0
mirror of https://github.com/minio/minio.git synced 2024-12-27 15:45:55 -05:00
Code Issues Packages Projects Releases Wiki Activity
1,650 Commits 4 Branches 506 Tags 189 MiB
9a4680475f
Commit Graph

325 Commits

Author SHA1 Message Date
Harshavardhana
e95604ff86 Finishing all the test case support for ACL and other fixes 2015-04-23 01:24:46 -07:00
Harshavardhana
1c0ff2c758 ACL driver/storage layer support 2015-04-22 19:31:45 -07:00
Frederick F. Kautz IV
c56f7380d6 Memory driver now limits by size 2015-04-22 17:50:26 -07:00
Harshavardhana
b121c8588f Remove bucketpolicy handlers and all its references 2015-04-22 14:46:51 -07:00
Harshavardhana
3f33643d39 File driver to take the high road 2015-04-16 02:16:39 -07:00
Frederick F. Kautz IV
59a81a73c6 Adding lru to memory driver, not wired to command line opts 2015-04-15 12:49:41 -07:00
Frederick F. Kautz IV
0cab3fd151 Better locking in memory driver 2015-04-15 11:48:56 -07:00
Harshavardhana
38155e86ac iodine is part of minio now 2015-04-08 18:05:48 -07:00
Harshavardhana
7fd1cc073c Add everything back into one project hood, still missing iodine 2015-04-08 17:47:56 -07:00
Frederick F. Kautz IV
6241c9880b Moving object storage drivers back to minio 2015-04-08 16:28:14 -07:00
Harshavardhana
0475d7d056 Migrate to external Donut package 2015-04-05 14:59:24 -07:00
Frederick F. Kautz IV
2f44438843 Reformatting imports 2015-04-04 13:59:59 -07:00
Frederick F. Kautz IV
be4bc44b4d Replacing gopkg.in/check.v1 with minio-io/check 2015-04-04 13:37:30 -07:00
Harshavardhana
92f408af5d Add erasure to godep 2015-04-01 16:56:43 -07:00
Harshavardhana
b748cf359a More simplification to erasure also add two more test functions 2015-03-29 14:52:13 -07:00
Frederick F. Kautz IV
c901d0fb32 Fix for readers closing in erasure early 2015-03-27 23:49:14 -07:00
Frederick F. Kautz IV
53439d7768 Updating iodine to match new api 2015-03-26 15:55:06 -07:00
Frederick F. Kautz IV
c5f10df957 Removing TODO 2015-03-26 11:33:50 -07:00
Frederick F. Kautz IV
eaeeb4bef0 Refactoring erasure, each block is encoded by a function. Iodine included. 2015-03-26 11:32:39 -07:00
Frederick F. Kautz IV
31746fdeb2 Adding iodine to donutObjectWriter.Write 2015-03-26 10:49:03 -07:00
Frederick F. Kautz IV
bfbb97c98f Merge pull request #393 from fkautz/pr_out_adding_iodine_to_object_writer 2015-03-26 10:42:57 -07:00
Frederick F. Kautz IV
f7a300400e Merge pull request #392 from fkautz/pr_out_adding_iodine_to_node_local 2015-03-26 10:42:50 -07:00
Frederick F. Kautz IV
caf7e41499 Adding iodine to object_writer 2015-03-26 10:41:40 -07:00
Frederick F. Kautz IV
5cdf2564a5 Adding iodine to node_local 2015-03-26 10:21:24 -07:00
Frederick F. Kautz IV
70a2217eb2 AddNode fails with proper error code when bucketid is malformed 2015-03-26 10:02:40 -07:00
Frederick F. Kautz IV
cc64417481 Adding iodine to donut 2015-03-26 09:57:21 -07:00
Frederick F. Kautz IV
40e311a919 Updating iodine to support new iodine.Error(error, map[string]string) error method 2015-03-25 19:12:55 -07:00
Frederick F. Kautz IV
eff48971ad Adding iodine to donut bucket 2015-03-25 17:05:26 -07:00
Harshavardhana
58082cd8dc Add gocyclo into source build, choosing cyclomatic complexity tolerance of 15 and below 2015-03-25 15:57:17 -07:00
Harshavardhana
c366fdd2d8 loop only on readers once 2015-03-25 14:48:46 -07:00
Frederick F. Kautz IV
be229473b5 Updating iodine 2015-03-25 00:43:48 -07:00
Harshavardhana
b3c3a8d6f2 Avoid empty if else body.. 2015-03-25 00:24:41 -07:00
Frederick F. Kautz IV
1ae74591f6 Adding iodine 2015-03-24 23:54:49 -07:00
Frederick F. Kautz IV
dd5ba865ff Merge remote-tracking branch 'origin/master' into HEAD 
Conflicts:
	pkg/drivers/donut/donut.go
	pkg/storage/donut/bucket.go
	pkg/storage/donut/donut.go
	pkg/storage/donut/donut_test.go
	pkg/storage/donut/donutdriver.go
 2015-03-24 21:04:08 -07:00
Frederick F. Kautz IV
cf2550b267 Populate buckets at start 2015-03-24 19:58:03 -07:00
Harshavardhana
653ea50a59 Erasure technique is now uint8, handle it appropriately 2015-03-24 19:12:52 -07:00
Anand Babu (AB) Periasamy
fa9b1f341b erasure encode janitor duty 2015-03-24 18:47:53 -07:00
Harshavardhana
6a87ed001c Handle errors properly during erasure Decoding, also populate technique and verify 2015-03-24 18:14:13 -07:00
Harshavardhana
0474439b43 Code restructuring, assigning proper subsystems to each 
- Drivers contain
    * donut/*
    * file/*
    * memory/*

 - Storage format contains
    * donut/*
      - GetObject() --> renamed to GetObjectReader()
      - Deleted stale objectwriter.go, renamed donutwriter.go to object_writer.go

Simplifying, and documenting codebase further
 2015-03-24 06:47:10 -07:00
Frederick F. Kautz IV
3c4012f1e7 Adding donut backend and setting as default 2015-03-23 20:07:22 -07:00
Harshavardhana
7c1c4b8a02 Merge pull request #351 from harshavardhana/pr_out_implement_log_package_as_drop_in_replacement_for_handling_debug_log_level 2015-03-23 18:55:14 -07:00
Harshavardhana
914962bd93 Implement log package as drop-in replacement for handling Debug log-level 2015-03-23 18:18:11 -07:00
Frederick F. Kautz IV
f91f5e59ee Running gofmt manually, auto gofmt didn't simplify code 2015-03-23 12:26:41 -07:00
Frederick F. Kautz IV
56a298fd6a Refactoring method variable names for erasureWriter struct 2015-03-23 12:25:20 -07:00
Frederick F. Kautz IV
d57e9d6c18 Adding sys. as reserved in erasureWriter 2015-03-23 11:56:24 -07:00
Frederick F. Kautz IV
b2f47ef446 Switching content hashing from sha512 to md5 2015-03-23 11:46:19 -07:00
Frederick F. Kautz IV
b2c3172095 Adding sha512 support for pre-encoded data and verification on decode 2015-03-22 20:11:25 -07:00
Frederick F. Kautz IV
befac7d047 Exorcising donutbox 2015-03-22 19:55:15 -07:00
Frederick F. Kautz IV
eb8005cc65 Reading correct block size in erasure coded data 2015-03-22 18:49:30 -07:00
Harshavardhana
3929135c34 Make sure we use O_EXCL with O_CREATE to make sure we don't trip over existing file 2015-03-22 15:55:40 -07:00
Harshavardhana
2fa9320df2 De-couple donut into smaller files, useful for ease in external integration 2015-03-22 15:43:37 -07:00
Frederick F. Kautz IV
55f3659bb3 Removing log tracers 2015-03-22 15:39:50 -07:00
Frederick F. Kautz IV
03f51eac29 Adding list objects to donut 2015-03-22 15:33:52 -07:00
Harshavardhana
4b9fbd3b3a DonutfileWriter should return value of file.Close() 2015-03-22 14:51:18 -07:00
Frederick F. Kautz IV
ced6edc9fe Minor change, removing unnecessary commented method 2015-03-22 12:44:45 -07:00
Frederick F. Kautz IV
99cf484488 Adding GetObjectMetadata to donut 2015-03-22 12:36:51 -07:00
Frederick F. Kautz IV
703af6c3ce Removing gob from donut driver 2015-03-22 12:17:12 -07:00
Harshavardhana
95affcb119 Fix build issues for new DonutDriver() 2015-03-22 02:53:21 -07:00
Frederick F. Kautz IV
933d44c656 New donut driver with local file encoding 2015-03-22 00:40:21 -07:00
Harshavardhana
2b0b5f014a Update license header Mini should be Minimalist really 2015-03-19 14:35:50 -07:00
Harshavardhana
c3602967ab Fix bucketpolicy to handle s3 and minio policy names separately. 
Also fix an apparent bug in GetBucketPolicy{}
 2015-03-19 12:22:14 -07:00
Frederick F. Kautz IV
2c69433588 Adding interfaces for donut driver 2015-03-18 14:27:10 -07:00
Frederick F. Kautz IV
bf86e78714 Renaming files to match 2015-03-18 12:26:44 -07:00
Frederick F. Kautz IV
1c9e550ee6 Renaming encoded driver to donutstorage 2015-03-18 12:25:37 -07:00
Frederick F. Kautz IV
982a52e0d4 Merge pull request #320 from fkautz/pr_out_removing_block_size_from_donut_writer_no_longer_necessary 2015-03-17 14:25:08 -07:00
Frederick F. Kautz IV
d6be8d3517 Removing block size from donut writer, no longer necessary 2015-03-17 14:24:59 -07:00
Harshavardhana
74b3d092f2 Now MinioAPI handles Content-MD5 set during putObject() 
- This change also facilitates proper error response
    in case of md5 mismatch or corruption
  - TODO a test function needs to be implemented
 2015-03-17 13:32:13 -07:00
Harshavardhana
42006c2ab0 Merge pull request #318 from harshavardhana/pr_out_remove_contenttype_pseudo_func_ 2015-03-17 00:47:46 -07:00
Harshavardhana
dccf32e5c4 Remove contenttype pseudo func() 2015-03-17 00:47:22 -07:00
Frederick F. Kautz IV
90b337a3e8 Fixing order of input and layout in time.Parse 2015-03-17 00:47:17 -07:00
Frederick F. Kautz IV
25725c442a Moving to more accurate RFC3339 2015-03-17 00:45:50 -07:00
Frederick F. Kautz IV
e36fa95f1f Moving from iso8601 to RFC3339 2015-03-17 00:42:08 -07:00
Frederick F. Kautz IV
598eb5192e Encoded now passes tests, enabling. 2015-03-17 00:33:49 -07:00
Harshavardhana
6d71935ea5 Encoded storage type now feature complete 2015-03-16 23:42:45 -07:00
Frederick F. Kautz IV
34cf6682fe list objects in buckets 2015-03-16 20:43:05 -07:00
Harshavardhana
b495f15178 Make blockSize as constant 2015-03-16 18:20:01 -07:00
Frederick F. Kautz IV
583c32c774 Metadata now stores hex md5, not base64. Minio API now converts md5 hex to base64. 2015-03-16 18:03:22 -07:00
Frederick F. Kautz IV
10ae089c46 Better handling of metadata in encoded 2015-03-16 17:40:59 -07:00
Harshavardhana
15f68972a5 Some necessary cleanup 
- Rename 'fs' as 'file' for brevity
  - Rename 'inmemory' as 'memory' for brevity
  - Change everywhere else
 2015-03-16 11:46:16 -07:00
Harshavardhana
bf04d125fd Merge pull request #304 from harshavardhana/pr_out_move_from_fmt_errorf_to_errors_new 2015-03-15 19:42:23 -07:00
Harshavardhana
596eeb097e Move from fmt.Errorf to errors.New 
In many cases fmt.Errorf is good enough, but since
error is an interface, you can use arbitrary data
structures as error values, to allow callers to
inspect the details of the error.
 2015-03-15 19:41:16 -07:00
Frederick F. Kautz IV
a589ddaddf Decoding and returning objects now works 2015-03-15 19:02:00 -07:00
Frederick F. Kautz IV
682bf085c4 Metadata and writing the object are now both committed together. 2015-03-15 17:42:33 -07:00
Frederick F. Kautz IV
96d0a089b8 Adding create bucket to encoded 2015-03-15 12:28:24 -07:00
Harshavardhana
3f95c55067 Skip uinimplemented tests and some cleanup 2015-03-15 01:23:57 -07:00
Frederick F. Kautz IV
3e453a394a Working in memory donut driver 2015-03-15 00:41:06 -07:00
Frederick F. Kautz IV
85da9fc627 Adding start of inmem donut 2015-03-14 18:02:27 -07:00
Frederick F. Kautz IV
54472c18fe Renaming singledisk to encoded 2015-03-14 11:37:43 -07:00
Frederick F. Kautz IV
348c4d9a88 Removing old donut format 2015-03-14 11:36:19 -07:00
Frederick F. Kautz IV
2f4b92eb8d Minor refactor to prepare for file version of donut 2015-03-14 11:34:45 -07:00
Harshavardhana
f7fd526b28 Cosmetic changes 2015-03-12 02:11:04 -07:00
Frederick F. Kautz IV
94fe03cfc1 More refactoring to meet new donut spec 2015-03-12 01:52:14 -07:00
Harshavardhana
67888cbaa9 Restructure and use human readable names 2015-03-12 00:36:33 -07:00
Harshavardhana
e21020b337 Merge pull request #280 from harshavardhana/pr_out_http_range_support_wip 
Get object range request added
 2015-03-12 00:11:38 -07:00
Harshavardhana
a42a48a455 Get object range request added 2015-03-12 00:07:27 -07:00
Frederick F. Kautz IV
24d5018037 Rename StoreBucket to CreateBucket 2015-03-11 20:14:40 -07:00
Frederick F. Kautz IV
8c6aa45458 Adding copyright 2015-03-11 19:59:20 -07:00
Frederick F. Kautz IV
7971b66e60 Adding object donut type 2015-03-11 19:56:18 -07:00
Frederick F. Kautz IV
ec9b20a94f Adding stubes for new single disk driver 2015-03-11 13:36:21 -07:00
Harshavardhana
41653774fc Simplify fs codebase, split them into separate files and more commenting 2015-03-08 18:01:48 -07:00
Harshavardhana
01e98eb326 Make BucketPolicy strictly typed 2015-03-08 15:57:20 -07:00
Harshavardhana
8866b2cc5f Add delimiter and prefix tests, also add inmemory to support delimiters 2015-03-08 14:51:31 -07:00
Harshavardhana
7a01db8005 Strict checks for delimiter and prefix if absent or present 2015-03-08 13:01:39 -07:00
Harshavardhana
9e89af0e3c Remove redundant uint32 2015-03-07 19:15:51 -08:00
Harshavardhana
9553eb6b4a Merge pull request #269 from harshavardhana/pr_out_get_object_metadata_needs_to_filter_based_of_prefix_when_delimiter_is_specified 2015-03-07 19:14:41 -08:00
Harshavardhana
f8e2b75e23 Get object metadata needs to filter based of Prefix when delimiter is specified 2015-03-07 19:13:15 -08:00
Frederick F. Kautz IV
b39eac36ea Merge pull request #268 from fkautz/pr_out_fragment_reader_implemented 2015-03-07 18:04:29 -08:00
Frederick F. Kautz IV
679c6ace3a Fragment reader implemented 2015-03-07 18:04:20 -08:00
Harshavardhana
df69038b04 Merge pull request #267 from harshavardhana/pr_out_add_md5_support_for_fsstorage_also_re_purpose_getobjectmetadata_for_listobjects 2015-03-07 17:16:00 -08:00
Harshavardhana
4751ed0e30 Add MD5 support for fsStorage, also re-purpose GetObjectMetadata for ListObjects 2015-03-07 17:15:00 -08:00
Frederick F. Kautz IV
ca1a4b616c Adding read header for erasure 2015-03-07 15:53:41 -08:00
Frederick F. Kautz IV
00381c887a Adding erasure package to wrap versioned erasure. Ensures we always write latest version. 2015-03-07 13:04:02 -08:00
Harshavardhana
d992bccd9a Move pkg/storage/erasure to pkg/encoding/erasure - and other cleanups 2015-03-07 00:16:48 -08:00
Frederick F. Kautz IV
856e0100c0 Erasure layer now writes using new technique 2015-03-06 16:37:44 -08:00
Harshavardhana
e5af8a3f5d Further fixes - 
- All test files have been renamed to their respective <package>_test name,
    this is done in accordance with
      - https://github.com/golang/go/wiki/CodeReviewComments#import-dot
        imports are largely used in testing, but to avoid namespace collision
        and circular dependencies

  - Never use _* in package names other than "_test" change fragment_v1 to expose
    fragment just like 'gopkg.in/check.v1'
 2015-03-06 02:04:21 -08:00
Harshavardhana
c00d1461b9 Golint cleanup from top level 2015-03-05 23:37:37 -08:00
Harshavardhana
540723d7ae Golint cleanup pkg/api 2015-03-05 21:09:19 -08:00
Harshavardhana
76e601b26b Golint cleanup pkg/storage 2015-03-05 21:09:19 -08:00
Frederick F. Kautz IV
1186c47603 Fragment now implements new version 2015-03-05 20:56:02 -08:00
Frederick F. Kautz IV
665af44c70 Renaming donut packages to match layout 2015-03-05 19:44:01 -08:00
Harshavardhana
f2e238199b Handle individual objects too 2015-03-04 01:54:35 -08:00
Harshavardhana
e400e85639 Add proper path prefix to be delimited 2015-03-04 00:54:39 -08:00
Harshavardhana
cc46475c86 Pathprefix is filtered out properly and delimited as well 2015-03-03 19:45:54 -08:00
Harshavardhana
3a3c8645fc Update comments across the codebase 2015-03-03 02:39:38 -08:00
Harshavardhana
137584d658 Add comments 2015-03-03 01:25:45 -08:00
Harshavardhana
c3ad0906e0 Add deadcode code which recursivley goes into all directories and verifies dangling variables. 2015-03-02 18:44:20 -08:00
Frederick F. Kautz IV
32f8c17174 Renaming AppendUstr to AppendUniqStr 2015-03-01 17:06:00 -08:00
Frederick F. Kautz IV
6f300c9ef0 Adding test at byte level 2015-03-01 15:13:09 -08:00
Frederick F. Kautz IV
0b08312ddd Specifying more exact types for data 2015-03-01 14:43:11 -08:00
Harshavardhana
9f43659135 Add tests, add NewHeader() for populating Header and ValidateHeader() 2015-03-01 13:19:06 -08:00
Anand Babu (AB) Periasamy
6515678e87 Merge pull request #236 from fkautz/pr_out_adding_data_writer 
Adding data writer
 2015-03-01 11:36:35 -08:00
Frederick F. Kautz IV
7414dc9b75 Adding data writer 2015-03-01 11:32:30 -08:00
Harshavardhana
eb9a4a83ac Move AppendUstr(), AppendUint() into helpers 2015-03-01 11:19:03 -08:00
Frederick F. Kautz IV
be12a6b3d9 Changing K and M in data.go to match new datatype in erasure package 2015-03-01 01:28:33 -08:00
Harshavardhana
33601ff567 Merge pull request #233 from harshavardhana/pr_out_make_k_m_to_be_uint8_and_technique_becomes_its_own_type 2015-03-01 01:26:42 -08:00
Harshavardhana
b77c62e25d Make K,M to be uint8 and Technique becomes its own type 2015-03-01 01:25:28 -08:00
Frederick F. Kautz IV
bb0441917c Making changes to data based on discussion 2015-03-01 01:16:22 -08:00
Harshavardhana
2f04a1299e Avoid allocating 1gig memory, avoid such mistakes 2015-03-01 00:43:12 -08:00
Frederick F. Kautz IV
2916b1278e Adding data.go, does nothing for now. 2015-03-01 00:27:52 -08:00
Frederick F. Kautz IV
b4ef522a9b Donut frame is now its own package 2015-03-01 00:02:14 -08:00
Frederick F. Kautz IV
6c2428833a Renaming Write to WriteFrame 2015-02-28 23:52:56 -08:00
Harshavardhana
6b397844b9 Add donut benchmark for 64MB, 128MB, 256MB, 512MB 
Current average values

~~~
> minio/pkg/storage/donut/v1 *> go test -bench .
OK: 2 passed
PASS
BenchmarkDonut64M              5         266249634 ns/op         252.05 MB/s
BenchmarkDonut128M             3         436507394 ns/op         307.48 MB/s
BenchmarkDonut256M             2         836116359 ns/op         321.05 MB/s
BenchmarkDonut512M             1        1676738951 ns/op         320.19 MB/s
ok      github.com/minio-io/minio/pkg/storage/donut/v1  8.430s
> minio/pkg/storage/donut/v1 *>
~~~
 2015-02-28 17:05:46 -08:00
Harshavardhana
65f22083ca donut_gen works with new Donut Frame 2015-02-28 16:30:18 -08:00
Frederick F. Kautz IV
ddc7cf835e Donut now tests amount written should match expected amount to write 2015-02-28 16:09:52 -08:00
Frederick F. Kautz IV
d3d1205eb1 Updating comment describing data type 2015-02-28 15:59:52 -08:00
Frederick F. Kautz IV
58a04ee831 New donut frame implemented 2015-02-28 15:54:13 -08:00
Harshavardhana
0c2d58bc6d Implement proper delimiter and prefix handling 
With this change Minio server now responds with, delimited
'object names' in conjunction with prefix filtering

~~~
<ListBucketResult>
  <Name>example-bucket</Name>
  <Prefix></Prefix>
  <Marker></Marker>
  <MaxKeys>1000</MaxKeys>
  <Delimiter>/</Delimiter>
  <IsTruncated>false</IsTruncated>
  <Contents>
    <Key>sample.html</Key>
    <LastModified>2011-02-26T01:56:20.000Z</LastModified>
    <ETag>example-bucket#sample.html</ETag>
    <Size>142863</Size>
    <Owner>
      <ID>minio</ID>
      <DisplayName>minio</DisplayName>
    </Owner>
    <StorageClass>STANDARD</StorageClass>
  </Contents>
  <CommonPrefixes>
    <Prefix>photos/</Prefix>
  </CommonPrefixes>
</ListBucketResult>
~~~

~~~
<ListBucketResult>
  <Name>example-bucket</Name>
  <Prefix>photos/2006/</Prefix>
  <Marker></Marker>
  <MaxKeys>1000</MaxKeys>
  <Delimiter>/</Delimiter>
  <IsTruncated>false</IsTruncated>

  <CommonPrefixes>
    <Prefix>photos/2006/feb/</Prefix>
  </CommonPrefixes>
  <CommonPrefixes>
    <Prefix>photos/2006/jan/</Prefix>
  </CommonPrefixes>
</ListBucketResult>
~~~
 2015-02-28 14:47:19 -08:00
Harshavardhana
d5e5a3b3e6 Use validString 2015-02-27 19:49:18 -08:00
Harshavardhana
6ebb48b4ea Add object name validation 2015-02-27 19:42:04 -08:00
Harshavardhana
53669a0854 Implement delimiter, path prefix 2015-02-27 16:23:05 -08:00