MyFavMirrors/minio
1
0
Fork 0
mirror of https://github.com/minio/minio.git synced 2024-12-26 23:25:54 -05:00
Code Issues Packages Projects Releases Wiki Activity
7,758 Commits 4 Branches 506 Tags 189 MiB
e96fdcd5ec
Commit Graph

1677 Commits

Author SHA1 Message Date
Frederick F. Kautz IV
b2c3172095 Adding sha512 support for pre-encoded data and verification on decode 2015-03-22 20:11:25 -07:00
Frederick F. Kautz IV
befac7d047 Exorcising donutbox 2015-03-22 19:55:15 -07:00
Frederick F. Kautz IV
eb8005cc65 Reading correct block size in erasure coded data 2015-03-22 18:49:30 -07:00
Anand Babu (AB) Periasamy
b9b0b35edf removed private k and m and some more cleanups 2015-03-22 18:22:09 -07:00
Frederick F. Kautz IV
293e438830 Merge pull request #341 from abperiasamy/janitor 2015-03-22 17:34:25 -07:00
Anand Babu (AB) Periasamy
3a76229551 encoded size function is now public + some janitor duty 2015-03-22 17:17:53 -07:00
Harshavardhana
3929135c34 Make sure we use O_EXCL with O_CREATE to make sure we don't trip over existing file 2015-03-22 15:55:40 -07:00
Harshavardhana
2fa9320df2 De-couple donut into smaller files, useful for ease in external integration 2015-03-22 15:43:37 -07:00
Frederick F. Kautz IV
55f3659bb3 Removing log tracers 2015-03-22 15:39:50 -07:00
Frederick F. Kautz IV
03f51eac29 Adding list objects to donut 2015-03-22 15:33:52 -07:00
Harshavardhana
4b9fbd3b3a DonutfileWriter should return value of file.Close() 2015-03-22 14:51:18 -07:00
Frederick F. Kautz IV
ced6edc9fe Minor change, removing unnecessary commented method 2015-03-22 12:44:45 -07:00
Frederick F. Kautz IV
99cf484488 Adding GetObjectMetadata to donut 2015-03-22 12:36:51 -07:00
Frederick F. Kautz IV
703af6c3ce Removing gob from donut driver 2015-03-22 12:17:12 -07:00
Harshavardhana
95affcb119 Fix build issues for new DonutDriver() 2015-03-22 02:53:21 -07:00
Frederick F. Kautz IV
933d44c656 New donut driver with local file encoding 2015-03-22 00:40:21 -07:00
Harshavardhana
a8056c4c12 Return error during JoinFiles back to the caller for inspection 2015-03-20 03:17:08 -07:00
Harshavardhana
2b0b5f014a Update license header Mini should be Minimalist really 2015-03-19 14:35:50 -07:00
Harshavardhana
c3602967ab Fix bucketpolicy to handle s3 and minio policy names separately. 
Also fix an apparent bug in GetBucketPolicy{}
 2015-03-19 12:22:14 -07:00
Harshavardhana
6eb5315249 Resource list updates 2015-03-19 00:47:34 -07:00
Harshavardhana
a1baed3bbc Refactor to move config handling into api/ 2015-03-18 22:49:02 -07:00
Harshavardhana
7ce3ab3193 Restructure minio api, move signature checks from utils to Api. 2015-03-18 18:16:19 -07:00
Frederick F. Kautz IV
2c69433588 Adding interfaces for donut driver 2015-03-18 14:27:10 -07:00
Frederick F. Kautz IV
bf86e78714 Renaming files to match 2015-03-18 12:26:44 -07:00
Frederick F. Kautz IV
1c9e550ee6 Renaming encoded driver to donutstorage 2015-03-18 12:25:37 -07:00
Frederick F. Kautz IV
d7f00194b5 Adding interfaces for implementing against 2015-03-17 22:33:59 -07:00
Frederick F. Kautz IV
982a52e0d4 Merge pull request #320 from fkautz/pr_out_removing_block_size_from_donut_writer_no_longer_necessary 2015-03-17 14:25:08 -07:00
Frederick F. Kautz IV
d6be8d3517 Removing block size from donut writer, no longer necessary 2015-03-17 14:24:59 -07:00
Harshavardhana
74b3d092f2 Now MinioAPI handles Content-MD5 set during putObject() 
- This change also facilitates proper error response
    in case of md5 mismatch or corruption
  - TODO a test function needs to be implemented
 2015-03-17 13:32:13 -07:00
Harshavardhana
42006c2ab0 Merge pull request #318 from harshavardhana/pr_out_remove_contenttype_pseudo_func_ 2015-03-17 00:47:46 -07:00
Harshavardhana
dccf32e5c4 Remove contenttype pseudo func() 2015-03-17 00:47:22 -07:00
Frederick F. Kautz IV
90b337a3e8 Fixing order of input and layout in time.Parse 2015-03-17 00:47:17 -07:00
Frederick F. Kautz IV
25725c442a Moving to more accurate RFC3339 2015-03-17 00:45:50 -07:00
Frederick F. Kautz IV
e36fa95f1f Moving from iso8601 to RFC3339 2015-03-17 00:42:08 -07:00
Frederick F. Kautz IV
598eb5192e Encoded now passes tests, enabling. 2015-03-17 00:33:49 -07:00
Harshavardhana
6d71935ea5 Encoded storage type now feature complete 2015-03-16 23:42:45 -07:00
Frederick F. Kautz IV
34cf6682fe list objects in buckets 2015-03-16 20:43:05 -07:00
Frederick F. Kautz IV
d99435963b Merge pull request #311 from fkautz/pr_out_switching_from_base64_to_hex_for_etag 2015-03-16 18:39:07 -07:00
Frederick F. Kautz IV
1a88b36c55 Switching from base64 to hex for etag 2015-03-16 18:38:42 -07:00
Harshavardhana
81f1d3c195 Merge pull request #310 from harshavardhana/pr_out_make_blocksize_as_constant 2015-03-16 18:20:22 -07:00
Harshavardhana
b495f15178 Make blockSize as constant 2015-03-16 18:20:01 -07:00
Frederick F. Kautz IV
211b084e50 Performing etag off of bytes rather than md5 string 2015-03-16 18:15:37 -07:00
Frederick F. Kautz IV
583c32c774 Metadata now stores hex md5, not base64. Minio API now converts md5 hex to base64. 2015-03-16 18:03:22 -07:00
Frederick F. Kautz IV
10ae089c46 Better handling of metadata in encoded 2015-03-16 17:40:59 -07:00
Harshavardhana
15f68972a5 Some necessary cleanup 
- Rename 'fs' as 'file' for brevity
  - Rename 'inmemory' as 'memory' for brevity
  - Change everywhere else
 2015-03-16 11:46:16 -07:00
Harshavardhana
765fde13f1 Remove x509 package, janitor continuation monad 2015-03-15 19:44:55 -07:00
Harshavardhana
bf04d125fd Merge pull request #304 from harshavardhana/pr_out_move_from_fmt_errorf_to_errors_new 2015-03-15 19:42:23 -07:00
Harshavardhana
596eeb097e Move from fmt.Errorf to errors.New 
In many cases fmt.Errorf is good enough, but since
error is an interface, you can use arbitrary data
structures as error values, to allow callers to
inspect the details of the error.
 2015-03-15 19:41:16 -07:00
Frederick F. Kautz IV
a589ddaddf Decoding and returning objects now works 2015-03-15 19:02:00 -07:00
Frederick F. Kautz IV
682bf085c4 Metadata and writing the object are now both committed together. 2015-03-15 17:42:33 -07:00
Frederick F. Kautz IV
96d0a089b8 Adding create bucket to encoded 2015-03-15 12:28:24 -07:00
Harshavardhana
3f95c55067 Skip uinimplemented tests and some cleanup 2015-03-15 01:23:57 -07:00
Frederick F. Kautz IV
3e453a394a Working in memory donut driver 2015-03-15 00:41:06 -07:00
Frederick F. Kautz IV
15d5e74837 More testing 2015-03-14 19:29:54 -07:00
Frederick F. Kautz IV
7e61696901 Inmemory donutbox is now thread safe 2015-03-14 19:21:34 -07:00
Frederick F. Kautz IV
85da9fc627 Adding start of inmem donut 2015-03-14 18:02:27 -07:00
Frederick F. Kautz IV
54472c18fe Renaming singledisk to encoded 2015-03-14 11:37:43 -07:00
Frederick F. Kautz IV
348c4d9a88 Removing old donut format 2015-03-14 11:36:19 -07:00
Frederick F. Kautz IV
2f4b92eb8d Minor refactor to prepare for file version of donut 2015-03-14 11:34:45 -07:00
Harshavardhana
f7b62caccc Update erasure README with supported architecture list 2015-03-12 16:37:08 -07:00
Harshavardhana
f7fd526b28 Cosmetic changes 2015-03-12 02:11:04 -07:00
Harshavardhana
1e4a47ac5f Merge pull request #290 from harshavardhana/pr_out_add_crc32c_license_headers_golang_and_minio 2015-03-12 02:09:06 -07:00
Harshavardhana
4c86e594d6 Add crc32c license headers Golang and Minio 2015-03-12 02:05:05 -07:00
Harshavardhana
e27ba44d94 Remove minio_calc_chunk_size() C code to write one in Golang 2015-03-12 02:03:04 -07:00
Frederick F. Kautz IV
94fe03cfc1 More refactoring to meet new donut spec 2015-03-12 01:52:14 -07:00
Harshavardhana
67888cbaa9 Restructure and use human readable names 2015-03-12 00:36:33 -07:00
Harshavardhana
e21020b337 Merge pull request #280 from harshavardhana/pr_out_http_range_support_wip 
Get object range request added
 2015-03-12 00:11:38 -07:00
Harshavardhana
a42a48a455 Get object range request added 2015-03-12 00:07:27 -07:00
Frederick F. Kautz IV
24d5018037 Rename StoreBucket to CreateBucket 2015-03-11 20:14:40 -07:00
Frederick F. Kautz IV
8c6aa45458 Adding copyright 2015-03-11 19:59:20 -07:00
Frederick F. Kautz IV
7971b66e60 Adding object donut type 2015-03-11 19:56:18 -07:00
Frederick F. Kautz IV
ec9b20a94f Adding stubes for new single disk driver 2015-03-11 13:36:21 -07:00
Harshavardhana
a24bfb007d Rename dateFormat with understandable name `iso8601Format` 2015-03-10 01:51:34 -07:00
Harshavardhana
b952855779 Add strict checks with vet, golint and gofmt 2015-03-09 16:17:38 -07:00
Harshavardhana
41653774fc Simplify fs codebase, split them into separate files and more commenting 2015-03-08 18:01:48 -07:00
Harshavardhana
01e98eb326 Make BucketPolicy strictly typed 2015-03-08 15:57:20 -07:00
Harshavardhana
8866b2cc5f Add delimiter and prefix tests, also add inmemory to support delimiters 2015-03-08 14:51:31 -07:00
Harshavardhana
7a01db8005 Strict checks for delimiter and prefix if absent or present 2015-03-08 13:01:39 -07:00
Harshavardhana
ba85ed7566 Revert "RFC1123 is wrong protocol format for AWS" 
This reverts commit c7b1f8ded8.
 2015-03-08 03:06:23 -07:00
Harshavardhana
c7b1f8ded8 RFC1123 is wrong protocol format for AWS 2015-03-08 01:55:25 -08:00
Harshavardhana
9e89af0e3c Remove redundant uint32 2015-03-07 19:15:51 -08:00
Harshavardhana
9553eb6b4a Merge pull request #269 from harshavardhana/pr_out_get_object_metadata_needs_to_filter_based_of_prefix_when_delimiter_is_specified 2015-03-07 19:14:41 -08:00
Harshavardhana
f8e2b75e23 Get object metadata needs to filter based of Prefix when delimiter is specified 2015-03-07 19:13:15 -08:00
Frederick F. Kautz IV
b39eac36ea Merge pull request #268 from fkautz/pr_out_fragment_reader_implemented 2015-03-07 18:04:29 -08:00
Frederick F. Kautz IV
679c6ace3a Fragment reader implemented 2015-03-07 18:04:20 -08:00
Harshavardhana
df69038b04 Merge pull request #267 from harshavardhana/pr_out_add_md5_support_for_fsstorage_also_re_purpose_getobjectmetadata_for_listobjects 2015-03-07 17:16:00 -08:00
Harshavardhana
4751ed0e30 Add MD5 support for fsStorage, also re-purpose GetObjectMetadata for ListObjects 2015-03-07 17:15:00 -08:00
Frederick F. Kautz IV
1ec0cd6683 Merge pull request #266 from fkautz/pr_out_adding_read_header_for_erasure 2015-03-07 15:54:17 -08:00
Frederick F. Kautz IV
ca1a4b616c Adding read header for erasure 2015-03-07 15:53:41 -08:00
Frederick F. Kautz IV
00381c887a Adding erasure package to wrap versioned erasure. Ensures we always write latest version. 2015-03-07 13:04:02 -08:00
Harshavardhana
bf5a314144 clang lacks proper gas support, implement stubs for sha256,sha512 for darwin. 
REF: http://llvm.org/bugs/show_bug.cgi?id=18918
 2015-03-07 02:55:48 -08:00
Harshavardhana
1637138f86 If incoming request IP bucketfromHostname needs to be empty string 2015-03-07 02:53:29 -08:00
Harshavardhana
a18bfa2fda Rename test-data to testdata, go will ignore it automatically 2015-03-07 01:32:35 -08:00
Harshavardhana
d992bccd9a Move pkg/storage/erasure to pkg/encoding/erasure - and other cleanups 2015-03-07 00:16:48 -08:00
Frederick F. Kautz IV
856e0100c0 Erasure layer now writes using new technique 2015-03-06 16:37:44 -08:00
Harshavardhana
856b7680df Now able to use crc32c and sha1 on Darwin, with OSX specific code 2015-03-06 02:38:29 -08:00
Harshavardhana
e5af8a3f5d Further fixes - 
- All test files have been renamed to their respective <package>_test name,
    this is done in accordance with
      - https://github.com/golang/go/wiki/CodeReviewComments#import-dot
        imports are largely used in testing, but to avoid namespace collision
        and circular dependencies

  - Never use _* in package names other than "_test" change fragment_v1 to expose
    fragment just like 'gopkg.in/check.v1'
 2015-03-06 02:04:21 -08:00
Harshavardhana
c00d1461b9 Golint cleanup from top level 2015-03-05 23:37:37 -08:00
Harshavardhana
540723d7ae Golint cleanup pkg/api 2015-03-05 21:09:19 -08:00
Harshavardhana
76e601b26b Golint cleanup pkg/storage 2015-03-05 21:09:19 -08:00
Harshavardhana
256faddab5 Golint cleanup utils/crypto/sha*,cpu,md5 2015-03-05 21:09:19 -08:00
Harshavardhana
66e31445ff Golint cleanup pkg/utils/crypto/keys 2015-03-05 21:09:19 -08:00
Harshavardhana
b33e2d2f9b Golint cleanup pkg/utils/config 2015-03-05 21:09:19 -08:00
Harshavardhana
861f13d5bd Golint cleanup pkg/utils/cpu 2015-03-05 21:09:19 -08:00
Harshavardhana
c36450a83a Golint cleanup pkg/utils/policy 2015-03-05 21:09:19 -08:00
Harshavardhana
3e321b6631 Golint cleanup pkg/utils/split 2015-03-05 21:09:19 -08:00
Frederick F. Kautz IV
1186c47603 Fragment now implements new version 2015-03-05 20:56:02 -08:00
Frederick F. Kautz IV
665af44c70 Renaming donut packages to match layout 2015-03-05 19:44:01 -08:00
Frederick F. Kautz IV
6ec7894312 Removing innerxml 2015-03-04 14:24:56 -08:00
Frederick F. Kautz IV
6f7840677a Fixing xml tags 2015-03-04 11:02:37 -08:00
Harshavardhana
f2e238199b Handle individual objects too 2015-03-04 01:54:35 -08:00
Harshavardhana
e400e85639 Add proper path prefix to be delimited 2015-03-04 00:54:39 -08:00
Harshavardhana
df135d7307 Merge pull request #249 from harshavardhana/pr_out_pathprefix_is_filtered_out_properly_and_delimited_as_well 2015-03-03 23:08:30 -08:00
Harshavardhana
cc46475c86 Pathprefix is filtered out properly and delimited as well 2015-03-03 19:45:54 -08:00
Frederick F. Kautz IV
34c1e58332 Renaming Package to goPackage to fix golint error 2015-03-03 12:56:19 -08:00
Harshavardhana
0ebaac3594 Janitor duty v2 2015-03-03 02:43:24 -08:00
Harshavardhana
3a3c8645fc Update comments across the codebase 2015-03-03 02:39:38 -08:00
Harshavardhana
137584d658 Add comments 2015-03-03 01:25:45 -08:00
Harshavardhana
c3ad0906e0 Add deadcode code which recursivley goes into all directories and verifies dangling variables. 2015-03-02 18:44:20 -08:00
Frederick F. Kautz IV
32f8c17174 Renaming AppendUstr to AppendUniqStr 2015-03-01 17:06:00 -08:00
Anand Babu (AB) Periasamy
81ede6c3fe Merge pull request #241 from abperiasamy/janitor 
On janitor duty.
 2015-03-01 16:43:04 -08:00
Frederick F. Kautz IV
de6131cc7c Renaming AppendUInt to AppendUniqInt 2015-03-01 16:39:45 -08:00
Anand Babu (AB) Periasamy
d9c9634138 removed custom Assert 2015-03-01 15:28:42 -08:00
Frederick F. Kautz IV
6f300c9ef0 Adding test at byte level 2015-03-01 15:13:09 -08:00
Anand Babu (AB) Periasamy
177121868b removed HomeDir 2015-03-01 15:13:08 -08:00
Frederick F. Kautz IV
0b08312ddd Specifying more exact types for data 2015-03-01 14:43:11 -08:00
Harshavardhana
9f43659135 Add tests, add NewHeader() for populating Header and ValidateHeader() 2015-03-01 13:19:06 -08:00
Anand Babu (AB) Periasamy
6515678e87 Merge pull request #236 from fkautz/pr_out_adding_data_writer 
Adding data writer
 2015-03-01 11:36:35 -08:00
Frederick F. Kautz IV
7414dc9b75 Adding data writer 2015-03-01 11:32:30 -08:00
Harshavardhana
eb9a4a83ac Move AppendUstr(), AppendUint() into helpers 2015-03-01 11:19:03 -08:00
Frederick F. Kautz IV
be12a6b3d9 Changing K and M in data.go to match new datatype in erasure package 2015-03-01 01:28:33 -08:00
Harshavardhana
33601ff567 Merge pull request #233 from harshavardhana/pr_out_make_k_m_to_be_uint8_and_technique_becomes_its_own_type 2015-03-01 01:26:42 -08:00
Harshavardhana
b77c62e25d Make K,M to be uint8 and Technique becomes its own type 2015-03-01 01:25:28 -08:00
Frederick F. Kautz IV
bb0441917c Making changes to data based on discussion 2015-03-01 01:16:22 -08:00
Harshavardhana
2f04a1299e Avoid allocating 1gig memory, avoid such mistakes 2015-03-01 00:43:12 -08:00
Frederick F. Kautz IV
2916b1278e Adding data.go, does nothing for now. 2015-03-01 00:27:52 -08:00
Frederick F. Kautz IV
b4ef522a9b Donut frame is now its own package 2015-03-01 00:02:14 -08:00
Frederick F. Kautz IV
6c2428833a Renaming Write to WriteFrame 2015-02-28 23:52:56 -08:00
Harshavardhana
6b397844b9 Add donut benchmark for 64MB, 128MB, 256MB, 512MB 
Current average values

~~~
> minio/pkg/storage/donut/v1 *> go test -bench .
OK: 2 passed
PASS
BenchmarkDonut64M              5         266249634 ns/op         252.05 MB/s
BenchmarkDonut128M             3         436507394 ns/op         307.48 MB/s
BenchmarkDonut256M             2         836116359 ns/op         321.05 MB/s
BenchmarkDonut512M             1        1676738951 ns/op         320.19 MB/s
ok      github.com/minio-io/minio/pkg/storage/donut/v1  8.430s
> minio/pkg/storage/donut/v1 *>
~~~
 2015-02-28 17:05:46 -08:00
Harshavardhana
65f22083ca donut_gen works with new Donut Frame 2015-02-28 16:30:18 -08:00
Frederick F. Kautz IV
4d98c355c9 Merge pull request #224 from abperiasamy/units 2015-02-28 16:12:10 -08:00
Frederick F. Kautz IV
ddc7cf835e Donut now tests amount written should match expected amount to write 2015-02-28 16:09:52 -08:00
Frederick F. Kautz IV
d3d1205eb1 Updating comment describing data type 2015-02-28 15:59:52 -08:00
Frederick F. Kautz IV
58a04ee831 New donut frame implemented 2015-02-28 15:54:13 -08:00
Anand Babu (AB) Periasamy
317a1141c0 removed unitconv 2015-02-28 15:53:44 -08:00
Frederick F. Kautz IV
90cd8b7b47 Adding streaming sha512 2015-02-28 15:51:33 -08:00
Harshavardhana
56bd413a16 Merge pull request #219 from harshavardhana/pr_out_add_1mb_block_benchmarks 2015-02-28 15:25:24 -08:00
Harshavardhana
7feca3f0d1 Add 1MB block benchmarks 2015-02-28 15:00:24 -08:00
Harshavardhana
0c2d58bc6d Implement proper delimiter and prefix handling 
With this change Minio server now responds with, delimited
'object names' in conjunction with prefix filtering

~~~
<ListBucketResult>
  <Name>example-bucket</Name>
  <Prefix></Prefix>
  <Marker></Marker>
  <MaxKeys>1000</MaxKeys>
  <Delimiter>/</Delimiter>
  <IsTruncated>false</IsTruncated>
  <Contents>
    <Key>sample.html</Key>
    <LastModified>2011-02-26T01:56:20.000Z</LastModified>
    <ETag>example-bucket#sample.html</ETag>
    <Size>142863</Size>
    <Owner>
      <ID>minio</ID>
      <DisplayName>minio</DisplayName>
    </Owner>
    <StorageClass>STANDARD</StorageClass>
  </Contents>
  <CommonPrefixes>
    <Prefix>photos/</Prefix>
  </CommonPrefixes>
</ListBucketResult>
~~~

~~~
<ListBucketResult>
  <Name>example-bucket</Name>
  <Prefix>photos/2006/</Prefix>
  <Marker></Marker>
  <MaxKeys>1000</MaxKeys>
  <Delimiter>/</Delimiter>
  <IsTruncated>false</IsTruncated>

  <CommonPrefixes>
    <Prefix>photos/2006/feb/</Prefix>
  </CommonPrefixes>
  <CommonPrefixes>
    <Prefix>photos/2006/jan/</Prefix>
  </CommonPrefixes>
</ListBucketResult>
~~~
 2015-02-28 14:47:19 -08:00
Harshavardhana
d5e5a3b3e6 Use validString 2015-02-27 19:49:18 -08:00
Harshavardhana
6ebb48b4ea Add object name validation 2015-02-27 19:42:04 -08:00
Harshavardhana
9acb1f728c Sort list of objects 2015-02-27 17:52:55 -08:00
Harshavardhana
53669a0854 Implement delimiter, path prefix 2015-02-27 16:23:05 -08:00
Harshavardhana
3f8b3462e6 Remove donut_gen binary and update crc32c function 2015-02-27 14:41:53 -08:00
Harshavardhana
9aea7fd97b Merge pull request #213 from harshavardhana/pr_out_crc32c_re_implementation_with_convenience_functions 2015-02-27 14:35:11 -08:00
Harshavardhana
8ce9b84b69 Crc32c re-implementation with Convenience functions 
- Sum32([]byte) --> uint32
   - Sum(io.Reader) --> uint32, error
 2015-02-27 13:39:17 -08:00
Frederick F. Kautz IV
2e37791681 Merge pull request #212 from fkautz/pr_out_adding_comparison_for_footer_s_inim 2015-02-27 12:40:10 -08:00
Frederick F. Kautz IV
1f7a3c13a5 Adding comparison for footer's INIM 2015-02-27 12:39:52 -08:00
Frederick F. Kautz IV
2ca62f13a5 Moving donut_gen to its own dir and renaming it as donut_gen_v1 2015-02-27 12:39:14 -08:00
Frederick F. Kautz IV
be768d01a2 Seek to beginning for tempbuffer 2015-02-27 12:18:27 -08:00
Frederick F. Kautz IV
5c52da6ebe Adding crc32c to write 2015-02-26 23:04:07 -08:00
Harshavardhana
a9aaff0ee0 Remove HTTP server timeouts write and read. 
Implement it later with a proper timer thread. Large file
transfers from 'mc' fail with i/o timeout without this change.
 2015-02-24 13:55:24 -08:00
Frederick F. Kautz IV
ef442289da Making donut use io.ReadWriteSeek instead of io.Writer 2015-02-23 20:27:25 -08:00
Harshavardhana
5e1e5ad786 More updates on documentation 2015-02-23 17:44:55 -08:00
Harshavardhana
b7868d23ac Update minioapi documentation 2015-02-23 16:46:48 -08:00
Harshavardhana
f4b5519451 Update erasure documentation 2015-02-23 13:39:16 -08:00
Anand Babu (AB) Periasamy
de41e465c8 move httpserver under server 2015-02-23 11:03:01 -08:00
Harshavardhana
51e80eaa6d Add domain and subdomain support for MinioAPI 
This change brings in domain and subdomain support

   - ./minio --domain "yourminiodomain.com"

This change brings in a much needed feature by keeping
bucketnames as part of your 'DNS' name.

All your existing applications can be migrated off from s3 to
Minio without little to no modifications.

NOTE: Setting up DNS for your `buckets` is out of scope of this feature
 2015-02-23 02:25:01 -08:00
Harshavardhana
a92c6cb2b2 Purge tiedot wrapper 2015-02-22 20:02:09 -08:00
Harshavardhana
49a714d1a4 Add license header for Minio modifications, improvements for sha1,sha256,sha512 implementations 
Also bring in SSE3 optimized public domain implementation of SHA1 from Intel ``FIPS PUB 180-1``
 2015-02-21 15:50:47 -08:00
Harshavardhana
12cff1be58 Implement sha1 as intel optimized set 2015-02-21 11:25:09 -08:00
Anis Elleuch
1d4d7ff274 Add missing asm instruction to correct sha256 calculation in avx mode 2015-02-21 19:34:36 +01:00
Anis Elleuch
e89271b8f5 Use Intel optimized code (ssse3, avx, avx2) to calculate sha256 2015-02-21 13:11:15 +01:00
Harshavardhana
3151f9b6c3 Add proper entries for GobHeader, Data copy. 
This commit also adds 'donut_gen.go' for generating sample donut file
 2015-02-21 01:44:27 -08:00
Frederick F. Kautz IV
8129d8fd0b Writing data out with error handling 2015-02-20 23:24:01 -08:00
Frederick F. Kautz IV
98ddbd840e Moving donut code to v1 in donut package 2015-02-20 23:07:43 -08:00
Frederick F. Kautz IV
9833647fb7 Fixing magic number for INIM 2015-02-20 22:42:58 -08:00
Anand Babu (AB) Periasamy
ec36a598c9 Magic as numbers are faster to encode and compare 2015-02-20 18:20:21 -08:00
Harshavardhana
741e14f749 Remove go cover and remove unncessary Makefiles 2015-02-20 17:44:26 -08:00
Harshavardhana
19a4998fcb SHA512 Implemention with Intel assembly code 2015-02-20 16:32:19 -08:00
Harshavardhana
c4f0aad9b5 Add donut tests, just for verification 2015-02-18 19:46:14 -08:00
Frederick F. Kautz IV
f7f34d08e8 Donut structure refined 2015-02-18 17:18:33 -08:00
Frederick F. Kautz IV
ab80c0ce12 Merge pull request #180 from abperiasamy/donut 2015-02-18 15:57:16 -08:00
Anand Babu (AB) Periasamy
9e304b53de donut spec v1 prototype 2015-02-18 15:52:12 -08:00
Harshavardhana
fab954f13f Run govet and fix 2015-02-18 15:33:55 -08:00
Harshavardhana
f7a06a5d1e Some more cleanup 2015-02-18 12:15:33 -08:00
Frederick F. Kautz IV
05ac09bf69 Adding header version 2015-02-18 10:21:02 -08:00
Harshavardhana
a3e5121f7b Make channels more aware of errors and shutdown on error - some cleanup 2015-02-18 00:24:19 -08:00
Harshavardhana
cf6d03b907 Expose policy code, for api router usage 2015-02-17 19:06:01 -08:00
Harshavardhana
5235badec9 Merge pull request #169 from harshavardhana/pr_out_add_policy_validation_code_more_rigorous 2015-02-17 18:10:00 -08:00
Frederick F. Kautz IV
fe9a5c14a0 Adding scaffolding for writing donut 2015-02-17 17:47:10 -08:00
Harshavardhana
980d229272 Add policy validation code more rigorous 2015-02-17 13:23:16 -08:00
Harshavardhana
f04beab7c2 Skip policy files 2015-02-16 18:37:09 -08:00
Harshavardhana
eeae64935e Implement bucket policy handler and with galore of cleanup 2015-02-16 18:20:40 -08:00
Harshavardhana
ba3e2d41f7 Add license and fix development scripts 2015-02-15 01:59:48 -08:00
Harshavardhana
966cdd4f05 Reply in xml from validate and ignore handlers 2015-02-11 22:00:45 -08:00
Harshavardhana
ac4f07906c Implement S3 Style ErrorCodes and Response 
This patchset also brings in lot of cleanup in terms of minioapi codebase
 2015-02-11 21:27:28 -08:00
Harshavardhana
680848bdcb Add license header 2015-02-08 03:00:32 -08:00
Harshavardhana
8087ca2450 Verify both 'x-amz-date' and 'date' header before discarding request 2015-02-08 02:37:19 -08:00
Harshavardhana
92f83982db Changes to tests pending from #142 2015-02-06 20:47:21 -08:00
Harshavardhana
ecb70a4c19 Avoid trailing '/' for buckets and also re-directs - in accordance with aws clients 2015-02-06 20:42:09 -08:00
Harshavardhana
81fc11ee5d Implement authorization support 2015-02-06 02:08:52 -08:00
Harshavardhana
0e8b16a55d Ignore EOF in ReadConfig() 2015-02-05 16:10:49 -08:00
Harshavardhana
72d0999dcb Reply back AccessKey, Secretkey through json 2015-02-05 15:59:44 -08:00
Harshavardhana
95f214c99b Append to Users list 2015-02-05 15:40:51 -08:00
Harshavardhana
b3b73ff4cb Merge pull request #136 from harshavardhana/pr_out_store_credentials_through_webui 2015-02-05 14:20:06 -08:00
Harshavardhana
4c90017b02 Store credentials through webui 2015-02-05 14:19:48 -08:00
Frederick F. Kautz IV
620435cea6 Content type is now passed from minioapi to storage 2015-02-05 14:05:43 -08:00
Frederick F. Kautz IV
288bc3fcc7 Adding content type to backend storage drivers 2015-02-04 17:32:40 -08:00
Frederick F. Kautz IV
816310825e Removing XMLName from json request 2015-02-04 14:21:47 -08:00
Frederick F. Kautz IV
7152712f34 Accessing dir as object should return ObjectNotFound 2015-02-03 16:45:00 -08:00
Harshavardhana
582ce3df36 Make sure config testing is done in tempdir 2015-02-03 00:25:33 -08:00
Harshavardhana
b1db70c807 Add json config reader/writer 2015-02-03 00:12:46 -08:00
Harshavardhana
aff4c5826b Use helper HomeDir() 2015-02-02 21:44:17 -08:00
Harshavardhana
dc47ca1431 Fix build issue 2015-02-01 17:21:17 -08:00
Frederick F. Kautz IV
a8eef13835 Minioapi TestPutObject properly creates bucket at start of test 2015-02-01 14:46:45 -08:00
Frederick F. Kautz IV
a092b44cf1 Adding test to ensure list objects returns an error for non existant buckets to match previous bug fix and fix for inmemory 2015-02-01 10:48:31 -08:00
Harshavardhana
5d197dd063 Reply back errors properly 2015-02-01 03:10:28 -08:00
Harshavardhana
b635501fb7 Add StrictSlash for path re-direction 2015-02-01 02:49:09 -08:00
Harshavardhana
e1b3836c60 Service a new WebUI server at port 8081. 
It is provided to avoid the router clutter on the Object API
end. This is just a first cut sample we are going to build on
top of this the new management console UI.

General interest comes from subsequently writing a small UI to
generate `AccessKey` and `SecretKey` to build authorization
layer.
 2015-02-01 02:44:53 -08:00
Frederick F. Kautz IV
54026dbc78 Setting order when retrieving buckets 2015-02-01 00:16:25 -08:00
Harshavardhana
26aac9a439 Add bucket resources to ignore handler 2015-01-31 22:45:52 -08:00
Frederick F. Kautz IV
94e124470c Converting unimplemented methods list into an http handler 2015-01-31 19:50:32 -08:00
Harshavardhana
46751f0b28 Add database query function, and some re-factoring 2015-01-31 15:51:18 -08:00
Frederick F. Kautz IV
0cd54031a7 Adding list api test 2015-01-30 17:36:53 -08:00
Harshavardhana
8a4128d304 Force clients to have certs 2015-01-30 16:51:33 -08:00
Frederick F. Kautz IV
84c5df8441 Using generic make test ./... and make build ./..., disabled signers code for now to fix build. 2015-01-30 10:58:47 -08:00
Frederick F. Kautz IV
7ca87e06aa Adding list bucket tests and fixing inmemory behavior. 2015-01-29 11:23:14 -08:00
Harshavardhana
82baccb8bf Prefix if empty skip it before IsValidBucket 2015-01-29 01:07:37 -08:00
Frederick F. Kautz IV
c6d537e66e Reducing code duplication relating to content type 2015-01-28 20:47:23 -08:00
Frederick F. Kautz IV
96a5fc0ee7 Updating godep 2015-01-28 20:26:52 -08:00
Harshavardhana
57d764778d Merge pull request #88 from harshavardhana/pr_out_add_x509_tests_and_should_be_part_of_the_build 2015-01-28 17:13:10 -08:00
Harshavardhana
ac99968796 Add x509 tests and should be part of the build 2015-01-28 17:12:59 -08:00
Frederick F. Kautz IV
b170f3f5fc Reordering minioapi functions 2015-01-28 16:51:44 -08:00
Frederick F. Kautz IV
452d46bfa6 Merge pull request #86 from fkautz/pr_out_adding_comments_to_explain_design_decisions_in_server_go 2015-01-28 16:33:28 -08:00
Frederick F. Kautz IV
66cc370e8b Adding comments to explain design decisions in server.go 2015-01-28 16:33:20 -08:00
Frederick F. Kautz IV
698f0b815f Merge pull request #85 from fkautz/pr_out_http_server_package_now_follows_convention 2015-01-28 16:30:02 -08:00
Frederick F. Kautz IV
f15e2c4e74 HTTP Server package now follows convention 2015-01-28 16:29:47 -08:00
Harshavardhana
1fdd62165e Merge pull request #83 from harshavardhana/pr_out_authorization_validation_in_accordance_with_s3_signer_awsv2 2015-01-28 16:14:39 -08:00
Harshavardhana
d78cd581c5 Authorization validation in accordance with S3 signer AWSv2 2015-01-28 16:14:08 -08:00
Frederick F. Kautz IV
e8399a6d05 Adding server config structure 2015-01-28 16:07:53 -08:00
Frederick F. Kautz IV
f3e31fe795 Fixing build break 2015-01-28 15:27:59 -08:00
Frederick F. Kautz IV
910aa176b4 Merge pull request #80 from fkautz/pr_out_fix_build 2015-01-28 14:00:41 -08:00
Frederick F. Kautz IV
ca7b1a7fc6 Fix build 2015-01-28 14:00:08 -08:00
Harshavardhana
fbafc98edb Add a new pkg/crypto/keys, first cut 
- provides three functions
     - GetRandomAlphaNumeric()
     - GetRandomAlphaNumericFull()
     - GetRandomBase64()
     - ValidAccessKey()
 2015-01-28 12:02:13 -08:00
Harshavardhana
2ccecd3f32 Add list buckets for 'fs' 2015-01-27 18:43:55 -08:00
Harshavardhana
3781734903 Merge pull request #77 from harshavardhana/pr_out_add_tiedot_integration_package_first_cut_more_improvements_needed 2015-01-27 18:15:50 -08:00
Harshavardhana
c63a6dba2f Add 'tiedot' integration package, first cut - more improvements needed 2015-01-27 18:15:23 -08:00
Frederick F. Kautz IV
5daa408e2c Creating subdirectories in fs now works 2015-01-27 17:47:11 -08:00
Frederick F. Kautz IV
044dc9af80 Adding more tests, including ordering of objects in prefix 2015-01-27 13:11:22 -08:00
Harshavardhana
aa4bc2549e Merge pull request #72 from harshavardhana/pr_out_add_erasure_package_in_its_full_form_v1_0 2015-01-27 12:57:51 -08:00
Harshavardhana
2547163cb2 Add erasure package in its full form v1.0 2015-01-27 12:55:17 -08:00
Frederick F. Kautz IV
74ef65daac Removing log from fs_test.go 2015-01-27 12:02:08 -08:00
Frederick F. Kautz IV
163a6c35db Adding fs storage 2015-01-27 11:09:55 -08:00
Frederick F. Kautz IV
49641e3410 Adding storage modifications for upcoming fs system 2015-01-26 15:45:10 -08:00
Harshavardhana
f3a27169fc Avoid 'clang' optimization, keep it -O0 2015-01-26 12:22:08 -08:00
Harshavardhana
b6627b4d6a Consolidate reply headers, simplify contentType 2015-01-25 18:23:39 -08:00
Harshavardhana
dfc3367e25 Add Server, Connection into HTTP header for all API replies 2015-01-25 17:40:22 -08:00
Harshavardhana
9e6eedfbbc Merge pull request #63 from harshavardhana/pr_out_implement_tls_server 2015-01-25 17:23:21 -08:00
Harshavardhana
063832baaf Implement TLS server 
$ ./minio --tls --cert <your_cert> --key <your_private_key>

This patchset also provides crypto/x509 - which is a wrapper package
to generate X509 certificates.

This is necessary to provide certificates later through management console
 2015-01-25 17:20:00 -08:00
Frederick F. Kautz IV
2368e7c936 Adding error support to all commands in storage interface 2015-01-25 15:35:08 -08:00
Frederick F. Kautz IV
d37bbfec41 Adding IsTruncated to minioapi 2015-01-25 13:32:39 -08:00
Frederick F. Kautz IV
39208601dc Adding more api suite tests 2015-01-25 13:06:36 -08:00
Frederick F. Kautz IV
0da193e6d6 Adding paging count tests 2015-01-25 12:57:58 -08:00
Frederick F. Kautz IV
82a69d3ab9 Adding initial test suites 2015-01-25 11:22:38 -08:00
Harshavardhana
15e832de64 For listobjects ignore resources should be for buckets 2015-01-24 18:46:23 -08:00
Harshavardhana
c751811dcb Return http.StatusNotImplemented for range of Bucket and Object resources 
Not Implemented bucket resources
================================
    "acl":            true
    "lifecycle":      true
    "policy":         true
    "location":       true
    "logging":        true
    "notification":   true
    "versions":       true
    "requestPayment": true
    "versioning":     true
    "website":        true
    "uploads":        true

Not Implemented object resources
================================
    "uploadId":       true
    "acl":            true
    "torrent":        true
    "uploads":        true
 2015-01-24 16:37:25 -08:00
Harshavardhana
704d6d8b0a Putbucket before PutObject in TestPutObject 2015-01-24 15:42:24 -08:00
Harshavardhana
77210fead3 Merge pull request #54 from harshavardhana/pr_out_add_bucket_to_be_part_of_object_struct_reply_back_with_bucketnotfound_for_erroneous_object_put_requests 2015-01-24 15:38:56 -08:00
Harshavardhana
357a81e879 Add bucket to be part of object struct, reply back with BucketNotFound 
for erroneous Object Put requests.

Other minor cleanups, to follow

  - https://github.com/golang/go/wiki/CodeReviewComments
 2015-01-24 15:35:05 -08:00
Frederick F. Kautz IV
af1ec94925 Adding more tests 2015-01-24 13:13:03 -08:00
Harshavardhana
01d15ca3b2 Implement crc32c for Darwin, update documentation 2015-01-23 19:43:51 -08:00
Frederick F. Kautz IV
dd0c833718 Adding tests for GET object in minioapi. 2015-01-23 19:33:56 -08:00
Frederick F. Kautz IV
71833b4513 Cleaning up more code. 2015-01-23 17:29:27 -08:00
Frederick F. Kautz IV
9224095420 Removing date format method 2015-01-23 17:19:38 -08:00
Harshavardhana
c516c74c51 Fix content-type in GetObjects 2015-01-23 13:16:31 -08:00
Frederick F. Kautz IV
574742ec14 Cleaning up minioapi code 2015-01-23 12:26:37 -08:00
Harshavardhana
d9d80f7c22 Get Bucket List objects compliance MaxKeys to 1000, Rename 
Content struct to Item for cosmetic reasons for xml decoding
on client side
 2015-01-23 12:15:01 -08:00
Harshavardhana
132e661136 List buckets response should be nested xml buckets 
Additionally check for 'Header' content-type before
referencing it, as it may be empty value
 2015-01-23 01:57:26 -08:00
Harshavardhana
d482c39341 Merge pull request #42 from harshavardhana/pr_out_make_gcc_mandatory_on_osx_since_clang_lacks_certain_gas_features 2015-01-22 18:33:49 -08:00
Harshavardhana
3d43305b1f Make gcc mandatory on OSX, since clang lacks certain GAS features 2015-01-22 18:33:35 -08:00
Frederick F. Kautz IV
8636e189e6 Cosmetic change, sorting routes 2015-01-22 16:19:55 -08:00
Frederick F. Kautz IV
8d1c447aac Adding HEAD verb to minioapi 2015-01-22 14:25:53 -08:00
Frederick F. Kautz IV
3a3d9989d1 Merge pull request #39 from fkautz/pr_out_making_in_memory_storage_data_type_private 2015-01-21 17:15:29 -08:00
Frederick F. Kautz IV
079cdcd909 Making in memory storage data type private 2015-01-21 17:15:05 -08:00
Frederick F. Kautz IV
8efd23b755 Migrating from unix ts to time.Time 2015-01-21 17:12:47 -08:00
Frederick F. Kautz IV
53a8536984 Fixing regex 2015-01-21 15:29:31 -08:00
Frederick F. Kautz IV
856781b2a4 Simplifying bucket name convention and making convention public 2015-01-21 15:22:15 -08:00
Frederick F. Kautz IV
ae0b88f319 Simplfying bucket naming convention 2015-01-21 15:20:35 -08:00
Harshavardhana
546e762abf Remove http unused package 2015-01-21 15:14:49 -08:00
Harshavardhana
bd1df87bd3 Merge pull request #34 from fkautz/pr_out_adding_etag_to_headers 2015-01-21 15:07:16 -08:00
Frederick F. Kautz IV
9ca1e37235 Adding etag to headers 2015-01-21 15:02:08 -08:00
Harshavardhana
5cb49175b1 Merge pull request #32 from harshavardhana/pr_out_remove_linux_specific_code_for_now_bring_them_in_later 2015-01-21 13:48:04 -08:00
Harshavardhana
9401262f75 Remove linux specific code for now, bring them in later 2015-01-21 13:47:50 -08:00
Frederick F. Kautz IV
c23fa26830 Extracting storage api to interface 2015-01-21 12:58:33 -08:00
Harshavardhana
d44404dd81 Provide ETag with sha256Sum of input object data 2015-01-21 10:55:33 -08:00
Harshavardhana
5b67da7d96 Add PutBucket and ListBuckets service 2015-01-21 00:52:59 -08:00
Harshavardhana
e22ae2475a Add license headers and other cleanup 2015-01-20 23:16:24 -08:00
Harshavardhana
9949a02e3e Merge pull request #2 from harshavardhana/pr_out_provide_package_http_as_a_http_server_wrapper 
Provide package http as a http.Server wrapper
 2015-01-20 23:15:16 -08:00
Frederick F. Kautz IV
761249934b Merge pull request #27 from fkautz/pr_out_adding_error_when_object_exists 2015-01-20 19:16:51 -08:00
Frederick F. Kautz IV
9260e6404a Adding error when object exists 2015-01-20 19:13:57 -08:00
Frederick F. Kautz IV
3026f77cfc Adding handler for listing objects to both trailing and non-trailing url 2015-01-20 19:10:52 -08:00
Frederick F. Kautz IV
cc4329fb12 List objects in a bucket. 2015-01-20 18:40:22 -08:00
Frederick F. Kautz IV
53190e1210 Initial work for xml list objects 2015-01-20 16:08:14 -08:00
Frederick F. Kautz IV
3c1580fcd9 Simplifying code 2015-01-20 14:07:39 -08:00
Frederick F. Kautz IV
2fedef04ac Moving scsi pkg away from errors.New 2015-01-20 11:34:05 -08:00
Frederick F. Kautz IV
36268f07ea Adding error handling to storage get call 2015-01-19 17:39:20 -08:00
Frederick F. Kautz IV
f3705c5d61 Moving web logic to minioapi package 2015-01-19 15:59:01 -08:00
Frederick F. Kautz IV
c791c7bca4 Store objects in memory map 2015-01-19 10:30:40 -08:00
Frederick F. Kautz IV
718794205f wip 2015-01-18 17:05:50 -08:00
Frederick F. Kautz IV
2ede0bff52 Closing channel when http server dies 
Fixing deadlock on error
 2015-01-18 17:04:09 -08:00
Frederick F. Kautz IV
1176d38e84 handler now only listens to GET requests 2015-01-18 16:23:17 -08:00
Frederick F. Kautz IV
c766f3617b Populating http handler with test scaffolding 2015-01-18 16:16:02 -08:00
Frederick F. Kautz IV
d3d1c1afb3 Minor refactoring, moving http request logic from storage to server 2015-01-18 15:50:53 -08:00
Frederick F. Kautz IV
f356599e22 Adding initial web server and storage server module infrastructure 2015-01-18 15:06:24 -08:00
Frederick F. Kautz IV
f550e84cf4 Removing old server 
Removing storage drivers
 2015-01-18 14:54:46 -08:00
Frederick F. Kautz IV
b8981ce946 Removing old server 2015-01-17 10:39:35 -08:00
Harshavardhana
b1266de212 Consolidate asm.S from crc32c, sha1, sha256 and sha512 2015-01-15 23:09:03 -08:00
Harshavardhana
3bf73642fd Remove assembly code bring back C code 2015-01-15 18:53:15 -08:00
Harshavardhana
6e84b1df97 Provide package http as a http.Server wrapper 2015-01-14 12:43:41 -08:00
Harshavardhana
68de9ac19e Further restructure 2015-01-14 12:40:43 -08:00
Harshavardhana
432275e966 Full restructure in accordance with 
- pkg/{subsystem}/{package} style
  - modify Makefile to reflect the new style,
    consolidate various entries
  - add a dummy ``main.go`` at top level
 2015-01-14 11:29:04 -08:00
Harshavardhana
f347a1e590 Merge with Intel ISAL changes from github.com/minio-io/isal 
- These changes bring in a much needed Mac OSX port for
    Intel ISAL library
  - At the current stage this MacOSX part of code is
    considered beta
  - pkg/cpu now supports OSX
  - pkg/checksum/crc32c - is still WIP, rest of the packages
    have been validated
 2015-01-11 00:39:39 -08:00
Harshavardhana
462808b87a Disable sha1,sha256,sha512 avx,avx2,sse3 crypto implementations. 
Re-implement them later, once stable
 2015-01-06 17:02:25 -08:00
Frederick F. Kautz IV
b5d84790a2 Further simplifying merging files. Eliminated a structure, switched to PipeWriter 2015-01-04 14:46:51 +13:00
Harshavardhana
0e4a26e3b4 Add Diskattrmap, Scsiattrmap for probed scsi devices 
Additional changes

- Use ``iota`` for constants
- Remove unncessary C header files
- ``new-cmd`` now depends on codegangsta cli
 2014-12-30 02:37:47 -08:00
Harshavardhana
6b36b5c551 A full restructure 2014-12-29 21:22:26 -08:00