Frederick F. Kautz IV
3c4012f1e7
Adding donut backend and setting as default
2015-03-23 20:07:22 -07:00
Harshavardhana
7c1c4b8a02
Merge pull request #351 from harshavardhana/pr_out_implement_log_package_as_drop_in_replacement_for_handling_debug_log_level
2015-03-23 18:55:14 -07:00
Harshavardhana
914962bd93
Implement log package as drop-in replacement for handling Debug log-level
2015-03-23 18:18:11 -07:00
Frederick F. Kautz IV
f91f5e59ee
Running gofmt manually, auto gofmt didn't simplify code
2015-03-23 12:26:41 -07:00
Frederick F. Kautz IV
56a298fd6a
Refactoring method variable names for erasureWriter struct
2015-03-23 12:25:20 -07:00
Frederick F. Kautz IV
d57e9d6c18
Adding sys. as reserved in erasureWriter
2015-03-23 11:56:24 -07:00
Frederick F. Kautz IV
b2f47ef446
Switching content hashing from sha512 to md5
2015-03-23 11:46:19 -07:00
Harshavardhana
b8e9834faa
Rename parms as --> params for brevity and misc cleanup
2015-03-22 20:26:01 -07:00
Frederick F. Kautz IV
b2c3172095
Adding sha512 support for pre-encoded data and verification on decode
2015-03-22 20:11:25 -07:00
Frederick F. Kautz IV
befac7d047
Exorcising donutbox
2015-03-22 19:55:15 -07:00
Frederick F. Kautz IV
eb8005cc65
Reading correct block size in erasure coded data
2015-03-22 18:49:30 -07:00
Anand Babu (AB) Periasamy
b9b0b35edf
removed private k and m and some more cleanups
2015-03-22 18:22:09 -07:00
Frederick F. Kautz IV
293e438830
Merge pull request #341 from abperiasamy/janitor
2015-03-22 17:34:25 -07:00
Anand Babu (AB) Periasamy
3a76229551
encoded size function is now public + some janitor duty
2015-03-22 17:17:53 -07:00
Harshavardhana
3929135c34
Make sure we use O_EXCL with O_CREATE to make sure we don't trip over existing file
2015-03-22 15:55:40 -07:00
Harshavardhana
2fa9320df2
De-couple donut into smaller files, useful for ease in external integration
2015-03-22 15:43:37 -07:00
Frederick F. Kautz IV
55f3659bb3
Removing log tracers
2015-03-22 15:39:50 -07:00
Frederick F. Kautz IV
03f51eac29
Adding list objects to donut
2015-03-22 15:33:52 -07:00
Harshavardhana
4b9fbd3b3a
DonutfileWriter should return value of file.Close()
2015-03-22 14:51:18 -07:00
Frederick F. Kautz IV
ced6edc9fe
Minor change, removing unnecessary commented method
2015-03-22 12:44:45 -07:00
Frederick F. Kautz IV
99cf484488
Adding GetObjectMetadata to donut
2015-03-22 12:36:51 -07:00
Frederick F. Kautz IV
703af6c3ce
Removing gob from donut driver
2015-03-22 12:17:12 -07:00
Harshavardhana
95affcb119
Fix build issues for new DonutDriver()
2015-03-22 02:53:21 -07:00
Frederick F. Kautz IV
933d44c656
New donut driver with local file encoding
2015-03-22 00:40:21 -07:00
Harshavardhana
a8056c4c12
Return error during JoinFiles back to the caller for inspection
2015-03-20 03:17:08 -07:00
Harshavardhana
2b0b5f014a
Update license header Mini should be Minimalist really
2015-03-19 14:35:50 -07:00
Harshavardhana
c3602967ab
Fix bucketpolicy to handle s3 and minio policy names separately.
...
Also fix an apparent bug in GetBucketPolicy{}
2015-03-19 12:22:14 -07:00
Harshavardhana
6eb5315249
Resource list updates
2015-03-19 00:47:34 -07:00
Harshavardhana
a1baed3bbc
Refactor to move config handling into api/
2015-03-18 22:49:02 -07:00
Harshavardhana
7ce3ab3193
Restructure minio api, move signature checks from utils to Api.
2015-03-18 18:16:19 -07:00
Frederick F. Kautz IV
2c69433588
Adding interfaces for donut driver
2015-03-18 14:27:10 -07:00
Frederick F. Kautz IV
bf86e78714
Renaming files to match
2015-03-18 12:26:44 -07:00
Frederick F. Kautz IV
1c9e550ee6
Renaming encoded driver to donutstorage
2015-03-18 12:25:37 -07:00
Frederick F. Kautz IV
d7f00194b5
Adding interfaces for implementing against
2015-03-17 22:33:59 -07:00
Frederick F. Kautz IV
982a52e0d4
Merge pull request #320 from fkautz/pr_out_removing_block_size_from_donut_writer_no_longer_necessary
2015-03-17 14:25:08 -07:00
Frederick F. Kautz IV
d6be8d3517
Removing block size from donut writer, no longer necessary
2015-03-17 14:24:59 -07:00
Harshavardhana
74b3d092f2
Now MinioAPI handles Content-MD5 set during putObject()
...
- This change also facilitates proper error response
in case of md5 mismatch or corruption
- TODO a test function needs to be implemented
2015-03-17 13:32:13 -07:00
Harshavardhana
42006c2ab0
Merge pull request #318 from harshavardhana/pr_out_remove_contenttype_pseudo_func_
2015-03-17 00:47:46 -07:00
Harshavardhana
dccf32e5c4
Remove contenttype pseudo func()
2015-03-17 00:47:22 -07:00
Frederick F. Kautz IV
90b337a3e8
Fixing order of input and layout in time.Parse
2015-03-17 00:47:17 -07:00
Frederick F. Kautz IV
25725c442a
Moving to more accurate RFC3339
2015-03-17 00:45:50 -07:00
Frederick F. Kautz IV
e36fa95f1f
Moving from iso8601 to RFC3339
2015-03-17 00:42:08 -07:00
Frederick F. Kautz IV
598eb5192e
Encoded now passes tests, enabling.
2015-03-17 00:33:49 -07:00
Harshavardhana
6d71935ea5
Encoded storage type now feature complete
2015-03-16 23:42:45 -07:00
Frederick F. Kautz IV
34cf6682fe
list objects in buckets
2015-03-16 20:43:05 -07:00
Frederick F. Kautz IV
d99435963b
Merge pull request #311 from fkautz/pr_out_switching_from_base64_to_hex_for_etag
2015-03-16 18:39:07 -07:00
Frederick F. Kautz IV
1a88b36c55
Switching from base64 to hex for etag
2015-03-16 18:38:42 -07:00
Harshavardhana
81f1d3c195
Merge pull request #310 from harshavardhana/pr_out_make_blocksize_as_constant
2015-03-16 18:20:22 -07:00
Harshavardhana
b495f15178
Make blockSize as constant
2015-03-16 18:20:01 -07:00
Frederick F. Kautz IV
211b084e50
Performing etag off of bytes rather than md5 string
2015-03-16 18:15:37 -07:00
Frederick F. Kautz IV
583c32c774
Metadata now stores hex md5, not base64. Minio API now converts md5 hex to base64.
2015-03-16 18:03:22 -07:00
Frederick F. Kautz IV
10ae089c46
Better handling of metadata in encoded
2015-03-16 17:40:59 -07:00
Harshavardhana
15f68972a5
Some necessary cleanup
...
- Rename 'fs' as 'file' for brevity
- Rename 'inmemory' as 'memory' for brevity
- Change everywhere else
2015-03-16 11:46:16 -07:00
Harshavardhana
765fde13f1
Remove x509 package, janitor continuation monad
2015-03-15 19:44:55 -07:00
Harshavardhana
bf04d125fd
Merge pull request #304 from harshavardhana/pr_out_move_from_fmt_errorf_to_errors_new
2015-03-15 19:42:23 -07:00
Harshavardhana
596eeb097e
Move from fmt.Errorf to errors.New
...
In many cases fmt.Errorf is good enough, but since
error is an interface, you can use arbitrary data
structures as error values, to allow callers to
inspect the details of the error.
2015-03-15 19:41:16 -07:00
Frederick F. Kautz IV
a589ddaddf
Decoding and returning objects now works
2015-03-15 19:02:00 -07:00
Frederick F. Kautz IV
682bf085c4
Metadata and writing the object are now both committed together.
2015-03-15 17:42:33 -07:00
Frederick F. Kautz IV
96d0a089b8
Adding create bucket to encoded
2015-03-15 12:28:24 -07:00
Harshavardhana
3f95c55067
Skip uinimplemented tests and some cleanup
2015-03-15 01:23:57 -07:00
Frederick F. Kautz IV
3e453a394a
Working in memory donut driver
2015-03-15 00:41:06 -07:00
Frederick F. Kautz IV
15d5e74837
More testing
2015-03-14 19:29:54 -07:00
Frederick F. Kautz IV
7e61696901
Inmemory donutbox is now thread safe
2015-03-14 19:21:34 -07:00
Frederick F. Kautz IV
85da9fc627
Adding start of inmem donut
2015-03-14 18:02:27 -07:00
Frederick F. Kautz IV
54472c18fe
Renaming singledisk to encoded
2015-03-14 11:37:43 -07:00
Frederick F. Kautz IV
348c4d9a88
Removing old donut format
2015-03-14 11:36:19 -07:00
Frederick F. Kautz IV
2f4b92eb8d
Minor refactor to prepare for file version of donut
2015-03-14 11:34:45 -07:00
Harshavardhana
f7b62caccc
Update erasure README with supported architecture list
2015-03-12 16:37:08 -07:00
Harshavardhana
f7fd526b28
Cosmetic changes
2015-03-12 02:11:04 -07:00
Harshavardhana
1e4a47ac5f
Merge pull request #290 from harshavardhana/pr_out_add_crc32c_license_headers_golang_and_minio
2015-03-12 02:09:06 -07:00
Harshavardhana
4c86e594d6
Add crc32c license headers Golang and Minio
2015-03-12 02:05:05 -07:00
Harshavardhana
e27ba44d94
Remove minio_calc_chunk_size() C code to write one in Golang
2015-03-12 02:03:04 -07:00
Frederick F. Kautz IV
94fe03cfc1
More refactoring to meet new donut spec
2015-03-12 01:52:14 -07:00
Harshavardhana
67888cbaa9
Restructure and use human readable names
2015-03-12 00:36:33 -07:00
Harshavardhana
e21020b337
Merge pull request #280 from harshavardhana/pr_out_http_range_support_wip
...
Get object range request added
2015-03-12 00:11:38 -07:00
Harshavardhana
a42a48a455
Get object range request added
2015-03-12 00:07:27 -07:00
Frederick F. Kautz IV
24d5018037
Rename StoreBucket to CreateBucket
2015-03-11 20:14:40 -07:00
Frederick F. Kautz IV
8c6aa45458
Adding copyright
2015-03-11 19:59:20 -07:00
Frederick F. Kautz IV
7971b66e60
Adding object donut type
2015-03-11 19:56:18 -07:00
Frederick F. Kautz IV
ec9b20a94f
Adding stubes for new single disk driver
2015-03-11 13:36:21 -07:00
Harshavardhana
a24bfb007d
Rename dateFormat with understandable name `iso8601Format
`
2015-03-10 01:51:34 -07:00
Harshavardhana
b952855779
Add strict checks with vet, golint and gofmt
2015-03-09 16:17:38 -07:00
Harshavardhana
41653774fc
Simplify fs codebase, split them into separate files and more commenting
2015-03-08 18:01:48 -07:00
Harshavardhana
01e98eb326
Make BucketPolicy strictly typed
2015-03-08 15:57:20 -07:00
Harshavardhana
8866b2cc5f
Add delimiter and prefix tests, also add inmemory to support delimiters
2015-03-08 14:51:31 -07:00
Harshavardhana
7a01db8005
Strict checks for delimiter and prefix if absent or present
2015-03-08 13:01:39 -07:00
Harshavardhana
ba85ed7566
Revert "RFC1123 is wrong protocol format for AWS"
...
This reverts commit c7b1f8ded8
.
2015-03-08 03:06:23 -07:00
Harshavardhana
c7b1f8ded8
RFC1123 is wrong protocol format for AWS
2015-03-08 01:55:25 -08:00
Harshavardhana
9e89af0e3c
Remove redundant uint32
2015-03-07 19:15:51 -08:00
Harshavardhana
9553eb6b4a
Merge pull request #269 from harshavardhana/pr_out_get_object_metadata_needs_to_filter_based_of_prefix_when_delimiter_is_specified
2015-03-07 19:14:41 -08:00
Harshavardhana
f8e2b75e23
Get object metadata needs to filter based of Prefix when delimiter is specified
2015-03-07 19:13:15 -08:00
Frederick F. Kautz IV
b39eac36ea
Merge pull request #268 from fkautz/pr_out_fragment_reader_implemented
2015-03-07 18:04:29 -08:00
Frederick F. Kautz IV
679c6ace3a
Fragment reader implemented
2015-03-07 18:04:20 -08:00
Harshavardhana
df69038b04
Merge pull request #267 from harshavardhana/pr_out_add_md5_support_for_fsstorage_also_re_purpose_getobjectmetadata_for_listobjects
2015-03-07 17:16:00 -08:00
Harshavardhana
4751ed0e30
Add MD5 support for fsStorage, also re-purpose GetObjectMetadata for ListObjects
2015-03-07 17:15:00 -08:00
Frederick F. Kautz IV
1ec0cd6683
Merge pull request #266 from fkautz/pr_out_adding_read_header_for_erasure
2015-03-07 15:54:17 -08:00
Frederick F. Kautz IV
ca1a4b616c
Adding read header for erasure
2015-03-07 15:53:41 -08:00
Frederick F. Kautz IV
00381c887a
Adding erasure package to wrap versioned erasure. Ensures we always write latest version.
2015-03-07 13:04:02 -08:00
Harshavardhana
bf5a314144
clang lacks proper gas support, implement stubs for sha256,sha512 for darwin.
...
REF: http://llvm.org/bugs/show_bug.cgi?id=18918
2015-03-07 02:55:48 -08:00
Harshavardhana
1637138f86
If incoming request IP bucketfromHostname needs to be empty string
2015-03-07 02:53:29 -08:00
Harshavardhana
a18bfa2fda
Rename test-data to testdata, go will ignore it automatically
2015-03-07 01:32:35 -08:00
Harshavardhana
d992bccd9a
Move pkg/storage/erasure to pkg/encoding/erasure - and other cleanups
2015-03-07 00:16:48 -08:00
Frederick F. Kautz IV
856e0100c0
Erasure layer now writes using new technique
2015-03-06 16:37:44 -08:00
Harshavardhana
856b7680df
Now able to use crc32c and sha1 on Darwin, with OSX specific code
2015-03-06 02:38:29 -08:00
Harshavardhana
e5af8a3f5d
Further fixes -
...
- All test files have been renamed to their respective <package>_test name,
this is done in accordance with
- https://github.com/golang/go/wiki/CodeReviewComments#import-dot
imports are largely used in testing, but to avoid namespace collision
and circular dependencies
- Never use _* in package names other than "_test" change fragment_v1 to expose
fragment just like 'gopkg.in/check.v1'
2015-03-06 02:04:21 -08:00
Harshavardhana
c00d1461b9
Golint cleanup from top level
2015-03-05 23:37:37 -08:00
Harshavardhana
540723d7ae
Golint cleanup pkg/api
2015-03-05 21:09:19 -08:00
Harshavardhana
76e601b26b
Golint cleanup pkg/storage
2015-03-05 21:09:19 -08:00
Harshavardhana
256faddab5
Golint cleanup utils/crypto/sha*,cpu,md5
2015-03-05 21:09:19 -08:00
Harshavardhana
66e31445ff
Golint cleanup pkg/utils/crypto/keys
2015-03-05 21:09:19 -08:00
Harshavardhana
b33e2d2f9b
Golint cleanup pkg/utils/config
2015-03-05 21:09:19 -08:00
Harshavardhana
861f13d5bd
Golint cleanup pkg/utils/cpu
2015-03-05 21:09:19 -08:00
Harshavardhana
c36450a83a
Golint cleanup pkg/utils/policy
2015-03-05 21:09:19 -08:00
Harshavardhana
3e321b6631
Golint cleanup pkg/utils/split
2015-03-05 21:09:19 -08:00
Frederick F. Kautz IV
1186c47603
Fragment now implements new version
2015-03-05 20:56:02 -08:00
Frederick F. Kautz IV
665af44c70
Renaming donut packages to match layout
2015-03-05 19:44:01 -08:00
Frederick F. Kautz IV
6ec7894312
Removing innerxml
2015-03-04 14:24:56 -08:00
Frederick F. Kautz IV
6f7840677a
Fixing xml tags
2015-03-04 11:02:37 -08:00
Harshavardhana
f2e238199b
Handle individual objects too
2015-03-04 01:54:35 -08:00
Harshavardhana
e400e85639
Add proper path prefix to be delimited
2015-03-04 00:54:39 -08:00
Harshavardhana
df135d7307
Merge pull request #249 from harshavardhana/pr_out_pathprefix_is_filtered_out_properly_and_delimited_as_well
2015-03-03 23:08:30 -08:00
Harshavardhana
cc46475c86
Pathprefix is filtered out properly and delimited as well
2015-03-03 19:45:54 -08:00
Frederick F. Kautz IV
34c1e58332
Renaming Package to goPackage to fix golint error
2015-03-03 12:56:19 -08:00
Harshavardhana
0ebaac3594
Janitor duty v2
2015-03-03 02:43:24 -08:00
Harshavardhana
3a3c8645fc
Update comments across the codebase
2015-03-03 02:39:38 -08:00
Harshavardhana
137584d658
Add comments
2015-03-03 01:25:45 -08:00
Harshavardhana
c3ad0906e0
Add deadcode code which recursivley goes into all directories and verifies dangling variables.
2015-03-02 18:44:20 -08:00
Frederick F. Kautz IV
32f8c17174
Renaming AppendUstr to AppendUniqStr
2015-03-01 17:06:00 -08:00
Anand Babu (AB) Periasamy
81ede6c3fe
Merge pull request #241 from abperiasamy/janitor
...
On janitor duty.
2015-03-01 16:43:04 -08:00
Frederick F. Kautz IV
de6131cc7c
Renaming AppendUInt to AppendUniqInt
2015-03-01 16:39:45 -08:00
Anand Babu (AB) Periasamy
d9c9634138
removed custom Assert
2015-03-01 15:28:42 -08:00
Frederick F. Kautz IV
6f300c9ef0
Adding test at byte level
2015-03-01 15:13:09 -08:00
Anand Babu (AB) Periasamy
177121868b
removed HomeDir
2015-03-01 15:13:08 -08:00
Frederick F. Kautz IV
0b08312ddd
Specifying more exact types for data
2015-03-01 14:43:11 -08:00
Harshavardhana
9f43659135
Add tests, add NewHeader() for populating Header and ValidateHeader()
2015-03-01 13:19:06 -08:00
Anand Babu (AB) Periasamy
6515678e87
Merge pull request #236 from fkautz/pr_out_adding_data_writer
...
Adding data writer
2015-03-01 11:36:35 -08:00
Frederick F. Kautz IV
7414dc9b75
Adding data writer
2015-03-01 11:32:30 -08:00
Harshavardhana
eb9a4a83ac
Move AppendUstr(), AppendUint() into helpers
2015-03-01 11:19:03 -08:00
Frederick F. Kautz IV
be12a6b3d9
Changing K and M in data.go to match new datatype in erasure package
2015-03-01 01:28:33 -08:00
Harshavardhana
33601ff567
Merge pull request #233 from harshavardhana/pr_out_make_k_m_to_be_uint8_and_technique_becomes_its_own_type
2015-03-01 01:26:42 -08:00
Harshavardhana
b77c62e25d
Make K,M to be uint8 and Technique becomes its own type
2015-03-01 01:25:28 -08:00
Frederick F. Kautz IV
bb0441917c
Making changes to data based on discussion
2015-03-01 01:16:22 -08:00
Harshavardhana
2f04a1299e
Avoid allocating 1gig memory, avoid such mistakes
2015-03-01 00:43:12 -08:00
Frederick F. Kautz IV
2916b1278e
Adding data.go, does nothing for now.
2015-03-01 00:27:52 -08:00
Frederick F. Kautz IV
b4ef522a9b
Donut frame is now its own package
2015-03-01 00:02:14 -08:00
Frederick F. Kautz IV
6c2428833a
Renaming Write to WriteFrame
2015-02-28 23:52:56 -08:00
Harshavardhana
6b397844b9
Add donut benchmark for 64MB, 128MB, 256MB, 512MB
...
Current average values
~~~
> minio/pkg/storage/donut/v1 *> go test -bench .
OK: 2 passed
PASS
BenchmarkDonut64M 5 266249634 ns/op 252.05 MB/s
BenchmarkDonut128M 3 436507394 ns/op 307.48 MB/s
BenchmarkDonut256M 2 836116359 ns/op 321.05 MB/s
BenchmarkDonut512M 1 1676738951 ns/op 320.19 MB/s
ok github.com/minio-io/minio/pkg/storage/donut/v1 8.430s
> minio/pkg/storage/donut/v1 *>
~~~
2015-02-28 17:05:46 -08:00
Harshavardhana
65f22083ca
donut_gen works with new Donut Frame
2015-02-28 16:30:18 -08:00
Frederick F. Kautz IV
4d98c355c9
Merge pull request #224 from abperiasamy/units
2015-02-28 16:12:10 -08:00
Frederick F. Kautz IV
ddc7cf835e
Donut now tests amount written should match expected amount to write
2015-02-28 16:09:52 -08:00
Frederick F. Kautz IV
d3d1205eb1
Updating comment describing data type
2015-02-28 15:59:52 -08:00
Frederick F. Kautz IV
58a04ee831
New donut frame implemented
2015-02-28 15:54:13 -08:00
Anand Babu (AB) Periasamy
317a1141c0
removed unitconv
2015-02-28 15:53:44 -08:00
Frederick F. Kautz IV
90cd8b7b47
Adding streaming sha512
2015-02-28 15:51:33 -08:00
Harshavardhana
56bd413a16
Merge pull request #219 from harshavardhana/pr_out_add_1mb_block_benchmarks
2015-02-28 15:25:24 -08:00
Harshavardhana
7feca3f0d1
Add 1MB block benchmarks
2015-02-28 15:00:24 -08:00
Harshavardhana
0c2d58bc6d
Implement proper delimiter and prefix handling
...
With this change Minio server now responds with, delimited
'object names' in conjunction with prefix filtering
~~~
<ListBucketResult>
<Name>example-bucket</Name>
<Prefix></Prefix>
<Marker></Marker>
<MaxKeys>1000</MaxKeys>
<Delimiter>/</Delimiter>
<IsTruncated>false</IsTruncated>
<Contents>
<Key>sample.html</Key>
<LastModified>2011-02-26T01:56:20.000Z</LastModified>
<ETag>example-bucket#sample.html</ETag>
<Size>142863</Size>
<Owner>
<ID>minio</ID>
<DisplayName>minio</DisplayName>
</Owner>
<StorageClass>STANDARD</StorageClass>
</Contents>
<CommonPrefixes>
<Prefix>photos/</Prefix>
</CommonPrefixes>
</ListBucketResult>
~~~
~~~
<ListBucketResult>
<Name>example-bucket</Name>
<Prefix>photos/2006/</Prefix>
<Marker></Marker>
<MaxKeys>1000</MaxKeys>
<Delimiter>/</Delimiter>
<IsTruncated>false</IsTruncated>
<CommonPrefixes>
<Prefix>photos/2006/feb/</Prefix>
</CommonPrefixes>
<CommonPrefixes>
<Prefix>photos/2006/jan/</Prefix>
</CommonPrefixes>
</ListBucketResult>
~~~
2015-02-28 14:47:19 -08:00
Harshavardhana
d5e5a3b3e6
Use validString
2015-02-27 19:49:18 -08:00
Harshavardhana
6ebb48b4ea
Add object name validation
2015-02-27 19:42:04 -08:00
Harshavardhana
9acb1f728c
Sort list of objects
2015-02-27 17:52:55 -08:00
Harshavardhana
53669a0854
Implement delimiter, path prefix
2015-02-27 16:23:05 -08:00
Harshavardhana
3f8b3462e6
Remove donut_gen binary and update crc32c function
2015-02-27 14:41:53 -08:00
Harshavardhana
9aea7fd97b
Merge pull request #213 from harshavardhana/pr_out_crc32c_re_implementation_with_convenience_functions
2015-02-27 14:35:11 -08:00
Harshavardhana
8ce9b84b69
Crc32c re-implementation with Convenience functions
...
- Sum32([]byte) --> uint32
- Sum(io.Reader) --> uint32, error
2015-02-27 13:39:17 -08:00
Frederick F. Kautz IV
2e37791681
Merge pull request #212 from fkautz/pr_out_adding_comparison_for_footer_s_inim
2015-02-27 12:40:10 -08:00
Frederick F. Kautz IV
1f7a3c13a5
Adding comparison for footer's INIM
2015-02-27 12:39:52 -08:00
Frederick F. Kautz IV
2ca62f13a5
Moving donut_gen to its own dir and renaming it as donut_gen_v1
2015-02-27 12:39:14 -08:00
Frederick F. Kautz IV
be768d01a2
Seek to beginning for tempbuffer
2015-02-27 12:18:27 -08:00
Frederick F. Kautz IV
5c52da6ebe
Adding crc32c to write
2015-02-26 23:04:07 -08:00
Harshavardhana
a9aaff0ee0
Remove HTTP server timeouts write and read.
...
Implement it later with a proper timer thread. Large file
transfers from 'mc' fail with i/o timeout without this change.
2015-02-24 13:55:24 -08:00
Frederick F. Kautz IV
ef442289da
Making donut use io.ReadWriteSeek instead of io.Writer
2015-02-23 20:27:25 -08:00
Harshavardhana
5e1e5ad786
More updates on documentation
2015-02-23 17:44:55 -08:00
Harshavardhana
b7868d23ac
Update minioapi documentation
2015-02-23 16:46:48 -08:00
Harshavardhana
f4b5519451
Update erasure documentation
2015-02-23 13:39:16 -08:00
Anand Babu (AB) Periasamy
de41e465c8
move httpserver under server
2015-02-23 11:03:01 -08:00
Harshavardhana
51e80eaa6d
Add domain and subdomain support for MinioAPI
...
This change brings in domain and subdomain support
- ./minio --domain "yourminiodomain.com"
This change brings in a much needed feature by keeping
bucketnames as part of your 'DNS' name.
All your existing applications can be migrated off from s3 to
Minio without little to no modifications.
NOTE: Setting up DNS for your `buckets` is out of scope of this feature
2015-02-23 02:25:01 -08:00
Harshavardhana
a92c6cb2b2
Purge tiedot wrapper
2015-02-22 20:02:09 -08:00
Harshavardhana
49a714d1a4
Add license header for Minio modifications, improvements for sha1,sha256,sha512 implementations
...
Also bring in SSE3 optimized public domain implementation of SHA1 from Intel ``FIPS PUB 180-1``
2015-02-21 15:50:47 -08:00
Harshavardhana
12cff1be58
Implement sha1 as intel optimized set
2015-02-21 11:25:09 -08:00
Anis Elleuch
1d4d7ff274
Add missing asm instruction to correct sha256 calculation in avx mode
2015-02-21 19:34:36 +01:00
Anis Elleuch
e89271b8f5
Use Intel optimized code (ssse3, avx, avx2) to calculate sha256
2015-02-21 13:11:15 +01:00
Harshavardhana
3151f9b6c3
Add proper entries for GobHeader, Data copy.
...
This commit also adds 'donut_gen.go' for generating sample donut file
2015-02-21 01:44:27 -08:00
Frederick F. Kautz IV
8129d8fd0b
Writing data out with error handling
2015-02-20 23:24:01 -08:00
Frederick F. Kautz IV
98ddbd840e
Moving donut code to v1 in donut package
2015-02-20 23:07:43 -08:00
Frederick F. Kautz IV
9833647fb7
Fixing magic number for INIM
2015-02-20 22:42:58 -08:00
Anand Babu (AB) Periasamy
ec36a598c9
Magic as numbers are faster to encode and compare
2015-02-20 18:20:21 -08:00
Harshavardhana
741e14f749
Remove go cover and remove unncessary Makefiles
2015-02-20 17:44:26 -08:00
Harshavardhana
19a4998fcb
SHA512 Implemention with Intel assembly code
2015-02-20 16:32:19 -08:00
Harshavardhana
c4f0aad9b5
Add donut tests, just for verification
2015-02-18 19:46:14 -08:00
Frederick F. Kautz IV
f7f34d08e8
Donut structure refined
2015-02-18 17:18:33 -08:00
Frederick F. Kautz IV
ab80c0ce12
Merge pull request #180 from abperiasamy/donut
2015-02-18 15:57:16 -08:00
Anand Babu (AB) Periasamy
9e304b53de
donut spec v1 prototype
2015-02-18 15:52:12 -08:00
Harshavardhana
fab954f13f
Run govet and fix
2015-02-18 15:33:55 -08:00
Harshavardhana
f7a06a5d1e
Some more cleanup
2015-02-18 12:15:33 -08:00
Frederick F. Kautz IV
05ac09bf69
Adding header version
2015-02-18 10:21:02 -08:00
Harshavardhana
a3e5121f7b
Make channels more aware of errors and shutdown on error - some cleanup
2015-02-18 00:24:19 -08:00
Harshavardhana
cf6d03b907
Expose policy code, for api router usage
2015-02-17 19:06:01 -08:00
Harshavardhana
5235badec9
Merge pull request #169 from harshavardhana/pr_out_add_policy_validation_code_more_rigorous
2015-02-17 18:10:00 -08:00
Frederick F. Kautz IV
fe9a5c14a0
Adding scaffolding for writing donut
2015-02-17 17:47:10 -08:00
Harshavardhana
980d229272
Add policy validation code more rigorous
2015-02-17 13:23:16 -08:00
Harshavardhana
f04beab7c2
Skip policy files
2015-02-16 18:37:09 -08:00
Harshavardhana
eeae64935e
Implement bucket policy handler and with galore of cleanup
2015-02-16 18:20:40 -08:00
Harshavardhana
ba3e2d41f7
Add license and fix development scripts
2015-02-15 01:59:48 -08:00
Harshavardhana
966cdd4f05
Reply in xml from validate and ignore handlers
2015-02-11 22:00:45 -08:00
Harshavardhana
ac4f07906c
Implement S3 Style ErrorCodes and Response
...
This patchset also brings in lot of cleanup in terms of minioapi codebase
2015-02-11 21:27:28 -08:00
Harshavardhana
680848bdcb
Add license header
2015-02-08 03:00:32 -08:00
Harshavardhana
8087ca2450
Verify both 'x-amz-date' and 'date' header before discarding request
2015-02-08 02:37:19 -08:00
Harshavardhana
92f83982db
Changes to tests pending from #142
2015-02-06 20:47:21 -08:00
Harshavardhana
ecb70a4c19
Avoid trailing '/' for buckets and also re-directs - in accordance with aws clients
2015-02-06 20:42:09 -08:00
Harshavardhana
81fc11ee5d
Implement authorization support
2015-02-06 02:08:52 -08:00
Harshavardhana
0e8b16a55d
Ignore EOF in ReadConfig()
2015-02-05 16:10:49 -08:00
Harshavardhana
72d0999dcb
Reply back AccessKey, Secretkey through json
2015-02-05 15:59:44 -08:00
Harshavardhana
95f214c99b
Append to Users list
2015-02-05 15:40:51 -08:00
Harshavardhana
b3b73ff4cb
Merge pull request #136 from harshavardhana/pr_out_store_credentials_through_webui
2015-02-05 14:20:06 -08:00
Harshavardhana
4c90017b02
Store credentials through webui
2015-02-05 14:19:48 -08:00
Frederick F. Kautz IV
620435cea6
Content type is now passed from minioapi to storage
2015-02-05 14:05:43 -08:00
Frederick F. Kautz IV
288bc3fcc7
Adding content type to backend storage drivers
2015-02-04 17:32:40 -08:00
Frederick F. Kautz IV
816310825e
Removing XMLName from json request
2015-02-04 14:21:47 -08:00
Frederick F. Kautz IV
7152712f34
Accessing dir as object should return ObjectNotFound
2015-02-03 16:45:00 -08:00
Harshavardhana
582ce3df36
Make sure config testing is done in tempdir
2015-02-03 00:25:33 -08:00
Harshavardhana
b1db70c807
Add json config reader/writer
2015-02-03 00:12:46 -08:00
Harshavardhana
aff4c5826b
Use helper HomeDir()
2015-02-02 21:44:17 -08:00
Harshavardhana
dc47ca1431
Fix build issue
2015-02-01 17:21:17 -08:00
Frederick F. Kautz IV
a8eef13835
Minioapi TestPutObject properly creates bucket at start of test
2015-02-01 14:46:45 -08:00
Frederick F. Kautz IV
a092b44cf1
Adding test to ensure list objects returns an error for non existant buckets to match previous bug fix and fix for inmemory
2015-02-01 10:48:31 -08:00
Harshavardhana
5d197dd063
Reply back errors properly
2015-02-01 03:10:28 -08:00
Harshavardhana
b635501fb7
Add StrictSlash for path re-direction
2015-02-01 02:49:09 -08:00
Harshavardhana
e1b3836c60
Service a new WebUI server at port 8081.
...
It is provided to avoid the router clutter on the Object API
end. This is just a first cut sample we are going to build on
top of this the new management console UI.
General interest comes from subsequently writing a small UI to
generate `AccessKey` and `SecretKey` to build authorization
layer.
2015-02-01 02:44:53 -08:00
Frederick F. Kautz IV
54026dbc78
Setting order when retrieving buckets
2015-02-01 00:16:25 -08:00
Harshavardhana
26aac9a439
Add bucket resources to ignore handler
2015-01-31 22:45:52 -08:00
Frederick F. Kautz IV
94e124470c
Converting unimplemented methods list into an http handler
2015-01-31 19:50:32 -08:00
Harshavardhana
46751f0b28
Add database query function, and some re-factoring
2015-01-31 15:51:18 -08:00
Frederick F. Kautz IV
0cd54031a7
Adding list api test
2015-01-30 17:36:53 -08:00
Harshavardhana
8a4128d304
Force clients to have certs
2015-01-30 16:51:33 -08:00
Frederick F. Kautz IV
84c5df8441
Using generic make test ./... and make build ./..., disabled signers code for now to fix build.
2015-01-30 10:58:47 -08:00
Frederick F. Kautz IV
7ca87e06aa
Adding list bucket tests and fixing inmemory behavior.
2015-01-29 11:23:14 -08:00
Harshavardhana
82baccb8bf
Prefix if empty skip it before IsValidBucket
2015-01-29 01:07:37 -08:00
Frederick F. Kautz IV
c6d537e66e
Reducing code duplication relating to content type
2015-01-28 20:47:23 -08:00
Frederick F. Kautz IV
96a5fc0ee7
Updating godep
2015-01-28 20:26:52 -08:00
Harshavardhana
57d764778d
Merge pull request #88 from harshavardhana/pr_out_add_x509_tests_and_should_be_part_of_the_build
2015-01-28 17:13:10 -08:00
Harshavardhana
ac99968796
Add x509 tests and should be part of the build
2015-01-28 17:12:59 -08:00
Frederick F. Kautz IV
b170f3f5fc
Reordering minioapi functions
2015-01-28 16:51:44 -08:00
Frederick F. Kautz IV
452d46bfa6
Merge pull request #86 from fkautz/pr_out_adding_comments_to_explain_design_decisions_in_server_go
2015-01-28 16:33:28 -08:00
Frederick F. Kautz IV
66cc370e8b
Adding comments to explain design decisions in server.go
2015-01-28 16:33:20 -08:00
Frederick F. Kautz IV
698f0b815f
Merge pull request #85 from fkautz/pr_out_http_server_package_now_follows_convention
2015-01-28 16:30:02 -08:00
Frederick F. Kautz IV
f15e2c4e74
HTTP Server package now follows convention
2015-01-28 16:29:47 -08:00
Harshavardhana
1fdd62165e
Merge pull request #83 from harshavardhana/pr_out_authorization_validation_in_accordance_with_s3_signer_awsv2
2015-01-28 16:14:39 -08:00
Harshavardhana
d78cd581c5
Authorization validation in accordance with S3 signer AWSv2
2015-01-28 16:14:08 -08:00
Frederick F. Kautz IV
e8399a6d05
Adding server config structure
2015-01-28 16:07:53 -08:00
Frederick F. Kautz IV
f3e31fe795
Fixing build break
2015-01-28 15:27:59 -08:00
Frederick F. Kautz IV
910aa176b4
Merge pull request #80 from fkautz/pr_out_fix_build
2015-01-28 14:00:41 -08:00
Frederick F. Kautz IV
ca7b1a7fc6
Fix build
2015-01-28 14:00:08 -08:00
Harshavardhana
fbafc98edb
Add a new pkg/crypto/keys, first cut
...
- provides three functions
- GetRandomAlphaNumeric()
- GetRandomAlphaNumericFull()
- GetRandomBase64()
- ValidAccessKey()
2015-01-28 12:02:13 -08:00
Harshavardhana
2ccecd3f32
Add list buckets for 'fs'
2015-01-27 18:43:55 -08:00
Harshavardhana
3781734903
Merge pull request #77 from harshavardhana/pr_out_add_tiedot_integration_package_first_cut_more_improvements_needed
2015-01-27 18:15:50 -08:00
Harshavardhana
c63a6dba2f
Add 'tiedot' integration package, first cut - more improvements needed
2015-01-27 18:15:23 -08:00
Frederick F. Kautz IV
5daa408e2c
Creating subdirectories in fs now works
2015-01-27 17:47:11 -08:00
Frederick F. Kautz IV
044dc9af80
Adding more tests, including ordering of objects in prefix
2015-01-27 13:11:22 -08:00
Harshavardhana
aa4bc2549e
Merge pull request #72 from harshavardhana/pr_out_add_erasure_package_in_its_full_form_v1_0
2015-01-27 12:57:51 -08:00
Harshavardhana
2547163cb2
Add erasure package in its full form v1.0
2015-01-27 12:55:17 -08:00
Frederick F. Kautz IV
74ef65daac
Removing log from fs_test.go
2015-01-27 12:02:08 -08:00
Frederick F. Kautz IV
163a6c35db
Adding fs storage
2015-01-27 11:09:55 -08:00
Frederick F. Kautz IV
49641e3410
Adding storage modifications for upcoming fs system
2015-01-26 15:45:10 -08:00
Harshavardhana
f3a27169fc
Avoid 'clang' optimization, keep it -O0
2015-01-26 12:22:08 -08:00
Harshavardhana
b6627b4d6a
Consolidate reply headers, simplify contentType
2015-01-25 18:23:39 -08:00
Harshavardhana
dfc3367e25
Add Server, Connection into HTTP header for all API replies
2015-01-25 17:40:22 -08:00
Harshavardhana
9e6eedfbbc
Merge pull request #63 from harshavardhana/pr_out_implement_tls_server
2015-01-25 17:23:21 -08:00
Harshavardhana
063832baaf
Implement TLS server
...
$ ./minio --tls --cert <your_cert> --key <your_private_key>
This patchset also provides crypto/x509 - which is a wrapper package
to generate X509 certificates.
This is necessary to provide certificates later through management console
2015-01-25 17:20:00 -08:00
Frederick F. Kautz IV
2368e7c936
Adding error support to all commands in storage interface
2015-01-25 15:35:08 -08:00
Frederick F. Kautz IV
d37bbfec41
Adding IsTruncated to minioapi
2015-01-25 13:32:39 -08:00
Frederick F. Kautz IV
39208601dc
Adding more api suite tests
2015-01-25 13:06:36 -08:00
Frederick F. Kautz IV
0da193e6d6
Adding paging count tests
2015-01-25 12:57:58 -08:00
Frederick F. Kautz IV
82a69d3ab9
Adding initial test suites
2015-01-25 11:22:38 -08:00
Harshavardhana
15e832de64
For listobjects ignore resources should be for buckets
2015-01-24 18:46:23 -08:00
Harshavardhana
c751811dcb
Return http.StatusNotImplemented for range of Bucket and Object resources
...
Not Implemented bucket resources
================================
"acl": true
"lifecycle": true
"policy": true
"location": true
"logging": true
"notification": true
"versions": true
"requestPayment": true
"versioning": true
"website": true
"uploads": true
Not Implemented object resources
================================
"uploadId": true
"acl": true
"torrent": true
"uploads": true
2015-01-24 16:37:25 -08:00
Harshavardhana
704d6d8b0a
Putbucket before PutObject in TestPutObject
2015-01-24 15:42:24 -08:00
Harshavardhana
77210fead3
Merge pull request #54 from harshavardhana/pr_out_add_bucket_to_be_part_of_object_struct_reply_back_with_bucketnotfound_for_erroneous_object_put_requests
2015-01-24 15:38:56 -08:00
Harshavardhana
357a81e879
Add bucket to be part of object struct, reply back with BucketNotFound
...
for erroneous Object Put requests.
Other minor cleanups, to follow
- https://github.com/golang/go/wiki/CodeReviewComments
2015-01-24 15:35:05 -08:00
Frederick F. Kautz IV
af1ec94925
Adding more tests
2015-01-24 13:13:03 -08:00
Harshavardhana
01d15ca3b2
Implement crc32c for Darwin, update documentation
2015-01-23 19:43:51 -08:00
Frederick F. Kautz IV
dd0c833718
Adding tests for GET object in minioapi.
2015-01-23 19:33:56 -08:00
Frederick F. Kautz IV
71833b4513
Cleaning up more code.
2015-01-23 17:29:27 -08:00
Frederick F. Kautz IV
9224095420
Removing date format method
2015-01-23 17:19:38 -08:00
Harshavardhana
c516c74c51
Fix content-type in GetObjects
2015-01-23 13:16:31 -08:00
Frederick F. Kautz IV
574742ec14
Cleaning up minioapi code
2015-01-23 12:26:37 -08:00
Harshavardhana
d9d80f7c22
Get Bucket List objects compliance MaxKeys to 1000, Rename
...
Content struct to Item for cosmetic reasons for xml decoding
on client side
2015-01-23 12:15:01 -08:00
Harshavardhana
132e661136
List buckets response should be nested xml buckets
...
Additionally check for 'Header' content-type before
referencing it, as it may be empty value
2015-01-23 01:57:26 -08:00
Harshavardhana
d482c39341
Merge pull request #42 from harshavardhana/pr_out_make_gcc_mandatory_on_osx_since_clang_lacks_certain_gas_features
2015-01-22 18:33:49 -08:00
Harshavardhana
3d43305b1f
Make gcc mandatory on OSX, since clang lacks certain GAS features
2015-01-22 18:33:35 -08:00
Frederick F. Kautz IV
8636e189e6
Cosmetic change, sorting routes
2015-01-22 16:19:55 -08:00
Frederick F. Kautz IV
8d1c447aac
Adding HEAD verb to minioapi
2015-01-22 14:25:53 -08:00
Frederick F. Kautz IV
3a3d9989d1
Merge pull request #39 from fkautz/pr_out_making_in_memory_storage_data_type_private
2015-01-21 17:15:29 -08:00
Frederick F. Kautz IV
079cdcd909
Making in memory storage data type private
2015-01-21 17:15:05 -08:00
Frederick F. Kautz IV
8efd23b755
Migrating from unix ts to time.Time
2015-01-21 17:12:47 -08:00
Frederick F. Kautz IV
53a8536984
Fixing regex
2015-01-21 15:29:31 -08:00
Frederick F. Kautz IV
856781b2a4
Simplifying bucket name convention and making convention public
2015-01-21 15:22:15 -08:00
Frederick F. Kautz IV
ae0b88f319
Simplfying bucket naming convention
2015-01-21 15:20:35 -08:00
Harshavardhana
546e762abf
Remove http unused package
2015-01-21 15:14:49 -08:00
Harshavardhana
bd1df87bd3
Merge pull request #34 from fkautz/pr_out_adding_etag_to_headers
2015-01-21 15:07:16 -08:00
Frederick F. Kautz IV
9ca1e37235
Adding etag to headers
2015-01-21 15:02:08 -08:00
Harshavardhana
5cb49175b1
Merge pull request #32 from harshavardhana/pr_out_remove_linux_specific_code_for_now_bring_them_in_later
2015-01-21 13:48:04 -08:00
Harshavardhana
9401262f75
Remove linux specific code for now, bring them in later
2015-01-21 13:47:50 -08:00
Frederick F. Kautz IV
c23fa26830
Extracting storage api to interface
2015-01-21 12:58:33 -08:00
Harshavardhana
d44404dd81
Provide ETag with sha256Sum of input object data
2015-01-21 10:55:33 -08:00
Harshavardhana
5b67da7d96
Add PutBucket and ListBuckets service
2015-01-21 00:52:59 -08:00
Harshavardhana
e22ae2475a
Add license headers and other cleanup
2015-01-20 23:16:24 -08:00
Harshavardhana
9949a02e3e
Merge pull request #2 from harshavardhana/pr_out_provide_package_http_as_a_http_server_wrapper
...
Provide package http as a http.Server wrapper
2015-01-20 23:15:16 -08:00
Frederick F. Kautz IV
761249934b
Merge pull request #27 from fkautz/pr_out_adding_error_when_object_exists
2015-01-20 19:16:51 -08:00
Frederick F. Kautz IV
9260e6404a
Adding error when object exists
2015-01-20 19:13:57 -08:00
Frederick F. Kautz IV
3026f77cfc
Adding handler for listing objects to both trailing and non-trailing url
2015-01-20 19:10:52 -08:00
Frederick F. Kautz IV
cc4329fb12
List objects in a bucket.
2015-01-20 18:40:22 -08:00
Frederick F. Kautz IV
53190e1210
Initial work for xml list objects
2015-01-20 16:08:14 -08:00
Frederick F. Kautz IV
3c1580fcd9
Simplifying code
2015-01-20 14:07:39 -08:00
Frederick F. Kautz IV
2fedef04ac
Moving scsi pkg away from errors.New
2015-01-20 11:34:05 -08:00
Frederick F. Kautz IV
36268f07ea
Adding error handling to storage get call
2015-01-19 17:39:20 -08:00
Frederick F. Kautz IV
f3705c5d61
Moving web logic to minioapi package
2015-01-19 15:59:01 -08:00
Frederick F. Kautz IV
c791c7bca4
Store objects in memory map
2015-01-19 10:30:40 -08:00
Frederick F. Kautz IV
718794205f
wip
2015-01-18 17:05:50 -08:00
Frederick F. Kautz IV
2ede0bff52
Closing channel when http server dies
...
Fixing deadlock on error
2015-01-18 17:04:09 -08:00
Frederick F. Kautz IV
1176d38e84
handler now only listens to GET requests
2015-01-18 16:23:17 -08:00
Frederick F. Kautz IV
c766f3617b
Populating http handler with test scaffolding
2015-01-18 16:16:02 -08:00
Frederick F. Kautz IV
d3d1c1afb3
Minor refactoring, moving http request logic from storage to server
2015-01-18 15:50:53 -08:00
Frederick F. Kautz IV
f356599e22
Adding initial web server and storage server module infrastructure
2015-01-18 15:06:24 -08:00
Frederick F. Kautz IV
f550e84cf4
Removing old server
...
Removing storage drivers
2015-01-18 14:54:46 -08:00
Frederick F. Kautz IV
b8981ce946
Removing old server
2015-01-17 10:39:35 -08:00
Harshavardhana
b1266de212
Consolidate asm.S from crc32c, sha1, sha256 and sha512
2015-01-15 23:09:03 -08:00
Harshavardhana
3bf73642fd
Remove assembly code bring back C code
2015-01-15 18:53:15 -08:00
Harshavardhana
6e84b1df97
Provide package http as a http.Server wrapper
2015-01-14 12:43:41 -08:00
Harshavardhana
68de9ac19e
Further restructure
2015-01-14 12:40:43 -08:00
Harshavardhana
432275e966
Full restructure in accordance with
...
- pkg/{subsystem}/{package} style
- modify Makefile to reflect the new style,
consolidate various entries
- add a dummy ``main.go`` at top level
2015-01-14 11:29:04 -08:00
Harshavardhana
f347a1e590
Merge with Intel ISAL changes from github.com/minio-io/isal
...
- These changes bring in a much needed Mac OSX port for
Intel ISAL library
- At the current stage this MacOSX part of code is
considered beta
- pkg/cpu now supports OSX
- pkg/checksum/crc32c - is still WIP, rest of the packages
have been validated
2015-01-11 00:39:39 -08:00
Harshavardhana
462808b87a
Disable sha1,sha256,sha512 avx,avx2,sse3 crypto implementations.
...
Re-implement them later, once stable
2015-01-06 17:02:25 -08:00
Frederick F. Kautz IV
b5d84790a2
Further simplifying merging files. Eliminated a structure, switched to PipeWriter
2015-01-04 14:46:51 +13:00
Harshavardhana
0e4a26e3b4
Add Diskattrmap, Scsiattrmap for probed scsi devices
...
Additional changes
- Use ``iota`` for constants
- Remove unncessary C header files
- ``new-cmd`` now depends on codegangsta cli
2014-12-30 02:37:47 -08:00
Harshavardhana
6b36b5c551
A full restructure
2014-12-29 21:22:26 -08:00