MyFavMirrors/minio
1
0
Fork 0
mirror of https://github.com/minio/minio.git synced 2024-12-26 23:25:54 -05:00
Code Issues Packages Projects Releases Wiki Activity
4,843 Commits 4 Branches 506 Tags 189 MiB
adf9a9d300
Commit Graph

1076 Commits

Author SHA1 Message Date
Krishna Srinivas
0010f0ee10 Remove empty directories while removing an oobject 2015-10-29 23:30:16 -07:00
Harshavardhana
53adfb38f4 Leverage sort Interface to provide sortUnique function 2015-10-23 15:55:41 -07:00
Harshavardhana
dbaa4d8643 Move ListObjects into its own file 2015-10-22 15:39:04 -07:00
Harshavardhana
1f66f4869b Fix all remaining windows path issues. 2015-10-22 00:05:10 -07:00
Harshavardhana
afa27b9847 On windows translate Prefix, Marker and Delimiter for paths 2015-10-21 22:00:03 -07:00
Harshavardhana
b74852116a Fix portability issues for arm on raspberry pi 2015-10-20 11:22:00 -07:00
Harshavardhana
dddb1650de Add 5% cumulative reduction in total size of the disk 
This is done due to filesystem holding additional metadata and inode space
which is unaccounted for during min-free-disk calculation.
 2015-10-19 10:33:34 -07:00
Harshavardhana
179d2d7dac Add initial cut of auto expiry of objects 2015-10-19 01:34:31 -07:00
Harshavardhana
c065be656c Implement min-free-disk as a subcommand, deprecate flag 2015-10-19 00:59:20 -07:00
Harshavardhana
5b2fa33bdb Implementing min-free-disk 2015-10-18 00:23:14 -07:00
Harshavardhana
a8a935f5fd Improve disk code to return back disk StatFS{} structure 
```
StatFS {
Total int64
Free int64
FSType string
}
```

Provides more information in a cross platform way.
 2015-10-17 20:19:26 -07:00
Harshavardhana
aee0845b2e Add disk package 
Implements

   - Stat returns total and free disk space supported across platforms
   - Type returns type of the filesystem underneath
 2015-10-17 16:48:24 -07:00
Harshavardhana
1256ca86d0 Add fs separator 2015-10-17 12:05:12 -07:00
Harshavardhana
2ec679a089 If directory already removed, return nil and move on 2015-10-17 00:13:46 -07:00
Harshavardhana
d534fc5a4f Implement delete bucket properly with proper error handlings 2015-10-17 00:01:12 -07:00
Harshavardhana
c9af01d807 Enhance listing further, this time handle cases related to common prefixes 2015-10-16 23:11:41 -07:00
Harshavardhana
0eb7f078f9 Implement Bucket ACL support 2015-10-16 19:47:30 -07:00
Harshavardhana
762b798767 Migrate this project to minio micro services code 2015-10-16 11:26:08 -07:00
Harshavardhana
3318cba132 Add windows support for minhttp library 2015-10-11 01:08:16 -07:00
Harshavardhana
7fea9cb550 Add rpc signature handler 2015-10-08 22:28:11 -07:00
Harshavardhana
11048708bb Implement GetBucketACL - fixes #893 2015-10-08 11:12:44 -07:00
Anand Babu (AB) Periasamy
bf901d3b9a Merge pull request #895 from abperiasamy/tasker 
new task model minio server
 2015-10-08 02:23:29 -07:00
Anand Babu (AB) Periasamy
b52697e6ad new task model minio server 2015-10-08 02:20:24 -07:00
Harshavardhana
a060b158c8 Add quick.CheckVersion() to verify config version quickly before unmarshalling the full struct 
This is needed during migration where we would need to verify the underlying version number
in a quick way.
 2015-10-07 17:44:33 -07:00
Harshavardhana
ab5ea997ab Make erasure Encode and Decode atomic to avoid races 2015-10-06 23:05:01 -07:00
Harshavardhana
1b42398e8b Canonicalize all the incoming input values, now PresignedPostPolicy works with minio-go 2015-10-06 10:21:28 -07:00
Harshavardhana
d5ce2f6944 Make erasure matrix type not optional choose automatically 
Remove option of providing Technique and handling errors based on that
choose a matrix type automatically based on number of data blocks.

INTEL recommends on using cauchy for consistent invertible matrices,
while vandermonde is faster we should default to cauchy for large
data blocks.
 2015-10-05 22:38:02 -07:00
Harshavardhana
4ed50a8004 Fix encoding bug in donut during encoding phase 
Stream reading needs to check for length parameter being non zero,
after Reading() a predefined set of buffer length an EOF might be returned
with length == 0.

Erasure taking this zeroed data in might wrongly encode it as part of existing
data blocks which leads to errors while decoding even when the other contents
are intact.
 2015-10-05 22:12:53 -07:00
Harshavardhana
f0a8dbecae Add --json output formatter for server 2015-10-05 00:20:49 -07:00
Harshavardhana
c4faf47e64 First time mode for controller 
- Upon first time invocation ``minio controller`` would create access keys and secret id
- Upon request passing 'keys' arg ``minio controller`` would provide the keys
- Add colorized notification
 2015-10-04 16:42:16 -07:00
Harshavardhana
cfdb29cac0 Simplify signature handling 
This change brings a new SignatureHandler where Presigned.
Requests without Payload are handled very early before even
going through the call.

This change simplifies Donut codebase to not have signature related
logic for all API's.

Simplification is still needed for Payload based signature eg. PUT/POST calls
, which are still part of the donut codebase, which will be done subsequently
after donut re-write.
 2015-10-04 13:15:33 -07:00
Harshavardhana
c8de5bad2f Implement presigned policy 2015-10-04 00:01:34 -07:00
Anis ELLEUCH
b5ea05d839 A better way to print prettified json syntax error msg 2015-10-03 12:25:44 -07:00
Harshavardhana
3b070dee16 Fix an important metadata getObject bug in donut 2015-10-01 10:18:03 -07:00
Harshavardhana
81cc017f91 Implement presigned signature v4 support 2015-10-01 10:17:47 -07:00
Harshavardhana
50750efb52 Reduce memory usage for memory multipart write by doing io.Pipe() streaming copy 2015-09-30 20:53:30 -07:00
Harshavardhana
8c7c5df770 Prober stringer should avoid frivolous newlines 2015-09-29 10:13:11 -07:00
Harshavardhana
cd52d7a11c Fix Linux/Mac OS X erasure decoding failure with new Golang version 1.5.1 
Fixes #871
 2015-09-24 21:42:59 -07:00
Harshavardhana
3785489153 Disable multipart for donut backend from being used 
Will enable it later after cleanup
 2015-09-24 18:54:30 -07:00
Harshavardhana
b938e40fb5 Move atomic package to the top and simplify its tests 2015-09-20 13:51:38 -07:00
Harshavardhana
d54488f144 Move all server and controller packages into top-level 2015-09-19 01:07:42 -07:00
Anand Babu (AB) Periasamy
89a86948b5 new version format and some cleanup 2015-09-18 23:33:28 -07:00
Anand Babu (AB) Periasamy
d1f1b7ac31 new version format and some cleanup 2015-09-18 23:27:04 -07:00
Harshavardhana
778f8cd222 Add new rpc tests for Server.Add and Server.List, improve Version.Get RPC to provide more details 2015-09-18 17:44:46 -07:00
Harshavardhana
2a15dd5eab Enhance signature handler - throw back valid error messages 2015-09-18 15:14:55 -07:00
Harshavardhana
ac93bbb41d Merge pull request #846 from harshavardhana/new-changes 
With new auth config changes, restructure the API code to use the new style
 2015-09-18 03:45:03 -07:00
Harshavardhana
6a5e5c1826 With new auth config changes, restructure the API code to use the new style 2015-09-18 03:41:05 -07:00
Harshavardhana
b4ce1e8c1d Add missing reply.Name and add possible failure tests 2015-09-18 03:15:19 -07:00
Harshavardhana
f8bb85aeb7 Enhance auth JSONRPC, now provides persistent output 
Implements

   - Auth.Generate("user")
   - Auth.Fetch("user")
   - Auth.Reset("user")

This patch also adds testing for each of these cases
 2015-09-18 03:02:39 -07:00
Harshavardhana
4bcd86408b Fix minio header in accordance with rfc2616.txt 2015-09-17 23:46:10 -07:00
Harshavardhana
7093a05ab1 Version is a package now, will be re-used across codebase. 2015-09-17 20:17:33 -07:00
Harshavardhana
1887114444 Fix all the golint complaints about newly added changes 
Do not use func(this *server), such generic names should not be used
for writing struct methods.
 2015-09-17 18:53:42 -07:00
Harshavardhana
03ef6533c8 Add more documentation for probe 2015-09-17 18:10:42 -07:00
Krishna Srinivas
c49407ced4 rename rpc/server.go -> rpc/rpc.go. rpc/server.go will accomodate ServerService 2015-09-17 15:46:51 -07:00
Harshavardhana
3f4b98ca4c Consolidate controller, move rpc package into controller - remove dangling code in pkg/server 2015-09-15 19:38:36 -07:00
Harshavardhana
8d5f6e0b96 Merge pull request #826 from krishnasrinivas/middleware-cleanup 
Remove unneeded functions in middleware init
 2015-09-15 18:11:45 -07:00
Krishna Srinivas
b1b387b157 Remove unneeded functions in middleware init 2015-09-15 18:09:09 -07:00
Harshavardhana
45d8898019 Merge branch 'parallel-read2' of https://github.com/krishnasrinivas/minio into krishnasrinivas-parallel-read2 
Make few more changes and rebased with current master
 2015-09-15 17:33:33 -07:00
Anand Babu (AB) Periasamy
6930e4d668 logrus logger 2015-09-09 20:28:09 -07:00
Harshavardhana
1e2c010174 Avoid shadowing variables and enable checks to avoid them during build 2015-09-09 15:14:55 -07:00
Harshavardhana
b649eff3fb Disable logging for now 2015-09-05 21:28:35 -07:00
Harshavardhana
d0f945f8e7 Simplify erasure package for OSX 2015-09-05 20:19:43 -07:00
Harshavardhana
afff3f8885 Revert "Enable controller to have CORS" 
This reverts commit f39ac24e99.
 2015-08-31 17:15:49 -07:00
Krishna Srinivas
1e82ee1192 Add CORS support to minio s3 server 2015-08-31 16:59:52 -07:00
Harshavardhana
f39ac24e99 Enable controller to have CORS 2015-08-31 01:47:05 -07:00
Harshavardhana
dcf0c71ca3 Merge pull request #810 from harshavardhana/restructure 
Restructure server code, controller now runs in silo
 2015-08-27 17:10:54 -07:00
Harshavardhana
025f95b1d6 Restructure server code, controller now runs in silo 2015-08-27 17:07:32 -07:00
Anand Babu (AB) Periasamy
c11aa1c892 return *probe.Error for Untrace() as well. 2015-08-24 03:35:24 -07:00
Harshavardhana
988d39a5b6 Migrate to golang1.5 release with GO15VENDOREXPERIMENT=1 enabled 2015-08-22 18:35:37 -07:00
Harshavardhana
0e416ea699 Tests were running 4 times due to multiple times the TestingT{} was being called 
Calling multiple times TestingT{} will hook up runner for Suites for that many times
which would lead to repeated running tests.

Fix it by only initializing it once for all the Suites
 2015-08-20 22:32:50 -07:00
Harshavardhana
068d1d1ba9 Add a new quick.Load() function to load directly any config file provided a quick compatible struct{} is also provided 2015-08-20 20:33:49 -07:00
Anand Babu (AB) Periasamy
b49b8cdbe8 return call stack in reverse 2015-08-19 22:40:27 -07:00
Anand Babu (AB) Periasamy
826202716f skip Env in json printinf if empty 2015-08-19 01:21:28 -07:00
Anand Babu (AB) Periasamy
76c40e075a make tracePoint public as well 2015-08-19 01:02:39 -07:00
Harshavardhana
ac928b5092 Fix typo in probe 2015-08-18 23:42:41 -07:00
Anand Babu (AB) Periasamy
cdf93e534c simplify probe APIs 2015-08-18 19:30:17 -07:00
Harshavardhana
e9c5a51bc6 Migrate pkg/quick from mc 2015-08-13 16:29:55 -07:00
Harshavardhana
61175ef091 Migrate to govendor to avoid limitations of godep 
- over the course of a project history every maintainer needs to update
  its dependency packages, the problem essentially with godep is manipulating
  GOPATH - this manipulation leads to static objects created at different locations
  which end up conflicting with the overall functionality of golang.

  This also leads to broken builds. There is no easier way out of this other than
  asking developers to do 'godep restore' all the time. Which perhaps as a practice
  doesn't sound like a clean solution. On the other hand 'godep restore' has its own
  set of problems.

- govendor is a right tool but a stop gap tool until we wait for golangs official
  1.5 version which fixes this vendoring issue once and for all.

- govendor provides consistency in terms of how import paths should be handled unlike
  manipulation GOPATH.

  This has advantages
    - no more compiled objects being referenced in GOPATH and build time GOPATH
      manging which leads to conflicts.
    - proper import paths referencing the exact package a project is dependent on.

 govendor is simple and provides the minimal necessary tooling to achieve this.

 For now this is the right solution.
 2015-08-12 19:24:57 -07:00
Harshavardhana
45b59b8456 Probe revamped to provide for a new WrappedError struct to wrap probes as error interface 
This convenience was necessary to be used for golang library functions like io.Copy and io.Pipe
where we shouldn't be writing proxies and alternatives returning *probe.Error

This change also brings more changes across code base for clear separation regarding where an error
interface should be passed encapsulating *probe.Error and where it should be used as is.
 2015-08-08 00:16:38 -07:00
Harshavardhana
f8141493bd use command not found helper 2015-08-03 18:06:19 -07:00
Harshavardhana
d09fd8b0a1 Migrate from iodine to probe 2015-08-03 16:33:44 -07:00
Harshavardhana
884e9771b2 More changes to probe to avoid nil dereferences 2015-08-03 01:47:37 -07:00
Harshavardhana
65e4aede82 Minor changes to probe 2015-08-02 20:33:49 -07:00
Anand Babu (AB) Periasamy
697009c0a1 trace on New and add read locks 2015-08-02 11:58:28 -07:00
Harshavardhana
d9493909d8 Merge pull request #788 from krishnasrinivas/open-and-openfile 
rename disk.OpenFile to Open which will do os.Open (which will be rea…
 2015-08-02 11:05:04 -07:00
Krishna Srinivas
ee4432bc40 rename disk.OpenFile to Open which will do os.Open (which will be read-only). disk.OpenFile will do os.OpenFile (which can be rw, append) 2015-08-02 17:34:29 +05:30
Anand Babu (AB) Periasamy
a728ddc027 probe package to trace & return errors 2015-08-02 02:38:08 -07:00
Harshavardhana
aabfd541e1 Merge cmd/donut into minio cmd, deprecate controller RPC request 2015-07-31 12:57:15 -07:00
Harshavardhana
2671b2dbf4 crypto/cleanup: remove unused functions 2015-07-29 13:09:55 -07:00
Krishna Srinivas
fdd2c22fa5 Do go fmt on bucket.go 2015-07-29 23:49:25 +05:30
Krishna Srinivas
ae8089c9b6 when signature is not available there is no need to compute sha256 2015-07-29 21:09:02 +05:30
Krishna Srinivas
bdc00624fd get erros from buffered channel. Return error during object read only if we have readers < k 2015-07-29 19:36:41 +05:30
Harshavardhana
d346250f1c Collapse GetPartialObject() into GetObject() 2015-07-28 19:33:56 -07:00
Krishna Srinivas
e1280779ed Read from the disks in parallel during object read 2015-07-28 18:04:17 +05:30
Krishna Srinivas
1ea91d2fa2 Use missingEncodedBlocksCount directly instead of "-1" workaround in missingEncodedBlocks[] 
Makes Code more readable
 2015-07-28 16:56:06 +05:30
Harshavardhana
b0ea64a04f Strip off quotes from ETag for verifying complete multipart upload 2015-07-25 16:10:05 -07:00
Harshavardhana
e082f26e10 Improving EncoderStream to return error only upon non io.EOF. 
io.EOF is okay since io.ReadFull will not have read any bytes at all.

Also making error channel receive only for go routine.
 2015-07-25 15:57:30 -07:00
Krishna Srinivas
bcfaa12a4d Encoder now directly reads from the object stream. Using split.Stream() was causing lot of redundant memory operations. 2015-07-26 03:54:39 +05:30
Harshavardhana
d6a0e0cc55 Rename more 2015-07-24 18:09:53 -07:00
Harshavardhana
63c9cf0c4b Move from Minimalist Object Storage to Minio Cloud Storage 2015-07-24 17:51:40 -07:00
Harshavardhana
43c908d5b9 ListObjects now considers multipart objects, also move to upstream check.v1 2015-07-18 15:49:41 -07:00
Harshavardhana
966786c78e remove unnecessary updateAt() 2015-07-17 19:48:09 -07:00
Harshavardhana
c1da10a207 Donut multipart support no get, listobjects() support yet 2015-07-17 16:23:51 -07:00
Harshavardhana
86a887f9d4 Add corresponding tests 2015-07-16 17:40:19 -07:00
Harshavardhana
5eae32f2b0 Return proper InvalidArgument messages like s3 for invalid data for ListObjects(), ListObjectParts(), ListMultipartUploads() 2015-07-16 17:22:45 -07:00
Harshavardhana
e605787e65 Method not allowed is right response for DELETE() operations and add tests 2015-07-16 14:15:24 -07:00
Harshavardhana
1f2e6a40a0 Fix an issue with reusing closed network connetion, changing the way rateLimitedListener is initialized 2015-07-16 10:56:14 -07:00
Harshavardhana
5507a39840 Heal buckets upon init if needed, adding new disks and HUP works 2015-07-16 09:59:05 -07:00
Harshavardhana
e4574c7d6f Add File() method to Ratelimitlistener for extracting underlying fd() 2015-07-16 00:11:17 -07:00
Harshavardhana
2e8f154f34 Iodine should indent with EmitJSON() 2015-07-16 00:10:55 -07:00
Harshavardhana
4498662c16 Remove scsi non portable code, instead "donut make" implements functionality to instantiate a donut 2015-07-15 11:55:57 -07:00
Harshavardhana
6baf45e360 Fix a crash during listObjects() populating NextMarker 2015-07-15 10:04:15 -07:00
Harshavardhana
2553654e24 Add initial version of heal, remove rebalance 2015-07-14 22:56:41 -07:00
Harshavardhana
e37c5315ec Live multiple disk removal works properly 2015-07-14 20:46:14 -07:00
Harshavardhana
e1e4908515 Wire up sha512 matching inside donut along with md5sum 2015-07-14 19:47:50 -07:00
Harshavardhana
e885259584 Handle removal of disks - getObject() now reads if disks are missing underneath, add initial stub healing code 2015-07-14 18:53:00 -07:00
Harshavardhana
efbf3eabb7 Return x-amz-request-id for all replies 2015-07-14 14:44:03 -07:00
Harshavardhana
da8b9fd112 Remove global custom config path variables, use get/set methods instead 2015-07-14 11:56:15 -07:00
Harshavardhana
c4cf7635bf Read req.Body for PutBucket() if any 2015-07-14 09:30:10 -07:00
Harshavardhana
45ddec925c Fix an ugly multipart bug 2015-07-13 21:38:01 -07:00
Harshavardhana
7ae60a6d10 For missing parts reply back as InvalidPart{} 2015-07-13 19:31:01 -07:00
Harshavardhana
634f70f3b0 WriteObject() encoded data in go routine, fix another multipart issue 2015-07-13 15:56:54 -07:00
Harshavardhana
1bad92356d Add a ratelimited listener than a ratelimited handler - more precise 2015-07-13 11:12:17 -07:00
Anand Babu (AB) Periasamy
8af5933b07 Merge pull request #743 from flandr/fix-osx 
Fix OS X build
 2015-07-13 10:15:53 -07:00
Nate Rosenblum
ec347f96fd Fix OS X build 
- Explicitly cast statfs_t members to int64 (this structure is
   platform-specific)
 - Add pass-through New methods to Darwin SHA package
 - Move scsi pkg types to common translation unit (package was empty)
 - Add stub implementations mount/disk ops for OS X
 2015-07-13 10:06:55 -07:00
Harshavardhana
55e4d0c6a5 mkdonut now creates a donut processing cli args 2015-07-12 21:21:31 -07:00
Harshavardhana
58a1d865a9 Renaming nimble to minhttp 2015-07-12 18:01:52 -07:00
Harshavardhana
847440196e Add signature v4 tests 2015-07-12 13:36:02 -07:00
Harshavardhana
7615a6bfe5 HEAD shouldn't have any body, handle it in writeErrorResponse() 2015-07-11 10:34:55 -07:00
Harshavardhana
8977f9a524 cached api test should also have a custom config path, would conflict with your .minio local path 2015-07-10 20:39:30 -07:00
Harshavardhana
97d4a27c7e Do not reply on ignoredHeaders for server, rely on SignedHeaders sent as part of Authorization header 2015-07-10 19:37:22 -07:00
Harshavardhana
53f5d2c32b nodejs http library sends Connection header during HTTP request, this clobbers up the signature handling ignore it 2015-07-10 15:44:47 -07:00
Harshavardhana
29838bb851 Cleanup temporary writers upon errors during putObject(), all metadata() write operations 2015-07-10 14:11:04 -07:00
Harshavardhana
15dd0df187 Support signature v4 at rest 2015-07-10 11:49:27 -07:00
Harshavardhana
7fa514351c Rename definitions to log.go, add valid prefixes 2015-07-10 11:11:20 -07:00
Harshavardhana
d5ffc16f25 Add abbreviated close response, to avoid any leaks 2015-07-10 10:20:00 -07:00
Harshavardhana
e5006c738d Handle both space and non-space characters, in signature v4 - add errors for all API's 2015-07-09 21:49:58 -07:00
Harshavardhana
84f427f14a All other API's now support signature v4 2015-07-09 19:45:56 -07:00
Harshavardhana
00890c254e CompleteMultipartUpload and CreateObjectPart now fully support signature v4 2015-07-09 19:01:15 -07:00
Harshavardhana
89c1215194 PutObject handler gets initial support for signature v4, working 2015-07-09 16:44:38 -07:00
Harshavardhana
d461fa5ab1 Add mountinfo functions for detecting mount disks, and other rpc changes 2015-07-09 12:25:29 -07:00
Harshavardhana
8654ddb566 Generate auth now saves in ${HOME}/.minio/users.json, also authHandler verifies request validity 2015-07-08 21:53:13 -07:00
Harshavardhana
375860077d HTTP header Content-Length signifies body length of the request, if its smaller reply appropriately 
This patch also handles large individual part sizes > 5MB by using less memory copies.
 2015-07-08 20:56:41 -07:00
Harshavardhana
ec33d79d57 Add server side signaturev4 check, not wired up to the readers yet. 2015-07-08 16:57:03 -07:00
Harshavardhana
396b728031 Add auth rpc service to generate access keys, add corresponding test 2015-07-08 14:40:39 -07:00
Harshavardhana
770fd23afa Renaming keys as auth, working towards signature v4 support for all put objects 2015-07-08 14:17:16 -07:00
Harshavardhana
2413a110e6 Fix SSL support, pointer indirection caused nil buffers 2015-07-08 11:02:15 -07:00
Harshavardhana
d1deda3a96 Add API tests for both donut on disk and donut cache 2015-07-07 19:39:46 -07:00
Harshavardhana
ece797c16e Add rpc tests 2015-07-07 17:27:34 -07:00
Harshavardhana
676b9058de Separate out memory statistics and system information into two different services 2015-07-07 16:59:20 -07:00
Harshavardhana
8abb96c030 If NodeDisks are not empty do not impose cache maxSize restriction 2015-07-07 16:41:40 -07:00
Harshavardhana
a50a44b0ca Add nimbleNet tests 2015-07-07 16:15:47 -07:00
Harshavardhana
317096a0c4 Add net.Addr wrapper with IsEqual() and use it. 2015-07-07 15:33:08 -07:00
Harshavardhana
11b893804c Moving os.MkdirAll() inside atomic for auto parent directory creates 2015-07-07 12:35:57 -07:00
Harshavardhana
52cd23ad9f Move atomic file writes into its own package, use them inside quick and disk packages 2015-07-07 12:29:14 -07:00
Harshavardhana
3622fbc87d Across donut, split, nimble some code cleanup 2015-07-06 21:55:21 -07:00
Harshavardhana
bbb89b5776 Add multi-thread protection and also allow atomic file creates, rename upon Close() 2015-07-06 18:54:32 -07:00
Harshavardhana
c2c7bdf0cd Cleanup nimble http 2015-07-06 18:51:20 -07:00
Harshavardhana
b029d0a5f0 Avoid config reload all the time, reload is manually triggerred from outside 2015-07-06 17:26:35 -07:00
Harshavardhana
8b94c53345 Fix issues with multipart upload 2015-07-06 16:22:27 -07:00
Harshavardhana
474954022e Add modified grace library from facebookgo, rename it as nimble 2015-07-06 15:40:12 -07:00
Harshavardhana
1d64e4b6c1 Add Donut rpc service for sending changes to configuration files 2015-07-06 11:10:06 -07:00
Harshavardhana
10b082144e Add updateConfig code to load config changes if possible for every function 2015-07-05 22:46:42 -07:00
Harshavardhana
ba0a5ed416 Add sighup, sigusr2 into trapping code, to trap signals for reloading configuration. 
Need to still figure out a way of graceful restarts - gave facebookgo/httpdown a shot,
but it is not suitable.
 2015-07-05 21:40:53 -07:00
Harshavardhana
4a27ab0e58 Fix another deadlock inside CreateObjectPart() code, premature return without Unlocking() 
Also this patch changes the cache key element to be interface{} type not string.
 2015-07-05 20:26:32 -07:00
Harshavardhana
75788c7a1d Fix go installation check on amazon instance 2015-07-05 18:12:58 -07:00
Harshavardhana
7f0c14f2b7 Add basic controller code, initiating json rpc connection getting list of disks and memstats for now. 2015-07-05 17:17:41 -07:00
Harshavardhana
a3ccb9d405 Rename stuttered service names and make them appropriate 2015-07-05 10:19:23 -07:00
Harshavardhana
e66a84242a Add disk detection for Linux, add new RPC service GetDiskInfoService(), remove dummy HelloService() 2015-07-05 02:08:33 -07:00
Harshavardhana
bab4a47525 Move to container/list datastructure from map[string][]byte 2015-07-04 17:08:23 -07:00
Harshavardhana
aa67a19e99 Implement new CPU detection using cpuid, cpuidex plan9 instructions from klauspost/cpuid project, remove C code 2015-07-04 14:28:16 -07:00
Harshavardhana
0a827305ad Implement metadata cache, metadata cache is used by top level donut right now. Rename trove as data cache 
We should use it internally everywhere.
 2015-07-03 21:09:57 -07:00
Harshavardhana
14844f48dd Make donut do everything as an atomic operation, this avoids all the deadlocks and races 2015-07-03 17:16:58 -07:00
Harshavardhana
30fc14e703 Restructure codebase move crypto, checksum to top-level, move `split` into donut, move crypto/keys into api/auth 2015-07-03 15:24:51 -07:00
Harshavardhana
8a4e7bcdcf Add full API tests, move storage/donut to donut, add disk tests as well 2015-07-03 14:36:29 -07:00
Harshavardhana
7c37e9d06a Make donut fully integrated back into API handlers 2015-07-02 21:04:04 -07:00
Harshavardhana
12bde7df30 Add simple Ticket Master which pro-actively sends messages on proceedChannel 
Handlers are going to wait on proceedChannel, this the initial step towards
providing priority for different set of API operations
 2015-07-02 21:04:04 -07:00
Harshavardhana
5cfb05465e Add cache, donut tests separately - fix behavior differences 
Remove priority queue, implement it using a simpler channels
 2015-07-02 21:04:04 -07:00
Harshavardhana
ebe61d99d9 Use cache Append() for saving objects in memory, GetObject() caches un-cached entries while reading 2015-07-02 21:04:04 -07:00
Harshavardhana
bce93c1b3a Integrate cache with donut, add tests 2015-07-02 21:04:04 -07:00
Harshavardhana
0533abf6a8 Make priority queue lambda function return error over a channel 2015-07-02 21:04:04 -07:00
Harshavardhana
38a6ce36e5 Remove slow AppendUniq code, rolling through over a slice is in-efficient 
Remove it and use map instead
 2015-07-02 21:04:04 -07:00
Harshavardhana
84810162f5 Add simple Version and GetSysInfo services 2015-07-02 21:04:04 -07:00
Harshavardhana
14ec42d646 Add initial implementation of priority queue, uses container/heap 2015-07-02 21:04:04 -07:00
Harshavardhana
eb5aa19dfa Remove custom Config, will use quick Config instead for user access keys 2015-07-02 21:04:04 -07:00
Harshavardhana
701c3e5242 Add new RPC helpers wrapping over regular rpc packages, add middleware chaining ability 2015-07-02 21:04:04 -07:00
Harshavardhana
4addf7a996 Restructure API handlers, add JSON RPC simple HelloService right now. 2015-07-02 21:04:04 -07:00
Harshavardhana
335c7827eb More donut, cache, api cleanup 2015-07-02 21:04:04 -07:00
Harshavardhana
dc0df3dc0e Breakaway from driver model, move cache into donut 2015-07-02 21:04:03 -07:00
Harshavardhana
72572d6c71 Remove some api server code bringing in new cleanup 2015-07-02 21:04:03 -07:00
Harshavardhana
c2031ca066 Add server and control command 2015-07-02 21:04:03 -07:00
Frederick F. Kautz IV
cfbc169034 Fixing API definitions 2015-07-02 13:14:21 -07:00
Harshavardhana
8f61d6b6be Move memory code out, add it as layer on top of existing donut code 
Just like how http.Handlers can be overlayed on top of each other
with each implementing ServeHTTP().

drivers.Driver can be overlayed on top of each other in similar manner
which would implement the drivers.Driver interface.

   API <----> cache <----> donut <----> donut(format)
 2015-06-30 10:09:12 -07:00
Harshavardhana
ab6e16bb41 Add dummy driver for community to submit new drivers 2015-06-29 16:43:50 -07:00
Harshavardhana
12de98fb62 Rename memory driver as cache 2015-06-29 16:43:50 -07:00
Harshavardhana
2571342451 Filesystem goes the high road *again* 2015-06-29 16:43:42 -07:00
Harshavardhana
f74d6138da IsValidBucket() is sufficient we don't need to verify for "." 2015-06-29 15:15:54 -07:00
Harshavardhana
3109909355 Handle couple of cases of OOM conditions, move caching to GetObject() rather than PutObject() 2015-06-29 12:28:50 -07:00
Harshavardhana
d07d0c670a Return back proper errors in writeObjectData(), rename few functions 2015-06-29 11:46:35 -07:00
Harshavardhana
10c807f233 Put object on successful write returns full metadata, to avoid subsequent GetObjectMetadata() calls in driver 2015-06-29 11:15:46 -07:00
Harshavardhana
6921328b93 Avoid frivolous GetObjectMetadata() calls at driver level, return back all the information in donut ListObjects() 2015-06-29 11:14:58 -07:00
Harshavardhana
d8f7896a43 Expand http server struct to store more values 2015-06-29 00:12:28 -07:00
Harshavardhana
42c0287943 Use errorChannels only for services not for drivers, reduce them to use simple functions 2015-06-28 23:59:47 -07:00
Harshavardhana
91e5f648cb Move to Set() not Append() due to large memory reference copy 2015-06-28 10:13:12 -07:00
Harshavardhana
ac4d8fe478 Add free() method for proxyReader to aggressively de-allocate Read data, to handle certain out of memory conditions 
There are still some more out there
 2015-06-27 20:43:25 -07:00
Harshavardhana
05f8654e3d Add Append() method to trove cache for appending data to an existing key 
This largely avoids a large buffer copy which would accumulate inside proxyReader{}

This patch also implements "initialize()" function to init and populate data
on all the existing buckets, avoiding the redundant ListBuckets() invoked by
every API call.
 2015-06-27 20:25:24 -07:00
Harshavardhana
367772b988 Make sure to populate on disk data into memory upon first API requests 2015-06-27 18:25:21 -07:00
Harshavardhana
45a7eab804 An attempt to bring in memory layer into donut driver 2015-06-27 17:23:34 -07:00
Harshavardhana
f3c25bcfc4 Keeping the lexical order same add optimizations, provide a comprehensive response from ListObjects() 2015-06-27 13:12:44 -07:00
Harshavardhana
ae66ae42c4 Rename functions for their purpose 2015-06-27 12:39:11 -07:00
Harshavardhana
9a4680475f Object metadata was wrongly misconstrued to be mutable, handle it 2015-06-26 23:22:53 -07:00
Harshavardhana
3aa6d90c5e Avoid racy maps, read from disk on success return quickly. Many more optimizations 2015-06-26 19:49:37 -07:00
Harshavardhana
68974918ac Donut moves to typed metadata, removing the necessity for strconv, parsing and string converstions 2015-06-26 16:23:12 -07:00
Harshavardhana
767d3743ee Add new metadata definitions and use them 2015-06-26 13:34:09 -07:00
Harshavardhana
9958e34772 Import quick key value store from Minio Client for persistent state files, primarily for donut 2015-06-25 19:57:31 -07:00
Harshavardhana
fb9adb5524 Donut cleanup another set 
- Make sure to close all readers
- Fix errors in api_testsuite c.Assert(err, IsNil) should be done right after each function call
 2015-06-25 18:54:34 -07:00
Harshavardhana
eec66f195a Take all the ListObjects into bucket handlers 
Earlier the listing would wait for all the objects to be processed
this is essentially very time consuming considering even for 100,000
files.
 2015-06-25 18:04:29 -07:00
Harshavardhana
45e9d25931 Remove more bloated code - simplify 2015-06-25 13:02:08 -07:00
Harshavardhana
82dcbf262d Add simple locking for donut API for now - fixes #671 2015-06-25 11:29:11 -07:00
Harshavardhana
03b4d3b308 Go vet fixes for donut 2015-06-24 21:07:03 -07:00
Harshavardhana
57a2b53178 Removing further bloated code simplifying 2015-06-24 21:03:39 -07:00
Harshavardhana
a2c205ff2e Use external package disk for donut. 2015-06-24 21:03:02 -07:00
Harshavardhana
841ff01412 Move disk into its own package, remove bloated code 2015-06-24 20:13:47 -07:00
Harshavardhana
1682c748ac Remove unnecessary interfaces from donut, cleanup 2015-06-24 19:43:38 -07:00
Harshavardhana
3498872467 Add sha256 and sha512 windows compatibility layer 2015-06-24 14:39:41 -07:00
Harshavardhana
77d35b87d4 Fix a bug on windows regarding blockSSE3 calculation 2015-06-24 14:24:33 -07:00
Harshavardhana
f1410731db Add windows code for sha1 and crc32c 2015-06-24 14:16:32 -07:00
Harshavardhana
ba2d3dea74 Fix wrong TMPFS listing in donut filesystem map 2015-06-23 23:16:06 -07:00
Harshavardhana
e818bc7187 Trim iodine path properly, so that now errors have github.com/minio/minio prefixed 2015-06-23 13:36:54 -07:00
Harshavardhana
2fd52ca284 Add missing strongly typed errors for Donut 2015-06-23 11:54:44 -07:00
Harshavardhana
7a060110ff Fix BUILDDEPS paths for golang installation 2015-06-22 12:07:23 -07:00
Harshavardhana
3bf64f5669 Verify d.donut to be non-nil, usually happens when multiple go versions compilations are linked - possible cause for #659 2015-06-20 11:03:17 -07:00
Harshavardhana
641f07cecf Use filepath everywhere instead of path.{} functions for portability - fixes #656 2015-06-18 16:02:45 -07:00
Harshavardhana
573a6134b2 Remove redundant !ok for map 2015-06-17 22:36:46 -07:00
Harshavardhana
e9a3fd677a Hold lock on GetGlobalStateKey() - fixes #652 2015-06-17 20:37:49 -07:00
Harshavardhana
b793f53d48 Minior change filter() to filterObjects() 2015-06-17 20:35:44 -07:00
Harshavardhana
d4e71beb54 Add time.Since on buckets created time to avoid deleting buckets spuriously during eviction of objects 2015-06-13 11:28:04 -07:00
Harshavardhana
636788997b Minor optimization avoiding one unnecessary copy 2015-06-11 14:23:05 -07:00
Harshavardhana
6d44274db6 Make sure to reply back ETag with quotes for s3 compliance 2015-06-09 01:13:02 -07:00
Harshavardhana
a947767339 Remove unnecessary tlsHelpers, SSL works with domain certificate 2015-06-08 20:54:41 -07:00
Harshavardhana
c6ecd94420 Enable TLS and use it 2015-06-08 20:10:59 -07:00
Harshavardhana
64b014369c Return back entity too large for createObject requests bigger than cache size 2015-06-08 17:10:07 -07:00
Harshavardhana
ab1976bc14 Load from persisted multipart session, now multipart uploads can resume even after restarting minio fs server 2015-06-08 15:15:37 -07:00
Harshavardhana
2976f109df More multipart fixes, filter out parts and multipart metadata from listObjects 2015-06-08 14:08:36 -07:00
Harshavardhana
e08d59be00 Make sure to delete uploadid's from active session file properly 2015-06-08 12:25:51 -07:00
Harshavardhana
0d3710cb9b Store in json file instead of gob for fs metadata 2015-06-08 12:08:01 -07:00
Harshavardhana
8efc842b59 Add delete handlers and reply back as 'NotImplemented' instead of 404 2015-06-08 11:06:06 -07:00
Harshavardhana
8a320c4447 Enhance logging handler to print in humanize form for ContentLength 2015-06-08 02:14:09 -07:00
Harshavardhana
cefd740002 Put back the older conn_limit 2015-06-06 18:11:38 -07:00
Harshavardhana
371651bde6 Add rate limiter instead of connection limit for now 2015-06-06 18:04:55 -07:00
Harshavardhana
21916dc0a1 Fix a deadlock in trove when evicting objects when we reach the maxSize used 2015-06-06 16:02:52 -07:00
Harshavardhana
8b43655d9f Make connection limit configurable 2015-06-06 14:31:50 -07:00
Harshavardhana
a344f5b34d Misc changes before capturing proper video 
- Disable management UI command option
- Remove featureflags option, filesystem and memory already support multipart
- Print informative messages after starting minio server
 2015-06-03 18:27:15 -07:00
Harshavardhana
097f70a3c5 Enable filesystem for API tests, this patch also implements ListMultipartUploads() 2015-06-01 20:57:13 -07:00
Harshavardhana
bfb9209689 Skip files and weird directories from listBuckets(), allowing this so that minio fs mode can be used as ftp service 2015-06-01 16:43:43 -07:00
Harshavardhana
d86ba5f336 Implement multipart support 
- implemented NewMultipartUpload()
 - implemented CreateObjectPart()
 - implemented ListObjectParts()
 - implemented AbortMultipartUpload()
 - implemented CompleteMultipartUpload()
 2015-06-01 16:25:11 -07:00
Harshavardhana
c07d48e7e8 ErrorResponse from minio server needed proper xml translations 2015-06-01 12:19:27 -07:00
Harshavardhana
a91e519692 Make caching a package trove, and use it inside memory driver 2015-05-31 18:41:56 -07:00
Anis Elleuch
7e5a26f42c Protect the exported method Delete() 2015-05-31 20:29:42 +01:00
Harshavardhana
82a0eac659 Add filesystem factory functions and add related cli options 2015-05-30 03:23:06 -07:00
Harshavardhana
0cc63706bb Resurrect back fileystem driver 2015-05-30 03:12:01 -07:00
Anis Elleuch
d360d0fec3 Forbid the upload of files bigger than the memory backend capacity 2015-05-28 22:07:02 +01:00
Harshavardhana
554a244b47 Add donut OSX support 2015-05-26 02:14:34 -07:00
Harshavardhana
e46660a966 Use http.TimeFormat to return, S3 expects us to return in GMT not UTC 2015-05-25 20:17:04 -07:00
Harshavardhana
19250296c6 Handle authorization header better 2015-05-24 20:59:53 -07:00
Harshavardhana
9666f2e5bf Relax minium object size limit, one can really upload a 0byte file 
For example
~~~
$ mc ls https://s3.amazonaws.com/ferenginar/test.go
[2015-05-23 12:02:06 PDT]     0B test.go

$ curl -i -X HEAD https://s3.amazonaws.com/ferenginar/test.go
HTTP/1.1 200 OK
x-amz-id-2: ZMWLriPH+uQJ8IsaMHmuNOU/FXvdSq+s6O7ugUI9hZ695XJTNAZ1utKxh03w5Jcf
x-amz-request-id: F5A11F533B74DD8F
Date: Sat, 23 May 2015 19:19:29 GMT
Last-Modified: Sat, 23 May 2015 19:02:06 GMT
ETag: "d41d8cd98f00b204e9800998ecf8427e"
Accept-Ranges: bytes
Content-Type: binary/octet-stream
Content-Length: 0
Server: AmazonS3
~~~
 2015-05-23 12:19:16 -07:00
Harshavardhana
e11f9110b6 add missing validate access keys after being extracted from AuthHeader 2015-05-21 19:23:47 -07:00
Harshavardhana
ad60b97d52 Fix some donut TODO's regarding bucket created date 2015-05-21 11:13:49 -07:00
Harshavardhana
cd817d34df Finish last bits of api testsuite on server 2015-05-21 02:57:03 -07:00
Harshavardhana
f00bfd38da Bump gocyclo back to 16 2015-05-19 18:32:50 -07:00
Harshavardhana
3d61ba5930 Fix duplicate map allocation 2015-05-18 13:14:01 -07:00
Harshavardhana
e4c0d574dc Update totalnumber of multiparts inside CreateObjectPart(), also introduce support for UploadIDMarker and Prefix for listMultipartUploads 2015-05-18 12:24:36 -07:00
Harshavardhana
7ce47130fd HEAD request should have Content-Length for only successful response, there is no response body for errors, just header is sufficient - fixes #603 
http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.13

"""  in the case of the HEAD method, the size of the entity-body that would have been sent had the request been a GET. """
 2015-05-15 21:55:59 -07:00
Harshavardhana
317637fcb8 Remove redundant WriteHeader calls 2015-05-15 20:58:28 -07:00
Harshavardhana
5c5761ae16 Remove unnecessary GetObjectMetadata prefix argument 2015-05-15 19:25:17 -07:00
Harshavardhana
6275238a47 For multiparts use its own cache avoids deadlocks during object expiration 
Using same object cache as regular objects for multiparts causes deadlocks during object expiration routine.

As a side affect this also fixes the bug of listing multiparts during regular 'listing' of objects.
 2015-05-14 20:26:59 -07:00
Harshavardhana
c41ebf7535 Fixing a race in NewMultipartUpload() session for simultaneous requests, observed at play.minio.io 2015-05-14 19:11:19 -07:00
Harshavardhana
8fb427a892 Add more tests for multipart now in driver api_testsuite 2015-05-14 17:28:39 -07:00
Harshavardhana
dba7c94571 Add tests for abortMultipartUpload, listobjectparts, listmultipartuploads 2015-05-14 17:07:30 -07:00
Harshavardhana
1bd94ec6ab An attempt to implement ListMultipartUploads() 2015-05-14 16:22:47 -07:00
Harshavardhana
82b043005a Avoid one last memory copy 2015-05-14 11:26:55 -07:00
Harshavardhana
5498c90788 Add a new validateContentTypeHandler{}, verify Accept header earlier 2015-05-13 12:22:15 -07:00
Harshavardhana
de56909d55 More scripts and code from Minio-io to minio 2015-05-11 16:43:38 -07:00
Harshavardhana
8d06504068 Change all minio-io path to minio 2015-05-11 16:23:10 -07:00
Harshavardhana
e39de4f603 Fix typo introduced in previous commit 2015-05-11 14:24:31 -07:00
Harshavardhana
f45635524f Remove SignatureV2 support, bring in SignatureV4 header only validation for now 2015-05-11 13:38:40 -07:00
Harshavardhana
b48b7f51ef Use convenience functions for url.Values over looping through keys, fix a bug in listObjectParts() 2015-05-10 11:32:41 -07:00
Harshavardhana
14b5828505 Handle partNumberMarker with listObjectParts now and other fixes 2015-05-09 19:39:00 -07:00
Frederick F. Kautz IV
cf559fa48e Limiting upload id size 2015-05-09 18:28:50 -07:00
Frederick F. Kautz IV
61a9ea0f4b Adding support for memory driver test 2015-05-09 18:15:27 -07:00
Frederick F. Kautz IV
058604aa21 Adding tests and fixes for multipart uploads uncovered from tests 2015-05-09 17:42:14 -07:00
Harshavardhana
8860aa0c8c Implement AbortMultipart 2015-05-09 16:09:50 -07:00
Harshavardhana
18c7f620cc Add listparts support 2015-05-09 13:55:55 -07:00
Harshavardhana
04e388d620 Handle two different styles of time format, s3cmd now compatible 2015-05-08 16:01:59 -07:00
Harshavardhana
01b474fca3 Move expiration code into its own and add ticker select case 2015-05-08 12:37:49 -07:00
Harshavardhana
23a545673d Verify incoming md5sum for invidual parts in complete multipart request and some simplification of the code 
Add two new functions for session cleanup
 2015-05-08 03:59:05 -07:00
Harshavardhana
616241a2c1 Re-enable bucket deletion, this time with uploadID sitting there 2015-05-08 02:28:38 -07:00
Harshavardhana
2ea10c798b Implement proper errors for Multipart 2015-05-08 02:02:51 -07:00
Harshavardhana
82c3656f79 Reply back CompleteMultipartUploadResult properly with final ETag computed 
- Now s3 libraries and also objectstorage-go work properly
 2015-05-07 23:01:20 -07:00
Frederick F. Kautz IV
ae5c99d7bb Router should work with ?uploads unescaped now 2015-05-07 21:37:34 -07:00