MyFavMirrors/minio
1
0
Fork 0
mirror of https://github.com/minio/minio.git synced 2024-12-26 23:25:54 -05:00
Code Issues Packages Projects Releases Wiki Activity
2,882 Commits 4 Branches 506 Tags 189 MiB
a9e778f460
Commit Graph

931 Commits

Author SHA1 Message Date
Harshavardhana
770fd23afa Renaming keys as auth, working towards signature v4 support for all put objects 2015-07-08 14:17:16 -07:00
Harshavardhana
2413a110e6 Fix SSL support, pointer indirection caused nil buffers 2015-07-08 11:02:15 -07:00
Harshavardhana
d1deda3a96 Add API tests for both donut on disk and donut cache 2015-07-07 19:39:46 -07:00
Harshavardhana
ece797c16e Add rpc tests 2015-07-07 17:27:34 -07:00
Harshavardhana
676b9058de Separate out memory statistics and system information into two different services 2015-07-07 16:59:20 -07:00
Harshavardhana
8abb96c030 If NodeDisks are not empty do not impose cache maxSize restriction 2015-07-07 16:41:40 -07:00
Harshavardhana
a50a44b0ca Add nimbleNet tests 2015-07-07 16:15:47 -07:00
Harshavardhana
317096a0c4 Add net.Addr wrapper with IsEqual() and use it. 2015-07-07 15:33:08 -07:00
Harshavardhana
11b893804c Moving os.MkdirAll() inside atomic for auto parent directory creates 2015-07-07 12:35:57 -07:00
Harshavardhana
52cd23ad9f Move atomic file writes into its own package, use them inside quick and disk packages 2015-07-07 12:29:14 -07:00
Harshavardhana
3622fbc87d Across donut, split, nimble some code cleanup 2015-07-06 21:55:21 -07:00
Harshavardhana
bbb89b5776 Add multi-thread protection and also allow atomic file creates, rename upon Close() 2015-07-06 18:54:32 -07:00
Harshavardhana
c2c7bdf0cd Cleanup nimble http 2015-07-06 18:51:20 -07:00
Harshavardhana
b029d0a5f0 Avoid config reload all the time, reload is manually triggerred from outside 2015-07-06 17:26:35 -07:00
Harshavardhana
8b94c53345 Fix issues with multipart upload 2015-07-06 16:22:27 -07:00
Harshavardhana
474954022e Add modified grace library from facebookgo, rename it as nimble 2015-07-06 15:40:12 -07:00
Harshavardhana
1d64e4b6c1 Add Donut rpc service for sending changes to configuration files 2015-07-06 11:10:06 -07:00
Harshavardhana
10b082144e Add updateConfig code to load config changes if possible for every function 2015-07-05 22:46:42 -07:00
Harshavardhana
ba0a5ed416 Add sighup, sigusr2 into trapping code, to trap signals for reloading configuration. 
Need to still figure out a way of graceful restarts - gave facebookgo/httpdown a shot,
but it is not suitable.
 2015-07-05 21:40:53 -07:00
Harshavardhana
4a27ab0e58 Fix another deadlock inside CreateObjectPart() code, premature return without Unlocking() 
Also this patch changes the cache key element to be interface{} type not string.
 2015-07-05 20:26:32 -07:00
Harshavardhana
75788c7a1d Fix go installation check on amazon instance 2015-07-05 18:12:58 -07:00
Harshavardhana
7f0c14f2b7 Add basic controller code, initiating json rpc connection getting list of disks and memstats for now. 2015-07-05 17:17:41 -07:00
Harshavardhana
a3ccb9d405 Rename stuttered service names and make them appropriate 2015-07-05 10:19:23 -07:00
Harshavardhana
e66a84242a Add disk detection for Linux, add new RPC service GetDiskInfoService(), remove dummy HelloService() 2015-07-05 02:08:33 -07:00
Harshavardhana
bab4a47525 Move to container/list datastructure from map[string][]byte 2015-07-04 17:08:23 -07:00
Harshavardhana
aa67a19e99 Implement new CPU detection using cpuid, cpuidex plan9 instructions from klauspost/cpuid project, remove C code 2015-07-04 14:28:16 -07:00
Harshavardhana
0a827305ad Implement metadata cache, metadata cache is used by top level donut right now. Rename trove as data cache 
We should use it internally everywhere.
 2015-07-03 21:09:57 -07:00
Harshavardhana
14844f48dd Make donut do everything as an atomic operation, this avoids all the deadlocks and races 2015-07-03 17:16:58 -07:00
Harshavardhana
30fc14e703 Restructure codebase move crypto, checksum to top-level, move `split` into donut, move crypto/keys into api/auth 2015-07-03 15:24:51 -07:00
Harshavardhana
8a4e7bcdcf Add full API tests, move storage/donut to donut, add disk tests as well 2015-07-03 14:36:29 -07:00
Harshavardhana
7c37e9d06a Make donut fully integrated back into API handlers 2015-07-02 21:04:04 -07:00
Harshavardhana
12bde7df30 Add simple Ticket Master which pro-actively sends messages on proceedChannel 
Handlers are going to wait on proceedChannel, this the initial step towards
providing priority for different set of API operations
 2015-07-02 21:04:04 -07:00
Harshavardhana
5cfb05465e Add cache, donut tests separately - fix behavior differences 
Remove priority queue, implement it using a simpler channels
 2015-07-02 21:04:04 -07:00
Harshavardhana
ebe61d99d9 Use cache Append() for saving objects in memory, GetObject() caches un-cached entries while reading 2015-07-02 21:04:04 -07:00
Harshavardhana
bce93c1b3a Integrate cache with donut, add tests 2015-07-02 21:04:04 -07:00
Harshavardhana
0533abf6a8 Make priority queue lambda function return error over a channel 2015-07-02 21:04:04 -07:00
Harshavardhana
38a6ce36e5 Remove slow AppendUniq code, rolling through over a slice is in-efficient 
Remove it and use map instead
 2015-07-02 21:04:04 -07:00
Harshavardhana
84810162f5 Add simple Version and GetSysInfo services 2015-07-02 21:04:04 -07:00
Harshavardhana
14ec42d646 Add initial implementation of priority queue, uses container/heap 2015-07-02 21:04:04 -07:00
Harshavardhana
eb5aa19dfa Remove custom Config, will use quick Config instead for user access keys 2015-07-02 21:04:04 -07:00
Harshavardhana
701c3e5242 Add new RPC helpers wrapping over regular rpc packages, add middleware chaining ability 2015-07-02 21:04:04 -07:00
Harshavardhana
4addf7a996 Restructure API handlers, add JSON RPC simple HelloService right now. 2015-07-02 21:04:04 -07:00
Harshavardhana
335c7827eb More donut, cache, api cleanup 2015-07-02 21:04:04 -07:00
Harshavardhana
dc0df3dc0e Breakaway from driver model, move cache into donut 2015-07-02 21:04:03 -07:00
Harshavardhana
72572d6c71 Remove some api server code bringing in new cleanup 2015-07-02 21:04:03 -07:00
Harshavardhana
c2031ca066 Add server and control command 2015-07-02 21:04:03 -07:00
Frederick F. Kautz IV
cfbc169034 Fixing API definitions 2015-07-02 13:14:21 -07:00
Harshavardhana
8f61d6b6be Move memory code out, add it as layer on top of existing donut code 
Just like how http.Handlers can be overlayed on top of each other
with each implementing ServeHTTP().

drivers.Driver can be overlayed on top of each other in similar manner
which would implement the drivers.Driver interface.

   API <----> cache <----> donut <----> donut(format)
 2015-06-30 10:09:12 -07:00
Harshavardhana
ab6e16bb41 Add dummy driver for community to submit new drivers 2015-06-29 16:43:50 -07:00
Harshavardhana
12de98fb62 Rename memory driver as cache 2015-06-29 16:43:50 -07:00
Harshavardhana
2571342451 Filesystem goes the high road *again* 2015-06-29 16:43:42 -07:00
Harshavardhana
f74d6138da IsValidBucket() is sufficient we don't need to verify for "." 2015-06-29 15:15:54 -07:00
Harshavardhana
3109909355 Handle couple of cases of OOM conditions, move caching to GetObject() rather than PutObject() 2015-06-29 12:28:50 -07:00
Harshavardhana
d07d0c670a Return back proper errors in writeObjectData(), rename few functions 2015-06-29 11:46:35 -07:00
Harshavardhana
10c807f233 Put object on successful write returns full metadata, to avoid subsequent GetObjectMetadata() calls in driver 2015-06-29 11:15:46 -07:00
Harshavardhana
6921328b93 Avoid frivolous GetObjectMetadata() calls at driver level, return back all the information in donut ListObjects() 2015-06-29 11:14:58 -07:00
Harshavardhana
d8f7896a43 Expand http server struct to store more values 2015-06-29 00:12:28 -07:00
Harshavardhana
42c0287943 Use errorChannels only for services not for drivers, reduce them to use simple functions 2015-06-28 23:59:47 -07:00
Harshavardhana
91e5f648cb Move to Set() not Append() due to large memory reference copy 2015-06-28 10:13:12 -07:00
Harshavardhana
ac4d8fe478 Add free() method for proxyReader to aggressively de-allocate Read data, to handle certain out of memory conditions 
There are still some more out there
 2015-06-27 20:43:25 -07:00
Harshavardhana
05f8654e3d Add Append() method to trove cache for appending data to an existing key 
This largely avoids a large buffer copy which would accumulate inside proxyReader{}

This patch also implements "initialize()" function to init and populate data
on all the existing buckets, avoiding the redundant ListBuckets() invoked by
every API call.
 2015-06-27 20:25:24 -07:00
Harshavardhana
367772b988 Make sure to populate on disk data into memory upon first API requests 2015-06-27 18:25:21 -07:00
Harshavardhana
45a7eab804 An attempt to bring in memory layer into donut driver 2015-06-27 17:23:34 -07:00
Harshavardhana
f3c25bcfc4 Keeping the lexical order same add optimizations, provide a comprehensive response from ListObjects() 2015-06-27 13:12:44 -07:00
Harshavardhana
ae66ae42c4 Rename functions for their purpose 2015-06-27 12:39:11 -07:00
Harshavardhana
9a4680475f Object metadata was wrongly misconstrued to be mutable, handle it 2015-06-26 23:22:53 -07:00
Harshavardhana
3aa6d90c5e Avoid racy maps, read from disk on success return quickly. Many more optimizations 2015-06-26 19:49:37 -07:00
Harshavardhana
68974918ac Donut moves to typed metadata, removing the necessity for strconv, parsing and string converstions 2015-06-26 16:23:12 -07:00
Harshavardhana
767d3743ee Add new metadata definitions and use them 2015-06-26 13:34:09 -07:00
Harshavardhana
9958e34772 Import quick key value store from Minio Client for persistent state files, primarily for donut 2015-06-25 19:57:31 -07:00
Harshavardhana
fb9adb5524 Donut cleanup another set 
- Make sure to close all readers
- Fix errors in api_testsuite c.Assert(err, IsNil) should be done right after each function call
 2015-06-25 18:54:34 -07:00
Harshavardhana
eec66f195a Take all the ListObjects into bucket handlers 
Earlier the listing would wait for all the objects to be processed
this is essentially very time consuming considering even for 100,000
files.
 2015-06-25 18:04:29 -07:00
Harshavardhana
45e9d25931 Remove more bloated code - simplify 2015-06-25 13:02:08 -07:00
Harshavardhana
82dcbf262d Add simple locking for donut API for now - fixes #671 2015-06-25 11:29:11 -07:00
Harshavardhana
03b4d3b308 Go vet fixes for donut 2015-06-24 21:07:03 -07:00
Harshavardhana
57a2b53178 Removing further bloated code simplifying 2015-06-24 21:03:39 -07:00
Harshavardhana
a2c205ff2e Use external package disk for donut. 2015-06-24 21:03:02 -07:00
Harshavardhana
841ff01412 Move disk into its own package, remove bloated code 2015-06-24 20:13:47 -07:00
Harshavardhana
1682c748ac Remove unnecessary interfaces from donut, cleanup 2015-06-24 19:43:38 -07:00
Harshavardhana
3498872467 Add sha256 and sha512 windows compatibility layer 2015-06-24 14:39:41 -07:00
Harshavardhana
77d35b87d4 Fix a bug on windows regarding blockSSE3 calculation 2015-06-24 14:24:33 -07:00
Harshavardhana
f1410731db Add windows code for sha1 and crc32c 2015-06-24 14:16:32 -07:00
Harshavardhana
ba2d3dea74 Fix wrong TMPFS listing in donut filesystem map 2015-06-23 23:16:06 -07:00
Harshavardhana
e818bc7187 Trim iodine path properly, so that now errors have github.com/minio/minio prefixed 2015-06-23 13:36:54 -07:00
Harshavardhana
2fd52ca284 Add missing strongly typed errors for Donut 2015-06-23 11:54:44 -07:00
Harshavardhana
7a060110ff Fix BUILDDEPS paths for golang installation 2015-06-22 12:07:23 -07:00
Harshavardhana
3bf64f5669 Verify d.donut to be non-nil, usually happens when multiple go versions compilations are linked - possible cause for #659 2015-06-20 11:03:17 -07:00
Harshavardhana
641f07cecf Use filepath everywhere instead of path.{} functions for portability - fixes #656 2015-06-18 16:02:45 -07:00
Harshavardhana
573a6134b2 Remove redundant !ok for map 2015-06-17 22:36:46 -07:00
Harshavardhana
e9a3fd677a Hold lock on GetGlobalStateKey() - fixes #652 2015-06-17 20:37:49 -07:00
Harshavardhana
b793f53d48 Minior change filter() to filterObjects() 2015-06-17 20:35:44 -07:00
Harshavardhana
d4e71beb54 Add time.Since on buckets created time to avoid deleting buckets spuriously during eviction of objects 2015-06-13 11:28:04 -07:00
Harshavardhana
636788997b Minor optimization avoiding one unnecessary copy 2015-06-11 14:23:05 -07:00
Harshavardhana
6d44274db6 Make sure to reply back ETag with quotes for s3 compliance 2015-06-09 01:13:02 -07:00
Harshavardhana
a947767339 Remove unnecessary tlsHelpers, SSL works with domain certificate 2015-06-08 20:54:41 -07:00
Harshavardhana
c6ecd94420 Enable TLS and use it 2015-06-08 20:10:59 -07:00
Harshavardhana
64b014369c Return back entity too large for createObject requests bigger than cache size 2015-06-08 17:10:07 -07:00
Harshavardhana
ab1976bc14 Load from persisted multipart session, now multipart uploads can resume even after restarting minio fs server 2015-06-08 15:15:37 -07:00
Harshavardhana
2976f109df More multipart fixes, filter out parts and multipart metadata from listObjects 2015-06-08 14:08:36 -07:00
Harshavardhana
e08d59be00 Make sure to delete uploadid's from active session file properly 2015-06-08 12:25:51 -07:00
Harshavardhana
0d3710cb9b Store in json file instead of gob for fs metadata 2015-06-08 12:08:01 -07:00
Harshavardhana
8efc842b59 Add delete handlers and reply back as 'NotImplemented' instead of 404 2015-06-08 11:06:06 -07:00
Harshavardhana
8a320c4447 Enhance logging handler to print in humanize form for ContentLength 2015-06-08 02:14:09 -07:00
Harshavardhana
cefd740002 Put back the older conn_limit 2015-06-06 18:11:38 -07:00
Harshavardhana
371651bde6 Add rate limiter instead of connection limit for now 2015-06-06 18:04:55 -07:00
Harshavardhana
21916dc0a1 Fix a deadlock in trove when evicting objects when we reach the maxSize used 2015-06-06 16:02:52 -07:00
Harshavardhana
8b43655d9f Make connection limit configurable 2015-06-06 14:31:50 -07:00
Harshavardhana
a344f5b34d Misc changes before capturing proper video 
- Disable management UI command option
- Remove featureflags option, filesystem and memory already support multipart
- Print informative messages after starting minio server
 2015-06-03 18:27:15 -07:00
Harshavardhana
097f70a3c5 Enable filesystem for API tests, this patch also implements ListMultipartUploads() 2015-06-01 20:57:13 -07:00
Harshavardhana
bfb9209689 Skip files and weird directories from listBuckets(), allowing this so that minio fs mode can be used as ftp service 2015-06-01 16:43:43 -07:00
Harshavardhana
d86ba5f336 Implement multipart support 
- implemented NewMultipartUpload()
 - implemented CreateObjectPart()
 - implemented ListObjectParts()
 - implemented AbortMultipartUpload()
 - implemented CompleteMultipartUpload()
 2015-06-01 16:25:11 -07:00
Harshavardhana
c07d48e7e8 ErrorResponse from minio server needed proper xml translations 2015-06-01 12:19:27 -07:00
Harshavardhana
a91e519692 Make caching a package trove, and use it inside memory driver 2015-05-31 18:41:56 -07:00
Anis Elleuch
7e5a26f42c Protect the exported method Delete() 2015-05-31 20:29:42 +01:00
Harshavardhana
82a0eac659 Add filesystem factory functions and add related cli options 2015-05-30 03:23:06 -07:00
Harshavardhana
0cc63706bb Resurrect back fileystem driver 2015-05-30 03:12:01 -07:00
Anis Elleuch
d360d0fec3 Forbid the upload of files bigger than the memory backend capacity 2015-05-28 22:07:02 +01:00
Harshavardhana
554a244b47 Add donut OSX support 2015-05-26 02:14:34 -07:00
Harshavardhana
e46660a966 Use http.TimeFormat to return, S3 expects us to return in GMT not UTC 2015-05-25 20:17:04 -07:00
Harshavardhana
19250296c6 Handle authorization header better 2015-05-24 20:59:53 -07:00
Harshavardhana
9666f2e5bf Relax minium object size limit, one can really upload a 0byte file 
For example
~~~
$ mc ls https://s3.amazonaws.com/ferenginar/test.go
[2015-05-23 12:02:06 PDT]     0B test.go

$ curl -i -X HEAD https://s3.amazonaws.com/ferenginar/test.go
HTTP/1.1 200 OK
x-amz-id-2: ZMWLriPH+uQJ8IsaMHmuNOU/FXvdSq+s6O7ugUI9hZ695XJTNAZ1utKxh03w5Jcf
x-amz-request-id: F5A11F533B74DD8F
Date: Sat, 23 May 2015 19:19:29 GMT
Last-Modified: Sat, 23 May 2015 19:02:06 GMT
ETag: "d41d8cd98f00b204e9800998ecf8427e"
Accept-Ranges: bytes
Content-Type: binary/octet-stream
Content-Length: 0
Server: AmazonS3
~~~
 2015-05-23 12:19:16 -07:00
Harshavardhana
e11f9110b6 add missing validate access keys after being extracted from AuthHeader 2015-05-21 19:23:47 -07:00
Harshavardhana
ad60b97d52 Fix some donut TODO's regarding bucket created date 2015-05-21 11:13:49 -07:00
Harshavardhana
cd817d34df Finish last bits of api testsuite on server 2015-05-21 02:57:03 -07:00
Harshavardhana
f00bfd38da Bump gocyclo back to 16 2015-05-19 18:32:50 -07:00
Harshavardhana
3d61ba5930 Fix duplicate map allocation 2015-05-18 13:14:01 -07:00
Harshavardhana
e4c0d574dc Update totalnumber of multiparts inside CreateObjectPart(), also introduce support for UploadIDMarker and Prefix for listMultipartUploads 2015-05-18 12:24:36 -07:00
Harshavardhana
7ce47130fd HEAD request should have Content-Length for only successful response, there is no response body for errors, just header is sufficient - fixes #603 
http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.13

"""  in the case of the HEAD method, the size of the entity-body that would have been sent had the request been a GET. """
 2015-05-15 21:55:59 -07:00
Harshavardhana
317637fcb8 Remove redundant WriteHeader calls 2015-05-15 20:58:28 -07:00
Harshavardhana
5c5761ae16 Remove unnecessary GetObjectMetadata prefix argument 2015-05-15 19:25:17 -07:00
Harshavardhana
6275238a47 For multiparts use its own cache avoids deadlocks during object expiration 
Using same object cache as regular objects for multiparts causes deadlocks during object expiration routine.

As a side affect this also fixes the bug of listing multiparts during regular 'listing' of objects.
 2015-05-14 20:26:59 -07:00
Harshavardhana
c41ebf7535 Fixing a race in NewMultipartUpload() session for simultaneous requests, observed at play.minio.io 2015-05-14 19:11:19 -07:00
Harshavardhana
8fb427a892 Add more tests for multipart now in driver api_testsuite 2015-05-14 17:28:39 -07:00
Harshavardhana
dba7c94571 Add tests for abortMultipartUpload, listobjectparts, listmultipartuploads 2015-05-14 17:07:30 -07:00
Harshavardhana
1bd94ec6ab An attempt to implement ListMultipartUploads() 2015-05-14 16:22:47 -07:00
Harshavardhana
82b043005a Avoid one last memory copy 2015-05-14 11:26:55 -07:00
Harshavardhana
5498c90788 Add a new validateContentTypeHandler{}, verify Accept header earlier 2015-05-13 12:22:15 -07:00
Harshavardhana
de56909d55 More scripts and code from Minio-io to minio 2015-05-11 16:43:38 -07:00
Harshavardhana
8d06504068 Change all minio-io path to minio 2015-05-11 16:23:10 -07:00
Harshavardhana
e39de4f603 Fix typo introduced in previous commit 2015-05-11 14:24:31 -07:00
Harshavardhana
f45635524f Remove SignatureV2 support, bring in SignatureV4 header only validation for now 2015-05-11 13:38:40 -07:00
Harshavardhana
b48b7f51ef Use convenience functions for url.Values over looping through keys, fix a bug in listObjectParts() 2015-05-10 11:32:41 -07:00
Harshavardhana
14b5828505 Handle partNumberMarker with listObjectParts now and other fixes 2015-05-09 19:39:00 -07:00
Frederick F. Kautz IV
cf559fa48e Limiting upload id size 2015-05-09 18:28:50 -07:00
Frederick F. Kautz IV
61a9ea0f4b Adding support for memory driver test 2015-05-09 18:15:27 -07:00
Frederick F. Kautz IV
058604aa21 Adding tests and fixes for multipart uploads uncovered from tests 2015-05-09 17:42:14 -07:00
Harshavardhana
8860aa0c8c Implement AbortMultipart 2015-05-09 16:09:50 -07:00
Harshavardhana
18c7f620cc Add listparts support 2015-05-09 13:55:55 -07:00
Harshavardhana
04e388d620 Handle two different styles of time format, s3cmd now compatible 2015-05-08 16:01:59 -07:00
Harshavardhana
01b474fca3 Move expiration code into its own and add ticker select case 2015-05-08 12:37:49 -07:00
Harshavardhana
23a545673d Verify incoming md5sum for invidual parts in complete multipart request and some simplification of the code 
Add two new functions for session cleanup
 2015-05-08 03:59:05 -07:00
Harshavardhana
616241a2c1 Re-enable bucket deletion, this time with uploadID sitting there 2015-05-08 02:28:38 -07:00
Harshavardhana
2ea10c798b Implement proper errors for Multipart 2015-05-08 02:02:51 -07:00
Harshavardhana
82c3656f79 Reply back CompleteMultipartUploadResult properly with final ETag computed 
- Now s3 libraries and also objectstorage-go work properly
 2015-05-07 23:01:20 -07:00
Frederick F. Kautz IV
ae5c99d7bb Router should work with ?uploads unescaped now 2015-05-07 21:37:34 -07:00
Frederick F. Kautz IV
9dc7b82b39 Merge pull request #576 from fkautz/pr_out_adding_multipart_support 2015-05-07 20:00:20 -07:00
Frederick F. Kautz IV
f050d5e974 Adding multipart support 2015-05-07 19:55:30 -07:00
Frederick F. Kautz IV
ed71c28c37 Fixing panic in intelligent, attempted to delete nonexistent key 2015-05-07 19:12:49 -07:00
Frederick F. Kautz IV
d0c4334834 Adding feature flags 2015-05-07 13:56:54 -07:00
Harshavardhana
1df28f3060 Get rid of sub-domain code and some other formatting changes 2015-05-07 00:21:40 -07:00
Harshavardhana
6d3d92694f Add Marker support for donut, also add a testing code 2015-05-05 22:18:24 -07:00
Harshavardhana
75028c2ad1 Add lexicographic Marker/NextMarker support for recursive listing of objects. 
Also update times when an object is accessed logic
 2015-05-05 18:05:34 -07:00
Frederick F. Kautz IV
24af2222c9 Adding xmlns 2015-05-05 12:54:34 -07:00
Harshavardhana
193a6606db Fix a potential race between ExpireObjects() and Set() over items map 2015-05-04 22:36:47 -07:00
Harshavardhana
ed1259d6f0 Deprecate LRU use just map[string]interface{} and call it intelligent 'cache' 2015-05-04 22:01:56 -07:00
Harshavardhana
63edb1e9a0 Nullify list memory to nil as we remove the element 2015-05-04 17:22:08 -07:00
Harshavardhana
d63064b8af Optimize memory usage in GetPartialObject() for memory driver 2015-05-04 15:26:56 -07:00
Harshavardhana
c8f31d97a8 Modify LRU further to add object expiration 2015-05-04 04:32:44 -07:00
Harshavardhana
670f997b07 Bring blockingWriter from client and use it here 2015-05-04 03:55:43 -07:00
Harshavardhana
d0df548eb5 Use new LRU inside memory driver 2015-05-04 03:55:43 -07:00
Harshavardhana
f7caef2d26 Change CreateObject() to take size argument from content-length 2015-05-04 03:55:43 -07:00
Harshavardhana
c342ce1588 New LRU based on GroupCache which keeps track of memory rather than entries 2015-05-04 03:55:19 -07:00
Frederick F. Kautz IV
3cf045fd1b Deleting key from lastAccessedObjects on eviction 2015-05-01 22:01:07 -07:00
Harshavardhana
9ec8925f1c Rename IsValidObject() to IsValidObjectName() 2015-05-01 20:31:26 -07:00
Frederick F. Kautz IV
33ecba8e49 Reimagining memory settings cli input 2015-05-01 18:12:58 -07:00
Frederick F. Kautz IV
25c268ac88 Attempt to fix memory lock on Get preventing write 2015-05-01 17:45:40 -07:00
Frederick F. Kautz IV
85f1b46492 Removing log message from test 2015-05-01 17:32:18 -07:00
Frederick F. Kautz IV
7cf86e6114 Fix docker support 2015-05-01 15:23:39 -07:00
Harshavardhana
2c7b30e5e1 Even mux vars don't help, handle it by looking at req.URL.Path 2015-05-01 12:41:17 -07:00
Frederick F. Kautz IV
b86b26e7e5 Exposing expiration for memory driver 2015-04-30 21:20:54 -07:00
Harshavardhana
c01fb92eca Merge pull request #557 from harshavardhana/pr_out_make_builddate_universal 2015-04-30 21:19:04 -07:00
Harshavardhana
bbc32d6ac0 Disable domain based routing for now 2015-04-30 21:15:53 -07:00
Harshavardhana
f3078d047c Get bucketName from mux.Vars instead of heuristic code, resolves subdomain issues 2015-04-30 21:11:32 -07:00
Frederick F. Kautz IV
e911cc332e Merge pull request #554 from fkautz/pr_out_disable_eviction_on_0_expiration_time 2015-04-30 19:40:46 -07:00
Harshavardhana
8e3d48bf35 Use UTC() everywhere 
ref - https://github.com/golang/go/issues/1988, no standard function to set timeZone
 2015-04-30 19:38:32 -07:00
Frederick F. Kautz IV
94e24f1a05 Disable eviction on 0 expiration time 2015-04-30 19:34:35 -07:00
Harshavardhana
72395798c8 Merge pull request #552 from harshavardhana/pr_out_minio_config_directory_should_be_0700_not_os_modedir 2015-04-30 17:40:15 -07:00
Harshavardhana
2de81c8a3c Minio config directory should be 0700 not os.ModeDir 2015-04-30 15:42:49 -07:00
Frederick F. Kautz IV
8e2f7f9493 Wrapping functions that require gc so that references are released before running 2015-04-30 12:38:12 -07:00
Harshavardhana
d815e6adfd Changes to CreateObject() now returns back md5 along with any error 
- This change is necessary to avoid the racy calls to GetObjectMetadata()
- This change is also necessary since one has to reply back md5sum with
  PUT object response header
 2015-04-30 03:38:14 -07:00
Harshavardhana
fd2203b1b7 Some more improvements to connection limit 2015-04-30 01:23:44 -07:00
Harshavardhana
f8a16dd22b Memory now evicts bucket if no more objects in memory struct 
- To avoid race in expiration while accessing memory driver structs with
  in two competing write locks. Use lru Len() instead to know exact length
  to schedule for eviction.

- squash both bucket and object structs, instead use a separate map to keep
  mutable info of lastAccessTime which can be independently used inside the
  expiration routine.
 2015-04-30 00:31:52 -07:00
Frederick F. Kautz IV
ed0a9a9e06 Removing entity too large errors from memory driver 2015-04-30 00:09:54 -07:00
Frederick F. Kautz IV
aff2b3b06a Adding debug statements for testing 2015-04-29 21:13:32 -07:00
Frederick F. Kautz IV
82a16b8848 Make minio work with curl and browsers again 2015-04-29 20:17:26 -07:00
Frederick F. Kautz IV
3a1386165f Fixing race condition 2015-04-29 18:54:59 -07:00
Harshavardhana
03bca9a001 Fix out of memory issue 2015-04-29 18:33:19 -07:00
Harshavardhana
25d4c0f6fa A small logical change in messaging and logging 2015-04-29 17:30:54 -07:00
Frederick F. Kautz IV
f596c744ed Adding error message on connection limit met 2015-04-29 17:21:23 -07:00
Harshavardhana
a89c4a57d3 Return resources as its available, not a new value 2015-04-29 16:47:14 -07:00
Harshavardhana
251c99176a Merge pull request #537 from harshavardhana/pr_out_add_proper_content_length_for_error_and_success_responses 2015-04-29 15:32:12 -07:00
Frederick F. Kautz IV
0da04c6f17 Expirations are now based on last accessed instead of time created 2015-04-29 15:28:45 -07:00
Harshavardhana
92e4301414 Add proper content-length for error and success responses 
- All compliance issues with S3 API for Put,Get,List (Bucket,Object) respectively
 - Encodes and returns back proper HTTP headers
 2015-04-29 15:28:28 -07:00
Frederick F. Kautz IV
5fc5f79ab7 Adding lru 2015-04-29 15:08:26 -07:00
Frederick F. Kautz IV
8fcb697844 Simplifying memory sleep logic 2015-04-29 13:53:09 -07:00
Frederick F. Kautz IV
5cc9418ca7 Adding expiration 2015-04-29 13:02:21 -07:00
Frederick F. Kautz IV
a608524ebf api logging is now in its own package 2015-04-29 12:24:53 -07:00
Harshavardhana
d1d5f7a78d Collate success response into writeSuccessResponse(), add docs 2015-04-29 10:52:04 -07:00
Harshavardhana
b18bb230da maxObjectSize and minObjectSize limitation added at putObjectHandler() 
Put() replies back with

   - EntityTooLarge with > 5GB in single PUT operation
   - EntityTooSmall with < 1B in single PUT operation
   - IncompleteBody with ho Content-Length found in HTTP request header
 2015-04-29 02:19:53 -07:00
Frederick F. Kautz IV
172cd18455 Adding initial logging framework to api_router 2015-04-28 20:37:08 -07:00
Frederick F. Kautz IV
88cca647ea Adding json logging to stderr 2015-04-28 18:11:53 -07:00
Frederick F. Kautz IV
de091068e8 Removing quotas, will handle with iptables 2015-04-28 12:14:34 -07:00
Frederick F. Kautz IV
f2257a643c Critical sections are now protected by lock 2015-04-27 19:59:58 -07:00
Harshavardhana
605edbc307 Add some logging with debug 2015-04-27 19:26:24 -07:00
Harshavardhana
7bdcf4bbf3 Merge pull request #515 from fkautz/pr_out_fix_multiple_response_writeheader_calls 
Fix multiple response.WriteHeader calls
 2015-04-27 19:21:47 -07:00
Frederick F. Kautz IV
db5870be28 Fix multiple response.WriteHeader calls 2015-04-27 18:30:21 -07:00
Frederick F. Kautz IV
fc2537eef9 Merge pull request #513 from fkautz/pr_out_adding_software_fallback 2015-04-27 14:41:41 -07:00
Frederick F. Kautz IV
289c859675 Adding software fallback 2015-04-27 13:55:40 -07:00
Harshavardhana
ade803a923 Verify if request date is 5minutes late, reject such a request as 
it could be a possible replay attack.

This commit also fixes #505, by returning MethodNotAllowed instead of NotImplemented
 2015-04-27 03:55:10 -07:00
Harshavardhana
9232ce3b4e Now client requests for ACL changes are honored through PutBucketACL API 2015-04-27 03:04:29 -07:00
Harshavardhana
c99d96dbc2 Now donut supports bucket ACL's, bringing in this support for dl.minio.io 2015-04-27 02:09:32 -07:00
Harshavardhana
8beca83090 Limit memory mode at 100 buckets and return error appropriately 
~~~
mc: <ERROR> Failed to create bucket for URL [http://localhost:9000/newbucket-101]. Reason: [You have attempted to create more buckets than allowed.].
~~~

Response from server
 2015-04-26 22:15:39 -07:00
Harshavardhana
2d96d5ad57 Enhance error responses for request limit and bring some code from api errors 
~~~
mc: <ERROR> Failed to create bucket for URL [http://localhost:9000/newbucket-101]. Reason: [Reduce your request rate.].
~~~

Client sees proper errors now.
 2015-04-26 22:10:59 -07:00
Frederick F. Kautz IV
8a57006b36 Merge pull request #504 from fkautz/pr_out_making_lock_much_less_aggressive_fixes_multiple_uploads_blocking 2015-04-26 20:11:50 -07:00
Frederick F. Kautz IV
a974e4d7c7 Making lock much less aggressive, fixes multiple uploads blocking 2015-04-26 20:08:37 -07:00
Frederick F. Kautz IV
258bf55e31 Bandwidth quota now supports 100-Continue 2015-04-26 20:07:05 -07:00
Frederick F. Kautz IV
9431a54a27 finish me 2015-04-26 19:25:48 -07:00
Frederick F. Kautz IV
202fc26719 Revert "Exporting WriteErrorResponse" 
This reverts commit e05d46a0b5.
 2015-04-26 17:02:49 -07:00
Frederick F. Kautz IV
e05d46a0b5 Exporting WriteErrorResponse 2015-04-26 16:28:54 -07:00
Frederick F. Kautz IV
64f7d04cd1 Adding connection limits 2015-04-26 16:14:51 -07:00
Frederick F. Kautz IV
7867ee89fa Bandwidth quotas now work on data returned from server 2015-04-26 15:49:55 -07:00
Frederick F. Kautz IV
e2403cbc56 Quotas are more accurate, occur on Read() 2015-04-26 15:07:57 -07:00
Frederick F. Kautz IV
bbb493080f Switching back to BigEndian 2015-04-25 18:16:21 -07:00
Frederick F. Kautz IV
3a48f9fe4d Adding quotas based upon type 2015-04-25 18:07:02 -07:00
Harshavardhana
b75f46aaa0 Use standard functions to achieve byteorder converstion 2015-04-25 13:18:15 -07:00
Frederick F. Kautz IV
e2475925ea quota handlers now log against ip properly against a duration 2015-04-24 20:49:33 -07:00
Harshavardhana
b3a9d8b058 Merge pull request #483 from harshavardhana/pr_out_memory_now_handles_submitting_large_files_fixes_482 2015-04-24 15:25:38 -07:00
Harshavardhana
c5d8ca245d Memory now handles submitting large files - fixes #482 2015-04-24 15:22:22 -07:00
Frederick F. Kautz IV
fda50d52c8 Add initial framework for quota 2015-04-24 00:16:29 -07:00
Frederick F. Kautz IV
5a607a6762 Moving factories and start minio method to server package 2015-04-23 16:13:50 -07:00
Frederick F. Kautz IV
5c515d3138 Simplifying server config handling 2015-04-23 12:44:22 -07:00
Harshavardhana
5ae8d34f76 Donut mode will now take multiple paths as argument, updated docs as well 2015-04-23 11:36:14 -07:00
Harshavardhana
a5729e8c02 Disable verifying ACL until we get webcli up and running 2015-04-23 03:21:08 -07:00
Harshavardhana
bcc4449de7 Add proper help and several other cleanup 2015-04-23 02:42:50 -07:00
Harshavardhana
e95604ff86 Finishing all the test case support for ACL and other fixes 2015-04-23 01:24:46 -07:00
Frederick F. Kautz IV
0f8626ab1a Refactoring cli to take commands 2015-04-23 01:05:44 -07:00
Harshavardhana
848c4ee31c Further fixes for ACL support, currently code is disabled in all the handlers 
Disabled because due to lack of testing support. Once we get that in we can
uncomment them back.
 2015-04-22 19:31:45 -07:00
Harshavardhana
1c0ff2c758 ACL driver/storage layer support 2015-04-22 19:31:45 -07:00
Harshavardhana
c8713fd650 Bring in full fledged acl support 2015-04-22 18:26:35 -07:00
Harshavardhana
2c1455af1b Implement x-amz-acl tests 2015-04-22 18:26:35 -07:00
Harshavardhana
107e077ec0 Implement x-amz-acl handling 2015-04-22 18:26:35 -07:00
Frederick F. Kautz IV
c56f7380d6 Memory driver now limits by size 2015-04-22 17:50:26 -07:00
Harshavardhana
b121c8588f Remove bucketpolicy handlers and all its references 2015-04-22 14:46:51 -07:00
Harshavardhana
3f33643d39 File driver to take the high road 2015-04-16 02:16:39 -07:00
Frederick F. Kautz IV
59a81a73c6 Adding lru to memory driver, not wired to command line opts 2015-04-15 12:49:41 -07:00
Frederick F. Kautz IV
0cab3fd151 Better locking in memory driver 2015-04-15 11:48:56 -07:00
Harshavardhana
148935392d Fix golint complaint 2015-04-08 23:42:59 -07:00
Frederick F. Kautz IV
26d8deee40 Removing unneeded files 2015-04-08 20:58:42 -07:00
Frederick F. Kautz IV
7212b3406a Adding comments to iodine 2015-04-08 19:44:44 -07:00
Frederick F. Kautz IV
c12d5ea8a9 Adding iodine error conversion 2015-04-08 19:32:37 -07:00
Harshavardhana
38155e86ac iodine is part of minio now 2015-04-08 18:05:48 -07:00
Harshavardhana
7fd1cc073c Add everything back into one project hood, still missing iodine 2015-04-08 17:47:56 -07:00
Frederick F. Kautz IV
6241c9880b Moving object storage drivers back to minio 2015-04-08 16:28:14 -07:00
Harshavardhana
2a21b7d639 Implement headBucketHandler() and its related tests 
Remove internal testify/mock and use upstream - update all godeps
 2015-04-07 01:16:32 -07:00
Frederick F. Kautz IV
ea961ef16d Updating to use objectdriver 2015-04-06 18:00:18 -07:00
Frederick F. Kautz IV
e71cc8d833 Adding Debug logger to log package 2015-04-06 12:15:19 -07:00
Harshavardhana
319feb52f1 erasure is now external pull it and update from minio-io/erasure 2015-04-05 23:51:24 -07:00
Frederick F. Kautz IV
052817ff91 Fix fd leak 2015-04-05 21:33:24 -07:00
Harshavardhana
d324c4b061 Return proper return for all drivers when BucketNotFound 2015-04-05 15:55:10 -07:00
Harshavardhana
0475d7d056 Migrate to external Donut package 2015-04-05 14:59:24 -07:00
Frederick F. Kautz IV
2f44438843 Reformatting imports 2015-04-04 13:59:59 -07:00
Frederick F. Kautz IV
be4bc44b4d Replacing gopkg.in/check.v1 with minio-io/check 2015-04-04 13:37:30 -07:00
Frederick F. Kautz IV
e6abca1a63 Adding iodine to object HEAD verb 2015-04-04 12:46:34 -07:00
Frederick F. Kautz IV
46badca07e Removing Godeps from pkg/api 2015-04-03 20:40:21 -07:00
Frederick F. Kautz IV
4171ad8b69 Adding test for invalid range in api 2015-04-03 20:35:39 -07:00
Harshavardhana
4079dd1d12 Commit testify,objx, gorilla/mux, minio-io/erasure gopkg.in/check.v1 2015-04-03 18:57:47 -07:00
Harshavardhana
4b25d4e8db Use verifyError instead of parsResponse, also get rid of mxj dependency 2015-04-03 18:56:47 -07:00
Frederick F. Kautz IV
9954e5c280 Better error handling + tests for get object 2015-04-03 18:53:27 -07:00
Frederick F. Kautz IV
db1e6e5988 Adding tests for put bucket errors, fixed mismatching error 2015-04-03 18:37:39 -07:00
Frederick F. Kautz IV
ebf1e784b3 Adding iodine and tests for for list buckets errors 2015-04-03 18:29:31 -07:00
Frederick F. Kautz IV
5865295f5d Adding better error support to api 2015-04-03 18:06:30 -07:00
Frederick F. Kautz IV
8757ccf9a3 Fixing race condition in api_test.go 2015-04-01 20:30:57 -07:00
Harshavardhana
85e547d854 Parse response headers in tests 2015-04-01 20:07:51 -07:00
Harshavardhana
92f408af5d Add erasure to godep 2015-04-01 16:56:43 -07:00
Frederick F. Kautz IV
818b6d2e33 Minor cleanup of memory get partial object 2015-04-01 15:50:59 -07:00
Frederick F. Kautz IV
0c5ed6f3ba Add partial object support to donut driver 2015-04-01 15:48:12 -07:00
Frederick F. Kautz IV
563a914046 Fix whitespace 2015-04-01 12:30:58 -07:00
Frederick F. Kautz IV
83f0f0971f Adding comments on why we subtract a second from date1 in an api test 2015-04-01 12:29:08 -07:00
Frederick F. Kautz IV
1f699a094b Minor modification for creating temp dir in api test 2015-04-01 12:07:33 -07:00
Frederick F. Kautz IV
5b1350b4bf api test now states which driver is currently running 2015-04-01 12:06:22 -07:00
Harshavardhana
a56098b8c9 Make sure in API we handle all error type exclusively, and return certain values exclusively 2015-03-31 20:26:16 -07:00
Frederick F. Kautz IV
6136960cfd Moving api_test to fixtures for drivers 2015-03-31 19:53:25 -07:00
Frederick F. Kautz IV
959d90fcf7 Adding test for partial objects 2015-03-31 19:07:45 -07:00
Frederick F. Kautz IV
62c4c997c2 Simplifying error handling in api bucket handlers 2015-03-30 14:27:32 -07:00
Frederick F. Kautz IV
318d9d5aa8 API Bucket Handlers errors simplified 2015-03-30 11:40:28 -07:00
Frederick F. Kautz IV
7d657eb55c Simplifying error response writing for object handlers 2015-03-29 22:03:13 -07:00
Frederick F. Kautz IV
c3cda5d9a8 Adding comments to errors 2015-03-29 22:03:13 -07:00
Frederick F. Kautz IV
ded090a8b0 Normalizing imports in api 2015-03-29 21:41:38 -07:00
Frederick F. Kautz IV
8cbc1c2346 More api cleanup 2015-03-29 21:38:23 -07:00
Frederick F. Kautz IV
452f0f8335 Cleaning up more api code 2015-03-29 21:06:51 -07:00