MyFavMirrors
/
minio
mirror of https://github.com/minio/minio.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
1,821 Commits 5 Branches 503 Tags 186 MiB
Commit Graph

710 Commits

Author SHA1 Message Date
Harshavardhana eec66f195a Take all the ListObjects into bucket handlers 
Earlier the listing would wait for all the objects to be processed
this is essentially very time consuming considering even for 100,000
files.
 2015-06-25 18:04:29 -07:00
Harshavardhana 45e9d25931 Remove more bloated code - simplify 2015-06-25 13:02:08 -07:00
Harshavardhana 82dcbf262d Add simple locking for donut API for now - fixes #671 2015-06-25 11:29:11 -07:00
Harshavardhana 03b4d3b308 Go vet fixes for donut 2015-06-24 21:07:03 -07:00
Harshavardhana 57a2b53178 Removing further bloated code simplifying 2015-06-24 21:03:39 -07:00
Harshavardhana a2c205ff2e Use external package disk for donut. 2015-06-24 21:03:02 -07:00
Harshavardhana 841ff01412 Move disk into its own package, remove bloated code 2015-06-24 20:13:47 -07:00
Harshavardhana 1682c748ac Remove unnecessary interfaces from donut, cleanup 2015-06-24 19:43:38 -07:00
Harshavardhana 3498872467 Add sha256 and sha512 windows compatibility layer 2015-06-24 14:39:41 -07:00
Harshavardhana 77d35b87d4 Fix a bug on windows regarding blockSSE3 calculation 2015-06-24 14:24:33 -07:00
Harshavardhana f1410731db Add windows code for sha1 and crc32c 2015-06-24 14:16:32 -07:00
Harshavardhana ba2d3dea74 Fix wrong TMPFS listing in donut filesystem map 2015-06-23 23:16:06 -07:00
Harshavardhana e818bc7187 Trim iodine path properly, so that now errors have github.com/minio/minio prefixed 2015-06-23 13:36:54 -07:00
Harshavardhana 2fd52ca284 Add missing strongly typed errors for Donut 2015-06-23 11:54:44 -07:00
Harshavardhana 7a060110ff Fix BUILDDEPS paths for golang installation 2015-06-22 12:07:23 -07:00
Harshavardhana 3bf64f5669 Verify d.donut to be non-nil, usually happens when multiple go versions compilations are linked - possible cause for #659 2015-06-20 11:03:17 -07:00
Harshavardhana 641f07cecf Use filepath everywhere instead of path.{} functions for portability - fixes #656 2015-06-18 16:02:45 -07:00
Harshavardhana 573a6134b2 Remove redundant !ok for map 2015-06-17 22:36:46 -07:00
Harshavardhana e9a3fd677a Hold lock on GetGlobalStateKey() - fixes #652 2015-06-17 20:37:49 -07:00
Harshavardhana b793f53d48 Minior change filter() to filterObjects() 2015-06-17 20:35:44 -07:00
Harshavardhana d4e71beb54 Add time.Since on buckets created time to avoid deleting buckets spuriously during eviction of objects 2015-06-13 11:28:04 -07:00
Harshavardhana 636788997b Minor optimization avoiding one unnecessary copy 2015-06-11 14:23:05 -07:00
Harshavardhana 6d44274db6 Make sure to reply back ETag with quotes for s3 compliance 2015-06-09 01:13:02 -07:00
Harshavardhana a947767339 Remove unnecessary tlsHelpers, SSL works with domain certificate 2015-06-08 20:54:41 -07:00
Harshavardhana c6ecd94420 Enable TLS and use it 2015-06-08 20:10:59 -07:00
Harshavardhana 64b014369c Return back entity too large for createObject requests bigger than cache size 2015-06-08 17:10:07 -07:00
Harshavardhana ab1976bc14 Load from persisted multipart session, now multipart uploads can resume even after restarting minio fs server 2015-06-08 15:15:37 -07:00
Harshavardhana 2976f109df More multipart fixes, filter out parts and multipart metadata from listObjects 2015-06-08 14:08:36 -07:00
Harshavardhana e08d59be00 Make sure to delete uploadid's from active session file properly 2015-06-08 12:25:51 -07:00
Harshavardhana 0d3710cb9b Store in json file instead of gob for fs metadata 2015-06-08 12:08:01 -07:00
Harshavardhana 8efc842b59 Add delete handlers and reply back as 'NotImplemented' instead of 404 2015-06-08 11:06:06 -07:00
Harshavardhana 8a320c4447 Enhance logging handler to print in humanize form for ContentLength 2015-06-08 02:14:09 -07:00
Harshavardhana cefd740002 Put back the older conn_limit 2015-06-06 18:11:38 -07:00
Harshavardhana 371651bde6 Add rate limiter instead of connection limit for now 2015-06-06 18:04:55 -07:00
Harshavardhana 21916dc0a1 Fix a deadlock in trove when evicting objects when we reach the maxSize used 2015-06-06 16:02:52 -07:00
Harshavardhana 8b43655d9f Make connection limit configurable 2015-06-06 14:31:50 -07:00
Harshavardhana a344f5b34d Misc changes before capturing proper video 
- Disable management UI command option
- Remove featureflags option, filesystem and memory already support multipart
- Print informative messages after starting minio server
 2015-06-03 18:27:15 -07:00
Harshavardhana 097f70a3c5 Enable filesystem for API tests, this patch also implements ListMultipartUploads() 2015-06-01 20:57:13 -07:00
Harshavardhana bfb9209689 Skip files and weird directories from listBuckets(), allowing this so that minio fs mode can be used as ftp service 2015-06-01 16:43:43 -07:00
Harshavardhana d86ba5f336 Implement multipart support 
- implemented NewMultipartUpload()
 - implemented CreateObjectPart()
 - implemented ListObjectParts()
 - implemented AbortMultipartUpload()
 - implemented CompleteMultipartUpload()
 2015-06-01 16:25:11 -07:00
Harshavardhana c07d48e7e8 ErrorResponse from minio server needed proper xml translations 2015-06-01 12:19:27 -07:00
Harshavardhana a91e519692 Make caching a package trove, and use it inside memory driver 2015-05-31 18:41:56 -07:00
Anis Elleuch 7e5a26f42c Protect the exported method Delete() 2015-05-31 20:29:42 +01:00
Harshavardhana 82a0eac659 Add filesystem factory functions and add related cli options 2015-05-30 03:23:06 -07:00
Harshavardhana 0cc63706bb Resurrect back fileystem driver 2015-05-30 03:12:01 -07:00
Anis Elleuch d360d0fec3 Forbid the upload of files bigger than the memory backend capacity 2015-05-28 22:07:02 +01:00
Harshavardhana 554a244b47 Add donut OSX support 2015-05-26 02:14:34 -07:00
Harshavardhana e46660a966 Use http.TimeFormat to return, S3 expects us to return in GMT not UTC 2015-05-25 20:17:04 -07:00
Harshavardhana 19250296c6 Handle authorization header better 2015-05-24 20:59:53 -07:00
Harshavardhana 9666f2e5bf Relax minium object size limit, one can really upload a 0byte file 
For example
~~~
$ mc ls https://s3.amazonaws.com/ferenginar/test.go
[2015-05-23 12:02:06 PDT]     0B test.go

$ curl -i -X HEAD https://s3.amazonaws.com/ferenginar/test.go
HTTP/1.1 200 OK
x-amz-id-2: ZMWLriPH+uQJ8IsaMHmuNOU/FXvdSq+s6O7ugUI9hZ695XJTNAZ1utKxh03w5Jcf
x-amz-request-id: F5A11F533B74DD8F
Date: Sat, 23 May 2015 19:19:29 GMT
Last-Modified: Sat, 23 May 2015 19:02:06 GMT
ETag: "d41d8cd98f00b204e9800998ecf8427e"
Accept-Ranges: bytes
Content-Type: binary/octet-stream
Content-Length: 0
Server: AmazonS3
~~~
 2015-05-23 12:19:16 -07:00
Harshavardhana e11f9110b6 add missing validate access keys after being extracted from AuthHeader 2015-05-21 19:23:47 -07:00
Harshavardhana ad60b97d52 Fix some donut TODO's regarding bucket created date 2015-05-21 11:13:49 -07:00
Harshavardhana cd817d34df Finish last bits of api testsuite on server 2015-05-21 02:57:03 -07:00
Harshavardhana f00bfd38da Bump gocyclo back to 16 2015-05-19 18:32:50 -07:00
Harshavardhana 3d61ba5930 Fix duplicate map allocation 2015-05-18 13:14:01 -07:00
Harshavardhana e4c0d574dc Update totalnumber of multiparts inside CreateObjectPart(), also introduce support for UploadIDMarker and Prefix for listMultipartUploads 2015-05-18 12:24:36 -07:00
Harshavardhana 7ce47130fd HEAD request should have Content-Length for only successful response, there is no response body for errors, just header is sufficient - fixes #603 
http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.13

"""  in the case of the HEAD method, the size of the entity-body that would have been sent had the request been a GET. """
 2015-05-15 21:55:59 -07:00
Harshavardhana 317637fcb8 Remove redundant WriteHeader calls 2015-05-15 20:58:28 -07:00
Harshavardhana 5c5761ae16 Remove unnecessary GetObjectMetadata prefix argument 2015-05-15 19:25:17 -07:00
Harshavardhana 6275238a47 For multiparts use its own cache avoids deadlocks during object expiration 
Using same object cache as regular objects for multiparts causes deadlocks during object expiration routine.

As a side affect this also fixes the bug of listing multiparts during regular 'listing' of objects.
 2015-05-14 20:26:59 -07:00
Harshavardhana c41ebf7535 Fixing a race in NewMultipartUpload() session for simultaneous requests, observed at play.minio.io 2015-05-14 19:11:19 -07:00
Harshavardhana 8fb427a892 Add more tests for multipart now in driver api_testsuite 2015-05-14 17:28:39 -07:00
Harshavardhana dba7c94571 Add tests for abortMultipartUpload, listobjectparts, listmultipartuploads 2015-05-14 17:07:30 -07:00
Harshavardhana 1bd94ec6ab An attempt to implement ListMultipartUploads() 2015-05-14 16:22:47 -07:00
Harshavardhana 82b043005a Avoid one last memory copy 2015-05-14 11:26:55 -07:00
Harshavardhana 5498c90788 Add a new validateContentTypeHandler{}, verify Accept header earlier 2015-05-13 12:22:15 -07:00
Harshavardhana de56909d55 More scripts and code from Minio-io to minio 2015-05-11 16:43:38 -07:00
Harshavardhana 8d06504068 Change all minio-io path to minio 2015-05-11 16:23:10 -07:00
Harshavardhana e39de4f603 Fix typo introduced in previous commit 2015-05-11 14:24:31 -07:00
Harshavardhana f45635524f Remove SignatureV2 support, bring in SignatureV4 header only validation for now 2015-05-11 13:38:40 -07:00
Harshavardhana b48b7f51ef Use convenience functions for url.Values over looping through keys, fix a bug in listObjectParts() 2015-05-10 11:32:41 -07:00
Harshavardhana 14b5828505 Handle partNumberMarker with listObjectParts now and other fixes 2015-05-09 19:39:00 -07:00
Frederick F. Kautz IV cf559fa48e Limiting upload id size 2015-05-09 18:28:50 -07:00
Frederick F. Kautz IV 61a9ea0f4b Adding support for memory driver test 2015-05-09 18:15:27 -07:00
Frederick F. Kautz IV 058604aa21 Adding tests and fixes for multipart uploads uncovered from tests 2015-05-09 17:42:14 -07:00
Harshavardhana 8860aa0c8c Implement AbortMultipart 2015-05-09 16:09:50 -07:00
Harshavardhana 18c7f620cc Add listparts support 2015-05-09 13:55:55 -07:00
Harshavardhana 04e388d620 Handle two different styles of time format, s3cmd now compatible 2015-05-08 16:01:59 -07:00
Harshavardhana 01b474fca3 Move expiration code into its own and add ticker select case 2015-05-08 12:37:49 -07:00
Harshavardhana 23a545673d Verify incoming md5sum for invidual parts in complete multipart request and some simplification of the code 
Add two new functions for session cleanup
 2015-05-08 03:59:05 -07:00
Harshavardhana 616241a2c1 Re-enable bucket deletion, this time with uploadID sitting there 2015-05-08 02:28:38 -07:00
Harshavardhana 2ea10c798b Implement proper errors for Multipart 2015-05-08 02:02:51 -07:00
Harshavardhana 82c3656f79 Reply back CompleteMultipartUploadResult properly with final ETag computed 
- Now s3 libraries and also objectstorage-go work properly
 2015-05-07 23:01:20 -07:00
Frederick F. Kautz IV ae5c99d7bb Router should work with ?uploads unescaped now 2015-05-07 21:37:34 -07:00
Frederick F. Kautz IV 9dc7b82b39 Merge pull request #576 from fkautz/pr_out_adding_multipart_support 2015-05-07 20:00:20 -07:00
Frederick F. Kautz IV f050d5e974 Adding multipart support 2015-05-07 19:55:30 -07:00
Frederick F. Kautz IV ed71c28c37 Fixing panic in intelligent, attempted to delete nonexistent key 2015-05-07 19:12:49 -07:00
Frederick F. Kautz IV d0c4334834 Adding feature flags 2015-05-07 13:56:54 -07:00
Harshavardhana 1df28f3060 Get rid of sub-domain code and some other formatting changes 2015-05-07 00:21:40 -07:00
Harshavardhana 6d3d92694f Add Marker support for donut, also add a testing code 2015-05-05 22:18:24 -07:00
Harshavardhana 75028c2ad1 Add lexicographic Marker/NextMarker support for recursive listing of objects. 
Also update times when an object is accessed logic
 2015-05-05 18:05:34 -07:00
Frederick F. Kautz IV 24af2222c9 Adding xmlns 2015-05-05 12:54:34 -07:00
Harshavardhana 193a6606db Fix a potential race between ExpireObjects() and Set() over items map 2015-05-04 22:36:47 -07:00
Harshavardhana ed1259d6f0 Deprecate LRU use just map[string]interface{} and call it intelligent 'cache' 2015-05-04 22:01:56 -07:00
Harshavardhana 63edb1e9a0 Nullify list memory to nil as we remove the element 2015-05-04 17:22:08 -07:00
Harshavardhana d63064b8af Optimize memory usage in GetPartialObject() for memory driver 2015-05-04 15:26:56 -07:00
Harshavardhana c8f31d97a8 Modify LRU further to add object expiration 2015-05-04 04:32:44 -07:00
Harshavardhana 670f997b07 Bring blockingWriter from client and use it here 2015-05-04 03:55:43 -07:00
Harshavardhana d0df548eb5 Use new LRU inside memory driver 2015-05-04 03:55:43 -07:00
Harshavardhana f7caef2d26 Change CreateObject() to take size argument from content-length 2015-05-04 03:55:43 -07:00
Harshavardhana c342ce1588 New LRU based on GroupCache which keeps track of memory rather than entries 2015-05-04 03:55:19 -07:00
Frederick F. Kautz IV 3cf045fd1b Deleting key from lastAccessedObjects on eviction 2015-05-01 22:01:07 -07:00
Harshavardhana 9ec8925f1c Rename IsValidObject() to IsValidObjectName() 2015-05-01 20:31:26 -07:00
Frederick F. Kautz IV 33ecba8e49 Reimagining memory settings cli input 2015-05-01 18:12:58 -07:00
Frederick F. Kautz IV 25c268ac88 Attempt to fix memory lock on Get preventing write 2015-05-01 17:45:40 -07:00
Frederick F. Kautz IV 85f1b46492 Removing log message from test 2015-05-01 17:32:18 -07:00
Frederick F. Kautz IV 7cf86e6114 Fix docker support 2015-05-01 15:23:39 -07:00
Harshavardhana 2c7b30e5e1 Even mux vars don't help, handle it by looking at req.URL.Path 2015-05-01 12:41:17 -07:00
Frederick F. Kautz IV b86b26e7e5 Exposing expiration for memory driver 2015-04-30 21:20:54 -07:00
Harshavardhana c01fb92eca Merge pull request #557 from harshavardhana/pr_out_make_builddate_universal 2015-04-30 21:19:04 -07:00
Harshavardhana bbc32d6ac0 Disable domain based routing for now 2015-04-30 21:15:53 -07:00
Harshavardhana f3078d047c Get bucketName from mux.Vars instead of heuristic code, resolves subdomain issues 2015-04-30 21:11:32 -07:00
Frederick F. Kautz IV e911cc332e Merge pull request #554 from fkautz/pr_out_disable_eviction_on_0_expiration_time 2015-04-30 19:40:46 -07:00
Harshavardhana 8e3d48bf35 Use UTC() everywhere 
ref - https://github.com/golang/go/issues/1988, no standard function to set timeZone
 2015-04-30 19:38:32 -07:00
Frederick F. Kautz IV 94e24f1a05 Disable eviction on 0 expiration time 2015-04-30 19:34:35 -07:00
Harshavardhana 72395798c8 Merge pull request #552 from harshavardhana/pr_out_minio_config_directory_should_be_0700_not_os_modedir 2015-04-30 17:40:15 -07:00
Harshavardhana 2de81c8a3c Minio config directory should be 0700 not os.ModeDir 2015-04-30 15:42:49 -07:00
Frederick F. Kautz IV 8e2f7f9493 Wrapping functions that require gc so that references are released before running 2015-04-30 12:38:12 -07:00
Harshavardhana d815e6adfd Changes to CreateObject() now returns back md5 along with any error 
- This change is necessary to avoid the racy calls to GetObjectMetadata()
- This change is also necessary since one has to reply back md5sum with
  PUT object response header
 2015-04-30 03:38:14 -07:00
Harshavardhana fd2203b1b7 Some more improvements to connection limit 2015-04-30 01:23:44 -07:00
Harshavardhana f8a16dd22b Memory now evicts bucket if no more objects in memory struct 
- To avoid race in expiration while accessing memory driver structs with
  in two competing write locks. Use lru Len() instead to know exact length
  to schedule for eviction.

- squash both bucket and object structs, instead use a separate map to keep
  mutable info of lastAccessTime which can be independently used inside the
  expiration routine.
 2015-04-30 00:31:52 -07:00
Frederick F. Kautz IV ed0a9a9e06 Removing entity too large errors from memory driver 2015-04-30 00:09:54 -07:00
Frederick F. Kautz IV aff2b3b06a Adding debug statements for testing 2015-04-29 21:13:32 -07:00
Frederick F. Kautz IV 82a16b8848 Make minio work with curl and browsers again 2015-04-29 20:17:26 -07:00
Frederick F. Kautz IV 3a1386165f Fixing race condition 2015-04-29 18:54:59 -07:00
Harshavardhana 03bca9a001 Fix out of memory issue 2015-04-29 18:33:19 -07:00
Harshavardhana 25d4c0f6fa A small logical change in messaging and logging 2015-04-29 17:30:54 -07:00
Frederick F. Kautz IV f596c744ed Adding error message on connection limit met 2015-04-29 17:21:23 -07:00
Harshavardhana a89c4a57d3 Return resources as its available, not a new value 2015-04-29 16:47:14 -07:00
Harshavardhana 251c99176a Merge pull request #537 from harshavardhana/pr_out_add_proper_content_length_for_error_and_success_responses 2015-04-29 15:32:12 -07:00
Frederick F. Kautz IV 0da04c6f17 Expirations are now based on last accessed instead of time created 2015-04-29 15:28:45 -07:00
Harshavardhana 92e4301414 Add proper content-length for error and success responses 
- All compliance issues with S3 API for Put,Get,List (Bucket,Object) respectively
 - Encodes and returns back proper HTTP headers
 2015-04-29 15:28:28 -07:00
Frederick F. Kautz IV 5fc5f79ab7 Adding lru 2015-04-29 15:08:26 -07:00
Frederick F. Kautz IV 8fcb697844 Simplifying memory sleep logic 2015-04-29 13:53:09 -07:00
Frederick F. Kautz IV 5cc9418ca7 Adding expiration 2015-04-29 13:02:21 -07:00
Frederick F. Kautz IV a608524ebf api logging is now in its own package 2015-04-29 12:24:53 -07:00
Harshavardhana d1d5f7a78d Collate success response into writeSuccessResponse(), add docs 2015-04-29 10:52:04 -07:00
Harshavardhana b18bb230da maxObjectSize and minObjectSize limitation added at putObjectHandler() 
Put() replies back with

   - EntityTooLarge with > 5GB in single PUT operation
   - EntityTooSmall with < 1B in single PUT operation
   - IncompleteBody with ho Content-Length found in HTTP request header
 2015-04-29 02:19:53 -07:00
Frederick F. Kautz IV 172cd18455 Adding initial logging framework to api_router 2015-04-28 20:37:08 -07:00
Frederick F. Kautz IV 88cca647ea Adding json logging to stderr 2015-04-28 18:11:53 -07:00
Frederick F. Kautz IV de091068e8 Removing quotas, will handle with iptables 2015-04-28 12:14:34 -07:00
Frederick F. Kautz IV f2257a643c Critical sections are now protected by lock 2015-04-27 19:59:58 -07:00
Harshavardhana 605edbc307 Add some logging with debug 2015-04-27 19:26:24 -07:00
Harshavardhana 7bdcf4bbf3 Merge pull request #515 from fkautz/pr_out_fix_multiple_response_writeheader_calls 
Fix multiple response.WriteHeader calls
 2015-04-27 19:21:47 -07:00
Frederick F. Kautz IV db5870be28 Fix multiple response.WriteHeader calls 2015-04-27 18:30:21 -07:00
Frederick F. Kautz IV fc2537eef9 Merge pull request #513 from fkautz/pr_out_adding_software_fallback 2015-04-27 14:41:41 -07:00
Frederick F. Kautz IV 289c859675 Adding software fallback 2015-04-27 13:55:40 -07:00
Harshavardhana ade803a923 Verify if request date is 5minutes late, reject such a request as 
it could be a possible replay attack.

This commit also fixes #505, by returning MethodNotAllowed instead of NotImplemented
 2015-04-27 03:55:10 -07:00
Harshavardhana 9232ce3b4e Now client requests for ACL changes are honored through PutBucketACL API 2015-04-27 03:04:29 -07:00
Harshavardhana c99d96dbc2 Now donut supports bucket ACL's, bringing in this support for dl.minio.io 2015-04-27 02:09:32 -07:00