Commit Graph

1811 Commits

Author SHA1 Message Date
Harshavardhana
9977888972 Merge pull request #698 from harshavardhana/pr_out_implement_metadata_cache_metadata_cache_is_used_by_top_level_donut_right_now_rename_trove_as_data_cache
Implement metadata cache, metadata cache is used by top level donut right now. Rename trove as data cache
2015-07-04 04:12:39 +00:00
Harshavardhana
0a827305ad Implement metadata cache, metadata cache is used by top level donut right now. Rename trove as data cache
We should use it internally everywhere.
2015-07-03 21:09:57 -07:00
Harshavardhana
7d2609856e Merge pull request #697 from harshavardhana/pr_out_make_donut_do_everything_as_an_atomic_operation_this_avoids_all_the_deadlocks_and_races
Make donut do everything as an atomic operation, this avoids all the deadlocks and races
2015-07-04 00:18:45 +00:00
Harshavardhana
14844f48dd Make donut do everything as an atomic operation, this avoids all the deadlocks and races 2015-07-03 17:16:58 -07:00
Harshavardhana
86bcfed2da Merge pull request #696 from minio/server-cleanup
Server cleanup
2015-07-03 22:58:08 +00:00
Harshavardhana
30fc14e703 Restructure codebase move crypto, checksum to top-level, move `split` into donut, move crypto/keys into api/auth 2015-07-03 15:24:51 -07:00
Harshavardhana
8a4e7bcdcf Add full API tests, move storage/donut to donut, add disk tests as well 2015-07-03 14:36:29 -07:00
Harshavardhana
7c37e9d06a Make donut fully integrated back into API handlers 2015-07-02 21:04:04 -07:00
Harshavardhana
12bde7df30 Add simple Ticket Master which pro-actively sends messages on proceedChannel
Handlers are going to wait on proceedChannel, this the initial step towards
providing priority for different set of API operations
2015-07-02 21:04:04 -07:00
Harshavardhana
5cfb05465e Add cache, donut tests separately - fix behavior differences
Remove priority queue, implement it using a simpler channels
2015-07-02 21:04:04 -07:00
Harshavardhana
ebe61d99d9 Use cache Append() for saving objects in memory, GetObject() caches un-cached entries while reading 2015-07-02 21:04:04 -07:00
Harshavardhana
bce93c1b3a Integrate cache with donut, add tests 2015-07-02 21:04:04 -07:00
Harshavardhana
0533abf6a8 Make priority queue lambda function return error over a channel 2015-07-02 21:04:04 -07:00
Harshavardhana
38a6ce36e5 Remove slow AppendUniq code, rolling through over a slice is in-efficient
Remove it and use map instead
2015-07-02 21:04:04 -07:00
Harshavardhana
84810162f5 Add simple Version and GetSysInfo services 2015-07-02 21:04:04 -07:00
Harshavardhana
14ec42d646 Add initial implementation of priority queue, uses container/heap 2015-07-02 21:04:04 -07:00
Harshavardhana
eb5aa19dfa Remove custom Config, will use quick Config instead for user access keys 2015-07-02 21:04:04 -07:00
Harshavardhana
701c3e5242 Add new RPC helpers wrapping over regular rpc packages, add middleware chaining ability 2015-07-02 21:04:04 -07:00
Harshavardhana
188785a886 Add and remove dependencies 2015-07-02 21:04:04 -07:00
Harshavardhana
4addf7a996 Restructure API handlers, add JSON RPC simple HelloService right now. 2015-07-02 21:04:04 -07:00
Harshavardhana
335c7827eb More donut, cache, api cleanup 2015-07-02 21:04:04 -07:00
Harshavardhana
dc0df3dc0e Breakaway from driver model, move cache into donut 2015-07-02 21:04:03 -07:00
Harshavardhana
72572d6c71 Remove some api server code bringing in new cleanup 2015-07-02 21:04:03 -07:00
Harshavardhana
c2031ca066 Add server and control command 2015-07-02 21:04:03 -07:00
Frederick F. Kautz IV
101784bc44 Merge pull request #695 from fkautz/pr_out_fixing_api_definitions 2015-07-02 13:16:49 -07:00
Frederick F. Kautz IV
cfbc169034 Fixing API definitions 2015-07-02 13:14:21 -07:00
Harshavardhana
1d31c76dd6 Merge pull request #694 from harshavardhana/pr_out_move_memory_code_out_add_it_as_layer_on_top_of_existing_cache_code_wip 2015-06-30 17:12:29 +00:00
Harshavardhana
8f61d6b6be Move memory code out, add it as layer on top of existing donut code
Just like how http.Handlers can be overlayed on top of each other
with each implementing ServeHTTP().

drivers.Driver can be overlayed on top of each other in similar manner
which would implement the drivers.Driver interface.

   API <----> cache <----> donut <----> donut(format)
2015-06-30 10:09:12 -07:00
Harshavardhana
fe3c618cc7 Merge pull request #693 from harshavardhana/pr_out_add_dummy_driver_for_community_to_submit_new_drivers 2015-06-29 23:48:37 +00:00
Harshavardhana
ab6e16bb41 Add dummy driver for community to submit new drivers 2015-06-29 16:43:50 -07:00
Harshavardhana
12de98fb62 Rename memory driver as cache 2015-06-29 16:43:50 -07:00
Harshavardhana
2571342451 Filesystem goes the high road *again* 2015-06-29 16:43:42 -07:00
Harshavardhana
c4c67581dc Merge pull request #692 from harshavardhana/pr_out_isvalidbucket_is_sufficient_we_don_t_need_to_verify_for_ 2015-06-29 22:27:11 +00:00
Harshavardhana
f74d6138da IsValidBucket() is sufficient we don't need to verify for "." 2015-06-29 15:15:54 -07:00
Harshavardhana
b5a5861c8f Merge pull request #691 from harshavardhana/pr_out_handle_couple_of_cases_of_oom_conditions_move_caching_to_getobject_rather_than_putobject_
Handle couple of cases of OOM conditions, move caching to GetObject() rather than PutObject()
2015-06-29 19:33:55 +00:00
Harshavardhana
3109909355 Handle couple of cases of OOM conditions, move caching to GetObject() rather than PutObject() 2015-06-29 12:28:50 -07:00
Harshavardhana
d07d0c670a Return back proper errors in writeObjectData(), rename few functions 2015-06-29 11:46:35 -07:00
Harshavardhana
be816145a9 Merge pull request #690 from harshavardhana/pr_out_put_object_on_successful_write_returns_full_metadata_to_avoid_subsequent_getobjectmetadata_calls_in_driver 2015-06-29 18:21:33 +00:00
Harshavardhana
10c807f233 Put object on successful write returns full metadata, to avoid subsequent GetObjectMetadata() calls in driver 2015-06-29 11:15:46 -07:00
Harshavardhana
6921328b93 Avoid frivolous GetObjectMetadata() calls at driver level, return back all the information in donut ListObjects() 2015-06-29 11:14:58 -07:00
Harshavardhana
f05ad062ee Merge pull request #689 from harshavardhana/pr_out_expand_http_server_struct_to_store_more_values 2015-06-29 07:17:31 +00:00
Harshavardhana
d8f7896a43 Expand http server struct to store more values 2015-06-29 00:12:28 -07:00
Harshavardhana
63f9647c80 Merge pull request #688 from harshavardhana/pr_out_use_errorchannels_only_for_services_not_for_drivers_reduce_them_to_use_simple_functions 2015-06-29 07:02:24 +00:00
Harshavardhana
42c0287943 Use errorChannels only for services not for drivers, reduce them to use simple functions 2015-06-28 23:59:47 -07:00
Harshavardhana
b2bf90afbd Merge pull request #687 from harshavardhana/pr_out_move_to_set_not_append_due_to_large_memory_reference_copy 2015-06-28 17:15:06 +00:00
Harshavardhana
91e5f648cb Move to Set() not Append() due to large memory reference copy 2015-06-28 10:13:12 -07:00
Harshavardhana
22abe1b397 Merge pull request #686 from harshavardhana/pr_out_add_free_method_for_proxyreader_to_aggressively_de_allocate_read_data_to_handle_certain_out_of_memory_conditions
Add free() method for proxyReader to aggressively de-allocate Read data, to handle certain out of memory conditions
2015-06-28 03:45:48 +00:00
Harshavardhana
ac4d8fe478 Add free() method for proxyReader to aggressively de-allocate Read data, to handle certain out of memory conditions
There are still some more out there
2015-06-27 20:43:25 -07:00
Harshavardhana
a4d20d1e75 Merge pull request #685 from harshavardhana/pr_out_add_append_method_to_trove_cache_for_appending_data_to_an_existing_key
Add Append() method to trove cache for appending data to an existing key
2015-06-28 03:28:06 +00:00
Harshavardhana
05f8654e3d Add Append() method to trove cache for appending data to an existing key
This largely avoids a large buffer copy which would accumulate inside proxyReader{}

This patch also implements "initialize()" function to init and populate data
on all the existing buckets, avoiding the redundant ListBuckets() invoked by
every API call.
2015-06-27 20:25:24 -07:00