1
0
mirror of https://github.com/minio/minio.git synced 2025-01-18 10:13:16 -05:00

117 Commits

Author SHA1 Message Date
Klaus Post
4bca62a0bd
crawler: Stream bucket usage cache data ()
Stream bucket caches to storage and through RPC calls.
2020-12-10 13:03:22 -08:00
Harshavardhana
9c53cc1b83
fix: heal multiple buckets in bulk ()
makes server startup, orders of magnitude
faster with large number of buckets
2020-12-05 13:00:44 -08:00
Klaus Post
e6ea5c2703
crawler: Missing folder heal check per set () 2020-12-01 12:07:39 -08:00
Harshavardhana
31e6f60847
fix: improve error handling in metacache () 2020-11-25 01:11:22 -08:00
Harshavardhana
73e308079a
fix: handle errors appropriately as they are wrapped () 2020-11-20 10:43:07 -08:00
Klaus Post
990d074f7d
metacache: Allow prefix filtering ()
Do listings with prefix filter when bloom filter is dirty.

This will forward the prefix filter to the lister which will make it 
only scan the folders/objects with the specified prefix.

If we have a clean bloom filter we try to build a more generally 
useful cache so in that case, we will list all objects/folders.
2020-11-18 10:44:18 -08:00
Klaus Post
e413f05397
Save listing error async ()
Since the RPC call may have to time out save an error state async 
to not hold up the listing returning.

Fixes 
2020-11-18 10:28:22 -08:00
Klaus Post
10099357b6
listcache: Wrap returned errors ()
To give an indication of where they happen
2020-11-17 09:11:59 -08:00
Klaus Post
b5a3d79bce
listobjectversions: Add shortcut for Veeam blocks ()
Add shortcut for `APN/1.0 Veeam/1.0 Backup/10.0`

It requests unique blocks with a specific prefix. We skip 
scanning the parent directory for more objects matching the prefix.
2020-11-13 16:58:20 -08:00
Klaus Post
a3017c724e
Sort directory objects correctly ()
Decode dir objects when listing and sort them correctly.
2020-11-12 13:09:34 -08:00
Harshavardhana
ca88ca753c
ignore typed errors correctly in list cache layer ()
bonus write bucket metadata cache with enough quorum

Possible fix for 
2020-11-12 09:28:56 -08:00
Klaus Post
b9277c8030
metacache: Add trashcan ()
Add trashcan that keeps recently updated lists after bucket deletion.
All caches were deleted once a bucket was deleted, so caches still running would report errors. Now they are canceled.
Fix `.minio.sys` not being transient.
2020-11-03 12:47:52 -08:00
Klaus Post
0a796505c1
metacache: Check only one disk for updates ()
Check only one disk for updates.

This will reduce IO while waiting for lists to finish.
2020-11-02 17:20:27 -08:00
Klaus Post
6135f072d2
Fix invalidated metacaches ()
* Fix caches having EOF marked as a failure.
* Simplify cache updates.
* Provide context for checkMetacacheState failures.
* Log 499 when the client disconnects.
2020-10-30 09:33:16 -07:00
Harshavardhana
4bf90ca67f
fix: handle a crash when AskDisks is set to -1 () 2020-10-29 09:25:43 -07:00
Klaus Post
bfc36aed89
Add update retry limit and compare error by string instead () 2020-10-28 13:19:53 -07:00
Klaus Post
a982baff27
ListObjects Metadata Caching ()
Design: https://gist.github.com/klauspost/025c09b48ed4a1293c917cecfabdf21c

Gist of improvements:

* Cross-server caching and listing will use the same data across servers and requests.
* Lists can be arbitrarily resumed at a constant speed.
* Metadata for all files scanned is stored for streaming retrieval.
* The existing bloom filters controlled by the crawler is used for validating caches.
* Concurrent requests for the same data (or parts of it) will not spawn additional walkers.
* Listing a subdirectory of an existing recursive cache will use the cache.
* All listing operations are fully streamable so the number of objects in a bucket no 
  longer dictates the amount of memory.
* Listings can be handled by any server within the cluster.
* Caches are cleaned up when out of date or superseded by a more recent one.
2020-10-28 09:18:35 -07:00