ListObjects Metadata Caching (#10648)

Design: https://gist.github.com/klauspost/025c09b48ed4a1293c917cecfabdf21c

Gist of improvements:

* Cross-server caching and listing will use the same data across servers and requests.
* Lists can be arbitrarily resumed at a constant speed.
* Metadata for all files scanned is stored for streaming retrieval.
* The existing bloom filters controlled by the crawler is used for validating caches.
* Concurrent requests for the same data (or parts of it) will not spawn additional walkers.
* Listing a subdirectory of an existing recursive cache will use the cache.
* All listing operations are fully streamable so the number of objects in a bucket no 
  longer dictates the amount of memory.
* Listings can be handled by any server within the cluster.
* Caches are cleaned up when out of date or superseded by a more recent one.
This commit is contained in:
Klaus Post
2020-10-28 09:18:35 -07:00
committed by GitHub
parent 51222cc664
commit a982baff27
65 changed files with 6328 additions and 742 deletions

View File

@@ -169,6 +169,11 @@ func (e InsufficientReadQuorum) Error() string {
return "Storage resources are insufficient for the read operation."
}
// Unwrap the error.
func (e InsufficientReadQuorum) Unwrap() error {
return errErasureReadQuorum
}
// InsufficientWriteQuorum storage cannot satisfy quorum for write operation.
type InsufficientWriteQuorum struct{}
@@ -176,6 +181,11 @@ func (e InsufficientWriteQuorum) Error() string {
return "Storage resources are insufficient for the write operation."
}
// Unwrap the error.
func (e InsufficientWriteQuorum) Unwrap() error {
return errErasureWriteQuorum
}
// GenericError - generic object layer error.
type GenericError struct {
Bucket string