Avoid excessive listing attempts in the daily sweep (#8081)

Add better dynamic timeouts for locks, also
add jitters before launching daily sweep to ensure
that not all the servers in distributed setup
are not trying to hold locks to begin the sweep
round.

Also, add enough delay for incoming requests based
on totalSetCount*totalDriveCount.

A possible fix for #8071
This commit is contained in:
Harshavardhana
2019-08-19 08:22:32 -10:00
committed by GitHub
parent 60f52f461f
commit b3ca304c01
3 changed files with 25 additions and 13 deletions

View File

@@ -103,14 +103,12 @@ func startDailyLifecycle() {
}
}
var lifecycleTimeout = newDynamicTimeout(60*time.Second, time.Second)
func lifecycleRound(ctx context.Context, objAPI ObjectLayer) error {
zeroDuration := time.Millisecond
zeroDynamicTimeout := newDynamicTimeout(zeroDuration, zeroDuration)
// Lock to avoid concurrent lifecycle ops from other nodes
sweepLock := globalNSMutex.NewNSLock(ctx, "system", "daily-lifecycle-ops")
if err := sweepLock.GetLock(zeroDynamicTimeout); err != nil {
if err := sweepLock.GetLock(lifecycleTimeout); err != nil {
return err
}
defer sweepLock.Unlock()