metrics: Add the number of requests in the waiting queue (#11580)

We can use this metric to check if there are too many S3 clients in the
queue and could explain why some of those S3 clients are timing out.

```
minio_s3_requests_waiting_total{server="127.0.0.1:9000"} 9981
```

If max_requests is 10000 then there is a strong possibility that clients
are timing out because of the queue deadline.
This commit is contained in:
Anis Elleuch
2021-02-20 09:21:55 +01:00
committed by GitHub
parent 173284903b
commit 98d3f94996
4 changed files with 27 additions and 4 deletions

View File

@@ -136,20 +136,25 @@ func maxClients(f http.HandlerFunc) http.HandlerFunc {
return
}
globalHTTPStats.addRequestsInQueue(1)
deadlineTimer := time.NewTimer(deadline)
defer deadlineTimer.Stop()
select {
case pool <- struct{}{}:
defer func() { <-pool }()
globalHTTPStats.addRequestsInQueue(-1)
f.ServeHTTP(w, r)
case <-deadlineTimer.C:
// Send a http timeout message
writeErrorResponse(r.Context(), w,
errorCodes.ToAPIErr(ErrOperationMaxedOut),
r.URL, guessIsBrowserReq(r))
globalHTTPStats.addRequestsInQueue(-1)
return
case <-r.Context().Done():
globalHTTPStats.addRequestsInQueue(-1)
return
}
}