Improve expiration of tiered objects (#18926)

- Use a shared worker pool for all ILM expiry tasks
- Free version cleanup executes in a separate goroutine
- Add a free version only if removing the remote object fails
- Add ILM expiry metrics to the node namespace
- Move tier journal tasks to expiryState
- Remove unused on-disk journal for tiered objects pending deletion
- Distribute expiry tasks across workers such that the expiry of versions of
  the same object serialized
- Ability to resize worker pool without server restart
- Make scaling down of expiryState workers' concurrency safe; Thanks
  @klauspost
- Add error logs when expiryState and transition state are not
  initialized (yet)
* metrics: Add missed tier journal entry tasks
* Initialize the ILM worker pool after the object layer

internal/config/errors.go

@@ -224,6 +224,11 @@ Examples:
 		"",
 		"MINIO_API_TRANSITION_WORKERS: should be >= GOMAXPROCS/2",
 	)
+	ErrInvalidExpiryWorkersValue = newErrFn(
+		"Invalid value for expiry workers",
+		"",
+		"MINIO_API_EXPIRY_WORKERS: should be between 1 and 500",
+	)
 	ErrInvalidBatchKeyRotationWorkersWait = newErrFn(
 		"Invalid value for batch key rotation workers wait",
 		"Please input a non-negative duration",