From 3bc39db34e90da666b07caa2bc38c4ab32e19fa7 Mon Sep 17 00:00:00 2001 From: Andrea Longo Date: Mon, 29 Jul 2024 12:48:51 -0600 Subject: [PATCH] Restructure metrics v3 readme for docs use (#20114) --- docs/metrics/prometheus/list.md | 8 +- docs/metrics/v3.md | 668 ++++++++++++++++---------------- 2 files changed, 348 insertions(+), 328 deletions(-) diff --git a/docs/metrics/prometheus/list.md b/docs/metrics/prometheus/list.md index 51d3b897e..d94137d37 100644 --- a/docs/metrics/prometheus/list.md +++ b/docs/metrics/prometheus/list.md @@ -157,9 +157,9 @@ For deployments with [bucket](https://min.io/docs/minio/linux/administration/buc | Name | Description | |:-----------------------------------------------|:--------------------------------------------------------------------------------------------------------------------------------------------| -| `minio_notify_current_send_in_progress` | Number of concurrent async Send calls active to all targets (deprecated, please use 'minio_notify_target_current_send_in_progress' instead) | -| `minio_notify_events_errors_total` | Events that were failed to be sent to the targets (deprecated, please use 'minio_notify_target_failed_events' instead) | -| `minio_notify_events_sent_total` | Total number of events sent to the targets (deprecated, please use 'minio_notify_target_total_events' instead) | +| `minio_notify_current_send_in_progress` | Number of concurrent async Send calls active to all targets (deprecated, please use `minio_notify_target_current_send_in_progress` instead) | +| `minio_notify_events_errors_total` | Events that were failed to be sent to the targets (deprecated, please use `minio_notify_target_failed_events` instead) | +| `minio_notify_events_sent_total` | Total number of events sent to the targets (deprecated, please use `minio_notify_target_total_events` instead) | | `minio_notify_events_skipped_total` | Events that were skipped to be sent to the targets due to the in-memory queue being full | | `minio_notify_target_current_send_in_progress` | Number of concurrent async Send calls active to the target | | `minio_notify_target_queue_length` | Number of events currently staged in the queue_dir configured for the target. | @@ -254,7 +254,7 @@ For deployments with [bucket](https://min.io/docs/minio/linux/administration/buc | `minio_node_io_read_bytes` | Total bytes read by the process from the underlying storage system, /proc/[pid]/io read_bytes. | | `minio_node_io_wchar_bytes` | Total bytes written by the process to the underlying storage system including page cache, /proc/[pid]/io wchar. | | `minio_node_io_write_bytes` | Total bytes written by the process to the underlying storage system, /proc/[pid]/io write_bytes. | -| `minio_node_process_cpu_total_seconds` | Total user and system CPU time spent in seconds. | +| `minio_node_process_cpu_total_seconds` | Total user and system CPU time spent in seconds by the process. | | `minio_node_process_resident_memory_bytes` | Resident memory size in bytes. | | `minio_node_process_virtual_memory_bytes` | Virtual memory size in bytes. | | `minio_node_process_starttime_seconds` | Start time for MinIO process per node, time in seconds since Unix epoc. | diff --git a/docs/metrics/v3.md b/docs/metrics/v3.md index 8c2d0e40f..8cbf46517 100644 --- a/docs/metrics/v3.md +++ b/docs/metrics/v3.md @@ -1,397 +1,417 @@ # Metrics Version 3 -In metrics version 3, all metrics are available under the endpoint: +In metrics version 3, all metrics are available under the following base endpoint: ``` /minio/metrics/v3 ``` -however, a specific path under this is required. +To query metrics of a specific type, append the appropriate path to the base endpoint. +Querying the base endpoint returns "404 Not Found." Metrics are organized into groups at paths **relative** to the top-level endpoint above. +Metrics are also available using the [MinIO Admin Client](https://min.io/docs/minio/linux/reference/minio-mc-admin.html) and the `mc admin prometheus metrics` command. For more information, see [Metrics and Alerts](https://min.io/docs/minio/linux/operations/monitoring/metrics-and-alerts.html) in the MinIO Documentation. + ## Metrics Request Handling -Each endpoint below can be queried at different intervals as needed via a scrape configuration in Prometheus or a compatible metrics collection tool. +Each endpoint can be queried as needed via a scrape configuration in Prometheus or a compatible metrics collection tool. You should schedule scrape operations so a prior scrape completes before the next one begins. -For ease of configuration, each (non-empty) parent of the path serves all metric endpoints that are at descendant paths. For example, to query all system metrics one needs to only scrape `/minio/metrics/v3/system/`. +For ease of configuration, each (non-empty) parent of the path serves all the metric endpoints at its child paths. For example, to query all system metrics scrape `/minio/metrics/v3/system/`. -Some metrics are bucket specific. These will have a `/bucket` component in their path. As the number of buckets can be large, the metrics scrape operation needs to be provided with a specific list of buckets via the `bucket` query parameter. Only metrics for the given buckets will be returned (with the bucket label set). For example to query API metrics for buckets `test1` and `test2`, make a scrape request to `/minio/metrics/v3/api/bucket?buckets=test1,test2`. +Each metric endpoint may support multiple child endpoints. For example, the `/v3/system/` metric has multiple child groups as `/v3/system/`. Querying the parent endpoint returns metrics for all child groups. Querying a child group returns only metrics for that child. -Instead of a metrics scrape, it is also possible to list the metrics that would be returned by a path. This is done by adding a `?list` query parameter. The MinIO server will then list all possible metrics that could be returned. During an actual metrics scrape, only available metrics are returned - not all of them. With the `list` query parameter, the output format can be selected - just set the request `Content-Type` to `application/json` for JSON output, or `text/plain` for a simple markdown formatted table. The latter is the default. +### Per-bucket Metrics -## Request, System and Cluster Metrics +Metrics with a `/bucket` component in the path return results for each specified bucket in the deployment. These endpoints **require** providing a list of buckets as a `bucket` query parameter. The endpoint then returns only metrics for the given buckets, with the bucket name in a `bucket` label. -At a high level metrics are grouped into three categories, listed in the following sub-sections. The path in each of the tables is relative to the top-level endpoint. +For example, to query API metrics for buckets `test1` and `test2`, make a scrape request to `/minio/metrics/v3/api/bucket?buckets=test1,test2`. -### Request metrics +### List Available Metrics -These are metrics about requests served by the (current) node. +Instead of a metrics scrape, you can list the metrics that would be returned by a path by adding a `list` query parameter. The MinIO server then lists all available metrics that could be returned. Note that during an actual metrics scrape only metrics with available _values_ are returned. Metrics with null values are omitted from the scrape results. -| Path | Description | -|-----------------|--------------------------------------------------| -| `/api/requests` | Metrics over all requests | -| `/bucket/api` | Metrics over all requests for a given bucket | -| | | +To set the output format, set the request `Content-Type` to the desired format. Accepted values are `application/json` for JSON output or `text/plain` for a Markdown-formatted table. The default is Markdown. + +For example, the the following returns a list of all available bucket metrics: +``` +/minio/metrics/v3/api/bucket?list +``` + +## Metric Categories + +At a high level, metrics are grouped into categories as described in the following sections. The path in each of the tables is relative to the top-level endpoint. Note that the standard GoCollector metrics are not shown. + +### Request metrics + +Metrics about requests served by the current node. + +| Path | Description | +|-----------------|-----------------------------------------------| +| `/api/requests` | Metrics over all requests. | +| `/bucket/api` | Metrics over all requests for a given bucket. | + +#### `/api/requests` + +| Name | Description | Labels | +|------------------------------------------------|--------------------------------------------------------------------------------|----------------------------------------------| +| `minio_api_requests_rejected_auth_total` | Total number of requests rejected for auth failure.

Type: counter | `type`, `pool_index`, `server` | +| `minio_api_requests_rejected_header_total` | Total number of requests rejected for invalid header.

Type: counter | `type`, `pool_index`, `server` | +| `minio_api_requests_rejected_timestamp_total` | Total number of requests rejected for invalid timestamp.

Type: counter | `type`, `pool_index`, `server` | +| `minio_api_requests_rejected_invalid_total` | Total number of invalid requests.

Type: counter | `type`, `pool_index`, `server` | +| `minio_api_requests_waiting_total` | Total number of requests in the waiting queue.

Type: gauge | `type`, `pool_index`, `server` | +| `minio_api_requests_incoming_total` | Total number of incoming requests.

Type: gauge | `type`, `pool_index`, `server` | +| `minio_api_requests_inflight_total` | Total number of requests currently in flight.

Type: gauge | `name`, `type`, `pool_index`, `server` | +| `minio_api_requests_total` | Total number of requests.

Type: counter | `name`, `type`, `pool_index`, `server` | +| `minio_api_requests_errors_total` | Total number of requests with 4xx or 5xx errors.

Type: counter | `name`, `type`, `pool_index`, `server` | +| `minio_api_requests_5xx_errors_total` | Total number of requests with 5xx errors.

Type: counter | `name`, `type`, `pool_index`, `server` | +| `minio_api_requests_4xx_errors_total` | Total number of requests with 4xx errors.

Type: counter | `name`, `type`, `pool_index`, `server` | +| `minio_api_requests_canceled_total` | Total number of requests canceled by the client.

Type: counter | `name`, `type`, `pool_index`, `server` | +| `minio_api_requests_ttfb_seconds_distribution` | Distribution of time to first byte across API calls.

Type: counter | `name`, `type`, `le`, `pool_index`, `server` | +| `minio_api_requests_traffic_sent_bytes` | Total number of bytes sent.

Type: counter | `type`, `pool_index`, `server` | +| `minio_api_requests_traffic_received_bytes` | Total number of bytes received.

Type: counter | `type`, `pool_index`, `server` | + +#### `/bucket/api` + +| Name | Description | Labels | +|----------------------------------------------|-----------------------------------------------------------------------------------------|--------------------------------------------------------| +| `minio_bucket_api_traffic_received_bytes` | Total number of bytes sent for a bucket.

Type: counter | `bucket`, `type`, `server`, `pool_index` | +| `minio_bucket_api_traffic_sent_bytes` | Total number of bytes received for a bucket.

Type: counter | `bucket`, `type`, `server`, `pool_index` | +| `minio_bucket_api_inflight_total` | Total number of requests currently in flight for a bucket.

Type: gauge | `bucket`, `name`, `type`, `server`, `pool_index` | +| `minio_bucket_api_total` | Total number of requests for a bucket.

Type: counter | `bucket`, `name`, `type`, `server`, `pool_index` | +| `minio_bucket_api_canceled_total` | Total number of requests canceled by the client for a bucket.

Type: counter | `bucket`, `name`, `type`, `server`, `pool_index` | +| `minio_bucket_api_4xx_errors_total` | Total number of requests with 4xx errors for a bucket.

Type: counter | `bucket`, `name`, `type`, `server`, `pool_index` | +| `minio_bucket_api_5xx_errors_total` | Total number of requests with 5xx errors for a bucket.

Type: counter | `bucket`, `name`, `type`, `server`, `pool_index` | +| `minio_bucket_api_ttfb_seconds_distribution` | Distribution of time to first byte across API calls for a bucket.

Type: counter | `bucket`, `name`, `le`, `type`, `server`, `pool_index` | ### Audit metrics -These are metrics about the minio audit functionality +Metrics about the MinIO audit functionality. -| Path | Description | -|----------|----------------------------------------| -| `/audit` | Metrics related to audit functionality | +| Path | Description | +|----------|-----------------------------------------| +| `/audit` | Metrics related to audit functionality. | -### ILM metrics +#### `/audit` -These are metrics about the minio ILM functionality - -| Path | Description | -|--------|--------------------------------------| -| `/ilm` | Metrics related to ILM functionality | - -### Logger webhook metrics - -These are metrics about the minio logger webhooks - -| Path | Description | -|-------------------|------------------------------------| -| `/logger/webhook` | Metrics related to logger webhooks | - -### Notification metrics - -These are metrics about the minio notification functionality - -| Path | Description | -|----------|------------------------------------------------------| -| `/notification` | Metrics related to notification functionality | - -### Scanner metrics - -These are metrics about the minio scanner - -| Path | Description | -|------------|--------------------------------------| -| `/scanner` | Metrics related to the MinIO scanner | - -### System metrics - -These are metrics about the minio process and the node. - -| Path | Description | -|-----------------------------|---------------------------------------------------| -| `/system/drive` | Metrics about drives on the system | -| `/system/memory` | Metrics about memory on the system | -| `/system/network/internode` | Metrics about internode requests made by the node | -| `/system/process` | Standard process metrics | -| | | - -### Debug metrics - -These are metrics for debugging - -| Path | Description | -|-----------------------------|---------------------------------------------------| -| `/debug/go` | Standard Go lang metrics | -| | | +| Name | Description | Labels | +|-----------------------------------|---------------------------------------------------------------------------------|-----------------------| +| `minio_audit_failed_messages` | Total number of messages that failed to send since start.

Type: counter | `target_id`, `server` | +| `minio_audit_target_queue_length` | Number of unsent messages in queue for target.

Type: gauge | `target_id`, `server` | +| `minio_audit_total_messages` | Total number of messages sent since start.

Type: counter | `target_id`, `server` | ### Cluster metrics -These present metrics about the whole MinIO cluster. +Metrics about an entire MinIO cluster. -| Path | Description | -|--------------------------|-----------------------------| -| `/cluster/health` | Cluster health metrics | -| `/cluster/usage/objects` | Object statistics | -| `/cluster/usage/buckets` | Object statistics by bucket | -| `/cluster/erasure-set` | Erasure set metrics | -| | | +| Path | Description | +|--------------------------|--------------------------------| +| `/cluster/config` | Cluster configuration metrics. | +| `/cluster/erasure-set` | Erasure set metrics. | +| `/cluster/health` | Cluster health metrics. | +| `/cluster/iam` | Cluster iam metrics. | +| `/cluster/usage/buckets` | Object statistics by bucket. | +| `/cluster/usage/objects` | Object statistics. | -## Metrics Listing +#### `/cluster/config` -Each of the following sub-sections list metrics returned by each of the endpoints. +| Name | Description | Labels | +|----------------------------------------|--------------------------------------------------------------|--------| +| `minio_cluster_config_rrs_parity` | Reduced redundancy storage class parity.

Type: gauge | | +| `minio_cluster_config_standard_parity` | Standard storage class parity.

Type: gauge | | -The standard metrics group for GoCollector is not shown below. +#### `/cluster/erasure-set` -### `/api/requests` +| Name | Description | Labels | +|--------------------------------------------------|---------------------------------------------------------------------------------------------------------|---------------------| +| `minio_cluster_erasure_set_overall_write_quorum` | Overall write quorum across pools and sets.

Type: gauge | | +| `minio_cluster_erasure_set_overall_health` | Overall health across pools and sets (1=healthy, 0=unhealthy).

Type: gauge | | +| `minio_cluster_erasure_set_read_quorum` | Read quorum for the erasure set in a pool.

Type: gauge | `pool_id`, `set_id` | +| `minio_cluster_erasure_set_write_quorum` | Write quorum for the erasure set in a pool.

Type: gauge | `pool_id`, `set_id` | +| `minio_cluster_erasure_set_online_drives_count` | Count of online drives in the erasure set in a pool.

Type: gauge | `pool_id`, `set_id` | +| `minio_cluster_erasure_set_healing_drives_count` | Count of healing drives in the erasure set in a pool.

Type: gauge | `pool_id`, `set_id` | +| `minio_cluster_erasure_set_health` | Health of the erasure set in a pool (1=healthy, 0=unhealthy).

Type: gauge | `pool_id`, `set_id` | +| `minio_cluster_erasure_set_read_tolerance` | Number of drive failures that can be tolerated without disrupting read operations.

Type: gauge | `pool_id`, `set_id` | +| `minio_cluster_erasure_set_write_tolerance` | Number of drive failures that can be tolerated without disrupting write operations.

Type: gauge | `pool_id`, `set_id` | +| `minio_cluster_erasure_set_read_health` | Health of the erasure set in a pool for read operations (1=healthy, 0=unhealthy).

Type: gauge | `pool_id`, `set_id` | +| `minio_cluster_erasure_set_write_health` | Health of the erasure set in a pool for write operations (1=healthy, 0=unhealthy).

Type: gauge | `pool_id`, `set_id` | -| Name | Type | Help | Labels | -|------------------------------------------------|-----------|---------------------------------------------------------|----------------------------------| -| `minio_api_requests_rejected_auth_total` | `counter` | Total number of requests rejected for auth failure | `type,pool_index,server` | -| `minio_api_requests_rejected_header_total` | `counter` | Total number of requests rejected for invalid header | `type,pool_index,server` | -| `minio_api_requests_rejected_timestamp_total` | `counter` | Total number of requests rejected for invalid timestamp | `type,pool_index,server` | -| `minio_api_requests_rejected_invalid_total` | `counter` | Total number of invalid requests | `type,pool_index,server` | -| `minio_api_requests_waiting_total` | `gauge` | Total number of requests in the waiting queue | `type,pool_index,server` | -| `minio_api_requests_incoming_total` | `gauge` | Total number of incoming requests | `type,pool_index,server` | -| `minio_api_requests_inflight_total` | `gauge` | Total number of requests currently in flight | `name,type,pool_index,server` | -| `minio_api_requests_total` | `counter` | Total number of requests | `name,type,pool_index,server` | -| `minio_api_requests_errors_total` | `counter` | Total number of requests with (4xx and 5xx) errors | `name,type,pool_index,server` | -| `minio_api_requests_5xx_errors_total` | `counter` | Total number of requests with 5xx errors | `name,type,pool_index,server` | -| `minio_api_requests_4xx_errors_total` | `counter` | Total number of requests with 4xx errors | `name,type,pool_index,server` | -| `minio_api_requests_canceled_total` | `counter` | Total number of requests canceled by the client | `name,type,pool_index,server` | -| `minio_api_requests_ttfb_seconds_distribution` | `counter` | Distribution of time to first byte across API calls | `name,type,le,pool_index,server` | -| `minio_api_requests_traffic_sent_bytes` | `counter` | Total number of bytes sent | `type,pool_index,server` | -| `minio_api_requests_traffic_received_bytes` | `counter` | Total number of bytes received | `type,pool_index,server` | +#### `/cluster/health` -### `/bucket/api` +| Name | Description | Labels | +|----------------------------------------------------|---------------------------------------------------------------------|--------| +| `minio_cluster_health_drives_offline_count` | Count of offline drives in the cluster.

Type: gauge | | +| `minio_cluster_health_drives_online_count` | Count of online drives in the cluster.

Type: gauge | | +| `minio_cluster_health_drives_count` | Count of all drives in the cluster.

Type: gauge | | +| `minio_cluster_health_nodes_offline_count` | Count of offline nodes in the cluster.

Type: gauge | | +| `minio_cluster_health_nodes_online_count` | Count of online nodes in the cluster.

Type: gauge | | +| `minio_cluster_health_capacity_raw_total_bytes` | Total cluster raw storage capacity in bytes.

Type: gauge | | +| `minio_cluster_health_capacity_raw_free_bytes` | Total cluster raw storage free in bytes.

Type: gauge | | +| `minio_cluster_health_capacity_usable_total_bytes` | Total cluster usable storage capacity in bytes.

Type: gauge | | +| `minio_cluster_health_capacity_usable_free_bytes` | Total cluster usable storage free in bytes.

Type: gauge | | -| Name | Type | Help | Labels | -|----------------------------------------------|-----------|------------------------------------------------------------------|-----------------------------------------| -| `minio_bucket_api_traffic_received_bytes` | `counter` | Total number of bytes sent for a bucket | `bucket,type,server,pool_index` | -| `minio_bucket_api_traffic_sent_bytes` | `counter` | Total number of bytes received for a bucket | `bucket,type,server,pool_index` | -| `minio_bucket_api_inflight_total` | `gauge` | Total number of requests currently in flight for a bucket | `bucket,name,type,server,pool_index` | -| `minio_bucket_api_total` | `counter` | Total number of requests for a bucket | `bucket,name,type,server,pool_index` | -| `minio_bucket_api_canceled_total` | `counter` | Total number of requests canceled by the client for a bucket | `bucket,name,type,server,pool_index` | -| `minio_bucket_api_4xx_errors_total` | `counter` | Total number of requests with 4xx errors for a bucket | `bucket,name,type,server,pool_index` | -| `minio_bucket_api_5xx_errors_total` | `counter` | Total number of requests with 5xx errors for a bucket | `bucket,name,type,server,pool_index` | -| `minio_bucket_api_ttfb_seconds_distribution` | `counter` | Distribution of time to first byte across API calls for a bucket | `bucket,name,le,type,server,pool_index` | +#### `/cluster/iam` -### `/bucket/replication` +| Name | Description | Labels | +|-----------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------|--------| +| `minio_cluster_iam_last_sync_duration_millis` | Last successful IAM data sync duration in milliseconds.

Type: counter | | +| `minio_cluster_iam_plugin_authn_service_failed_requests_minute` | When plugin authentication is configured, returns failed requests count in the last full minute.

Type: counter | | +| `minio_cluster_iam_plugin_authn_service_last_fail_seconds` | When plugin authentication is configured, returns time (in seconds) since the last failed request to the service.

Type: counter | | +| `minio_cluster_iam_plugin_authn_service_last_succ_seconds` | When plugin authentication is configured, returns time (in seconds) since the last successful request to the service.

Type: counter | | +| `minio_cluster_iam_plugin_authn_service_succ_avg_rtt_ms_minute` | When plugin authentication is configured, returns average round-trip time of successful requests in the last full minute.

Type: counter | | +| `minio_cluster_iam_plugin_authn_service_succ_max_rtt_ms_minute` | When plugin authentication is configured, returns maximum round-trip time of successful requests in the last full minute.

Type: counter | | +| `minio_cluster_iam_plugin_authn_service_total_requests_minute` | When plugin authentication is configured, returns total requests count in the last full minute.

Type: counter | | +| `minio_cluster_iam_since_last_sync_millis` | Time (in milliseconds) since last successful IAM data sync.

Type: counter | | +| `minio_cluster_iam_sync_failures` | Number of failed IAM data syncs since server start.

Type: counter | | +| `minio_cluster_iam_sync_successes` | Number of successful IAM data syncs since server start.

Type: counter | | -| Name | Type | Help | Labels | -|---------------------------------------------------------------------|-----------|---------------------------------------------------------------------------------------------|-------------------------------------------| -| `minio_bucket_replication_last_hour_failed_bytes` | `gauge` | Total number of bytes failed at least once to replicate in the last hour on a bucket | `bucket,server` | -| `minio_bucket_replication_last_hour_failed_count` | `gauge` | Total number of objects which failed replication in the last hour on a bucket | `bucket,server` | -| `minio_bucket_replication_last_minute_failed_bytes` | `gauge` | Total number of bytes failed at least once to replicate in the last full minute on a bucket | `bucket,server` | -| `minio_bucket_replication_last_minute_failed_count` | `gauge` | Total number of objects which failed replication in the last full minute on a bucket | `bucket,server` | -| `minio_bucket_replication_latency_ms` | `gauge` | Replication latency on a bucket in milliseconds | `bucket,operation,range,targetArn,server` | -| `minio_bucket_replication_proxied_delete_tagging_requests_total` | `counter` | Number of DELETE tagging requests proxied to replication target | `bucket,server` | -| `minio_bucket_replication_proxied_get_requests_failures` | `counter` | Number of failures in GET requests proxied to replication target | `bucket,server` | -| `minio_bucket_replication_proxied_get_requests_total` | `counter` | Number of GET requests proxied to replication target | `bucket,server` | -| `minio_bucket_replication_proxied_get_tagging_requests_failures` | `counter` | Number of failures in GET tagging requests proxied to replication target | `bucket,server` | -| `minio_bucket_replication_proxied_get_tagging_requests_total` | `counter` | Number of GET tagging requests proxied to replication target | `bucket,server` | -| `minio_bucket_replication_proxied_head_requests_failures` | `counter` | Number of failures in HEAD requests proxied to replication target | `bucket,server` | -| `minio_bucket_replication_proxied_head_requests_total` | `counter` | Number of HEAD requests proxied to replication target | `bucket,server` | -| `minio_bucket_replication_proxied_put_tagging_requests_failures` | `counter` | Number of failures in PUT tagging requests proxied to replication target | `bucket,server` | -| `minio_bucket_replication_proxied_put_tagging_requests_total` | `counter` | Number of PUT tagging requests proxied to replication target | `bucket,server` | -| `minio_bucket_replication_sent_bytes` | `counter` | Total number of bytes replicated to the target | `bucket,server` | -| `minio_bucket_replication_sent_count` | `counter` | Total number of objects replicated to the target | `bucket,server` | -| `minio_bucket_replication_total_failed_bytes` | `counter` | Total number of bytes failed at least once to replicate since server start | `bucket,server` | -| `minio_bucket_replication_total_failed_count` | `counter` | Total number of objects which failed replication since server start | `bucket,server` | -| `minio_bucket_replication_proxied_delete_tagging_requests_failures` | `counter` | Number of failures in DELETE tagging requests proxied to replication target | `bucket,server` | +#### `/cluster/usage/buckets` -### `/audit` +| Name | Description | Labels | +|-----------------------------------------------------------------|--------------------------------------------------------------------------------------|-------------------| +| `minio_cluster_usage_buckets_since_last_update_seconds` | Time since last update of usage metrics in seconds.

Type: gauge | | +| `minio_cluster_usage_buckets_total_bytes` | Total bucket size in bytes.

Type: gauge | `bucket` | +| `minio_cluster_usage_buckets_objects_count` | Total object count in bucket.

Type: gauge | `bucket` | +| `minio_cluster_usage_buckets_versions_count` | Total object versions count in bucket, including delete markers.

Type: gauge | `bucket` | +| `minio_cluster_usage_buckets_delete_markers_count` | Total delete markers count in bucket.

Type: gauge | `bucket` | +| `minio_cluster_usage_buckets_quota_total_bytes` | Total bucket quota in bytes.

Type: gauge | `bucket` | +| `minio_cluster_usage_buckets_object_size_distribution` | Bucket object size distribution.

Type: gauge | `range`, `bucket` | +| `minio_cluster_usage_buckets_object_version_count_distribution` | Bucket object version count distribution.

Type: gauge | `range`, `bucket` | -| Name | Type | Help | Labels | -|-----------------------------------|-----------|----------------------------------------------------------|--------------------| -| `minio_audit_failed_messages` | `counter` | Total number of messages that failed to send since start | `target_id,server` | -| `minio_audit_target_queue_length` | `gauge` | Number of unsent messages in queue for target | `target_id,server` | -| `minio_audit_total_messages` | `counter` | Total number of messages sent since start | `target_id,server` | +#### `/cluster/usage/objects` -### `/system/drive` +| Name | Description | Labels | +|----------------------------------------------------------|------------------------------------------------------------------------------------|---------| +| `minio_cluster_usage_objects_since_last_update_seconds` | Time since last update of usage metrics in seconds.

Type: gauge | | +| `minio_cluster_usage_objects_total_bytes` | Total cluster usage in bytes.

Type: gauge | | +| `minio_cluster_usage_objects_count` | Total cluster objects count.

Type: gauge | | +| `minio_cluster_usage_objects_versions_count` | Total cluster object versions count, including delete markers.

Type: gauge | | +| `minio_cluster_usage_objects_delete_markers_count` | Total cluster delete markers count.

Type: gauge | | +| `minio_cluster_usage_objects_buckets_count` | Total cluster buckets count.

Type: gauge | | +| `minio_cluster_usage_objects_size_distribution` | Cluster object size distribution.

Type: gauge | `range` | +| `minio_cluster_usage_objects_version_count_distribution` | Cluster object version count distribution.

Type: gauge | `range` | -| Name | Type | Help | Labels | -|------------------------------------------------|-----------|--------------------------------------------------------------------|-----------------------------------------------------| -| `minio_system_drive_used_bytes` | `gauge` | Total storage used on a drive in bytes | `drive,set_index,drive_index,pool_index,server` | -| `minio_system_drive_free_bytes` | `gauge` | Total storage free on a drive in bytes | `drive,set_index,drive_index,pool_index,server` | -| `minio_system_drive_total_bytes` | `gauge` | Total storage available on a drive in bytes | `drive,set_index,drive_index,pool_index,server` | -| `minio_system_drive_used_inodes` | `gauge` | Total used inodes on a drive | `drive,set_index,drive_index,pool_index,server` | -| `minio_system_drive_free_inodes` | `gauge` | Total free inodes on a drive | `drive,set_index,drive_index,pool_index,server` | -| `minio_system_drive_total_inodes` | `gauge` | Total inodes available on a drive | `drive,set_index,drive_index,pool_index,server` | -| `minio_system_drive_timeout_errors_total` | `counter` | Total timeout errors on a drive | `drive,set_index,drive_index,pool_index,server` | -| `minio_system_drive_io_errors_total` | `counter` | Total I/O errors on a drive | `drive,set_index,drive_index,pool_index,server` | -| `minio_system_drive_availability_errors_total` | `counter` | Total availability errors (I/O errors, timeouts) on a drive | `drive,set_index,drive_index,pool_index,server` | -| `minio_system_drive_waiting_io` | `gauge` | Total waiting I/O operations on a drive | `drive,set_index,drive_index,pool_index,server` | -| `minio_system_drive_api_latency_micros` | `gauge` | Average last minute latency in µs for drive API storage operations | `drive,api,set_index,drive_index,pool_index,server` | -| `minio_system_drive_offline_count` | `gauge` | Count of offline drives | `pool_index,server` | -| `minio_system_drive_online_count` | `gauge` | Count of online drives | `pool_index,server` | -| `minio_system_drive_count` | `gauge` | Count of all drives | `pool_index,server` | -| `minio_system_drive_health` | `gauge` | Drive health (0 = offline, 1 = healthy, 2 = healing) | `drive,set_index,drive_index,pool_index,server` | -| `minio_system_drive_reads_per_sec` | `gauge` | Reads per second on a drive | `drive,set_index,drive_index,pool_index,server` | -| `minio_system_drive_reads_kb_per_sec` | `gauge` | Kilobytes read per second on a drive | `drive,set_index,drive_index,pool_index,server` | -| `minio_system_drive_reads_await` | `gauge` | Average time for read requests served on a drive | `drive,set_index,drive_index,pool_index,server` | -| `minio_system_drive_writes_per_sec` | `gauge` | Writes per second on a drive | `drive,set_index,drive_index,pool_index,server` | -| `minio_system_drive_writes_kb_per_sec` | `gauge` | Kilobytes written per second on a drive | `drive,set_index,drive_index,pool_index,server` | -| `minio_system_drive_writes_await` | `gauge` | Average time for write requests served on a drive | `drive,set_index,drive_index,pool_index,server` | -| `minio_system_drive_perc_util` | `gauge` | Percentage of time the disk was busy | `drive,set_index,drive_index,pool_index,server` | +### Debug metrics -### `/system/memory` +Standard Go runtime metrics. -| Name | Type | Help | Labels | -|----------------------------------|---------|------------------------------------|----------| -| `minio_system_memory_used` | `gauge` | Used memory on the node | `server` | -| `minio_system_memory_used_perc` | `gauge` | Used memory percentage on the node | `server` | -| `minio_system_memory_free` | `gauge` | Free memory on the node | `server` | -| `minio_system_memory_total` | `gauge` | Total memory on the node | `server` | -| `minio_system_memory_buffers` | `gauge` | Buffers memory on the node | `server` | -| `minio_system_memory_cache` | `gauge` | Cache memory on the node | `server` | -| `minio_system_memory_shared` | `gauge` | Shared memory on the node | `server` | -| `minio_system_memory_available` | `gauge` | Available memory on the node | `server` | +| Path | Description | +|-------------|---------------------| +| `/debug/go` | Go runtime metrics. | -### `/system/cpu` +### ILM metrics -| Name | Type | Help | Labels | -|-------------------------------|---------|------------------------------------|----------| -| `minio_system_cpu_avg_idle` | `gauge` | Average CPU idle time | `server` | -| `minio_system_cpu_avg_iowait` | `gauge` | Average CPU IOWait time | `server` | -| `minio_system_cpu_load` | `gauge` | CPU load average 1min | `server` | -| `minio_system_cpu_load_perc` | `gauge` | CPU load average 1min (percentage) | `server` | -| `minio_system_cpu_nice` | `gauge` | CPU nice time | `server` | -| `minio_system_cpu_steal` | `gauge` | CPU steal time | `server` | -| `minio_system_cpu_system` | `gauge` | CPU system time | `server` | -| `minio_system_cpu_user` | `gauge` | CPU user time | `server` | +Metrics about the MinIO ILM functionality. -### `/system/network/internode` +| Path | Description | +|--------|---------------------------------------| +| `/ilm` | Metrics related to ILM functionality. | -| Name | Type | Help | Labels | -|------------------------------------------------------|-----------|----------------------------------------------------------|---------------------| -| `minio_system_network_internode_errors_total` | `counter` | Total number of failed internode calls | `server,pool_index` | -| `minio_system_network_internode_dial_errors_total` | `counter` | Total number of internode TCP dial timeouts and errors | `server,pool_index` | -| `minio_system_network_internode_dial_avg_time_nanos` | `gauge` | Average dial time of internodes TCP calls in nanoseconds | `server,pool_index` | -| `minio_system_network_internode_sent_bytes_total` | `counter` | Total number of bytes sent to other peer nodes | `server,pool_index` | -| `minio_system_network_internode_recv_bytes_total` | `counter` | Total number of bytes received from other peer nodes | `server,pool_index` | +#### `/ilm` -### `/system/process` +| Name | Description | Labels | +|-------------------------------------------------------|---------------------------------------------------------------------------------------------------|----------| +| `minio_cluster_ilm_expiry_pending_tasks` | Number of pending ILM expiry tasks in the queue.

Type: gauge | `server` | +| `minio_cluster_ilm_transition_active_tasks` | Number of active ILM transition tasks.

Type: gauge | `server` | +| `minio_cluster_ilm_transition_pending_tasks` | Number of pending ILM transition tasks in the queue.

Type: gauge | `server` | +| `minio_cluster_ilm_transition_missed_immediate_tasks` | Number of missed immediate ILM transition tasks.

Type: counter | `server` | +| `minio_cluster_ilm_versions_scanned` | Total number of object versions checked for ILM actions since server start.

Type: counter | `server` | -| Name | Type | Help | Labels | -|----------------------------------------------------|-----------|----------------------------------------------------------------------------------------------------------------|----------| -| `minio_system_process_locks_read_total` | `gauge` | Number of current READ locks on this peer | `server` | -| `minio_system_process_locks_write_total` | `gauge` | Number of current WRITE locks on this peer | `server` | -| `minio_system_process_cpu_total_seconds` | `counter` | Total user and system CPU time spent in seconds | `server` | -| `minio_system_process_go_routine_total` | `gauge` | Total number of go routines running | `server` | -| `minio_system_process_io_rchar_bytes` | `counter` | Total bytes read by the process from the underlying storage system including cache, /proc/[pid]/io rchar | `server` | -| `minio_system_process_io_read_bytes` | `counter` | Total bytes read by the process from the underlying storage system, /proc/[pid]/io read_bytes | `server` | -| `minio_system_process_io_wchar_bytes` | `counter` | Total bytes written by the process to the underlying storage system including page cache, /proc/[pid]/io wchar | `server` | -| `minio_system_process_io_write_bytes` | `counter` | Total bytes written by the process to the underlying storage system, /proc/[pid]/io write_bytes | `server` | -| `minio_system_process_start_time_seconds` | `gauge` | Start time for MinIO process in seconds since Unix epoc | `server` | -| `minio_system_process_uptime_seconds` | `gauge` | Uptime for MinIO process in seconds | `server` | -| `minio_system_process_file_descriptor_limit_total` | `gauge` | Limit on total number of open file descriptors for the MinIO Server process | `server` | -| `minio_system_process_file_descriptor_open_total` | `gauge` | Total number of open file descriptors by the MinIO Server process | `server` | -| `minio_system_process_syscall_read_total` | `counter` | Total read SysCalls to the kernel. /proc/[pid]/io syscr | `server` | -| `minio_system_process_syscall_write_total` | `counter` | Total write SysCalls to the kernel. /proc/[pid]/io syscw | `server` | -| `minio_system_process_resident_memory_bytes` | `gauge` | Resident memory size in bytes | `server` | -| `minio_system_process_virtual_memory_bytes` | `gauge` | Virtual memory size in bytes | `server` | -| `minio_system_process_virtual_memory_max_bytes` | `gauge` | Maximum virtual memory size in bytes | `server` | +### Logger webhook metrics -### `/cluster/health` +Metrics about MinIO logger webhooks. -| Name | Type | Help | Labels | -|----------------------------------------------------|---------|------------------------------------------------|--------| -| `minio_cluster_health_drives_offline_count` | `gauge` | Count of offline drives in the cluster | | -| `minio_cluster_health_drives_online_count` | `gauge` | Count of online drives in the cluster | | -| `minio_cluster_health_drives_count` | `gauge` | Count of all drives in the cluster | | -| `minio_cluster_health_nodes_offline_count` | `gauge` | Count of offline nodes in the cluster | | -| `minio_cluster_health_nodes_online_count` | `gauge` | Count of online nodes in the cluster | | -| `minio_cluster_health_capacity_raw_total_bytes` | `gauge` | Total cluster raw storage capacity in bytes | | -| `minio_cluster_health_capacity_raw_free_bytes` | `gauge` | Total cluster raw storage free in bytes | | -| `minio_cluster_health_capacity_usable_total_bytes` | `gauge` | Total cluster usable storage capacity in bytes | | -| `minio_cluster_health_capacity_usable_free_bytes` | `gauge` | Total cluster usable storage free in bytes | | +| Path | Description | +|-------------------|-------------------------------------| +| `/logger/webhook` | Metrics related to logger webhooks. | -### `/cluster/config` +#### `/logger/webhook` -| Name | Type | Help | Labels | -|----------------------------------------|---------|------------------------------------------------|--------| -| `minio_cluster_config_rrs_parity` | `gauge` | Reduced redundancy storage class parity | | -| `minio_cluster_config_standard_parity` | `gauge` | Standard storage class parity | | +| Name | Description | Labels | +|-----------------------------------------|---------------------------------------------------------------------|------------------------------| +| `minio_logger_webhook_failed_messages` | Number of messages that failed to send.

Type: counter | `server`, `name`, `endpoint` | +| `minio_logger_webhook_queue_length` | Webhook queue length.

Type: gauge | `server`, `name`, `endpoint` | +| `minio_logger_webhook_total_message` | Total number of messages sent to this target.

Type: counter | `server`, `name`, `endpoint` | -### `/cluster/usage/objects` +### Notification metrics -| Name | Type | Help | Labels | -|----------------------------------------------------------|---------|----------------------------------------------------------------|---------| -| `minio_cluster_usage_objects_since_last_update_seconds` | `gauge` | Time since last update of usage metrics in seconds | | -| `minio_cluster_usage_objects_total_bytes` | `gauge` | Total cluster usage in bytes | | -| `minio_cluster_usage_objects_count` | `gauge` | Total cluster objects count | | -| `minio_cluster_usage_objects_versions_count` | `gauge` | Total cluster object versions (including delete markers) count | | -| `minio_cluster_usage_objects_delete_markers_count` | `gauge` | Total cluster delete markers count | | -| `minio_cluster_usage_objects_buckets_count` | `gauge` | Total cluster buckets count | | -| `minio_cluster_usage_objects_size_distribution` | `gauge` | Cluster object size distribution | `range` | -| `minio_cluster_usage_objects_version_count_distribution` | `gauge` | Cluster object version count distribution | `range` | +Metrics about the MinIO notification functionality. -### `/cluster/usage/buckets` +| Path | Description | +|-----------------|------------------------------------------------| +| `/notification` | Metrics related to notification functionality. | -| Name | Type | Help | Labels | -|-----------------------------------------------------------------|---------|------------------------------------------------------------------|----------------| -| `minio_cluster_usage_buckets_since_last_update_seconds` | `gauge` | Time since last update of usage metrics in seconds | | -| `minio_cluster_usage_buckets_total_bytes` | `gauge` | Total bucket size in bytes | `bucket` | -| `minio_cluster_usage_buckets_objects_count` | `gauge` | Total objects count in bucket | `bucket` | -| `minio_cluster_usage_buckets_versions_count` | `gauge` | Total object versions (including delete markers) count in bucket | `bucket` | -| `minio_cluster_usage_buckets_delete_markers_count` | `gauge` | Total delete markers count in bucket | `bucket` | -| `minio_cluster_usage_buckets_quota_total_bytes` | `gauge` | Total bucket quota in bytes | `bucket` | -| `minio_cluster_usage_buckets_object_size_distribution` | `gauge` | Bucket object size distribution | `range,bucket` | -| `minio_cluster_usage_buckets_object_version_count_distribution` | `gauge` | Bucket object version count distribution | `range,bucket` | +#### `/notification` -### `/cluster/erasure-set` +| Name | Description | Labels | +|-----------------------------------------------|-------------------------------------------------------------------------------------------------------|----------| +| `minio_notification_current_send_in_progress` | Number of concurrent async Send calls active to all targets.

Type: counter | `server` | +| `minio_notification_events_errors_total` | Total number of events that failed to send to the targets.

Type: counter | `server` | +| `minio_notification_events_sent_total` | Total number of events sent to the targets.

Type: counter | `server` | +| `minio_notification_events_skipped_total` | Number of events not sent to the targets due to the in-memory queue being full.

Type: counter | `server` | -| Name | Type | Help | Labels | -|--------------------------------------------------|---------|-----------------------------------------------------------------------------------|------------------| -| `minio_cluster_erasure_set_overall_write_quorum` | `gauge` | Overall write quorum across pools and sets | | -| `minio_cluster_erasure_set_overall_health` | `gauge` | Overall health across pools and sets (1=healthy, 0=unhealthy) | | -| `minio_cluster_erasure_set_read_quorum` | `gauge` | Read quorum for the erasure set in a pool | `pool_id,set_id` | -| `minio_cluster_erasure_set_write_quorum` | `gauge` | Write quorum for the erasure set in a pool | `pool_id,set_id` | -| `minio_cluster_erasure_set_online_drives_count` | `gauge` | Count of online drives in the erasure set in a pool | `pool_id,set_id` | -| `minio_cluster_erasure_set_healing_drives_count` | `gauge` | Count of healing drives in the erasure set in a pool | `pool_id,set_id` | -| `minio_cluster_erasure_set_health` | `gauge` | Health of the erasure set in a pool (1=healthy, 0=unhealthy) | `pool_id,set_id` | -| `minio_cluster_erasure_set_read_tolerance` | `gauge` | No of drive failures that can be tolerated without disrupting read operations | `pool_id,set_id` | -| `minio_cluster_erasure_set_write_tolerance` | `gauge` | No of drive failures that can be tolerated without disrupting write operations | `pool_id,set_id` | -| `minio_cluster_erasure_set_read_health` | `gauge` | Health of the erasure set in a pool for read operations (1=healthy, 0=unhealthy) | `pool_id,set_id` | -| `minio_cluster_erasure_set_write_health` | `gauge` | Health of the erasure set in a pool for write operations (1=healthy, 0=unhealthy) | `pool_id,set_id` | +### Replication metrics -### `/cluster/iam` +Metrics about MinIO site and bucket replication. -| Name | Type | Help | Labels | -|-----------------------------------------------------------------|-----------|--------------------------------------------------------------------------------------------------------------------------|--------| -| `minio_cluster_iam_last_sync_duration_millis` | `counter` | Last successful IAM data sync duration in milliseconds | | -| `minio_cluster_iam_plugin_authn_service_failed_requests_minute` | `counter` | When plugin authentication is configured, returns failed requests count in the last full minute | | -| `minio_cluster_iam_plugin_authn_service_last_fail_seconds` | `counter` | When plugin authentication is configured, returns time (in seconds) since the last failed request to the service | | -| `minio_cluster_iam_plugin_authn_service_last_succ_seconds` | `counter` | When plugin authentication is configured, returns time (in seconds) since the last successful request to the service | | -| `minio_cluster_iam_plugin_authn_service_succ_avg_rtt_ms_minute` | `counter` | When plugin authentication is configured, returns average round-trip-time of successful requests in the last full minute | | -| `minio_cluster_iam_plugin_authn_service_succ_max_rtt_ms_minute` | `counter` | When plugin authentication is configured, returns maximum round-trip-time of successful requests in the last full minute | | -| `minio_cluster_iam_plugin_authn_service_total_requests_minute` | `counter` | When plugin authentication is configured, returns total requests count in the last full minute | | -| `minio_cluster_iam_since_last_sync_millis` | `counter` | Time (in milliseconds) since last successful IAM data sync | | -| `minio_cluster_iam_sync_failures` | `counter` | Number of failed IAM data syncs since server start | | -| `minio_cluster_iam_sync_successes` | `counter` | Number of successful IAM data syncs since server start | | +| Path | Description | +|-----------------------|----------------------------------------| +| `/bucket/replication` | Metrics related to bucket replication. | +| `/replication` | Metrics related to site replication. | -### `/logger/webhook` +#### `/replication` -| Name | Type | Help | Labels | -|-----------------------------------------|-----------|----------------------------------------------|------------------------| -| `minio_logger_webhook_failed_messages` | `counter` | Number of messages that failed to send | `server,name,endpoint` | -| `minio_logger_webhook_queue_length` | `gauge` | Webhook queue length | `server,name,endpoint` | -| `minio_logger_webhook_total_message` | `counter` | Total number of messages sent to this target | `server,name,endpoint` | +| Name | Description | Labels | +|---------------------------------------------------|---------------------------------------------------------------------------------------------|----------| +| `minio_replication_average_active_workers` | Average number of active replication workers.

Type: gauge | `server` | +| `minio_replication_average_queued_bytes` | Average number of bytes queued for replication since server start.

Type: gauge | `server` | +| `minio_replication_average_queued_count` | Average number of objects queued for replication since server start.

Type: gauge | `server` | +| `minio_replication_average_data_transfer_rate` | Average replication data transfer rate in bytes/sec.

Type: gauge | `server` | +| `minio_replication_current_active_workers` | Total number of active replication workers.

Type: gauge | `server` | +| `minio_replication_current_data_transfer_rate` | Current replication data transfer rate in bytes/sec.

Type: gauge | `server` | +| `minio_replication_last_minute_queued_bytes` | Number of bytes queued for replication in the last full minute.

Type: gauge | `server` | +| `minio_replication_last_minute_queued_count` | Number of objects queued for replication in the last full minute.

Type: gauge | `server` | +| `minio_replication_max_active_workers` | Maximum number of active replication workers seen since server start.

Type: gauge | `server` | +| `minio_replication_max_queued_bytes` | Maximum number of bytes queued for replication since server start.

Type: gauge | `server` | +| `minio_replication_max_queued_count` | Maximum number of objects queued for replication since server start.

Type: gauge | `server` | +| `minio_replication_max_data_transfer_rate` | Maximum replication data transfer rate in bytes/sec since server start.

Type: gauge | `server` | -### `/replication` +#### `/bucket/replication` -| Name | Type | Help | Labels | -|---------------------------------------------------|---------|-----------------------------------------------------------------------------|----------| -| `minio_replication_average_active_workers` | `gauge` | Average number of active replication workers | `server` | -| `minio_replication_average_queued_bytes` | `gauge` | Average number of bytes queued for replication since server start | `server` | -| `minio_replication_average_queued_count` | `gauge` | Average number of objects queued for replication since server start | `server` | -| `minio_replication_average_data_transfer_rate` | `gauge` | Average replication data transfer rate in bytes/sec | `server` | -| `minio_replication_current_active_workers` | `gauge` | Total number of active replication workers | `server` | -| `minio_replication_current_data_transfer_rate` | `gauge` | Current replication data transfer rate in bytes/sec | `server` | -| `minio_replication_last_minute_queued_bytes` | `gauge` | Number of bytes queued for replication in the last full minute | `server` | -| `minio_replication_last_minute_queued_count` | `gauge` | Number of objects queued for replication in the last full minute | `server` | -| `minio_replication_max_active_workers` | `gauge` | Maximum number of active replication workers seen since server start | `server` | -| `minio_replication_max_queued_bytes` | `gauge` | Maximum number of bytes queued for replication since server start | `server` | -| `minio_replication_max_queued_count` | `gauge` | Maximum number of objects queued for replication since server start | `server` | -| `minio_replication_max_data_transfer_rate` | `gauge` | Maximum replication data transfer rate in bytes/sec seen since server start | `server` | +| Name | Description | Labels | +|---------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------|-------------------------------------------------------| +| `minio_bucket_replication_last_hour_failed_bytes` | Total number of bytes on a bucket which failed to replicate at least once in the last hour.

Type: gauge | `bucket`, `server` | +| `minio_bucket_replication_last_hour_failed_count` | Total number of objects on a bucket which failed to replicate in the last hour.

Type: gauge | `bucket`, `server` | +| `minio_bucket_replication_last_minute_failed_bytes` | Total number of bytes on a bucket which failed at least once in the last full minute.

Type: gauge | `bucket`, `server` | +| `minio_bucket_replication_last_minute_failed_count` | Total number of objects on a bucket which failed to replicate in the last full minute.

Type: gauge | `bucket`, `server` | +| `minio_bucket_replication_latency_ms` | Replication latency on a bucket in milliseconds.

Type: gauge | `bucket`, `operation`, `range`, `targetArn`, `server` | +| `minio_bucket_replication_proxied_delete_tagging_requests_total` | Number of DELETE tagging requests proxied to replication target.

Type: counter | `bucket`, `server` | +| `minio_bucket_replication_proxied_get_requests_failures` | Number of failures in GET requests proxied to replication target.

Type: counter | `bucket`, `server` | +| `minio_bucket_replication_proxied_get_requests_total` | Number of GET requests proxied to replication target.

Type: counter | `bucket`, `server` | +| `minio_bucket_replication_proxied_get_tagging_requests_failures` | Number of failures in GET tagging requests proxied to replication target.

Type: counter | `bucket`, `server` | +| `minio_bucket_replication_proxied_get_tagging_requests_total` | Number of GET tagging requests proxied to replication target.

Type: counter | `bucket`, `server` | +| `minio_bucket_replication_proxied_head_requests_failures` | Number of failures in HEAD requests proxied to replication target.

Type: counter | `bucket`, `server` | +| `minio_bucket_replication_proxied_head_requests_total` | Number of HEAD requests proxied to replication target.

Type: counter | `bucket`, `server` | +| `minio_bucket_replication_proxied_put_tagging_requests_failures` | Number of failures in PUT tagging requests proxied to replication target.

Type: counter | `bucket`, `server` | +| `minio_bucket_replication_proxied_put_tagging_requests_total` | Number of PUT tagging requests proxied to replication target.

Type: counter | `bucket`, `server` | +| `minio_bucket_replication_sent_bytes` | Total number of bytes replicated to the target.

Type: counter | `bucket`, `server` | +| `minio_bucket_replication_sent_count` | Total number of objects replicated to the target.

Type: counter | `bucket`, `server` | +| `minio_bucket_replication_total_failed_bytes` | Total number of bytes failed to replicate at least once since server start.

Type: counter | `bucket`, `server` | +| `minio_bucket_replication_total_failed_count` | Total number of objects that failed to replicate since server start.

Type: counter | `bucket`, `server` | +| `minio_bucket_replication_proxied_delete_tagging_requests_failures` | Number of failures in DELETE tagging requests proxied to replication target.

Type: counter | `bucket`, `server` | -### `/notification` +### Scanner metrics -| Name | Type | Help | Labels | -|-----------------------------------------------|-----------|------------------------------------------------------------------------------------------|----------| -| `minio_notification_current_send_in_progress` | `counter` | Number of concurrent async Send calls active to all targets | `server` | -| `minio_notification_events_errors_total` | `counter` | Events that were failed to be sent to the targets | `server` | -| `minio_notification_events_sent_total` | `counter` | Total number of events sent to the targets | `server` | -| `minio_notification_events_skipped_total` | `counter` | Events that were skipped to be sent to the targets due to the in-memory queue being full | `server` | +Metrics about the MinIO scanner. -### `/scanner` +| Path | Description | +|------------|---------------------------------------| +| `/scanner` | Metrics related to the MinIO scanner. | -| Name | Type | Help | Labels | -|--------------------------------------------|-----------|------------------------------------------------------------|----------| -| `minio_scanner_bucket_scans_finished` | `counter` | Total number of bucket scans finished since server start | `server` | -| `minio_scanner_bucket_scans_started` | `counter` | Total number of bucket scans started since server start | `server` | -| `minio_scanner_directories_scanned` | `counter` | Total number of directories scanned since server start | `server` | -| `minio_scanner_last_activity_seconds` | `gauge` | Time elapsed (in seconds) since last scan activity | `server` | -| `minio_scanner_objects_scanned` | `counter` | Total number of unique objects scanned since server start | `server` | -| `minio_scanner_versions_scanned` | `counter` | Total number of object versions scanned since server start | `server` | +#### `/scanner` -### `/ilm` +| Name | Description | Labels | +|--------------------------------------------|-----------------------------------------------------------------------------------|----------| +| `minio_scanner_bucket_scans_finished` | Total number of bucket scans completed since server start.

Type: counter | `server` | +| `minio_scanner_bucket_scans_started` | Total number of bucket scans started since server start.

Type: counter | `server` | +| `minio_scanner_directories_scanned` | Total number of directories scanned since server start.

Type: counter | `server` | +| `minio_scanner_last_activity_seconds` | Time elapsed (in seconds) since last scan activity.

Type: gauge | `server` | +| `minio_scanner_objects_scanned` | Total number of unique objects scanned since server start.

Type: counter | `server` | +| `minio_scanner_versions_scanned` | Total number of object versions scanned since server start.

Type: counter | `server` | -| Name | Type | Help | Labels | -|-------------------------------------------------------|-----------|----------------------------------------------------------------------------|----------| -| `minio_cluster_ilm_expiry_pending_tasks` | `gauge` | Number of pending ILM expiry tasks in the queue | `server` | -| `minio_cluster_ilm_transition_active_tasks` | `gauge` | Number of active ILM transition tasks | `server` | -| `minio_cluster_ilm_transition_pending_tasks` | `gauge` | Number of pending ILM transition tasks in the queue | `server` | -| `minio_cluster_ilm_transition_missed_immediate_tasks` | `counter` | Number of missed immediate ILM transition tasks | `server` | -| `minio_cluster_ilm_versions_scanned` | `counter` | Total number of object versions checked for ILM actions since server start | `server` | +### System metrics + +Metrics about the MinIO process and the node. + +| Path | Description | +|-----------------------------|----------------------------------------------------| +| `/system/cpu` | Metrics about CPUs on the system. | +| `/system/drive` | Metrics about drives on the system. | +| `/system/network/internode` | Metrics about internode requests made by the node. | +| `/system/memory` | Metrics about memory on the system. | +| `/system/process` | Standard process metrics. | + +#### `/system/drive` + +| Name | Description | Labels | +|------------------------------------------------|-----------------------------------------------------------------------------------------|--------------------------------------------------------------------| +| `minio_system_drive_used_bytes` | Total storage used on a drive in bytes.

Type: gauge | `drive`, `set_index`, `drive_index`, `pool_index`, `server` | +| `minio_system_drive_free_bytes` | Total storage free on a drive in bytes.

Type: gauge | `drive`, `set_index`, `drive_index`, `pool_index`, `server` | +| `minio_system_drive_total_bytes` | Total storage available on a drive in bytes.

Type: gauge | `drive`, `set_index`, `drive_index`, `pool_index`, `server` | +| `minio_system_drive_used_inodes` | Total used inodes on a drive.

Type: gauge | `drive`, `set_index`, `drive_index`, `pool_index`, `server` | +| `minio_system_drive_free_inodes` | Total free inodes on a drive.

Type: gauge | `drive`, `set_index`, `drive_index`, `pool_index`, `server` | +| `minio_system_drive_total_inodes` | Total inodes available on a drive.

Type: gauge | `drive`, `set_index`, `drive_index`, `pool_index`, `server` | +| `minio_system_drive_timeout_errors_total` | Total timeout errors on a drive.

Type: counter | `drive`, `set_index`, `drive_index`, `pool_index`, `server` | +| `minio_system_drive_io_errors_total` | Total I/O errors on a drive.

Type: counter | `drive`, `set_index`, `drive_index`, `pool_index`, `server` | +| `minio_system_drive_availability_errors_total` | Total availability errors (I/O errors, timeouts) on a drive.

Type: counter | `drive`, `set_index`, `drive_index`, `pool_index`, `server` | +| `minio_system_drive_waiting_io` | Total waiting I/O operations on a drive.

Type: gauge | `drive`, `set_index`, `drive_index`, `pool_index`, `server` | +| `minio_system_drive_api_latency_micros` | Average last minute latency in µs for drive API storage operations.

Type: gauge | `drive`, `api`, `set_index`, `drive_index`, `pool_index`, `server` | +| `minio_system_drive_offline_count` | Count of offline drives.

Type: gauge | `pool_index`, `server` | +| `minio_system_drive_online_count` | Count of online drives.

Type: gauge | `pool_index`, `server` | +| `minio_system_drive_count` | Count of all drives.

Type: gauge | `pool_index`, `server` | +| `minio_system_drive_health` | Drive health (0 = offline, 1 = healthy, 2 = healing).

Type: gauge | `drive`, `set_index`, `drive_index`, `pool_index`, `server` | +| `minio_system_drive_reads_per_sec` | Reads per second on a drive.

Type: gauge | `drive`, `set_index`, `drive_index`, `pool_index`, `server` | +| `minio_system_drive_reads_kb_per_sec` | Kilobytes read per second on a drive.

Type: gauge | `drive`, `set_index`, `drive_index`, `pool_index`, `server` | +| `minio_system_drive_reads_await` | Average time for read requests served on a drive.

Type: gauge | `drive`, `set_index`, `drive_index`, `pool_index`, `server` | +| `minio_system_drive_writes_per_sec` | Writes per second on a drive.

Type: gauge | `drive`, `set_index`, `drive_index`, `pool_index`, `server` | +| `minio_system_drive_writes_kb_per_sec` | Kilobytes written per second on a drive.

Type: gauge | `drive`, `set_index`, `drive_index`, `pool_index`, `server` | +| `minio_system_drive_writes_await` | Average time for write requests served on a drive.

Type: gauge | `drive`, `set_index`, `drive_index`, `pool_index`, `server` | +| `minio_system_drive_perc_util` | Percentage of time the disk was busy.

Type: gauge | `drive`, `set_index`, `drive_index`, `pool_index`, `server` | + +#### `/system/memory` + +| Name | Description | Labels | +|----------------------------------|---------------------------------------------------------|----------| +| `minio_system_memory_used` | Used memory on the node.

Type: gauge | `server` | +| `minio_system_memory_used_perc` | Used memory percentage on the node.

Type: gauge | `server` | +| `minio_system_memory_free` | Free memory on the node.

Type: gauge | `server` | +| `minio_system_memory_total` | Total memory on the node.

Type: gauge | `server` | +| `minio_system_memory_buffers` | Buffers memory on the node.

Type: gauge | `server` | +| `minio_system_memory_cache` | Cache memory on the node.

Type: gauge | `server` | +| `minio_system_memory_shared` | Shared memory on the node.

Type: gauge | `server` | +| `minio_system_memory_available` | Available memory on the node.

Type: gauge | `server` | + +#### `/system/cpu` + +| Name | Description | Labels | +|-------------------------------|---------------------------------------------------------|----------| +| `minio_system_cpu_avg_idle` | Average CPU idle time.

Type: gauge | `server` | +| `minio_system_cpu_avg_iowait` | Average CPU IOWait time.

Type: gauge | `server` | +| `minio_system_cpu_load` | CPU load average 1min.

Type: gauge | `server` | +| `minio_system_cpu_load_perc` | CPU load average 1min (percentage).

Type: gauge | `server` | +| `minio_system_cpu_nice` | CPU nice time.

Type: gauge | `server` | +| `minio_system_cpu_steal` | CPU steal time.

Type: gauge | `server` | +| `minio_system_cpu_system` | CPU system time.

Type: gauge | `server` | +| `minio_system_cpu_user` | CPU user time.

Type: gauge | `server` | + +#### `/system/network/internode` + +| Name | Description | Labels | +|------------------------------------------------------|-------------------------------------------------------------------------------|------------------------| +| `minio_system_network_internode_errors_total` | Total number of failed internode calls.

Type: counter | `server`, `pool_index` | +| `minio_system_network_internode_dial_errors_total` | Total number of internode TCP dial timeouts and errors.

Type: counter | `server`, `pool_index` | +| `minio_system_network_internode_dial_avg_time_nanos` | Average dial time of internodes TCP calls in nanoseconds.

Type: gauge | `server`, `pool_index` | +| `minio_system_network_internode_sent_bytes_total` | Total number of bytes sent to other peer nodes.

Type: counter | `server`, `pool_index` | +| `minio_system_network_internode_recv_bytes_total` | Total number of bytes received from other peer nodes.

Type: counter | `server`, `pool_index` | + +#### `/system/process` + +| Name | Description | Labels | +|----------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------|----------| +| `minio_system_process_locks_read_total` | Number of current READ locks on this peer.

Type: gauge | `server` | +| `minio_system_process_locks_write_total` | Number of current WRITE locks on this peer.

Type: gauge | `server` | +| `minio_system_process_cpu_total_seconds` | Total user and system CPU time spent in seconds.

Type: counter | `server` | +| `minio_system_process_go_routine_total` | Total number of go routines running.

Type: gauge | `server` | +| `minio_system_process_io_rchar_bytes` | Total bytes read by the process from the underlying storage system including cache, /proc/[pid]/io rchar.

Type: counter | `server` | +| `minio_system_process_io_read_bytes` | Total bytes read by the process from the underlying storage system, /proc/[pid]/io read_bytes.

Type: counter | `server` | +| `minio_system_process_io_wchar_bytes` | Total bytes written by the process to the underlying storage system including page cache, /proc/[pid]/io wchar.

Type: counter | `server` | +| `minio_system_process_io_write_bytes` | Total bytes written by the process to the underlying storage system, /proc/[pid]/io write_bytes.

Type: counter | `server` | +| `minio_system_process_start_time_seconds` | Start time for MinIO process in seconds since Unix epoch.

Type: gauge | `server` | +| `minio_system_process_uptime_seconds` | Uptime for MinIO process in seconds.

Type: gauge | `server` | +| `minio_system_process_file_descriptor_limit_total` | Limit on total number of open file descriptors for the MinIO Server process.

Type: gauge | `server` | +| `minio_system_process_file_descriptor_open_total` | Total number of open file descriptors by the MinIO Server process.

Type: gauge | `server` | +| `minio_system_process_syscall_read_total` | Total read SysCalls to the kernel. /proc/[pid]/io syscr.

Type: counter | `server` | +| `minio_system_process_syscall_write_total` | Total write SysCalls to the kernel. /proc/[pid]/io syscw.

Type: counter | `server` | +| `minio_system_process_resident_memory_bytes` | Resident memory size in bytes.

Type: gauge | `server` | +| `minio_system_process_virtual_memory_bytes` | Virtual memory size in bytes.

Type: gauge | `server` | +| `minio_system_process_virtual_memory_max_bytes` | Maximum virtual memory size in bytes.

Type: gauge | `server` |