mirror of https://github.com/minio/minio.git
26 KiB
26 KiB
Metrics Version 3
In metrics version 3, all metrics are available under the endpoint:
/minio/metrics/v3
however, a specific path under this is required.
Metrics are organized into groups at paths relative to the top-level endpoint above.
Metrics Request Handling
Each endpoint below can be queried at different intervals as needed via a scrape configuration in Prometheus or a compatible metrics collection tool.
For ease of configuration, each (non-empty) parent of the path serves all metric endpoints that are at descendant paths. For example, to query all system metrics one needs to only scrape /minio/metrics/v3/system/.
Some metrics are bucket specific. These will have a /bucket component in their path. As the number of buckets can be large, the metrics scrape operation needs to be provided with a specific list of buckets via the bucket query parameter. Only metrics for the given buckets will be returned (with the bucket label set). For example to query API metrics for buckets test1 and test2, make a scrape request to /minio/metrics/v3/api/bucket?buckets=test1,test2.
Instead of a metrics scrape, it is also possible to list the metrics that would be returned by a path. This is done by adding a ?list query parameter. The MinIO server will then list all possible metrics that could be returned. During an actual metrics scrape, only available metrics are returned - not all of them. With the list query parameter, the output format can be selected - just set the request Content-Type to application/json for JSON output, or text/plain for a simple markdown formatted table. The latter is the default.
Request, System and Cluster Metrics
At a high level metrics are grouped into three categories, listed in the following sub-sections. The path in each of the tables is relative to the top-level endpoint.
Request metrics
These are metrics about requests served by the (current) node.
| Path | Description |
|---|---|
/api/requests |
Metrics over all requests |
/api/bucket |
Metrics over all requests split by bucket labels |
System metrics
These are metrics about the minio process and the node.
| Path | Description |
|---|---|
/system/drive |
Metrics about drives on the system |
/system/memory |
Metrics about memory on the system |
/system/network/internode |
Metrics about internode requests made by the node |
/system/process |
Standard process metrics |
/system/go |
Standard Go lang metrics |
Cluster metrics
These present metrics about the whole MinIO cluster.
| Path | Description |
|---|---|
/cluster/health |
Cluster health metrics |
/cluster/usage/objects |
Object statistics |
/cluster/usage/buckets |
Object statistics by bucket |
/cluster/erasure-set |
Erasure set metrics |
Metrics Listing
Each of the following sub-sections list metrics returned by each of the endpoints.
The standard metrics group for GoCollector is not shown below.
/api/requests
| Name | Type | Help | Labels |
|---|---|---|---|
minio_api_requests_rejected_auth_total |
counter |
Total number of requests rejected for auth failure | type,pool_index,server |
minio_api_requests_rejected_header_total |
counter |
Total number of requests rejected for invalid header | type,pool_index,server |
minio_api_requests_rejected_timestamp_total |
counter |
Total number of requests rejected for invalid timestamp | type,pool_index,server |
minio_api_requests_rejected_invalid_total |
counter |
Total number of invalid requests | type,pool_index,server |
minio_api_requests_waiting_total |
gauge |
Total number of requests in the waiting queue | type,pool_index,server |
minio_api_requests_incoming_total |
gauge |
Total number of incoming requests | type,pool_index,server |
minio_api_requests_inflight_total |
gauge |
Total number of requests currently in flight | name,type,pool_index,server |
minio_api_requests_total |
counter |
Total number of requests | name,type,pool_index,server |
minio_api_requests_errors_total |
counter |
Total number of requests with (4xx and 5xx) errors | name,type,pool_index,server |
minio_api_requests_5xx_errors_total |
counter |
Total number of requests with 5xx errors | name,type,pool_index,server |
minio_api_requests_4xx_errors_total |
counter |
Total number of requests with 4xx errors | name,type,pool_index,server |
minio_api_requests_canceled_total |
counter |
Total number of requests canceled by the client | name,type,pool_index,server |
minio_api_requests_ttfb_seconds_distribution |
counter |
Distribution of time to first byte across API calls | name,type,le,pool_index,server |
minio_api_requests_traffic_sent_bytes |
counter |
Total number of bytes sent | type,pool_index,server |
minio_api_requests_traffic_received_bytes |
counter |
Total number of bytes received | type,pool_index,server |
/api/bucket
| Name | Type | Help | Labels |
|---|---|---|---|
minio_api_bucket_traffic_received_bytes |
counter |
Total number of bytes sent for a bucket | bucket,type,server,pool_index |
minio_api_bucket_traffic_sent_bytes |
counter |
Total number of bytes received for a bucket | bucket,type,server,pool_index |
minio_api_bucket_inflight_total |
gauge |
Total number of requests currently in flight for a bucket | bucket,name,type,server,pool_index |
minio_api_bucket_total |
counter |
Total number of requests for a bucket | bucket,name,type,server,pool_index |
minio_api_bucket_canceled_total |
counter |
Total number of requests canceled by the client for a bucket | bucket,name,type,server,pool_index |
minio_api_bucket_4xx_errors_total |
counter |
Total number of requests with 4xx errors for a bucket | bucket,name,type,server,pool_index |
minio_api_bucket_5xx_errors_total |
counter |
Total number of requests with 5xx errors for a bucket | bucket,name,type,server,pool_index |
minio_api_bucket_ttfb_seconds_distribution |
counter |
Distribution of time to first byte across API calls for a bucket | bucket,name,le,type,server,pool_index |
/system/drive
| Name | Type | Help | Labels |
|---|---|---|---|
minio_system_drive_used_bytes |
gauge |
Total storage used on a drive in bytes | drive,set_index,drive_index,pool_index,server |
minio_system_drive_free_bytes |
gauge |
Total storage free on a drive in bytes | drive,set_index,drive_index,pool_index,server |
minio_system_drive_total_bytes |
gauge |
Total storage available on a drive in bytes | drive,set_index,drive_index,pool_index,server |
minio_system_drive_used_inodes |
gauge |
Total used inodes on a drive | drive,set_index,drive_index,pool_index,server |
minio_system_drive_free_inodes |
gauge |
Total free inodes on a drive | drive,set_index,drive_index,pool_index,server |
minio_system_drive_total_inodes |
gauge |
Total inodes available on a drive | drive,set_index,drive_index,pool_index,server |
minio_system_drive_timeout_errors_total |
counter |
Total timeout errors on a drive | drive,set_index,drive_index,pool_index,server |
minio_system_drive_io_errors_total |
counter |
Total I/O errors on a drive | drive,set_index,drive_index,pool_index,server |
minio_system_drive_availability_errors_total |
counter |
Total availability errors (I/O errors, timeouts) on a drive | drive,set_index,drive_index,pool_index,server |
minio_system_drive_waiting_io |
gauge |
Total waiting I/O operations on a drive | drive,set_index,drive_index,pool_index,server |
minio_system_drive_api_latency_micros |
gauge |
Average last minute latency in µs for drive API storage operations | drive,api,set_index,drive_index,pool_index,server |
minio_system_drive_offline_count |
gauge |
Count of offline drives | pool_index,server |
minio_system_drive_online_count |
gauge |
Count of online drives | pool_index,server |
minio_system_drive_count |
gauge |
Count of all drives | pool_index,server |
minio_system_drive_healing |
gauge |
Is it healing? | drive,set_index,drive_index,pool_index,server |
minio_system_drive_online |
gauge |
Is it online? | drive,set_index,drive_index,pool_index,server |
minio_system_drive_reads_per_sec |
gauge |
Reads per second on a drive | drive,set_index,drive_index,pool_index,server |
minio_system_drive_reads_kb_per_sec |
gauge |
Kilobytes read per second on a drive | drive,set_index,drive_index,pool_index,server |
minio_system_drive_reads_await |
gauge |
Average time for read requests served on a drive | drive,set_index,drive_index,pool_index,server |
minio_system_drive_writes_per_sec |
gauge |
Writes per second on a drive | drive,set_index,drive_index,pool_index,server |
minio_system_drive_writes_kb_per_sec |
gauge |
Kilobytes written per second on a drive | drive,set_index,drive_index,pool_index,server |
minio_system_drive_writes_await |
gauge |
Average time for write requests served on a drive | drive,set_index,drive_index,pool_index,server |
minio_system_drive_perc_util |
gauge |
Percentage of time the disk was busy | drive,set_index,drive_index,pool_index,server |
/system/memory
| Name | Type | Help | Labels |
|---|---|---|---|
minio_system_memory_used |
gauge |
Used memory on the node | server |
minio_system_memory_used_perc |
gauge |
Used memory percentage on the node | server |
minio_system_memory_free |
gauge |
Free memory on the node | server |
minio_system_memory_total |
gauge |
Total memory on the node | server |
minio_system_memory_buffers |
gauge |
Buffers memory on the node | server |
minio_system_memory_cache |
gauge |
Cache memory on the node | server |
minio_system_memory_shared |
gauge |
Shared memory on the node | server |
minio_system_memory_available |
gauge |
Available memory on the node | server |
/system/cpu
| Name | Type | Help | Labels |
|---|---|---|---|
minio_system_cpu_avg_idle |
gauge |
Average CPU idle time | server |
minio_system_cpu_avg_iowait |
gauge |
Average CPU IOWait time | server |
minio_system_cpu_load |
gauge |
CPU load average 1min | server |
minio_system_cpu_load_perc |
gauge |
CPU load average 1min (percentage) | server |
minio_system_cpu_nice |
gauge |
CPU nice time | server |
minio_system_cpu_steal |
gauge |
CPU steal time | server |
minio_system_cpu_system |
gauge |
CPU system time | server |
minio_system_cpu_user |
gauge |
CPU user time | server |
/system/network/internode
| Name | Type | Help | Labels |
|---|---|---|---|
minio_system_network_internode_errors_total |
counter |
Total number of failed internode calls | server,pool_index |
minio_system_network_internode_dial_errors_total |
counter |
Total number of internode TCP dial timeouts and errors | server,pool_index |
minio_system_network_internode_dial_avg_time_nanos |
gauge |
Average dial time of internodes TCP calls in nanoseconds | server,pool_index |
minio_system_network_internode_sent_bytes_total |
counter |
Total number of bytes sent to other peer nodes | server,pool_index |
minio_system_network_internode_recv_bytes_total |
counter |
Total number of bytes received from other peer nodes | server,pool_index |
/system/process
| Name | Type | Help | Labels |
|---|---|---|---|
locks_read_total |
gauge |
Number of current READ locks on this peer | server |
locks_write_total |
gauge |
Number of current WRITE locks on this peer | server |
cpu_total_seconds |
counter |
Total user and system CPU time spent in seconds | server |
go_routine_total |
gauge |
Total number of go routines running | server |
io_rchar_bytes |
counter |
Total bytes read by the process from the underlying storage system including cache, /proc/[pid]/io rchar | server |
io_read_bytes |
counter |
Total bytes read by the process from the underlying storage system, /proc/[pid]/io read_bytes | server |
io_wchar_bytes |
counter |
Total bytes written by the process to the underlying storage system including page cache, /proc/[pid]/io wchar | server |
io_write_bytes |
counter |
Total bytes written by the process to the underlying storage system, /proc/[pid]/io write_bytes | server |
start_time_seconds |
gauge |
Start time for MinIO process in seconds since Unix epoc | server |
uptime_seconds |
gauge |
Uptime for MinIO process in seconds | server |
file_descriptor_limit_total |
gauge |
Limit on total number of open file descriptors for the MinIO Server process | server |
file_descriptor_open_total |
gauge |
Total number of open file descriptors by the MinIO Server process | server |
syscall_read_total |
counter |
Total read SysCalls to the kernel. /proc/[pid]/io syscr | server |
syscall_write_total |
counter |
Total write SysCalls to the kernel. /proc/[pid]/io syscw | server |
resident_memory_bytes |
gauge |
Resident memory size in bytes | server |
virtual_memory_bytes |
gauge |
Virtual memory size in bytes | server |
virtual_memory_max_bytes |
gauge |
Maximum virtual memory size in bytes | server |
/cluster/health
| Name | Type | Help | Labels |
|---|---|---|---|
minio_cluster_health_drives_offline_count |
gauge |
Count of offline drives in the cluster | |
minio_cluster_health_drives_online_count |
gauge |
Count of online drives in the cluster | |
minio_cluster_health_drives_count |
gauge |
Count of all drives in the cluster | |
minio_cluster_health_nodes_offline_count |
gauge |
Count of offline nodes in the cluster | |
minio_cluster_health_nodes_online_count |
gauge |
Count of online nodes in the cluster | |
minio_cluster_health_capacity_raw_total_bytes |
gauge |
Total cluster raw storage capacity in bytes | |
minio_cluster_health_capacity_raw_free_bytes |
gauge |
Total cluster raw storage free in bytes | |
minio_cluster_health_capacity_usable_total_bytes |
gauge |
Total cluster usable storage capacity in bytes | |
minio_cluster_health_capacity_usable_free_bytes |
gauge |
Total cluster usable storage free in bytes |
/cluster/audit
| Name | Type | Help | Labels |
|---|---|---|---|
minio_cluster_audit_failed_messages |
counter |
Total number of messages that failed to send since start | target_id |
minio_cluster_audit_target_queue_length |
gauge |
Number of unsent messages in queue for target | target_id |
minio_cluster_audit_total_messages |
counter |
Total number of messages sent since start | target_id |
/cluster/usage/objects
| Name | Type | Help | Labels |
|---|---|---|---|
minio_cluster_usage_objects_since_last_update_seconds |
gauge |
Time since last update of usage metrics in seconds | |
minio_cluster_usage_objects_total_bytes |
gauge |
Total cluster usage in bytes | |
minio_cluster_usage_objects_count |
gauge |
Total cluster objects count | |
minio_cluster_usage_objects_versions_count |
gauge |
Total cluster object versions (including delete markers) count | |
minio_cluster_usage_objects_delete_markers_count |
gauge |
Total cluster delete markers count | |
minio_cluster_usage_objects_buckets_count |
gauge |
Total cluster buckets count | |
minio_cluster_usage_objects_size_distribution |
gauge |
Cluster object size distribution | range |
minio_cluster_usage_objects_version_count_distribution |
gauge |
Cluster object version count distribution | range |
/cluster/usage/buckets
| Name | Type | Help | Labels |
|---|---|---|---|
minio_cluster_usage_buckets_since_last_update_seconds |
gauge |
Time since last update of usage metrics in seconds | |
minio_cluster_usage_buckets_total_bytes |
gauge |
Total bucket size in bytes | bucket |
minio_cluster_usage_buckets_objects_count |
gauge |
Total objects count in bucket | bucket |
minio_cluster_usage_buckets_versions_count |
gauge |
Total object versions (including delete markers) count in bucket | bucket |
minio_cluster_usage_buckets_delete_markers_count |
gauge |
Total delete markers count in bucket | bucket |
minio_cluster_usage_buckets_quota_total_bytes |
gauge |
Total bucket quota in bytes | bucket |
minio_cluster_usage_buckets_object_size_distribution |
gauge |
Bucket object size distribution | range,bucket |
minio_cluster_usage_buckets_object_version_count_distribution |
gauge |
Bucket object version count distribution | range,bucket |
/cluster/erasure-set
| Name | Type | Help | Labels |
|---|---|---|---|
minio_cluster_erasure_set_overall_write_quorum |
gauge |
Overall write quorum across pools and sets | |
minio_cluster_erasure_set_overall_health |
gauge |
Overall health across pools and sets (1=healthy, 0=unhealthy) | |
minio_cluster_erasure_set_read_quorum |
gauge |
Read quorum for the erasure set in a pool | pool_id,set_id |
minio_cluster_erasure_set_write_quorum |
gauge |
Write quorum for the erasure set in a pool | pool_id,set_id |
minio_cluster_erasure_set_online_drives_count |
gauge |
Count of online drives in the erasure set in a pool | pool_id,set_id |
minio_cluster_erasure_set_healing_drives_count |
gauge |
Count of healing drives in the erasure set in a pool | pool_id,set_id |
minio_cluster_erasure_set_health |
gauge |
Health of the erasure set in a pool (1=healthy, 0=unhealthy) | pool_id,set_id |
/cluster/notification
| Name | Type | Help | Labels |
|---|---|---|---|
minio_cluster_notification_current_send_in_progress |
counter |
Number of concurrent async Send calls active to all targets | |
minio_cluster_notification_events_errors_total |
counter |
Events that were failed to be sent to the targets | |
minio_cluster_notification_events_sent_total |
counter |
Total number of events sent to the targets | |
minio_cluster_notification_events_skipped_total |
counter |
Events that were skipped to be sent to the targets due to the in-memory queue being full |