Commit Graph

1558 Commits

Author SHA1 Message Date
Shubhendu
468a9fae83
Enable replication of SSE-C objects (#19107)
If site replication enabled across sites, replicate the SSE-C
objects as well. These objects could be read from target sites
using the same client encryption keys.

Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
2024-03-28 10:44:56 -07:00
Shubhendu
d87f91720b
Split the replication dashboard in cluster and node level (#19374)
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
2024-03-28 10:15:39 -07:00
Shubhendu
d63e603040
Pre populate the server names using a query (#19367)
User doesn't need to remember and enter the server values,
rather they can select from the pre populated list.

Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
2024-03-28 08:14:26 -07:00
Shubhendu
3d4fc28ec9
Render node graphs by node (#19356)
As total drives count, online vs offline are per node basis, its
corect to select node for which graphs need to be rendered.

Set prometheus scrape jobs to fetch metrics from all nodes. A sample
scrape job for node metrics could be as below

```
- job_name: minio-job-node
  bearer_token: <token>
  metrics_path: /minio/v2/metrics/node
  scheme: https
  tls_config:
    insecure_skip_verify: true
  static_configs:
  - targets: [tenant1-ss-0-0.tenant1-hl.tenant-ns.svc.cluster.local:9000,tenant1-ss-0-1.tenant1-hl.tenant-ns.svc.cluster.local:9000,tenant1-ss-0-2.tenant1-hl.tenant-ns.svc.cluster.local:9000,tenant1-ss-0-3.tenant1-hl.tenant-ns.svc.cluster.local:9000]
```

Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
2024-03-27 10:41:08 -07:00
Minio Trusted
428155add9 Update yaml files to latest version RELEASE.2024-03-26T22-10-45Z 2024-03-26 23:12:03 +00:00
Shubhendu
53a14c7301
Adding dashboard for MinIO node metrics (#19329)
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
2024-03-26 08:01:28 -07:00
Minio Trusted
15b930be1f Update yaml files to latest version RELEASE.2024-03-21T23-13-43Z 2024-03-22 20:08:28 +00:00
Minio Trusted
d2373d5d6c Update yaml files to latest version RELEASE.2024-03-15T01-07-19Z 2024-03-15 02:47:20 +00:00
Harshavardhana
062f0cffad
fix: do not look for non-existent bucket in decom tests (#19261) 2024-03-14 08:54:11 -07:00
Klaus Post
a2f6252b2f
xl-meta: Add inline data bitrot check (#19240)
When using `-data` also perform a bitrot check on the data.

Example:

```
λ xl-meta -data net.zip
{
        "minio-1.com:9000/data/minio1/p40/44b6/44b612e9a7294856bd2b5fe6f6cdeb0d.pdf/xl.meta": {"null":{"bitrot_valid":true,"bytes":11710}},
        "minio-1.com:9000/data/minio2/p40/44b6/44b612e9a7294856bd2b5fe6f6cdeb0d.pdf/xl.meta": {"null":{"bitrot_valid":true,"bytes":11710}},
        "minio-1.com:9000/data/minio3/p40/44b6/44b612e9a7294856bd2b5fe6f6cdeb0d.pdf/xl.meta": {"null":{"bitrot_valid":true,"bytes":11710}},
        "minio-1.com:9000/data/minio4/p40/44b6/44b612e9a7294856bd2b5fe6f6cdeb0d.pdf/xl.meta": {"null":{"bitrot_valid":true,"bytes":11710}},
        "minio-2.com:9000/data/minio1/p40/44b6/44b612e9a7294856bd2b5fe6f6cdeb0d.pdf/xl.meta": {"null":{"bitrot_valid":true,"bytes":11710}},
        "minio-2.com:9000/data/minio2/p40/44b6/44b612e9a7294856bd2b5fe6f6cdeb0d.pdf/xl.meta": {"null":{"bitrot_valid":true,"bytes":11710}},
        "minio-2.com:9000/data/minio3/p40/44b6/44b612e9a7294856bd2b5fe6f6cdeb0d.pdf/xl.meta": {"null":{"bitrot_valid":true,"bytes":11710}},
        "minio-2.com:9000/data/minio4/p40/44b6/44b612e9a7294856bd2b5fe6f6cdeb0d.pdf/xl.meta": {"null":{"bitrot_valid":true,"bytes":11710}},
        "minio-3.com:9000/data/minio1/p40/44b6/44b612e9a7294856bd2b5fe6f6cdeb0d.pdf/xl.meta": {"null":{"bitrot_valid":true,"bytes":11710}},
        "minio-3.com:9000/data/minio2/p40/44b6/44b612e9a7294856bd2b5fe6f6cdeb0d.pdf/xl.meta": {"null":{"bitrot_valid":true,"bytes":11710}},
        "minio-3.com:9000/data/minio3/p40/44b6/44b612e9a7294856bd2b5fe6f6cdeb0d.pdf/xl.meta": {"null":{"bitrot_valid":true,"bytes":11710}},
        "minio-3.com:9000/data/minio4/p40/44b6/44b612e9a7294856bd2b5fe6f6cdeb0d.pdf/xl.meta": {"null":{"bitrot_valid":true,"bytes":11710}},
        "minio-4.com:9000/data/minio1/p40/44b6/44b612e9a7294856bd2b5fe6f6cdeb0d.pdf/xl.meta": {"null":{"bitrot_valid":true,"bytes":11710}},
        "minio-4.com:9000/data/minio2/p40/44b6/44b612e9a7294856bd2b5fe6f6cdeb0d.pdf/xl.meta": {"null":{"bitrot_valid":true,"bytes":11710}},
        "minio-4.com:9000/data/minio3/p40/44b6/44b612e9a7294856bd2b5fe6f6cdeb0d.pdf/xl.meta": {"null":{"bitrot_valid":true,"bytes":11710}},
        "minio-4.com:9000/data/minio4/p40/44b6/44b612e9a7294856bd2b5fe6f6cdeb0d.pdf/xl.meta": {"null":{"bitrot_valid":true,"bytes":11710}}
}
```
2024-03-11 10:57:11 -07:00
Dennis Marttinen
6c964fede5
Improve handling of compression inclusion for objects (#19234) 2024-03-11 04:55:34 -07:00
Aditya Manthramurthy
b2c5b75efa
feat: Add Metrics V3 API (#19068)
Metrics v3 is mainly a reorganization of metrics into smaller groups of
metrics and the removal of internal aggregation of metrics received from
peer nodes in a MinIO cluster.

This change adds the endpoint `/minio/metrics/v3` as the top-level metrics
endpoint and under this, various sub-endpoints are implemented. These
are currently documented in `docs/metrics/v3.md`

The handler will serve metrics at any path
`/minio/metrics/v3/PATH`, as follows:

when PATH is a sub-endpoint listed above => serves the group of
metrics under that path; or when PATH is a (non-empty) parent 
directory of the sub-endpoints listed above => serves metrics
from each child sub-endpoint of PATH. otherwise, returns a no 
resource found error

All available metrics are listed in the `docs/metrics/v3.md`. More will
be added subsequently.
2024-03-10 01:15:15 -08:00
Minio Trusted
2dfa9adc5d Update yaml files to latest version RELEASE.2024-03-10T02-53-48Z 2024-03-10 08:42:35 +00:00
Minio Trusted
8ab3dac4f2 Update yaml files to latest version RELEASE.2024-03-07T00-43-48Z 2024-03-07 06:25:40 +00:00
Poorna
837a2a3d4b
sr: use service account cred for claims check (#19209)
PR #19111 overlaid service account secret with site replicator secret
during token claims check.

Fixes : #19206
2024-03-06 16:19:24 -08:00
Harshavardhana
74ccee6619
avoid too much auditing during decom/rebalance make it more robust (#19174)
there can be a sudden spike in tiny allocations,
due to too much auditing being done, also don't hang
on the

```
h.logCh <- entry
```

after initializing workers if you do not have a way to
dequeue for some reason.
2024-03-06 03:43:16 -08:00
Minio Trusted
e3e3d92241 Update yaml files to latest version RELEASE.2024-03-05T04-48-44Z 2024-03-05 06:20:43 +00:00
Aditya Manthramurthy
9a4d003ac7
Add common middleware to S3 API handlers (#19171)
The middleware sets up tracing, throttling, gzipped responses and
collecting API stats.

Additionally, this change updates the names of handler functions in
metric labels to be the same as the name derived from Go lang reflection
on the handler name.

The metric api labels are now stored in memory the same as the handler
name - they will be camelcased, e.g. `GetObject` instead of `getobject`.

For compatibility, we lowercase the metric api label values when emitting the metrics.
2024-03-04 10:05:56 -08:00
Minio Trusted
18b0b7299a Update yaml files to latest version RELEASE.2024-03-03T17-50-39Z 2024-03-03 18:33:43 +00:00
Ravind Kumar
f3e7c42425
Update metrics list.md with new metrics from RELEASE.2024-01-05 (#19161) 2024-02-29 14:53:54 -08:00
Shubhendu
f46bee242c
Re-organized grafana dashboards (#19157)
Moved different dashboards to their specific directories. Also
mentioned that these dashbards are examples of how to create
graphs using MinIO provided and metrics and customers should
change / add graphs on their specific need basis.

Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
2024-02-29 10:35:20 -08:00
Anis Eleuch
828d4df6f0
debug: Add --search to print only specific goroutines (#19158)
Easier to filter goroutines belonging to a specific subsystem
2024-02-29 08:28:18 -08:00
Poorna
b1351e2dee sr: use site replicator svcacct to sign STS session tokens (#19111)
This change is to decouple need for root credentials to match between
 site replication deployments.

 Also ensuring site replication config initialization is re-tried until
 it succeeds, this deoendency is critical to STS flow in site replication
 scenario.
2024-02-26 13:30:28 -08:00
Minio Trusted
85bcb5874a Update yaml files to latest version RELEASE.2024-02-26T09-33-48Z 2024-02-26 10:28:02 +00:00
Minio Trusted
b49ce1713f Update yaml files to latest version RELEASE.2024-02-24T17-11-14Z 2024-02-25 05:23:21 +00:00
schmittey
c44f311c4f
Add missing yaml syntax highlighting in prometheus README.md (#19087) 2024-02-20 16:22:37 -08:00
Shubhendu
cb7dab17cb
Graph cluster and bucket replication proxied requests (#19078)
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
2024-02-20 01:45:00 -08:00
Minio Trusted
23c10350f3 Update yaml files to latest version RELEASE.2024-02-17T01-15-57Z 2024-02-17 01:37:10 +00:00
Harshavardhana
b6e98aed01
fix: found races in accessing globalLocalDrives (#19069)
make a copy before accessing globalLocalDrives

Bonus: update console v0.46.0

Signed-off-by: Harshavardhana <harsha@minio.io>
2024-02-16 17:15:57 -08:00
Minio Trusted
b5791e6f28 Update yaml files to latest version RELEASE.2024-02-14T21-36-02Z 2024-02-14 21:51:12 +00:00
Harshavardhana
00cb58eaf3
add customer specific hotfixes to 'registry.min.dev' (#19057)
```
REPO="registry.min.dev/<customer>" CRED_DIR=/media/builder/minio make docker-hotfix-push
```
2024-02-14 13:36:02 -08:00
Poorna
912a0031b7
fix sr tests to capture all server logs (#19051) 2024-02-13 20:51:23 -08:00
Minio Trusted
f8e15e7d09 Update yaml files to latest version RELEASE.2024-02-13T15-35-11Z 2024-02-13 16:01:38 +00:00
Harshavardhana
0c068b15c7
add missing handler for reloading site replication config on peers (#19042) 2024-02-13 06:55:54 -08:00
Minio Trusted
6b63123ca9 Update yaml files to latest version RELEASE.2024-02-12T21-02-27Z 2024-02-12 21:40:49 +00:00
Harshavardhana
eac4e4b279
honor replaced disk properly by updating globalLocalDrives (#19038)
globalLocalDrives seem to be not updated during the
HealFormat() leads to a requirement where the server
needs to be restarted for the healing to continue.
2024-02-12 13:00:20 -08:00
Minio Trusted
8e69f3cb89 Update yaml files to latest version RELEASE.2024-02-09T21-25-16Z 2024-02-09 22:48:07 +00:00
Harshavardhana
997ba3a574
introduce reader deadlines for net.Conn (#19023)
Bonus: set "retry-after" header for AWS SDKs if possible to honor them.
2024-02-09 13:25:16 -08:00
Aditya Manthramurthy
a29c66ed74
Update IAM access manager plugin demo (#19007)
Now prints the JSON payload for easier debugging.
2024-02-08 09:15:20 -08:00
Minio Trusted
eead4db1d2 Update yaml files to latest version RELEASE.2024-02-06T21-36-22Z 2024-02-07 12:11:36 +00:00
Poorna
27d02ea6f7
metrics: add replication metrics on proxied requests (#18957) 2024-02-05 22:00:45 -08:00
Minio Trusted
8414aff424 Update yaml files to latest version RELEASE.2024-02-04T22-36-13Z 2024-02-05 00:46:41 +00:00
Minio Trusted
ee0055b929 Update yaml files to latest version RELEASE.2024-01-31T20-20-33Z 2024-01-31 20:35:33 +00:00
Daniel Valdivia
403ec7cf21
fix: metrics URI path in prometheus docs (#18907) 2024-01-29 14:34:21 -08:00
Minio Trusted
ff4f4d4649 Update yaml files to latest version RELEASE.2024-01-29T03-56-32Z 2024-01-29 05:33:23 +00:00
Minio Trusted
ff8269575a Update yaml files to latest version RELEASE.2024-01-28T22-35-53Z 2024-01-29 01:22:56 +00:00
Harshavardhana
944f3c1477
remove local disk metrics from cluster metrics (#18886)
local disk metrics were polluting cluster metrics
Please remove them instead of adding relevant ones.

- batch job metrics were incorrectly kept at bucket
  metrics endpoint, move it to cluster metrics.

- add tier metrics to cluster peer metrics from the node.

- fix missing set level cluster health metrics
2024-01-28 12:53:59 -08:00
Cesar N
1a91edecae
Update list.md node_cpu wording (#18878) 2024-01-26 18:57:58 -08:00
Harshavardhana
e11d851aee
add new drive I/O waiting/tokens metric (#18836)
Bonus: add virtual memory used as well part of the system resource metrics.
2024-01-19 14:51:36 -08:00
Minio Trusted
c66c5828ea Update yaml files to latest version RELEASE.2024-01-18T22-51-28Z 2024-01-19 11:35:32 +00:00