MyFavMirrors/minio
1
0
Fork 0
mirror of https://github.com/minio/minio.git synced 2025-11-10 05:59:43 -05:00
Code Issues Packages Projects Releases Wiki Activity

feat: Add support for kakfa audit logger target (#12678)

Browse Source
This commit is contained in:
Harshavardhana
2021-07-13 09:39:13 -07:00
committed by GitHub
parent 559d075627
commit e316873f84
14 changed files with 811 additions and 202 deletions
Download Patch File Download Diff File Expand all files Collapse all files

81
docs/logging/README.md
View File

@@ -36,9 +36,11 @@ minio server /mnt/data
## Audit Targets
Assuming `mc` is already [configured](https://docs.min.io/docs/minio-client-quickstart-guide.html)
### HTTP Target
```
mc admin config get myminio/ audit_webhook
audit_webhook:name1 enable=off endpoint= auth_token= client_cert= client_key=
audit_webhook:name1 enable=off endpoint= auth_token= client_cert= client_key=
```
```
@@ -119,6 +121,83 @@ NOTE:
}
```
### Kafka Target
Assuming that you already have Apache Kafka configured and running.
```
mc admin config set myminio/ audit_kafka
KEY:
audit_kafka[:name] send audit logs to kafka endpoints
ARGS:
brokers* (csv) comma separated list of Kafka broker addresses
topic (string) Kafka topic used for bucket notifications
sasl_username (string) username for SASL/PLAIN or SASL/SCRAM authentication
sasl_password (string) password for SASL/PLAIN or SASL/SCRAM authentication
sasl_mechanism (string) sasl authentication mechanism, default 'plain'
tls_client_auth (string) clientAuth determines the Kafka server's policy for TLS client auth
sasl (on|off) set to 'on' to enable SASL authentication
tls (on|off) set to 'on' to enable TLS
tls_skip_verify (on|off) trust server TLS without verification, defaults to "on" (verify)
client_tls_cert (path) path to client certificate for mTLS auth
client_tls_key (path) path to client key for mTLS auth
version (string) specify the version of the Kafka cluster
comment (sentence) optionally add a comment to this setting
```
Configure MinIO to send audit logs to locally running Kafka brokers
```
mc admin config set myminio/ audit_kafka:target1 brokers=localhost:29092 topic=auditlog
mc admin service restart myminio/
```
On another terminal assuming you have `kafkacat` installed
```
kafkacat -b localhost:29092 -t auditlog -C
{"version":"1","deploymentid":"8a1d8091-b874-45df-b9ea-e044eede6ace","time":"2021-07-13T02:00:47.020547414Z","trigger":"incoming","api":{"name":"ListBuckets","status":"OK","statusCode":200,"timeToFirstByte":"261795ns","timeToResponse":"312490ns"},"remotehost":"127.0.0.1","requestID":"16913736591C237F","userAgent":"MinIO (linux; amd64) minio-go/v7.0.11 mc/DEVELOPMENT.2021-07-09T02-22-26Z","requestHeader":{"Authorization":"AWS4-HMAC-SHA256 Credential=minio/20210713/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=7fe65c5467e05ca21de64094688da43f96f34fec82e8955612827079f4600527","User-Agent":"MinIO (linux; amd64) minio-go/v7.0.11 mc/DEVELOPMENT.2021-07-09T02-22-26Z","X-Amz-Content-Sha256":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855","X-Amz-Date":"20210713T020047Z"},"responseHeader":{"Accept-Ranges":"bytes","Content-Length":"547","Content-Security-Policy":"block-all-mixed-content","Content-Type":"application/xml","Server":"MinIO","Vary":"Origin,Accept-Encoding","X-Amz-Request-Id":"16913736591C237F","X-Xss-Protection":"1; mode=block"}}
```
MinIO also honors environment variable for Kafka target Audit logging as shown below, this setting will override the endpoint settings in the MinIO server config.
```
mc admin config set myminio/ audit_kafka --env
KEY:
audit_kafka[:name] send audit logs to kafka endpoints
ARGS:
MINIO_AUDIT_KAFKA_ENABLE* (on|off) enable audit_kafka target, default is 'off'
MINIO_AUDIT_KAFKA_BROKERS* (csv) comma separated list of Kafka broker addresses
MINIO_AUDIT_KAFKA_TOPIC (string) Kafka topic used for bucket notifications
MINIO_AUDIT_KAFKA_SASL_USERNAME (string) username for SASL/PLAIN or SASL/SCRAM authentication
MINIO_AUDIT_KAFKA_SASL_PASSWORD (string) password for SASL/PLAIN or SASL/SCRAM authentication
MINIO_AUDIT_KAFKA_SASL_MECHANISM (string) sasl authentication mechanism, default 'plain'
MINIO_AUDIT_KAFKA_TLS_CLIENT_AUTH (string) clientAuth determines the Kafka server's policy for TLS client auth
MINIO_AUDIT_KAFKA_SASL (on|off) set to 'on' to enable SASL authentication
MINIO_AUDIT_KAFKA_TLS (on|off) set to 'on' to enable TLS
MINIO_AUDIT_KAFKA_TLS_SKIP_VERIFY (on|off) trust server TLS without verification, defaults to "on" (verify)
MINIO_AUDIT_KAFKA_CLIENT_TLS_CERT (path) path to client certificate for mTLS auth
MINIO_AUDIT_KAFKA_CLIENT_TLS_KEY (path) path to client key for mTLS auth
MINIO_AUDIT_KAFKA_VERSION (string) specify the version of the Kafka cluster
MINIO_AUDIT_KAFKA_COMMENT (sentence) optionally add a comment to this setting
```
```
export MINIO_AUDIT_KAFKA_ENABLE_target1="on"
export MINIO_AUDIT_KAFKA_BROKERS_target1="localhost:29092"
export MINIO_AUDIT_KAFKA_TOPIC_target1="auditlog"
minio server /mnt/data
```
Setting this environment variable automatically enables audit logging to the Kafka target. The audit logging is in JSON format as described below.
NOTE:
- `timeToFirstByte` and `timeToResponse` will be expressed in Nanoseconds.
- Additionally in the case of the erasure coded setup `tags.objectErasureMap` provides per object details about
- Pool number the object operation was performed on.
- Set number the object operation was performed on.
- The list of disks participating in this operation belong to the set.
## Explore Further
* [MinIO Quickstart Guide](https://docs.min.io/docs/minio-quickstart-guide)
* [Configure MinIO Server with TLS](https://docs.min.io/docs/how-to-secure-access-to-minio-server-with-tls)