mirror of
https://github.com/minio/minio.git
synced 2024-12-24 06:05:55 -05:00
docs: added event-notification back end doc. (#3510)
This commit is contained in:
koolhead17
Harshavardhana
parent
1b30a3be2b
commit
4dea4f3b89
477
docs/bucket-notifications/README.md
Normal file
477
docs/bucket-notifications/README.md
Normal file
@ -0,0 +1,477 @@
|
|||||||
|
[](https://slack.minio.io)
|
||||||
|
|
||||||
|
Minio server supports Amazon S3 compatible bucket event notification for the following targets [AMQP](https://www.amqp.org/about/what), [Elasticsearch](https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started.html) , [Redis](http://redis.io/documentation), [nats.io](http://nats.io/), [PostgreSQL](https://www.postgresql.org) and [kafka](https://kafka.apache.org/).
|
||||||
|
|
||||||
|
## Prerequisites
|
||||||
|
|
||||||
|
* Install and configure Minio Server from [here](http://docs.minio.io/docs/minio).
|
||||||
|
* Install and configure Minio Client from [here](https://docs.minio.io/docs/minio-client-quickstart-guide).
|
||||||
|
|
||||||
|
| Notification Targets|
|
||||||
|
|:--
|
||||||
|
| [`RabbitMQ`](#RabbitMQ) |
|
||||||
|
| [`Elasticsearch`](#Elasticsearch) |
|
||||||
|
| [`Redis`](#Redis) |
|
||||||
|
| [`NATS`](#NATS) |
|
||||||
|
| [`PostgreSQL`](#PostgreSQL) |
|
||||||
|
| [`kafka`](#kafka) |
|
||||||
|
|
||||||
|
<a name="RabbitMQ"></a>
|
||||||
|
## Publish Minio events via RabbitMQ
|
||||||
|
|
||||||
|
Install RabbitMQ from [here](https://www.rabbitmq.com/).
|
||||||
|
|
||||||
|
### Recipe steps
|
||||||
|
|
||||||
|
### Step 1: Add RabbitMQ endpoint to Minio
|
||||||
|
|
||||||
|
The default location of Minio server configuration file is ``~/.minio/config.json``. Update the RabbitMQ configuration block in ``config.json`` as follows:
|
||||||
|
|
||||||
|
```
|
||||||
|
"amqp": {
|
||||||
|
"1": {
|
||||||
|
"enable": true,
|
||||||
|
"url": "amqp://myuser:mypassword@localhost:5672",
|
||||||
|
"exchange": "bucketevents",
|
||||||
|
"routingKey": "bucketlogs",
|
||||||
|
"exchangeType": "fanout",
|
||||||
|
"mandatory": false,
|
||||||
|
"immediate": false,
|
||||||
|
"durable": false,
|
||||||
|
"internal": false,
|
||||||
|
"noWait": false,
|
||||||
|
"autoDeleted": false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
Restart Minio server to reflect config changes. Minio supports all the exchange available in [RabbitMQ](https://www.rabbitmq.com/). For this setup, we are using ``fanout`` exchange.
|
||||||
|
|
||||||
|
### Step 2: Enable bucket notification using Minio client
|
||||||
|
|
||||||
|
We will enable bucket event notification to trigger whenever a JPEG image is uploaded or deleted ``images`` bucket on ``myminio`` server. Here ARN value is ``arn:minio:sqs:us-east-1:1:amqp``. To understand more about ARN please follow [AWS ARN](http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html) documentation.
|
||||||
|
|
||||||
|
```
|
||||||
|
mc mb myminio/images
|
||||||
|
mc events add myminio/images arn:minio:sqs:us-east-1:1:amqp --suffix .jpg
|
||||||
|
mc events list myminio/images
|
||||||
|
arn:minio:sqs:us-east-1:1:amqp s3:ObjectCreated:*,s3:ObjectRemoved:* Filter: suffix=”.jpg”
|
||||||
|
```
|
||||||
|
|
||||||
|
### Step 3: Test on RabbitMQ
|
||||||
|
|
||||||
|
The python program below waits on the queue exchange ``bucketevents`` and prints event notifications on the console. We use [Pika Python Client](https://www.rabbitmq.com/tutorials/tutorial-three-python.html) library to do this.
|
||||||
|
|
||||||
|
```py
|
||||||
|
#!/usr/bin/env python
|
||||||
|
import pika
|
||||||
|
|
||||||
|
connection = pika.BlockingConnection(pika.ConnectionParameters(
|
||||||
|
host='localhost'))
|
||||||
|
channel = connection.channel()
|
||||||
|
|
||||||
|
channel.exchange_declare(exchange='bucketevents',
|
||||||
|
type='fanout')
|
||||||
|
|
||||||
|
result = channel.queue_declare(exclusive=False)
|
||||||
|
queue_name = result.method.queue
|
||||||
|
|
||||||
|
channel.queue_bind(exchange='bucketevents',
|
||||||
|
queue=queue_name)
|
||||||
|
|
||||||
|
print(' [*] Waiting for logs. To exit press CTRL+C')
|
||||||
|
|
||||||
|
def callback(ch, method, properties, body):
|
||||||
|
print(" [x] %r" % body)
|
||||||
|
|
||||||
|
channel.basic_consume(callback,
|
||||||
|
queue=queue_name,
|
||||||
|
no_ack=False)
|
||||||
|
|
||||||
|
channel.start_consuming()
|
||||||
|
```
|
||||||
|
|
||||||
|
Execute this example python program to watch for RabbitMQ events on the console.
|
||||||
|
|
||||||
|
```py
|
||||||
|
python rabbit.py
|
||||||
|
```
|
||||||
|
|
||||||
|
Open another terminal and upload a JPEG image into ``images`` bucket.
|
||||||
|
|
||||||
|
```
|
||||||
|
mc cp myphoto.jpg myminio/images
|
||||||
|
```
|
||||||
|
|
||||||
|
You should receive the following event notification via RabbitMQ once the upload completes.
|
||||||
|
|
||||||
|
```py
|
||||||
|
python rabbit.py
|
||||||
|
‘{“Records”:[{“eventVersion”:”2.0",”eventSource”:”aws:s3",”awsRegion”:”us-east-1",”eventTime”:”2016–09–08T22:34:38.226Z”,”eventName”:”s3:ObjectCreated:Put”,”userIdentity”:{“principalId”:”minio”},”requestParameters”:{“sourceIPAddress”:”10.1.10.150:44576"},”responseElements”:{},”s3":{“s3SchemaVersion”:”1.0",”configurationId”:”Config”,”bucket”:{“name”:”images”,”ownerIdentity”:{“principalId”:”minio”},”arn”:”arn:aws:s3:::images”},”object”:{“key”:”myphoto.jpg”,”size”:200436,”sequencer”:”147279EAF9F40933"}}}],”level”:”info”,”msg”:””,”time”:”2016–09–08T15:34:38–07:00"}\n
|
||||||
|
```
|
||||||
|
<a name="Elasticsearch"></a>
|
||||||
|
## Publish Minio events via Elasticsearch
|
||||||
|
|
||||||
|
Install Elasticsearch 2.4 from [here](https://www.elastic.co/downloads/past-releases/elasticsearch-2-4-0).
|
||||||
|
|
||||||
|
## Recipe steps
|
||||||
|
|
||||||
|
### Step 1: Add Elasticsearch endpoint to Minio
|
||||||
|
|
||||||
|
The default location of Minio server configuration file is ``~/.minio/config.json``. Update the Elasticsearch configuration block in ``config.json`` as follows:
|
||||||
|
|
||||||
|
```
|
||||||
|
"elasticsearch": {
|
||||||
|
"1": {
|
||||||
|
"enable": true,
|
||||||
|
"url": "http://127.0.0.1:9200",
|
||||||
|
"index": "bucketevents"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
```
|
||||||
|
Restart Minio server to reflect config changes. ``bucketevents`` is the index used by Elasticsearch.
|
||||||
|
|
||||||
|
### Step 2: Enable bucket notification using Minio client
|
||||||
|
|
||||||
|
We will enable bucket event notification to trigger whenever a JPEG image is uploaded or deleted from ``images`` bucket on ``myminio`` server. Here ARN value is ``arn:minio:sqs:us-east-1:1:elasticsearch``. To understand more about ARN please follow [AWS ARN](http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html) documentation.
|
||||||
|
|
||||||
|
```
|
||||||
|
mc mb myminio/images
|
||||||
|
mc events add myminio/images arn:minio:sqs:us-east-1:1:elasticsearch --suffix .jpg
|
||||||
|
mc events list myminio/images
|
||||||
|
arn:minio:sqs:us-east-1:1:elasticsearch s3:ObjectCreated:*,s3:ObjectRemoved:* Filter: suffix=”.jpg”
|
||||||
|
```
|
||||||
|
|
||||||
|
### Step 3: Test on Elasticsearch
|
||||||
|
|
||||||
|
Upload a JPEG image into ``images`` bucket, this is the bucket which has been configured for event notification.
|
||||||
|
|
||||||
|
```
|
||||||
|
mc cp myphoto.jpg myminio/images
|
||||||
|
```
|
||||||
|
|
||||||
|
Run ``curl`` to see new index name ``bucketevents`` in your Elasticsearch setup.
|
||||||
|
|
||||||
|
```
|
||||||
|
curl -XGET '127.0.0.1:9200/_cat/indices?v'
|
||||||
|
health status index pri rep docs.count docs.deleted store.size pri.store.size
|
||||||
|
yellow open bucketevents 5 1 1 0 7.8kb 7.8kb
|
||||||
|
```
|
||||||
|
|
||||||
|
Use curl to view contents of ``bucketevents`` index.
|
||||||
|
|
||||||
|
```
|
||||||
|
curl -XGET '127.0.0.1:9200/bucketevents/_search?pretty=1'
|
||||||
|
{
|
||||||
|
"took" : 3,
|
||||||
|
"timed_out" : false,
|
||||||
|
"_shards" : {
|
||||||
|
"total" : 5,
|
||||||
|
"successful" : 5,
|
||||||
|
"failed" : 0
|
||||||
|
},
|
||||||
|
"hits" : {
|
||||||
|
"total" : 1,
|
||||||
|
"max_score" : 1.0,
|
||||||
|
"hits" : [ {
|
||||||
|
"_index" : "bucketevents",
|
||||||
|
"_type" : "event",
|
||||||
|
"_id" : "AVcRVOlwe-uNB1tfj6bx",
|
||||||
|
"_score" : 1.0,
|
||||||
|
"_source" : {
|
||||||
|
"Records" : [ {
|
||||||
|
"eventVersion" : "2.0",
|
||||||
|
"eventSource" : "aws:s3",
|
||||||
|
"awsRegion" : "us-east-1",
|
||||||
|
"eventTime" : "2016-09-09T23:42:39.977Z",
|
||||||
|
"eventName" : "s3:ObjectCreated:Put",
|
||||||
|
"userIdentity" : {
|
||||||
|
"principalId" : "minio"
|
||||||
|
},
|
||||||
|
"requestParameters" : {
|
||||||
|
"sourceIPAddress" : "10.1.10.150:52140"
|
||||||
|
},
|
||||||
|
"responseElements" : { },
|
||||||
|
"s3" : {
|
||||||
|
"s3SchemaVersion" : "1.0",
|
||||||
|
"configurationId" : "Config",
|
||||||
|
"bucket" : {
|
||||||
|
"name" : "images",
|
||||||
|
"ownerIdentity" : {
|
||||||
|
"principalId" : "minio"
|
||||||
|
},
|
||||||
|
"arn" : "arn:aws:s3:::images"
|
||||||
|
},
|
||||||
|
"object" : {
|
||||||
|
"key" : "myphoto.jpg",
|
||||||
|
"size" : 200436,
|
||||||
|
"sequencer" : "1472CC35E6971AF3"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} ]
|
||||||
|
}
|
||||||
|
} ]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
``curl`` output above states that an Elasticsearch index has been successfully created with notification contents.
|
||||||
|
|
||||||
|
<a name="Redis"></a>
|
||||||
|
## Publish Minio events via Redis
|
||||||
|
|
||||||
|
Install Redis from [here](http://redis.io/download).
|
||||||
|
|
||||||
|
## Recipe steps
|
||||||
|
|
||||||
|
### Step 1: Add Redis endpoint to Minio
|
||||||
|
|
||||||
|
The default location of Minio server configuration file is ``~/.minio/config.json``. Update the Redis configuration block in ``config.json`` as follows:
|
||||||
|
|
||||||
|
```
|
||||||
|
"redis": {
|
||||||
|
"1": {
|
||||||
|
"enable": true,
|
||||||
|
"address": "127.0.0.1:6379",
|
||||||
|
"password": "yoursecret",
|
||||||
|
"key": "bucketevents"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
Restart Minio server to reflect config changes. ``bucketevents`` is the key used by Redis in this example.
|
||||||
|
|
||||||
|
### Step 2: Enable bucket notification using Minio client
|
||||||
|
|
||||||
|
We will enable bucket event notification to trigger whenever a JPEG image is uploaded or deleted from ``images`` bucket on ``myminio`` server. Here ARN value is ``arn:minio:sqs:us-east-1:1:redis``. To understand more about ARN please follow [AWS ARN](http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html) documentation.
|
||||||
|
|
||||||
|
```
|
||||||
|
mc mb myminio/images
|
||||||
|
mc events add myminio/images arn:minio:sqs:us-east-1:1:redis --suffix .jpg
|
||||||
|
mc events list myminio/images
|
||||||
|
arn:minio:sqs:us-east-1:1:redis s3:ObjectCreated:*,s3:ObjectRemoved:* Filter: suffix=”.jpg”
|
||||||
|
```
|
||||||
|
|
||||||
|
### Step 3: Test on Redis
|
||||||
|
|
||||||
|
Redis comes with a handy command line interface ``redis-cli`` to print all notifications on the console.
|
||||||
|
|
||||||
|
```
|
||||||
|
redis-cli -a yoursecret
|
||||||
|
```
|
||||||
|
|
||||||
|
Open another terminal and upload a JPEG image into ``images`` bucket.
|
||||||
|
|
||||||
|
```
|
||||||
|
mc cp myphoto.jpg myminio/images
|
||||||
|
```
|
||||||
|
|
||||||
|
``redis-cli`` prints event notification to the console.
|
||||||
|
|
||||||
|
```
|
||||||
|
redis-cli -a yoursecret
|
||||||
|
127.0.0.1:6379> monitor
|
||||||
|
OK
|
||||||
|
1474321638.556108 [0 127.0.0.1:40190] "AUTH" "yoursecret"
|
||||||
|
1474321638.556477 [0 127.0.0.1:40190] "RPUSH" "bucketevents" "{\"Records\":[{\"eventVersion\":\"2.0\",\"eventSource\":\"aws:s3\",\"awsRegion\":\"us-east-1\",\"eventTime\":\"2016-09-19T21:47:18.555Z\",\"eventName\":\"s3:ObjectCreated:Put\",\"userIdentity\":{\"principalId\":\"minio\"},\"requestParameters\":{\"sourceIPAddress\":\"[::1]:39250\"},\"responseElements\":{},\"s3\":{\"s3SchemaVersion\":\"1.0\",\"configurationId\":\"Config\",\"bucket\":{\"name\":\"images\",\"ownerIdentity\":{\"principalId\":\"minio\"},\"arn\":\"arn:aws:s3:::images\"},\"object\":{\"key\":\"myphoto.jpg\",\"size\":23745,\"sequencer\":\"1475D7B80ECBD853\"}}}],\"level\":\"info\",\"msg\":\"\",\"time\":\"2016-09-19T14:47:18-07:00\"}\n"
|
||||||
|
```
|
||||||
|
<a name="NATS"></a>
|
||||||
|
## Publish Minio events via NATS
|
||||||
|
|
||||||
|
Install NATS from [here](http://nats.io/).
|
||||||
|
|
||||||
|
## Recipe steps
|
||||||
|
|
||||||
|
### Step 1: Add NATS endpoint to Minio
|
||||||
|
|
||||||
|
The default location of Minio server configuration file is ``~/.minio/config.json``. Update the NATS configuration block in ``config.json`` as follows:
|
||||||
|
|
||||||
|
```
|
||||||
|
"nats": {
|
||||||
|
"1": {
|
||||||
|
"enable": true,
|
||||||
|
"address": "0.0.0.0:4222",
|
||||||
|
"subject": "bucketevents",
|
||||||
|
"username": "yourusername",
|
||||||
|
"password": "yoursecret",
|
||||||
|
"token": "",
|
||||||
|
"secure": false,
|
||||||
|
"pingInterval": 0
|
||||||
|
}
|
||||||
|
},
|
||||||
|
```
|
||||||
|
Restart Minio server to reflect config changes. ``bucketevents`` is the subject used by NATS in this example.
|
||||||
|
|
||||||
|
### Step 2: Enable bucket notification using Minio client
|
||||||
|
|
||||||
|
We will enable bucket event notification to trigger whenever a JPEG image is uploaded or deleted from ``images`` bucket on ``myminio`` server. Here ARN value is ``arn:minio:sqs:us-east-1:1:nats``. To understand more about ARN please follow [AWS ARN](http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html) documentation.
|
||||||
|
|
||||||
|
```
|
||||||
|
mc mb myminio/images
|
||||||
|
mc events add myminio/images arn:minio:sqs:us-east-1:1:nats --suffix .jpg
|
||||||
|
mc events list myminio/images
|
||||||
|
arn:minio:sqs:us-east-1:1:nats s3:ObjectCreated:*,s3:ObjectRemoved:* Filter: suffix=”.jpg”
|
||||||
|
```
|
||||||
|
|
||||||
|
### Step 3: Test on NATS
|
||||||
|
|
||||||
|
Using this program below we can log the bucket notification added to NATS.
|
||||||
|
|
||||||
|
```go
|
||||||
|
package main
|
||||||
|
|
||||||
|
// Import Go and NATS packages
|
||||||
|
import (
|
||||||
|
"runtime"
|
||||||
|
"log"
|
||||||
|
"github.com/nats-io/nats"
|
||||||
|
)
|
||||||
|
|
||||||
|
func main() {
|
||||||
|
|
||||||
|
// Create server connection
|
||||||
|
natsConnection, _ := nats.Connect("nats://yourusername:yoursecret@localhost:4222")
|
||||||
|
log.Println("Connected")
|
||||||
|
|
||||||
|
// Subscribe to subject
|
||||||
|
log.Printf("Subscribing to subject 'bucketevents'\n")
|
||||||
|
natsConnection.Subscribe("bucketevents", func(msg *nats.Msg) {
|
||||||
|
|
||||||
|
// Handle the message
|
||||||
|
log.Printf("Received message '%s\n", string(msg.Data) + "'")
|
||||||
|
})
|
||||||
|
|
||||||
|
// Keep the connection alive
|
||||||
|
runtime.Goexit()
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
```
|
||||||
|
go run nats.go
|
||||||
|
2016/10/12 06:39:18 Connected
|
||||||
|
2016/10/12 06:39:18 Subscribing to subject 'bucketevents'
|
||||||
|
```
|
||||||
|
|
||||||
|
Open another terminal and upload a JPEG image into ``images`` bucket.
|
||||||
|
|
||||||
|
```
|
||||||
|
mc cp myphoto.jpg myminio/images
|
||||||
|
```
|
||||||
|
|
||||||
|
The example ``nats.go`` program prints event notification to console.
|
||||||
|
|
||||||
|
```
|
||||||
|
go run nats.go
|
||||||
|
2016/10/12 06:51:26 Connected
|
||||||
|
2016/10/12 06:51:26 Subscribing to subject 'bucketevents'
|
||||||
|
2016/10/12 06:51:33 Received message '{"EventType":"s3:ObjectCreated:Put","Key":"images/myphoto.jpg","Records":[{"eventVersion":"2.0","eventSource":"aws:s3","awsRegion":"us-east-1","eventTime":"2016-10-12T13:51:33Z","eventName":"s3:ObjectCreated:Put","userIdentity":{"principalId":"minio"},"requestParameters":{"sourceIPAddress":"[::1]:57106"},"responseElements":{},"s3":{"s3SchemaVersion":"1.0","configurationId":"Config","bucket":{"name":"images","ownerIdentity":{"principalId":"minio"},"arn":"arn:aws:s3:::images"},"object":{"key":"myphoto.jpg","size":56060,"eTag":"1d97bf45ecb37f7a7b699418070df08f","sequencer":"147CCD1AE054BFD0"}}}],"level":"info","msg":"","time":"2016-10-12T06:51:33-07:00"}
|
||||||
|
```
|
||||||
|
|
||||||
|
<a name="PostgreSQL"></a>
|
||||||
|
## Publish Minio events via PostgreSQL
|
||||||
|
|
||||||
|
Install PostgreSQL from [here](https://www.postgresql.org/).
|
||||||
|
|
||||||
|
## Recipe steps
|
||||||
|
|
||||||
|
### Step 1: Add PostgreSQL endpoint to Minio
|
||||||
|
|
||||||
|
The default location of Minio server configuration file is ``~/.minio/config.json``. Update the PostgreSQL configuration block in ``config.json`` as follows:
|
||||||
|
|
||||||
|
```
|
||||||
|
"postgresql": {
|
||||||
|
"1": {
|
||||||
|
"enable": true,
|
||||||
|
"connectionString": "",
|
||||||
|
"table": "bucketevents",
|
||||||
|
"host": "127.0.0.1",
|
||||||
|
"port": "5432",
|
||||||
|
"user": "postgres",
|
||||||
|
"password": "mypassword",
|
||||||
|
"database": "bucketevents_db"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
Restart Minio server to reflect config changes. ``bucketevents`` is the database table used by PostgreSQL in this example.
|
||||||
|
|
||||||
|
### Step 2: Enable bucket notification using Minio client
|
||||||
|
|
||||||
|
We will enable bucket event notification to trigger whenever a JPEG image is uploaded or deleted from ``images`` bucket on ``myminio`` server. Here ARN value is ``arn:minio:sqs:us-east-1:1:postgresql``. To understand more about ARN please follow [AWS ARN](http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html) documentation.
|
||||||
|
|
||||||
|
```
|
||||||
|
mc mb myminio/images
|
||||||
|
mc events add myminio/images arn:minio:sqs:us-east-1:1:postgresql --suffix .jpg
|
||||||
|
mc events list myminio/images
|
||||||
|
arn:minio:sqs:us-east-1:1:postgresql s3:ObjectCreated:*,s3:ObjectRemoved:* Filter: suffix=”.jpg”
|
||||||
|
```
|
||||||
|
|
||||||
|
### Step 3: Test on PostgreSQL
|
||||||
|
|
||||||
|
Open another terminal and upload a JPEG image into ``images`` bucket.
|
||||||
|
|
||||||
|
```
|
||||||
|
mc cp myphoto.jpg myminio/images
|
||||||
|
```
|
||||||
|
|
||||||
|
Open PostgreSQL terminal to list the saved event notification logs.
|
||||||
|
|
||||||
|
```
|
||||||
|
bucketevents_db=# select * from bucketevents;
|
||||||
|
|
||||||
|
key | value
|
||||||
|
--------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|
||||||
|
images/myphoto.jpg | {"Records": [{"s3": {"bucket": {"arn": "arn:aws:s3:::images", "name": "images", "ownerIdentity": {"principalId": "minio"}}, "object": {"key": "myphoto.jpg", "eTag": "1d97bf45ecb37f7a7b699418070df08f", "size": 56060, "sequencer": "147CE57C70B31931"}, "configurationId": "Config", "s3SchemaVersion": "1.0"}, "awsRegion": "us-east-1", "eventName": "s3:ObjectCreated:Put", "eventTime": "2016-10-12T21:18:20Z", "eventSource": "aws:s3", "eventVersion": "2.0", "userIdentity": {"principalId": "minio"}, "responseElements": {}, "requestParameters": {"sourceIPAddress": "[::1]:39706"}}]}
|
||||||
|
(1 row)
|
||||||
|
```
|
||||||
|
|
||||||
|
<a name="kafka"></a>
|
||||||
|
## Publish Minio events via kafka
|
||||||
|
|
||||||
|
Install kafka from [here](http://kafka.apache.org/).
|
||||||
|
|
||||||
|
## Recipe steps
|
||||||
|
|
||||||
|
### Step 1: Add kafka endpoint to Minio
|
||||||
|
|
||||||
|
The default location of Minio server configuration file is ``~/.minio/config.json``. Update the kafka configuration block in ``config.json`` as follows:
|
||||||
|
|
||||||
|
```
|
||||||
|
"kafka": {
|
||||||
|
"1": {
|
||||||
|
"enable": true,
|
||||||
|
"brokers": ["localhost:9092"],
|
||||||
|
"topic": "bucketevents"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
Restart Minio server to reflect config changes. ``bucketevents`` is the topic used by kafka in this example.
|
||||||
|
|
||||||
|
### Step 2: Enable bucket notification using Minio client
|
||||||
|
|
||||||
|
We will enable bucket event notification to trigger whenever a JPEG image is uploaded or deleted from ``images`` bucket on ``myminio`` server. Here ARN value is ``arn:minio:sqs:us-east-1:1:kafka``. To understand more about ARN please follow [AWS ARN](http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html) documentation.
|
||||||
|
|
||||||
|
```
|
||||||
|
mc mb myminio/images
|
||||||
|
mc events add myminio/images arn:minio:sqs:us-east-1:1:kafka --suffix .jpg
|
||||||
|
mc events list myminio/images
|
||||||
|
arn:minio:sqs:us-east-1:1:kafka s3:ObjectCreated:*,s3:ObjectRemoved:* Filter: suffix=”.jpg”
|
||||||
|
```
|
||||||
|
|
||||||
|
### Step 3: Test on kafka
|
||||||
|
|
||||||
|
We used [kafkacat](https://github.com/edenhill/kafkacat) to print all notifications on the console.
|
||||||
|
|
||||||
|
```
|
||||||
|
kafkacat -b localhost:9092 -t bucketevents
|
||||||
|
```
|
||||||
|
|
||||||
|
Open another terminal and upload a JPEG image into ``images`` bucket.
|
||||||
|
|
||||||
|
```
|
||||||
|
mc cp myphoto.jpg myminio/images
|
||||||
|
```
|
||||||
|
|
||||||
|
``kafkacat`` prints the event notification to the console.
|
||||||
|
|
||||||
|
```
|
||||||
|
kafkacat -b localhost:9092 -t bucketevents
|
||||||
|
{"EventType":"s3:ObjectCreated:Put","Key":"images/myphoto.jpg","Records":[{"eventVersion":"2.0","eventSource":"aws:s3","awsRegion":"us-east-1","eventTime":"2017-01-31T10:01:51Z","eventName":"s3:ObjectCreated:Put","userIdentity":{"principalId":"88QR09S7IOT4X1IBAQ9B"},"requestParameters":{"sourceIPAddress":"192.173.5.2:57904"},"responseElements":{"x-amz-request-id":"149ED2FD25589220","x-minio-origin-endpoint":"http://192.173.5.2:9000"},"s3":{"s3SchemaVersion":"1.0","configurationId":"Config","bucket":{"name":"images","ownerIdentity":{"principalId":"88QR09S7IOT4X1IBAQ9B"},"arn":"arn:aws:s3:::images"},"object":{"key":"myphoto.jpg","size":541596,"eTag":"04451d05b4faf4d62f3d538156115e2a","sequencer":"149ED2FD25589220"}}}],"level":"info","msg":"","time":"2017-01-31T15:31:51+05:30"}
|
||||||
|
```
|
||||||
|
*NOTE* If you are running [distributed Minio](https://docs.minio.io/docs/distributed-minio-quickstart-guide), modify ``~/.minio/config.json`` on all the nodes with your bucket event notification backend configuration.
|
||||||
Loading…
Reference in New Issue
Block a user