MyFavMirrors/minio
1
0
Fork 0
mirror of https://github.com/minio/minio.git synced 2025-11-07 21:02:58 -05:00
Code Issues Packages Projects Releases Wiki Activity

feature: added nsq as broker for events (#6740)

Browse Source
This commit is contained in:
Matthias Schneider
2018-11-07 19:23:13 +01:00
committed by kannappanr
parent bf414068a3
commit 71c66464c1
30 changed files with 4884 additions and 45 deletions
Download Patch File Download Diff File Expand all files Collapse all files

73
docs/bucket/notifications/README.md
View File

@@ -17,6 +17,7 @@ Bucket events can be published to the following targets:
| [`AMQP`](#AMQP) | [`Redis`](#Redis) | [`MySQL`](#MySQL) |
| [`MQTT`](#MQTT) | [`NATS`](#NATS) | [`Apache Kafka`](#apache-kafka) |
| [`Elasticsearch`](#Elasticsearch) | [`PostgreSQL`](#PostgreSQL) | [`Webhooks`](#webhooks) |
| [`NSQ`](#NSQ) | | |
## Prerequisites
@@ -1007,5 +1008,77 @@ mc ls myminio/images-thumbnail
[2017-02-08 11:39:40 IST] 992B images-thumbnail.jpg
```
<a name="NSQ"></a>
## Publish Minio events to NSQ
Install an NSQ Daemon from [here](https://nsq.io/). Or use the following Docker
command for starting an nsq daemon:
```
docker run --rm -p 4150-4151:4150-4151 nsqio/nsq /nsqd
```
### Step 1: Add NSQ endpoint to Minio
The Minio server configuration file is stored on the backend in json format. The NSQ configuration is located in the `nsq` key under the `notify` top-level key. Create a configuration key-value pair here for your NSQ instance. The key is a name for your NSQ endpoint, and the value is a collection of key-value parameters.
An example configuration for NSQ is shown below:
```json
"nsq": {
"1": {
"enable": true,
"nsqdAddress": "127.0.0.1:4150",
"topic": "minio",
"tls": {
"enable": false,
"skipVerify": true
}
}
}
```
To update the configuration, use `mc admin config get` command to get the current configuration file for the minio deployment in json format, and save it locally.
```sh
$ mc admin config get myminio/ > /tmp/myconfig
```
After updating the NSQ configuration in /tmp/myconfig , use `mc admin config set` command to update the configuration for the deployment.Restart the Minio server to put the changes into effect. The server will print a line like `SQS ARNs: arn:minio:sqs::1:nsq` at start-up if there were no errors.
```sh
$ mc admin config set myminio < /tmp/myconfig
```
Note that, you can add as many NSQ daemon endpoint configurations as needed by providing an identifier (like "1" in the example above) for the NSQ instance and an object of per-server configuration parameters.
### Step 2: Enable bucket notification using Minio client
We will enable bucket event notification to trigger whenever a JPEG image is uploaded or deleted ``images`` bucket on ``myminio`` server. Here ARN value is ``arn:minio:sqs::1:nsq``.
```
mc mb myminio/images
mc events add myminio/images arn:minio:sqs::1:nsq --suffix .jpg
mc events list myminio/images
arn:minio:sqs::1:nsq s3:ObjectCreated:*,s3:ObjectRemoved:* Filter: suffix=”.jpg”
```
### Step 3: Test on NSQ
The simplest test is to download `nsq_tail` from [nsq github](https://github.com/nsqio/nsq/releases)
```
./nsq_tail -nsqd-tcp-address 127.0.0.1:4150 -topic minio
```
Open another terminal and upload a JPEG image into ``images`` bucket.
```
mc cp gopher.jpg myminio/images
```
You should receive the following event notification via NSQ once the upload completes.
```
{"EventName":"s3:ObjectCreated:Put","Key":"images/gopher.jpg","Records":[{"eventVersion":"2.0","eventSource":"minio:s3","awsRegion":"","eventTime":"2018-10-31T09:31:11Z","eventName":"s3:ObjectCreated:Put","userIdentity":{"principalId":"21EJ9HYV110O8NVX2VMS"},"requestParameters":{"sourceIPAddress":"10.1.1.1"},"responseElements":{"x-amz-request-id":"1562A792DAA53426","x-minio-origin-endpoint":"http://10.0.3.1:9000"},"s3":{"s3SchemaVersion":"1.0","configurationId":"Config","bucket":{"name":"images","ownerIdentity":{"principalId":"21EJ9HYV110O8NVX2VMS"},"arn":"arn:aws:s3:::images"},"object":{"key":"gopher.jpg","size":162023,"eTag":"5337769ffa594e742408ad3f30713cd7","contentType":"image/jpeg","userMetadata":{"content-type":"image/jpeg"},"versionId":"1","sequencer":"1562A792DAA53426"}},"source":{"host":"","port":"","userAgent":"Minio (linux; amd64) minio-go/v6.0.8 mc/DEVELOPMENT.GOGET"}}]}
```
*NOTE* If you are running [distributed Minio](https://docs.minio.io/docs/distributed-minio-quickstart-guide), modify ``~/.minio/config.json`` on all the nodes with your bucket event notification backend configuration.