MyFavMirrors/minio
1
0
Fork 0
mirror of https://github.com/minio/minio.git synced 2025-11-07 21:02:58 -05:00
Code Issues Packages Projects Releases Wiki Activity

Add access format support for Elasticsearch notification target (#4006)

Browse Source 
This change adds `access` format support for notifications to a
Elasticsearch server, and it refactors `namespace` format support.

In the case of `access` format, for each event in Minio, a JSON
document is inserted into Elasticsearch with its timestamp set to the
event's timestamp, and with the ID generated automatically by
elasticsearch. No events are modified or deleted in this mode.

In the case of `namespace` format, for each event in Minio, a JSON
document is keyed together by the bucket and object name is updated in
Elasticsearch. In the case of an object being created or over-written
in Minio, a new document or an existing document is inserted into the
Elasticsearch index. If an object is deleted in Minio, the
corresponding document is deleted from the Elasticsearch index.

Additionally, this change upgrades Elasticsearch support to the 5.x
series. This is a breaking change, and users of previous elasticsearch
versions should upgrade.

Also updates documentation on Elasticsearch notification target usage
and has a link to an elasticsearch upgrade guide.

This is the last patch that finally resolves #3928.
This commit is contained in:
Aditya Manthramurthy
2017-04-01 02:41:27 +05:30
committed by Harshavardhana
parent 2040d32ef8
commit a2a8d54bb6
218 changed files with 7041 additions and 4510 deletions
Download Patch File Download Diff File Expand all files Collapse all files

137
docs/bucket/notifications/README.md
View File

@@ -137,29 +137,52 @@ python rabbit.py
<a name="Elasticsearch"></a>
## Publish Minio events via Elasticsearch
Install Elasticsearch 2.4 from [here](https://www.elastic.co/downloads/past-releases/elasticsearch-2-4-0).
Install [Elasticsearch](https://www.elastic.co/downloads/elasticsearch) server. Minio server supports the latest major release series 5.x. Elasticsearch provides version upgrade migration guidelines [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-upgrade.html).
This notification target supports two formats: _namespace_ and _access_.
When the _namespace_ format is used, Minio synchronizes objects in the bucket with documents in the index. For each event in the Minio, the server creates a document with the bucket and object name from the event as the document ID. Other details of the event are stored in the body of the document. Thus if an existing object is over-written in Minio, the corresponding document in the Elasticsearch index is updated. If an object is deleted, the corresponding document is deleted from the index.
When the _access_ format is used, Minio appends events as documents in an Elasticsearch index. For each event, a document with the event details, with the timestamp of document set to the event's timestamp is appended to an index. The ID of the documented is randomly generated by Elasticsearch. No documents are deleted or modified in this format.
The steps below show how to use this notification target in `namespace` format. The other format is very similar and is omitted for brevity.
## Recipe steps
### Step 1: Add Elasticsearch endpoint to Minio
The default location of Minio server configuration file is ``~/.minio/config.json``. Update the Elasticsearch configuration block in ``config.json`` as follows:
The default location of Minio server configuration file is ``~/.minio/config.json``. The Elasticsearch configuration is located in the `elasticsearch` key under the `notify` top-level key. Create a configuration key-value pair here for your Elasticsearch instance. The key is a name for your Elasticsearch endpoint, and the value is a collection of key-value parameters described in the table below.
| Parameter | Type | Description |
|:---|:---|:---|
| `enable` | _bool_ | (Required) Is this server endpoint configuration active/enabled? |
| `format` | _string_ | (Required) Either `namespace` or `access`. |
| `url` | _string_ | (Required) The Elasticsearch server's address. For example: `http://localhost:9200`. |
| `index` | _string_ | (Required) The name of an Elasticsearch index in which Minio will store documents. |
An example of Elasticsearch configuration is as follows:
```json
"elasticsearch": {
"1": {
"enable": true,
"format": "namespace",
"url": "http://127.0.0.1:9200",
"index": "bucketevents"
"index": "minio_events"
}
},
```
Restart Minio server to reflect config changes. ``bucketevents`` is the index used by Elasticsearch.
After updating the configuration file, restart the Minio server to put the changes into effect. The server will print a line like `SQS ARNs: arn:minio:sqs:us-east-1:1:elasticsearch` at start-up if there were no errors.
Note that, you can add as many Elasticsearch server endpoint configurations as needed by providing an identifier (like "1" in the example above) for the Elasticsearch instance and an object of per-server configuration parameters.
### Step 2: Enable bucket notification using Minio client
We will enable bucket event notification to trigger whenever a JPEG image is uploaded or deleted from ``images`` bucket on ``myminio`` server. Here ARN value is ``arn:minio:sqs:us-east-1:1:elasticsearch``. To understand more about ARN please follow [AWS ARN](http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html) documentation.
We will now enable bucket event notifications on a bucket named `images`. Whenever a JPEG image is created/overwritten, a new document is added or an existing document is updated in the Elasticsearch index configured above. When an existing object is deleted, the corresponding document is deleted from the index. Thus, the rows in the Elasticsearch index, reflect the `.jpg` objects in the `images` bucket.
To configure this bucket notification, we need the ARN printed by Minio in the previous step. Additional information about ARN is available [here](http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html).
With the `mc` tool, the configuration is very simple to add. Let us say that the Minio server is aliased as `myminio` in our mc configuration. Execute the following:
```
mc mb myminio/images
@@ -170,26 +193,18 @@ arn:minio:sqs:us-east-1:1:elasticsearch s3:ObjectCreated:*,s3:ObjectRemoved:* Fi
### Step 3: Test on Elasticsearch
Upload a JPEG image into ``images`` bucket, this is the bucket which has been configured for event notification.
Upload a JPEG image into ``images`` bucket.
```
mc cp myphoto.jpg myminio/images
```
Run ``curl`` to see new index name ``bucketevents`` in your Elasticsearch setup.
Use curl to view contents of ``minio_events`` index.
```
curl -XGET '127.0.0.1:9200/_cat/indices?v'
health status index pri rep docs.count docs.deleted store.size pri.store.size
yellow open bucketevents 5 1 1 0 7.8kb 7.8kb
```
Use curl to view contents of ``bucketevents`` index.
```
curl -XGET '127.0.0.1:9200/bucketevents/_search?pretty=1'
$ curl "http://localhost:9200/minio_events/_search?pretty=true"
{
"took" : 3,
"took" : 40,
"timed_out" : false,
"_shards" : {
"total" : 5,
@@ -199,49 +214,64 @@ curl -XGET '127.0.0.1:9200/bucketevents/_search?pretty=1'
"hits" : {
"total" : 1,
"max_score" : 1.0,
"hits" : [ {
"_index" : "bucketevents",
"_type" : "event",
"_id" : "AVcRVOlwe-uNB1tfj6bx",
"_score" : 1.0,
"_source" : {
"Records" : [ {
"eventVersion" : "2.0",
"eventSource" : "aws:s3",
"awsRegion" : "us-east-1",
"eventTime" : "2016-09-09T23:42:39.977Z",
"eventName" : "s3:ObjectCreated:Put",
"userIdentity" : {
"principalId" : "minio"
},
"requestParameters" : {
"sourceIPAddress" : "10.1.10.150:52140"
},
"responseElements" : { },
"s3" : {
"s3SchemaVersion" : "1.0",
"configurationId" : "Config",
"bucket" : {
"name" : "images",
"ownerIdentity" : {
"hits" : [
{
"_index" : "minio_events",
"_type" : "event",
"_id" : "images/myphoto.jpg",
"_score" : 1.0,
"_source" : {
"Records" : [
{
"eventVersion" : "2.0",
"eventSource" : "minio:s3",
"awsRegion" : "us-east-1",
"eventTime" : "2017-03-30T08:00:41Z",
"eventName" : "s3:ObjectCreated:Put",
"userIdentity" : {
"principalId" : "minio"
},
"arn" : "arn:aws:s3:::images"
},
"object" : {
"key" : "myphoto.jpg",
"size" : 200436,
"sequencer" : "1472CC35E6971AF3"
"requestParameters" : {
"sourceIPAddress" : "127.0.0.1:38062"
},
"responseElements" : {
"x-amz-request-id" : "14B09A09703FC47B",
"x-minio-origin-endpoint" : "http://192.168.86.115:9000"
},
"s3" : {
"s3SchemaVersion" : "1.0",
"configurationId" : "Config",
"bucket" : {
"name" : "images",
"ownerIdentity" : {
"principalId" : "minio"
},
"arn" : "arn:aws:s3:::images"
},
"object" : {
"key" : "myphoto.jpg",
"size" : 6474,
"eTag" : "a3410f4f8788b510d6f19c5067e60a90",
"sequencer" : "14B09A09703FC47B"
}
},
"source" : {
"host" : "127.0.0.1",
"port" : "38062",
"userAgent" : "Minio (linux; amd64) minio-go/2.0.3 mc/2017-02-15T17:57:25Z"
}
}
}
} ]
]
}
}
} ]
]
}
}
```
``curl`` output above states that an Elasticsearch index has been successfully created with notification contents.
This output shows that a document has been created for the event in Elasticsearch.
Here we see that the document ID is the bucket and object name. In case `access` format was used, the document ID would be automatically generated by Elasticsearch.
<a name="Redis"></a>
## Publish Minio events via Redis
@@ -270,7 +300,6 @@ The default location of Minio server configuration file is ``~/.minio/config.jso
An example of Redis configuration is as follows:
```json
"redis": {
"1": {