XWiki Macro - Elastic Search
Last modified by Admin on 2024/07/08 00:08
 | Elastic Search and Kibana4 macros |
| Type | XAR |
| Category | |
| Developed by | ldubost |
| Active Installs | 12 |
| Rating | |
| License | GNU Lesser General Public License 2.1 |
| Compatibility | Tested on XE 7.4.2 but this extension should work from 5.4+ |
Table of contents
Description
Provide some macros and groovy code to allow to interact with Elastic Search and Kibana (v4).
Here are some sample code and results.
Elastic Search Query
{{esquery server="http://extensions.xwiki.org/activeinstalls/installs/_search?pretty" model="[[ 'name' : 'key', 'isKey' : true, 'prettyName' : 'Week', 'type' : 'date', 'format' : 'E dd/MM/yyyy'], [ 'name' : 'instanceId_count', 'prettyName' : 'Install count', 'type' : 'number' ] ]" key="period" reverse="true" }}
{
"query": {
"filtered": {
"query": {
"query_string": {
"query": "distributionVersion:* AND -distributionVersion:8.1-SNAPSHOT AND -distributionVersion:8.0-SNAPSHOT AND -distributionVersion:7.4-SNAPSHOT AND -distributionVersion:7.3-SNAPSHOT AND -distributionVersion:7.2-SNAPSHOT AND -distributionVersion:7.1-SNAPSHOT AND -distributionVersion:7.0-SNAPSHOT AND -distributionVersion:*SNAPSHOT AND -distributionVersion:*rc* AND -distributionVersion:*milestone*"
}
},
"filter": {
"script" : {
"script" : "((doc['_timestamp'].value - doc['firstPingDate'].value) / 1000 >= 86400)"
}
}
}},
"aggregations" : {
"period": {
"date_histogram": {
"field": "firstPingDate",
"interval": "month"
},
"aggregations": {
"instanceId_count": {
"cardinality": {
"field": "instanceId"
}
}
}
}
}
}
{{/esquery}}
{
"query": {
"filtered": {
"query": {
"query_string": {
"query": "distributionVersion:* AND -distributionVersion:8.1-SNAPSHOT AND -distributionVersion:8.0-SNAPSHOT AND -distributionVersion:7.4-SNAPSHOT AND -distributionVersion:7.3-SNAPSHOT AND -distributionVersion:7.2-SNAPSHOT AND -distributionVersion:7.1-SNAPSHOT AND -distributionVersion:7.0-SNAPSHOT AND -distributionVersion:*SNAPSHOT AND -distributionVersion:*rc* AND -distributionVersion:*milestone*"
}
},
"filter": {
"script" : {
"script" : "((doc['_timestamp'].value - doc['firstPingDate'].value) / 1000 >= 86400)"
}
}
}},
"aggregations" : {
"period": {
"date_histogram": {
"field": "firstPingDate",
"interval": "month"
},
"aggregations": {
"instanceId_count": {
"cardinality": {
"field": "instanceId"
}
}
}
}
}
}
{{/esquery}}
Result:
Another sample macro is:
{{esquery server="http://extensions.xwiki.org/activeinstalls/installs/_search?pretty" model="[[ 'name' : 'key', 'isKey' : true, 'prettyName' : 'Extension'], [ 'name' : 'nbinstalls', 'prettyName' : 'Install count', 'type' : 'number' ] ]" key="extension"}}
{
"query": {
"query_string": {
"lowercase_expanded_terms": false,
"query": "_timestamp:[now-1d TO *]"
}
},
"aggregations": {
"extension": {
"terms": {
"field": "extensions.id",
"size": 500,
"order": {
"nbinstalls" : "desc"
}
},
"aggs": {
"nbinstalls" : {
"cardinality": {
"field": "instanceId"
}
}
}
}
}
}
{{/esquery}}
{
"query": {
"query_string": {
"lowercase_expanded_terms": false,
"query": "_timestamp:[now-1d TO *]"
}
},
"aggregations": {
"extension": {
"terms": {
"field": "extensions.id",
"size": 500,
"order": {
"nbinstalls" : "desc"
}
},
"aggs": {
"nbinstalls" : {
"cardinality": {
"field": "instanceId"
}
}
}
}
}
}
{{/esquery}}
Elastic Search Graph
{{chart type="xy_line_and_shape" params="range:A2-B29;dataset:timetable_xy;domain_axis_type:date;domain_axis_date_format:dd/MM/yyyy;date_format:dd/MM/yyyy;time_period:month;range_axis_lower:0;range_axis_upper:1000" width="1000"}}
{{esquery server="http://extensions.xwiki.org/activeinstalls/installs/_search?pretty" model="[[ 'name' : 'key', 'isKey' : true, 'prettyName' : 'Week', 'type' : 'date', 'format' : 'dd/MM/yyyy'], [ 'name' : 'instanceId_count', 'prettyName' : 'Install count', 'type' : 'number' ] ]" key="period" reverse="true" }}
{
"query": {
"filtered": {
"query": {
"query_string": {
"query": "distributionVersion:* AND -distributionVersion:8.1-SNAPSHOT AND -distributionVersion:8.0-SNAPSHOT AND -distributionVersion:7.4-SNAPSHOT AND -distributionVersion:7.3-SNAPSHOT AND -distributionVersion:7.2-SNAPSHOT AND -distributionVersion:7.1-SNAPSHOT AND -distributionVersion:7.0-SNAPSHOT AND -distributionVersion:*SNAPSHOT AND -distributionVersion:*rc* AND -distributionVersion:*milestone*"
}
},
"filter": {
"script" : {
"script" : "((doc['_timestamp'].value - doc['firstPingDate'].value) / 1000 >= 86400)"
}
}
}},
"aggregations" : {
"period": {
"date_histogram": {
"field": "firstPingDate",
"interval": "month"
},
"aggregations": {
"instanceId_count": {
"cardinality": {
"field": "instanceId"
}
}
}
}
}
}
{{/esquery}}
{{/chart}}
{{esquery server="http://extensions.xwiki.org/activeinstalls/installs/_search?pretty" model="[[ 'name' : 'key', 'isKey' : true, 'prettyName' : 'Week', 'type' : 'date', 'format' : 'dd/MM/yyyy'], [ 'name' : 'instanceId_count', 'prettyName' : 'Install count', 'type' : 'number' ] ]" key="period" reverse="true" }}
{
"query": {
"filtered": {
"query": {
"query_string": {
"query": "distributionVersion:* AND -distributionVersion:8.1-SNAPSHOT AND -distributionVersion:8.0-SNAPSHOT AND -distributionVersion:7.4-SNAPSHOT AND -distributionVersion:7.3-SNAPSHOT AND -distributionVersion:7.2-SNAPSHOT AND -distributionVersion:7.1-SNAPSHOT AND -distributionVersion:7.0-SNAPSHOT AND -distributionVersion:*SNAPSHOT AND -distributionVersion:*rc* AND -distributionVersion:*milestone*"
}
},
"filter": {
"script" : {
"script" : "((doc['_timestamp'].value - doc['firstPingDate'].value) / 1000 >= 86400)"
}
}
}},
"aggregations" : {
"period": {
"date_histogram": {
"field": "firstPingDate",
"interval": "month"
},
"aggregations": {
"instanceId_count": {
"cardinality": {
"field": "instanceId"
}
}
}
}
}
}
{{/esquery}}
{{/chart}}
Result:
Kibana4 widgets
A simple way to integrate powerful graphs in XWiki is by using Kibana4 widgets. First you make the widget in Kibana4 and using the share button get the information to use to embed in XWiki:
{{kibana4 server="http://parlement.letemps.ch/" filters="" panels="(col:1,id:Count-per-name,row:1,size_x:12,size_y:4,type:visualization)" height="600" width="800" /}}
Result:
{{kibana4 url="http://parlement.letemps.ch/#/dashboard/Default1?embed&_g=()&_a=(filters:!((meta:(disabled:!f,index:parlament,key:canton,negate:!f,value:zh),query:(match:(canton:(query:zh,type:phrase))))),panels:!((col:1,id:Distribution-par-canton,row:1,size_x:6,size_y:5,type:visualization),(col:1,id:Count-per-name,row:6,size_x:6,size_y:4,type:visualization),(col:7,id:Top-10-Surnames,row:6,size_x:6,size_y:4,type:visualization),(col:7,id:Profession-per-Party,row:1,size_x:6,size_y:5,type:visualization)),query:(query_string:(analyze_wildcard:!t,query:'*')),title:Default1)" height="600" width="800" /}}
Result:
Write Groovy code
It is possible to write groovy code to gather elastic search data and then do a custom displaying in XWiki. Here is a sample code:
{{groovy}}
def es = xwiki.parseGroovyFromPage("ElasticSearch.Groovy");
es.setServerURL("http://extensions.xwiki.org/activeinstalls/installs/_search?pretty")
def model = [ [ name : "key", "isKey" : true, "prettyName" : "Week", "type" : "date", "format" : "E dd/MM/yyyy"], [ "name" : "instanceId_count", "prettyName" : "Install count", "type" : "number" ] ];
def esquery = """
{
"query": {
"filtered": {
"query": {
"query_string": {
"query": "distributionVersion:* AND -distributionVersion:8.1-SNAPSHOT AND -distributionVersion:8.0-SNAPSHOT AND -distributionVersion:7.4-SNAPSHOT AND -distributionVersion:7.3-SNAPSHOT AND -distributionVersion:7.2-SNAPSHOT AND -distributionVersion:7.1-SNAPSHOT AND -distributionVersion:7.0-SNAPSHOT AND -distributionVersion:*SNAPSHOT AND -distributionVersion:*rc* AND -distributionVersion:*milestone*"
}
},
"filter": {
"script" : {
"script" : "((doc['_timestamp'].value - doc['firstPingDate'].value) / 1000 >= 86400)"
}
}
}},
"aggregations" : {
"period": {
"date_histogram": {
"field": "firstPingDate",
"interval": "month"
},
"aggregations": {
"instanceId_count": {
"cardinality": {
"field": "instanceId"
}
}
}
}
}
}
"""
def data = es.getData(esquery);
println es.showBucketsData(data, model, "period", true)
if (request.debug)
println es.debugStr;
{{/groovy}}
def es = xwiki.parseGroovyFromPage("ElasticSearch.Groovy");
es.setServerURL("http://extensions.xwiki.org/activeinstalls/installs/_search?pretty")
def model = [ [ name : "key", "isKey" : true, "prettyName" : "Week", "type" : "date", "format" : "E dd/MM/yyyy"], [ "name" : "instanceId_count", "prettyName" : "Install count", "type" : "number" ] ];
def esquery = """
{
"query": {
"filtered": {
"query": {
"query_string": {
"query": "distributionVersion:* AND -distributionVersion:8.1-SNAPSHOT AND -distributionVersion:8.0-SNAPSHOT AND -distributionVersion:7.4-SNAPSHOT AND -distributionVersion:7.3-SNAPSHOT AND -distributionVersion:7.2-SNAPSHOT AND -distributionVersion:7.1-SNAPSHOT AND -distributionVersion:7.0-SNAPSHOT AND -distributionVersion:*SNAPSHOT AND -distributionVersion:*rc* AND -distributionVersion:*milestone*"
}
},
"filter": {
"script" : {
"script" : "((doc['_timestamp'].value - doc['firstPingDate'].value) / 1000 >= 86400)"
}
}
}},
"aggregations" : {
"period": {
"date_histogram": {
"field": "firstPingDate",
"interval": "month"
},
"aggregations": {
"instanceId_count": {
"cardinality": {
"field": "instanceId"
}
}
}
}
}
}
"""
def data = es.getData(esquery);
println es.showBucketsData(data, model, "period", true)
if (request.debug)
println es.debugStr;
{{/groovy}}
This second example shows the same result but with full custom code:
{{groovy}}
import org.joda.time.format.DateTimeFormat;
import org.joda.time.DateTime;
def es = xwiki.parseGroovyFromPage("ElasticSearch.Groovy");
es.setServerURL("http://extensions.xwiki.org/activeinstalls/installs/_search?pretty")
def esquery = """
{
"query": {
"filtered": {
"query": {
"query_string": {
"query": "distributionVersion:* AND -distributionVersion:8.1-SNAPSHOT AND -distributionVersion:8.0-SNAPSHOT AND -distributionVersion:7.4-SNAPSHOT AND -distributionVersion:7.3-SNAPSHOT AND -distributionVersion:7.2-SNAPSHOT AND -distributionVersion:7.1-SNAPSHOT AND -distributionVersion:7.0-SNAPSHOT AND -distributionVersion:*SNAPSHOT AND -distributionVersion:*rc* AND -distributionVersion:*milestone*"
}
},
"filter": {
"script" : {
"script" : "((doc['_timestamp'].value - doc['firstPingDate'].value) / 1000 >= 86400)"
}
}
}},
"aggregations" : {
"period": {
"date_histogram": {
"field": "firstPingDate",
"interval": "month"
},
"aggregations": {
"instanceId_count": {
"cardinality": {
"field": "instanceId"
}
}
}
}
}
}
"""
def data = es.getData(esquery);
def dateFormatter = DateTimeFormat.forPattern("E dd/MM/yyyy");;
try {
if (data==null || data.aggregations==null || data.aggregations.period==null) {
println "No data"
} else {
def buckets = data.aggregations.period.buckets;
buckets = buckets.reverse();
println "|= Week |= Install count"
for (item in buckets) {
println "| ${dateFormatter.print(new DateTime(item.key))} | ${item.instanceId_count.value}"
}
}
} catch (e) {
println e.getMessage();
}
if (request.debug)
println es.debugStr;
{{/groovy}}
import org.joda.time.format.DateTimeFormat;
import org.joda.time.DateTime;
def es = xwiki.parseGroovyFromPage("ElasticSearch.Groovy");
es.setServerURL("http://extensions.xwiki.org/activeinstalls/installs/_search?pretty")
def esquery = """
{
"query": {
"filtered": {
"query": {
"query_string": {
"query": "distributionVersion:* AND -distributionVersion:8.1-SNAPSHOT AND -distributionVersion:8.0-SNAPSHOT AND -distributionVersion:7.4-SNAPSHOT AND -distributionVersion:7.3-SNAPSHOT AND -distributionVersion:7.2-SNAPSHOT AND -distributionVersion:7.1-SNAPSHOT AND -distributionVersion:7.0-SNAPSHOT AND -distributionVersion:*SNAPSHOT AND -distributionVersion:*rc* AND -distributionVersion:*milestone*"
}
},
"filter": {
"script" : {
"script" : "((doc['_timestamp'].value - doc['firstPingDate'].value) / 1000 >= 86400)"
}
}
}},
"aggregations" : {
"period": {
"date_histogram": {
"field": "firstPingDate",
"interval": "month"
},
"aggregations": {
"instanceId_count": {
"cardinality": {
"field": "instanceId"
}
}
}
}
}
}
"""
def data = es.getData(esquery);
def dateFormatter = DateTimeFormat.forPattern("E dd/MM/yyyy");;
try {
if (data==null || data.aggregations==null || data.aggregations.period==null) {
println "No data"
} else {
def buckets = data.aggregations.period.buckets;
buckets = buckets.reverse();
println "|= Week |= Install count"
for (item in buckets) {
println "| ${dateFormatter.print(new DateTime(item.key))} | ${item.instanceId_count.value}"
}
}
} catch (e) {
println e.getMessage();
}
if (request.debug)
println es.debugStr;
{{/groovy}}
Prerequisites & Installation Instructions
We recommend using the Extension Manager to install this extension (Make sure that the text "Installable with the Extension Manager" is displayed at the top right location on this page to know if this extension can be installed with the Extension Manager). Note that installing Extensions when being offline is currently not supported and you'd need to use some complex manual method.
You can also use the following manual method, which is useful if this extension cannot be installed with the Extension Manager or if you're using an old version of XWiki that doesn't have the Extension Manager:
- Log in the wiki with a user having Administration rights
- Go to the Administration page and select the Import category
- Follow the on-screen instructions to upload the downloaded XAR
- Click on the uploaded XAR and follow the instructions
- You'll also need to install all dependent Extensions that are not already installed in your wiki
Install the XAR and visit the ElasticSearch space to see some samples.
