XWiki Macro - Elastic Search

Last modified by Admin on 2022/02/28 00:24

cogElastic Search and Kibana4 macros
TypeXAR
Category
Developed by

ldubost

Active Installs12
Rating
0 Votes
LicenseGNU Lesser General Public License 2.1
Compatibility

Tested on XE 7.4.2 but this extension should work from 5.4+

Installable with the Extension Manager

Description

Provide some macros and groovy code to allow to interact with Elastic Search and Kibana (v4).

Here are some sample code and results.

Elastic Search Query

{{esquery server="http://extensions.xwiki.org/activeinstalls/installs/_search?pretty" model="[[ 'name' : 'key', 'isKey' : true, 'prettyName' : 'Week', 'type' : 'date', 'format' : 'E dd/MM/yyyy'], [ 'name' : 'instanceId_count', 'prettyName' : 'Install count', 'type' : 'number' ] ]" key="period" reverse="true" }}
{
    "query": {
        "filtered": {
            "query": {
                "query_string": {
                  "query": "distributionVersion:* AND -distributionVersion:8.1-SNAPSHOT AND -distributionVersion:8.0-SNAPSHOT AND -distributionVersion:7.4-SNAPSHOT AND -distributionVersion:7.3-SNAPSHOT AND -distributionVersion:7.2-SNAPSHOT AND -distributionVersion:7.1-SNAPSHOT AND -distributionVersion:7.0-SNAPSHOT AND -distributionVersion:*SNAPSHOT AND -distributionVersion:*rc* AND -distributionVersion:*milestone*"
                }
            },
            "filter": {
                   "script" : {
                      "script" : "((doc['_timestamp'].value - doc['firstPingDate'].value) / 1000 >= 86400)"
                   }
            }
        }},
    "aggregations" : {
        "period": {
            "date_histogram": {
                "field":    "firstPingDate",
                "interval": "month"
            },
            "aggregations": {
                "instanceId_count": {
                    "cardinality": {
                        "field": "instanceId"
                    }
                }
           }
       }
    }
  }
{{/esquery}}

Result:

elastic-query.png

Another sample macro is:

{{esquery server="http://extensions.xwiki.org/activeinstalls/installs/_search?pretty" model="[[ 'name' : 'key', 'isKey' : true, 'prettyName' : 'Extension'], [ 'name' : 'nbinstalls', 'prettyName' : 'Install count', 'type' : 'number' ] ]" key="extension"}}
    {
      "query": {
        "query_string": {
          "lowercase_expanded_terms": false,
           "query": "_timestamp:[now-1d TO *]"
        }
      },
      "aggregations": {
        "extension": {
          "terms": {
           "field": "extensions.id",
           "size": 500,
           "order": {
                "nbinstalls" : "desc"
              }
          },
          "aggs": {
           "nbinstalls" : {
           "cardinality": {
            "field": "instanceId"
          }
         }
        }
       }
      }
    }
{{/esquery}}

Elastic Search Graph

{{chart type="xy_line_and_shape" params="range:A2-B29;dataset:timetable_xy;domain_axis_type:date;domain_axis_date_format:dd/MM/yyyy;date_format:dd/MM/yyyy;time_period:month;range_axis_lower:0;range_axis_upper:1000" width="1000"}}
{{esquery server="http://extensions.xwiki.org/activeinstalls/installs/_search?pretty" model="[[ 'name' : 'key', 'isKey' : true, 'prettyName' : 'Week', 'type' : 'date', 'format' : 'dd/MM/yyyy'], [ 'name' : 'instanceId_count', 'prettyName' : 'Install count', 'type' : 'number' ] ]" key="period" reverse="true" }}
{
    "query": {
        "filtered": {
            "query": {
                "query_string": {
                  "query": "distributionVersion:* AND -distributionVersion:8.1-SNAPSHOT AND -distributionVersion:8.0-SNAPSHOT AND -distributionVersion:7.4-SNAPSHOT AND -distributionVersion:7.3-SNAPSHOT AND -distributionVersion:7.2-SNAPSHOT AND -distributionVersion:7.1-SNAPSHOT AND -distributionVersion:7.0-SNAPSHOT AND -distributionVersion:*SNAPSHOT AND -distributionVersion:*rc* AND -distributionVersion:*milestone*"
                }
            },
            "filter": {
                   "script" : {
                      "script" : "((doc['_timestamp'].value - doc['firstPingDate'].value) / 1000 >= 86400)"
                   }
            }
        }},
    "aggregations" : {
        "period": {
            "date_histogram": {
                "field":    "firstPingDate",
                "interval": "month"
            },
            "aggregations": {
                "instanceId_count": {
                    "cardinality": {
                        "field": "instanceId"
                    }
                }
           }
       }
    }
  }
{{/esquery}}
{{/chart}}

Result:

elastic-graph.png

Kibana4 widgets

A simple way to integrate powerful graphs in XWiki is by using Kibana4 widgets. First you make the widget in Kibana4 and using the share button get the information to use to embed in XWiki:

{{kibana4 server="http://parlement.letemps.ch/" filters="" panels="(col:1,id:Count-per-name,row:1,size_x:12,size_y:4,type:visualization)" height="600" width="800" /}}

Result:

kibana4a.png

{{kibana4 url="http://parlement.letemps.ch/#/dashboard/Default1?embed&_g=()&_a=(filters:!((meta:(disabled:!f,index:parlament,key:canton,negate:!f,value:zh),query:(match:(canton:(query:zh,type:phrase))))),panels:!((col:1,id:Distribution-par-canton,row:1,size_x:6,size_y:5,type:visualization),(col:1,id:Count-per-name,row:6,size_x:6,size_y:4,type:visualization),(col:7,id:Top-10-Surnames,row:6,size_x:6,size_y:4,type:visualization),(col:7,id:Profession-per-Party,row:1,size_x:6,size_y:5,type:visualization)),query:(query_string:(analyze_wildcard:!t,query:'*')),title:Default1)" height="600" width="800" /}}

Result:

kibana4b.png

Write Groovy code

It is possible to write groovy code to gather elastic search data and then do a custom displaying in XWiki. Here is a sample code:

{{groovy}}
def es = xwiki.parseGroovyFromPage("ElasticSearch.Groovy");
 es.setServerURL("http://extensions.xwiki.org/activeinstalls/installs/_search?pretty")

def model = [ [ name : "key", "isKey" : true, "prettyName" : "Week", "type" : "date", "format" : "E dd/MM/yyyy"], [ "name" : "instanceId_count", "prettyName" : "Install count", "type" : "number" ] ];
def esquery = """
{
    "query": {
        "filtered": {
            "query": {
                "query_string": {
                  "query": "distributionVersion:* AND -distributionVersion:8.1-SNAPSHOT AND -distributionVersion:8.0-SNAPSHOT AND -distributionVersion:7.4-SNAPSHOT AND -distributionVersion:7.3-SNAPSHOT AND -distributionVersion:7.2-SNAPSHOT AND -distributionVersion:7.1-SNAPSHOT AND -distributionVersion:7.0-SNAPSHOT AND -distributionVersion:*SNAPSHOT AND -distributionVersion:*rc* AND -distributionVersion:*milestone*"
                }
            },
            "filter": {
                   "script" : {
                      "script" : "((doc['_timestamp'].value - doc['firstPingDate'].value) / 1000 >= 86400)"
                   }
            }
        }},
    "aggregations" : {
        "period": {
            "date_histogram": {
                "field":    "firstPingDate",
                "interval": "month"
            },
            "aggregations": {
                "instanceId_count": {
                    "cardinality": {
                        "field": "instanceId"
                    }
                }
           }
       }
    }
  }
 """


def data = es.getData(esquery);
 println es.showBucketsData(data, model, "period", true)

if (request.debug)
 println es.debugStr;

{{/groovy}}

This second example shows the same result but with full custom code:

{{groovy}}
import org.joda.time.format.DateTimeFormat;
import org.joda.time.DateTime;

def es = xwiki.parseGroovyFromPage("ElasticSearch.Groovy");
 es.setServerURL("http://extensions.xwiki.org/activeinstalls/installs/_search?pretty")

def esquery = """
{
    "query": {
        "filtered": {
            "query": {
                "query_string": {
                  "query": "distributionVersion:* AND -distributionVersion:8.1-SNAPSHOT AND -distributionVersion:8.0-SNAPSHOT AND -distributionVersion:7.4-SNAPSHOT AND -distributionVersion:7.3-SNAPSHOT AND -distributionVersion:7.2-SNAPSHOT AND -distributionVersion:7.1-SNAPSHOT AND -distributionVersion:7.0-SNAPSHOT AND -distributionVersion:*SNAPSHOT AND -distributionVersion:*rc* AND -distributionVersion:*milestone*"
                }
            },
            "filter": {
                   "script" : {
                      "script" : "((doc['_timestamp'].value - doc['firstPingDate'].value) / 1000 >= 86400)"
                   }
            }
        }},
    "aggregations" : {
        "period": {
            "date_histogram": {
                "field":    "firstPingDate",
                "interval": "month"
            },
            "aggregations": {
                "instanceId_count": {
                    "cardinality": {
                        "field": "instanceId"
                    }
                }
           }
       }
    }
  }
 """


def data = es.getData(esquery);
def dateFormatter = DateTimeFormat.forPattern("E dd/MM/yyyy");;

try {
 if (data==null ||  data.aggregations==null || data.aggregations.period==null) {
   println "No data"
 } else {
  def buckets = data.aggregations.period.buckets;
   buckets = buckets.reverse();

   println "|= Week |= Install count"

  for (item in buckets) {
     println "| ${dateFormatter.print(new DateTime(item.key))} | ${item.instanceId_count.value}"
  }
 }
} catch (e) {
   println e.getMessage();
}

if (request.debug)
 println es.debugStr;
{{/groovy}}

Prerequisites & Installation Instructions

We recommend using the Extension Manager to install this extension (Make sure that the text "Installable with the Extension Manager" is displayed at the top right location on this page to know if this extension can be installed with the Extension Manager). Note that installing Extensions when being offline is currently not supported and you'd need to use some complex manual method.

You can also use the following manual method, which is useful if this extension cannot be installed with the Extension Manager or if you're using an old version of XWiki that doesn't have the Extension Manager:

  1. Log in the wiki with a user having Administration rights
  2. Go to the Administration page and select the Import category
  3. Follow the on-screen instructions to upload the downloaded XAR
  4. Click on the uploaded XAR and follow the instructions
  5. You'll also need to install all dependent Extensions that are not already installed in your wiki

Install the XAR and visit the ElasticSearch space to see some samples.

Tags:
    

Get Connected