Plugin installation was unsuccessful due to error "Plugin elastalert-kibana-plugin [7.5.0] is incompatible with Kibana [7.5.2]"

[Yunamii]

Hello,

I have kibana 7.5.2 and I whould install elastalert on Kibana.
When I install the latest version of elastalert kibana plugin, I have the error message below:

./kibana-plugin install https://github.com/bitsensor/elastalert-kibana-plugin/releases/download/1.1.0/elastalert-kibana-plugin-1.1.0-7.5.0.zip --allow-root
Attempting to transfer from https://github.com/bitsensor/elastalert-kibana-plugin/releases/download/1.1.0/elastalert-kibana-plugin-1.1.0-7.5.0.zip
Transferring 25617568 bytes....................
Transfer complete
Retrieving metadata from plugin archive
Extracting plugin archive
Extraction complete
Plugin installation was unsuccessful due to error "Plugin elastalert-kibana-plugin [7.5.0] is incompatible with Kibana [7.5.2]"

Is there a future deployement for the 7.5.2 version?

[whataboutpereira]

Unfortunately the same issue with any newer version (like the current 7.6.1).

[nsano-rururu]

@Yunamii

Currently, zip files for Kibana 7.5.1 and Kibana 7.5.2 are not released, so you need to download the zip for Kibana 7.5.0 and edit package.json.

How to create and install a zip file for kiban 7.5.2 is as follows

cd /tmp
curl -L -O https://github.com/bitsensor/elastalert-kibana-plugin/releases/download/1.1.0/elastalert-kibana-plugin-1.1.0-7.5.0.zip
mv elastalert-kibana-plugin-1.1.0-7.5.0.zip elastalert-kibana-plugin-1.1.0-7.5.2.zip
unzip elastalert-kibana-plugin-1.1.0-7.5.2.zip kibana/elastalert-kibana-plugin/package.json
sed -i "s/7\.5\.0/7\.5\.2/g" kibana/elastalert-kibana-plugin/package.json
zip elastalert-kibana-plugin-1.1.0-7.5.2.zip kibana/elastalert-kibana-plugin/package.json
rm -rf kibana
cd /usr/share/kibana/bin
./kibana-plugin install --allow-root file:///tmp/elastalert-kibana-plugin-1.1.0-7.5.2.zip

[shijeshk]

Still running into the same problem

Attempting to transfer from file:///tmp/new/elastalert-kibana-plugin-1.1.0-7.5.2.zip
Transferring 25617568 bytes....................
Transfer complete
Retrieving metadata from plugin archive
Extracting plugin archive
Extraction complete
Plugin installation was unsuccessful due to error "Plugin elastalert-kibana-plugin [7.5.0] is incompatible with Kibana [7.5.2]"

[nsano-rururu]

@shijeshk

I put the elastalert-kibana-plugin-1.1.0-7.5.2.zip file in my github repository, can you check if the problem occurs again?
https://github.com/nsano-rururu/docker-sample/blob/master/elastalert-kibana-plugin/kibana/plugin/elastalert-kibana-plugin-1.1.0-7.5.2.zip

[shijeshk]

Thanks @nsano-rururu I am able to install now.

[nsano-rururu]

@shijeshk

Thank you for confirming the install.

[nsano-rururu]

@Yunamii

Is the problem solved? . If there are no additional questions, please close.

[markgaolei]

@nsano-rururu Could i use above command to build 7.6.0 version?

[nsano-rururu]

@nsano-rururu Could i use above command to build 7.6.0 version?

@markgaolei

cd /tmp
curl -L -O https://github.com/bitsensor/elastalert-kibana-plugin/releases/download/1.1.0/elastalert-kibana-plugin-1.1.0-7.5.0.zip
curl -L -O https://raw.githubusercontent.com/mmguero-dev/Malcolm/development/kibana/elastalert-kibana-plugin/server/routes/elastalert.js
mv elastalert.js elastalert-server-routes.js
mv elastalert-kibana-plugin-1.1.0-7.5.0.zip elastalert-kibana-plugin-1.1.0-7.6.0.zip
unzip elastalert-kibana-plugin-1.1.0-7.6.0.zip kibana/elastalert-kibana-plugin/package.json
sed -i "s/7\.5\.0/7\.6\.0/g" kibana/elastalert-kibana-plugin/package.json
mkdir -p kibana/elastalert-kibana-plugin/server/routes/
cp /tmp/elastalert-server-routes.js kibana/elastalert-kibana-plugin/server/routes/elastalert.js
zip elastalert-kibana-plugin-1.1.0-7.6.0.zip kibana/elastalert-kibana-plugin/package.json kibana/elastalert-kibana-plugin/server/routes/elastalert.js
rm -rf kibana
rm elastalert-server-routes.js
cd /usr/share/kibana/bin
./kibana-plugin install --allow-root file:///tmp/elastalert-kibana-plugin-1.1.0-7.6.0.zip

[mismailzz]

@nsano-rururu After successful installation of Elastalert for kibana 7.6.2, I got an error on kibana.
Error: Kibana server is not ready yet

[nsano-rururu]

@mismailzz

・Is Elasticsearch also known as 7.6.2?

・Is Elasticsearch started?

・Is setting added to kibana.yml?

# defaults to localhost
elastalert-kibana-plugin.serverHost: ip address or hostname
# defaults to 3030
elastalert-kibana-plugin.serverPort: 3030
# defaults to true
#elastalert-kibana-plugin.enabled
# defaults to false 
#elastalert-kibana-plugin.serverSsl  

・Did the elastalert kibana plugin execute the following commands for Kibana 7.6.2? . Did you execute something different from the following command?

cd /tmp
curl -L -O https://github.com/bitsensor/elastalert-kibana-plugin/releases/download/1.1.0/elastalert-kibana-plugin-1.1.0-7.5.0.zip
curl -L -O https://raw.githubusercontent.com/mmguero-dev/Malcolm/development/kibana/elastalert-kibana-plugin/server/routes/elastalert.js
mv elastalert.js elastalert-server-routes.js
mv elastalert-kibana-plugin-1.1.0-7.5.0.zip elastalert-kibana-plugin-1.1.0-7.6.2.zip
unzip elastalert-kibana-plugin-1.1.0-7.6.2.zip kibana/elastalert-kibana-plugin/package.json
sed -i "s/7\.5\.0/7\.6\.2/g" kibana/elastalert-kibana-plugin/package.json
mkdir -p kibana/elastalert-kibana-plugin/server/routes/
cp /tmp/elastalert-server-routes.js kibana/elastalert-kibana-plugin/server/routes/elastalert.js
zip elastalert-kibana-plugin-1.1.0-7.6.2.zip kibana/elastalert-kibana-plugin/package.json kibana/elastalert-kibana-plugin/server/routes/elastalert.js
rm -rf kibana
rm elastalert-server-routes.js
cd /usr/share/kibana/bin
./kibana-plugin install --allow-root file:///tmp/elastalert-kibana-plugin-1.1.0-7.6.2.zip

[whataboutpereira]

After successful installation of Elastalert for kibana 7.6.2, I got an error on kibana.
Error: Kibana server is not ready yet

That usually implies Kibana is still starting up.

[nsano-rururu]

If the message "Kibana server is not ready yet" is displayed, it is possible that there is a problem with the config or other items that are still starting up, or the startup is interrupted.

I've often run into an issue where I forget to add the settings to kibana.yml and it doesn't start. . .

After that, even if you are in trouble if you do not work with machine specifications (CPU, memory) and JVM memory settings set too little. . .

[mismailzz]

@nsano-rururu, Yes I'm using Elasticsearch version 7.6.2, it is running perfectly and during the installation of Elastalert I changed to version 7.6.0 to 7.6.2. The only thing that I skipped was the Elastic plugin information in the Kibana.yml file.
Now when I added this configuration in Kibana.yml file then I got the same issue. I added the configuration like this at the end of file

# defaults to localhost
elastalert-kibana-plugin.serverHost: "localhost"
# defaults to 3030
elastalert-kibana-plugin.serverPort: 5601
# defaults to true
#elastalert-kibana-plugin.enabled
# defaults to false 
#elastalert-kibana-plugin.serverSsl 

One last thing is that after some time while kibana is running the machine get stuck.

[nsano-rururu]

@mismailzz

As a result of trying with Docker, the plugin of ElastAlert worked with Kibana 7.6.2

Elasticsearch 7.6.2
Kibana 7.6.2
kibana-elastalert-plugin
bitsensor/elastalert:3.0.0-beta.0

AWS Workspace
Amazon Linux2
2 vCPU、7.5GiB

/home/user/docker-wk2
|--docker-compose.yml
|--Dockerfiles
|  |--Dockerfile-elastalert
|
|--es
|  |--config
|  |  |--elasticsearch.yml
|  |--data
|
|--kibana
|  |--config
|  |  |--kibana.yml
|  |--plugin
|  |  |--kibana-elastealert-plugin-1.1.0-7.6.2.zip
|
|--elastalert
|  |--bin
|  |  |--elastalert-start.sh
|  |  |--elastic_search_status.sh
|  |--config
|  |  |--config.json
|  |  |--elastalert-test.yaml
|  |  |--elastalert.yaml
|  |--rule_templates
|  |--rules

cd 
mkdir docker-wk2
cd docker-wk2
touch docker-compose.yml
mkdir Dockerfiles
touch Dockerfiles/Dockerfile-elastalert
touch Dockerfiles/Dockerfile-kibana
mkdir -p es/config
mkdir -p es/data
touch es/config/elasticsearch.yml
mkdir -p kibana/config
mkdir -p kibana/plugin
touch kibana/config/kibana.yml
mkdir -p elastalert/bin
mkdir -p elastalert/config
mkdir -p elastalert/rule_templates
mkdir -p elastalert/rules
touch elastalert/bin/elastalert-start.sh
touch elastalert/bin/elastic_search_status.sh
touch elastalert/config/config.json
touch elastalert/config/elastalert-test.yaml
touch elastalert/config/elastalert.yaml

# Download necessary files
cd /tmp
curl -L -O https://github.com/bitsensor/elastalert-kibana-plugin/releases/download/1.1.0/elastalert-kibana-plugin-1.1.0-7.5.0.zip
curl -L -O https://raw.githubusercontent.com/mmguero-dev/Malcolm/development/kibana/elastalert-kibana-plugin/server/routes/elastalert.js

# update elasticsearch package to 7.6.2
mv elastalert.js elastalert-server-routes.js
mv elastalert-kibana-plugin-1.1.0-7.5.0.zip elastalert-kibana-plugin-1.1.0-7.6.2.zip
unzip elastalert-kibana-plugin-1.1.0-7.6.2.zip kibana/elastalert-kibana-plugin/package.json
sed -i "s/7\.5\.0/7\.6\.2/g" kibana/elastalert-kibana-plugin/package.json
mkdir -p kibana/elastalert-kibana-plugin/server/routes/
mv /tmp/elastalert-server-routes.js kibana/elastalert-kibana-plugin/server/routes/elastalert.js
zip elastalert-kibana-plugin-1.1.0-7.6.2.zip kibana/elastalert-kibana-plugin/package.json kibana/elastalert-kibana-plugin/server/routes/elastalert.js

# delete remaining directory
rm -rf kibana

# copy the created package to your workspace
mv /tmp/elastalert-kibana-plugin-1.1.0-7.6.2.zip /home/user/docker-wk2/kibana/plugin


cd
cd docker-wk2

docker-compose.yml

version: "3.7"
services:
  elasticsearch:
    container_name: elasticsearch
    image: docker.elastic.co/elasticsearch/elasticsearch:7.6.2
    ports:
      - 9200:9200
      - 9300:9300
    environment:
      - ES_JAVA_OPTS=-Xms256m -Xmx256m
      - discovery.type=single-node
    restart: always
    volumes:
      - ./es/data:/usr/share/elasticsearch/data
      - ./es/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
    healthcheck:
        test: ["CMD-SHELL", "curl -f http://localhost:9200 || exit 1"]
        interval: 30s
        timeout: 15s
        retries: 3
        start_period: 180s

  kibana:
    container_name: kibana
    image: docker.elastic.co/kibana/kibana:7.6.2
    command: sh -c './bin/kibana-plugin list | grep [email protected]; result=`echo $$?`; if [ $$result = 1 ]; then  ./bin/kibana-plugin install file:///usr/share/kibana/work/elastalert-kibana-plugin-1.1.0-7.6.2.zip && exec /usr/local/bin/kibana-docker; else exec /usr/local/bin/kibana-docker; fi'
    ports:
      - 5601:5601
    depends_on:
      - elasticsearch
    restart: always
    volumes:
      - ./kibana/config/kibana.yml:/usr/share/kibana/config/kibana.yml
      - ./kibana/plugin:/usr/share/kibana/work
    healthcheck:
        test: ["CMD-SHELL", "curl -f http://localhost:5601/api/status || exit 1"]
        interval: 30s
        timeout: 15s
        retries: 3
        start_period: 200s

  elastalert:
    container_name: elastalert
    build:
      context: .
      dockerfile: Dockerfiles/Dockerfile-elastalert
    image: elastalert:3.0.0-beta.0
    ports:
      - 3030:3030
      - 3333:3333
    depends_on:
      - elasticsearch
      - kibana
    restart: always
    volumes:
      - ./elastalert/config/elastalert.yaml:/opt/elastalert/config.yaml
      - ./elastalert/config/elastalert-test.yaml:/opt/elastalert/config-test.yaml
      - ./elastalert/config/config.json:/opt/elastalert-server/config/config.json
      - ./elastalert/rules:/opt/elastalert/rules
      - ./elastalert/rule_templates:/opt/elastalert/rule_templates
    healthcheck:
        test: ["CMD-SHELL", "curl -f http://localhost:3030 || exit 1"]
        interval: 30s
        timeout: 15s
        retries: 3
        start_period: 200s

es/config/elasticsearch.yml

cluster.name: "docker-cluster"
network.host: 0.0.0.0
discovery.zen.minimum_master_nodes: 1

kibana/config/kibana.yml

server.name: kibana
server.host: "0"
elasticsearch.hosts: http://elasticsearch:9200
xpack.monitoring.ui.container.elasticsearch.enabled: true

# elastalert-kibana-plugin
elastalert-kibana-plugin.serverHost: elastalert
elastalert-kibana-plugin.serverPort: 3030

Dockerfiles/Dockerfile-elastalert

FROM bitsensor/elastalert:3.0.0-beta.0

USER root

RUN apk update && \
    apk add bash curl && \
    rm -rf /var/cache/apk/*

ADD elastalert/bin/elastalert-start.sh /usr/local/bin/
ADD elastalert/bin/elastic_search_status.sh /usr/local/bin/

RUN chmod +x /usr/local/bin/elastalert-start.sh 
RUN chmod +x /usr/local/bin/elastic_search_status.sh

USER node

ENTRYPOINT ["/usr/local/bin/elastalert-start.sh"]

elastalert/bin/elastic_search_status.sh

#!/bin/bash

set -e

if [ $# -gt 0 ]; then
  ES_URL="$1"
elif [[ -n $ELASTICSEARCH_URL ]]; then
  ES_URL="$ELASTICSEARCH_URL"
elif [[ -n $ES_HOST ]] && [[ -n $ES_PORT ]]; then
  ES_URL="http://$ES_HOST:$ES_PORT"
else
  ES_URL="http://elasticsearch:9200"
fi

until [[ "$(curl -fsSL "$ES_URL/_cat/health?h=status" | sed -r 's/^[[:space:]]+|[[:space:]]+$//g')" =~ ^(yellow|green)$ ]]; do
  # printf '+' >&2
  sleep 1
done

echo "Elasticsearch is up and healthy at "$ES_URL"" >&2

elastalert/bin/elastalert-start.sh

#!/bin/bash

set -e

echo "Giving Elasticsearch at $ELASTICSEARCH_URL time to start..."

elastic_search_status.sh

echo "Starting ElastAlert!"
npm start

elastalert/config/config.json

{
  "appName": "elastalert-server",
  "port": 3030,
  "wsport": 3333,
  "elastalertPath": "/opt/elastalert",
  "verbose": false,
  "es_debug": false,
  "debug": false,
  "rulesPath": {
    "relative": true,
    "path": "/rules"
  },
  "templatesPath": {
    "relative": true,
    "path": "/rule_templates"
  },
  "es_host": "elasticsearch",
  "es_port": 9200,
  "writeback_index": "elastalert_status"
}

elastalert/config/elastalert-test.yml

# NOTE: This config is used when testing a rule

# The elasticsearch hostname for metadata writeback
# Note that every rule can have its own elasticsearch host
es_host: elasticsearch

# The elasticsearch port
es_port: 9200

# This is the folder that contains the rule yaml files
# Any .yaml file will be loaded as a rule
rules_folder: rules

# How often ElastAlert will query elasticsearch
# The unit can be anything from weeks to seconds
run_every:
  seconds: 5

# ElastAlert will buffer results from the most recent
# period of time, in case some log sources are not in real time
buffer_time:
  minutes: 1

# Optional URL prefix for elasticsearch
#es_url_prefix: elasticsearch

# Connect with TLS to elasticsearch
#use_ssl: True

# Verify TLS certificates
#verify_certs: True

# GET request with body is the default option for Elasticsearch.
# If it fails for some reason, you can pass 'GET', 'POST' or 'source'.
# See http://elasticsearch-py.readthedocs.io/en/master/connection.html?highlight=send_get_body_as#transport
# for details
#es_send_get_body_as: GET

# Option basic-auth username and password for elasticsearch
#es_username: someusername
#es_password: somepassword

# The index on es_host which is used for metadata storage
# This can be a unmapped index, but it is recommended that you run
# elastalert-create-index to set a mapping
writeback_index: elastalert_status

# If an alert fails for some reason, ElastAlert will retry
# sending the alert until this time period has elapsed
alert_time_limit:
  days: 2

elastalert/config/elastalert.yml

# The elasticsearch hostname for metadata writeback
# Note that every rule can have its own elasticsearch host
es_host: elasticsearch

# The elasticsearch port
es_port: 9200

# This is the folder that contains the rule yaml files
# Any .yaml file will be loaded as a rule
rules_folder: rules

# How often ElastAlert will query elasticsearch
# The unit can be anything from weeks to seconds
run_every:
  seconds: 5

# ElastAlert will buffer results from the most recent
# period of time, in case some log sources are not in real time
buffer_time:
  minutes: 1

# Optional URL prefix for elasticsearch
#es_url_prefix: elasticsearch

# Connect with TLS to elasticsearch
#use_ssl: True

# Verify TLS certificates
#verify_certs: True

# GET request with body is the default option for Elasticsearch.
# If it fails for some reason, you can pass 'GET', 'POST' or 'source'.
# See http://elasticsearch-py.readthedocs.io/en/master/connection.html?highlight=send_get_body_as#transport
# for details
#es_send_get_body_as: GET

# Option basic-auth username and password for elasticsearch
#es_username: someusername
#es_password: somepassword

# The index on es_host which is used for metadata storage
# This can be a unmapped index, but it is recommended that you run
# elastalert-create-index to set a mapping
writeback_index: elastalert_status

# If an alert fails for some reason, ElastAlert will retry
# sending the alert until this time period has elapsed
alert_time_limit:
  days: 2

$ chmod 777 es/data
$ chmod 777 elastalert/rules
$ chmod 777 elastalert/rule_templates

$ docker-compose up -d

$ docker ps

CONTAINER ID        IMAGE                                                 COMMAND                  CREATED             STATUS                   PORTS                                            NAMES
b2f0a31c5b36        elastalert:3.0.0-beta.0                               "/usr/local/bin/elas…"   6 minutes ago       Up 6 minutes (healthy)   0.0.0.0:3030->3030/tcp, 0.0.0.0:3333->3333/tcp   elastalert
321dc2d22b00        docker.elastic.co/kibana/kibana:7.6.2                 "/usr/local/bin/dumb…"   6 minutes ago       Up 6 minutes (healthy)   0.0.0.0:5601->5601/tcp                           kibana
2ecd8dc1934a        docker.elastic.co/elasticsearch/elasticsearch:7.6.2   "/usr/local/bin/dock…"   6 minutes ago       Up 6 minutes (healthy)   0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp   elasticsearch

[mismailzz]

@nsano-rururu buddy, Thank you so much. I will try through docker and let you know, soon. Allah bless you!

[nsano-rururu]

@mismailzz

Reference information. You may already know. .

Elasticsearch will run out of Java heap memory if you continue to input data with the index open.
circuit_breaking_exception may occur
→ Elasticsearch heap size change
→ Elasticsearch indices.breaker.total.limit changed
→ Close unnecessary index data Or Delete unnecessary Index data
ILM: Manage the index lifecycle

ElastAlert also falls out of memory. Probably, it is possible that the open data of the index to be searched once is expanded in the memory and searched.

[nsano-rururu]

@mismailzz

I could also make a Docker image with Kibana's elastalert-kibana-plugin installed in advance, so I will share the information.

・docker image create (install elastalert-kibana-plugin on Kibana)

cd
mkdir -p docker-image-build/kibana
cd docker-image-build/kibana
mkdir -p kibana/elastalert-kibana-plugin/server/routes
cd kibana/elastalert-kibana-plugin/server/routes
curl -L -O https://raw.githubusercontent.com/mmguero-dev/Malcolm/development/kibana/elastalert-kibana-plugin/server/routes/elastalert.js
cd 
cd docker-image-build/kibana
mkdir -p elastalert-kibana-plugin
cd elastalert-kibana-plugin
curl -L -O https://github.com/bitsensor/elastalert-kibana-plugin/releases/download/1.1.0/elastalert-kibana-plugin-1.1.0-7.5.0.zip
cd
cd docker-image-build/kibana
touch Dockerfile-elastalert-kibana-plugin-1.1.0-7.6.2

・Dockerfile-elastalert-kibana-plugin-1.1.0-7.6.2

FROM docker.elastic.co/kibana/kibana:7.6.2

USER root

RUN yum install -y \
    zip \
    unzip && \
    rm -rf /var/cache/yum/* && \
    yum clean all

ADD elastalert-kibana-plugin/elastalert-kibana-plugin-1.1.0-7.5.0.zip /tmp/elastalert-kibana-plugin.zip
ADD kibana/elastalert-kibana-plugin/server/routes/elastalert.js /tmp/elastalert-server-routes.js

RUN cd /tmp && \
      unzip elastalert-kibana-plugin.zip kibana/elastalert-kibana-plugin/package.json && \
      sed -i "s/7\.5\.0/7\.6\.2/g" kibana/elastalert-kibana-plugin/package.json && \
      mkdir -p kibana/elastalert-kibana-plugin/server/routes/ && \
      cp /tmp/elastalert-server-routes.js kibana/elastalert-kibana-plugin/server/routes/elastalert.js && \
      zip elastalert-kibana-plugin.zip kibana/elastalert-kibana-plugin/package.json kibana/elastalert-kibana-plugin/server/routes/elastalert.js && \
      cd /usr/share/kibana/plugins && \
      /usr/share/kibana/bin/kibana-plugin install file:///tmp/elastalert-kibana-plugin.zip --allow-root && \
      rm -rf /tmp/elastalert-kibana-plugin.zip /tmp/elastalert.js /tmp/kibana

USER kibana

docker build -t kibana:7.6.2 -f Dockerfile-elastalert-kibana-plugin-1.1.0-7.6.2 .

・docker-compose.yml update

add「image: kibana:7.6.2」
coment out「image: docker.elastic.co/kibana/kibana:7.6.2」 and 「command:・・・」

kibana:
    container_name: kibana
    image: kibana:7.6.2
    # image: docker.elastic.co/kibana/kibana:7.6.2
    # command: sh -c './bin/kibana-plugin list | grep [email protected]; result=`echo $$?`; if [ $$result = 1 ]; then  ./bin/kibana-plugin install file:///usr/share/kibana/work/elastalert-kibana-plugin-1.1.0-7.6.2.zip && exec /usr/local/bin/kibana-docker; else exec /usr/local/bin/kibana-docker; fi'

[mismailzz]

@nsano-rururu Issue resolved, Thank You So Much!