Skip to content

Commit

Permalink
Improve indexer package generation guide
Browse files Browse the repository at this point in the history
  • Loading branch information
@f-galland
f-galland committed Nov 26, 2024
1 parent eb43ffc commit b7ea426
Showing 1 changed file with 101 additions and 187 deletions.
288 changes: 101 additions & 187 deletions source/development/packaging/generate-indexer-package.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,258 +3,172 @@
.. meta::
:description: Wazuh provides an automated way of building packages for the Wazuh components. Learn how to build your own Wazuh indexer package in this section of our documentation.

=============
Wazuh indexer
=============

The packages' generation process is orchestrated by two scripts, which are found within the ``packaging_scripts`` folder of the repository:
Local packages generation
#########################

While official ``Wazuh Indexer`` packages are generated in a GitHub Actions pipeline, packages can also be compiled locally or within a docker container.

The packages' generation process is orchestrated by two scripts, found under the ``build-scripts`` directory of the repository:

- ``build.sh``: compiles the Java application and bundles it into a package.
- ``assemble.sh``: uses the package from the previous step and inflates it with plugins and configuration files, ready for production deployment.

Official packages are built through a GitHub Actions pipeline, however, the process is designed to be independent enough for maximum portability.
.. contents:: Table of contents:
:depth: 2
:local:

The building process is self-contained in the application code.
Docker environment
******************
.. raw:: html

<div class="accordion-section open">

Pre-requisistes:
Pre-requisistes
===============

- Clone the ``wazuh-indexer`` repository and switch to the appropriate branch:
1. Clone the ``Wazuh Indexer`` GitHub repository and switch to the ``v|WAZUH_CURRENT|`` tag:

.. code:: console
# git clone https://github.com/wazuh/wazuh-indexer
# git clone https://github.com/wazuh/wazuh-indexer/
# git checkout v|WAZUH_CURRENT|
Build stage
-----------
2. Bring the docker environment up:

Docker environment
^^^^^^^^^^^^^^^^^^
.. code:: console
Using the provided `Docker environment <https://www.github.com/wazuh/wazuh-indexer/tree/v|WAZUH_CURRENT|/docker>`__:
# cd wazuh-indexer/docker/ci
# bash ci.sh up
# cd ../..
.. tabs::
Build a minimal package
=======================

.. group-tab:: RPM
A basic package including only the ``Wazuh Indexer`` engine without extra plugin is generated first.

.. code-block:: console
# docker exec -it wi-build_|WAZUH_CURRENT| bash packaging_scripts/build.sh -a x64 -d rpm
.. group-tab:: DEB
1. Set the environment variables:

.. code-block:: console
# docker exec -it wi-build_|WAZUH_CURRENT| bash packaging_scripts/build.sh -a x64 -d deb
.. group-tab:: TAR
.. note::

.. code-block:: console
# docker exec -it wi-build_|WAZUH_CURRENT| bash packaging_scripts/build.sh -a x64 -d tar
Replace ``<arch>`` with one of ``x64`` or ``arm64`` and ``<package-type>`` with one of ``rpm``, ``deb`` or ``tar``

Local package generation
^^^^^^^^^^^^^^^^^^^^^^^^
.. code:: console
For local package generation, use the ``build.sh`` script.
# ARCHITECTURE=<arch>
# PACKAGE_TYPE=<package-type>
Take a look at the ``build.yml`` workflow file for an example of usage.
2. Run the build script:

.. code:: console
# bash packaging_scripts/build.sh -a x64 -d tar -n $(bash packaging_scripts/baptizer.sh -a x64 -d tar -m)
# docker exec -it wi-build_$(<VERSION) bash build-scripts/build.sh -a $ARCHITECTURE -d $PACKAGE_TYPE -n $(bash build-scripts/baptizer.sh -a $ARCHITECTURE -d $PACKAGE_TYPE -m)
The generated package is sent to the ``wazuh-indexer/artifacts`` folder.
After this step, a minimal package (without plugins) will be present under the ``artifacts`` directory.

.. _full-package-assemble-stage-1:
Full package assembly
=====================

Assembly stage
--------------
1. Set the environment variables:

Docker environment
^^^^^^^^^^^^^^^^^^
.. note::

Replace ``<arch>`` with one of ``x64`` or ``arm64`` and ``<package-type>`` with one of ``rpm``, ``deb`` or ``tar``.

.. code:: console
# ARCHITECTURE=<arch>
# PACKAGE_TYPE=<package-type>
Pre-requisites:
2. Run the assembly process:

- Current directory: ``wazuh-indexer/``
- Existing package in ``wazuh-indexer/artifacts/dist/{rpm|deb}``, as a result of the *Build* stage.
- Using the `Docker environment <https://www.github.com/wazuh/wazuh-indexer/tree/v|WAZUH_CURRENT|/docker>`__:
.. code:: console
.. tabs::
# docker exec -it wi-assemble_$(<VERSION) bash build-scripts/assemble.sh -a $ARCHITECTURE -d $PACKAGE_TYPE -r 1
.. group-tab:: RPM
Native environment
******************
.. raw:: html

.. code-block:: console
<div class="accordion-section open">

# docker exec -it wi-assemble_|WAZUH_CURRENT| bash packaging_scripts/assemble.sh -a x64 -d rpm

.. group-tab:: DEB
Pre-requisistes
================

.. code-block:: console
1. Install build dependencies

# docker exec -it wi-assemble_|WAZUH_CURRENT| bash packaging_scripts/assemble.sh -a x64 -d deb
.. tabs::

.. group-tab:: RPM

.. code-block:: console
.. group-tab:: TAR
# yum install -y git curl gnupg2 gcc gcc-c++ make cpio rpm-build mesa-libGLU freeglut alsa-lib atk at-spi2-core cairo cairo-devel cups-libs libdrm libgbm nspr nspr-devel nss pango libXcomposite libXdamage libXfixes libXfixes-devel libXi libxkbcommon libXrandr libXrender libXtst rpm rpm-build maven
.. group-tab:: DEB

.. code-block:: console
.. code-block:: console
# apt-get update
# apt-get install -y git curl gnupg2 y build-essential cpio debhelper-compat debmake freeglut3 libasound2 libatk-bridge2.0-0 libatk1.0-0 libatspi2.0-dev libcairo2 libcairo2-dev libcups2 libdrm2 libgbm-dev libgconf-2-4 libnspr4 libnspr4-dev libnss3 libpangocairo-1.0-0 libxcomposite-dev libxdamage1 libxfixes-dev libxfixes3 libxi6 libxkbcommon-x11-0 libxrandr2 libxrender1 libxtst6 rpm rpm2cpio maven
# docker exec -it wi-assemble_|WAZUH_CURRENT| bash packaging_scripts/assemble.sh -a x64 -d tar
2. Clone the ``wazuh-indexer`` repository and switch to the appropriate branch

Local package generation
^^^^^^^^^^^^^^^^^^^^^^^^
.. code:: console
.. note::
# git clone https://github.com/wazuh/wazuh-indexer
# git checkout v|WAZUH_CURRENT|
Set the environment variable ``TEST=true`` to assemble a package with a minimal set of plugins, speeding up the assembly process.
.. tabs::
Build a minimal package
=======================

.. group-tab:: RPM
A basic package including only the ``Wazuh Indexer`` engine without extra plugin is generated first.

The ``assemble.sh`` script will use the output from the ``build.sh`` script and use it as a base to bundle together a final package containing the plugins, the production configuration and the service files.

The script will:

#. Extract the RPM package using ``rpm2cpio`` and ``cpio`` tools.

By default, ``rpm2cpio`` and ``cpio`` tools expect the package to be in ``wazuh-indexer/artifacts/tmp/rpm``.
The script takes care of creating the required folder structure, copying also the min package and the SPEC file.

Current folder loadout at this stage:

.. code-block:: none
/rpm/$ARCH
/etc
/usr
/var
wazuh-indexer-min-*.rpm
wazuh-indexer.rpm.spec
``usr``, ``etc`` and ``var`` folders contain ``wazuh-indexer`` files, extracted from ``wazuh-indexer-min-*.rpm``.
1. Set the environment variables:

``wazuh-indexer.rpm.spec`` is copied over from ``wazuh-indexer/distribution/packages/src/rpm/wazuh-indexer.rpm.spec``.
.. note::

The ``wazuh-indexer-performance-analyzer.service`` file is also copied from the same folder.
Replace ``<arch>`` with one of ``x64`` or ``arm64`` and ``<package-type>`` with one of ``rpm``, ``deb`` or ``tar``

It is a dependency of the SPEC file.

#. Install the plugins using the ``opensearch-plugin`` CLI tool.

#. Set up configuration files.

Included in ``min-package``. Default files are overwritten.

#. Bundle an RPM file with ``rpmbuild`` and the SPEC file ``wazuh-indexer.rpm.spec``.

``rpmbuild`` is part of the ``rpm`` OS package.
.. code:: console
``rpmbuild`` is invoked from ``wazuh-indexer/artifacts/tmp/rpm``.
# ARCHITECTURE=<arch>
# PACKAGE_TYPE=<package-type>
It creates the ``{BUILD,RPMS,SOURCES,SRPMS,SPECS,TMP}`` folders and applies the rules in the SPEC file.
2. Run the build script:

If successful, ``rpmbuild`` will generate the package in the ``RPMS/`` folder.
.. code:: console
The script will copy it to ``wazuh-indexer/artifacts/dist`` and clean: remove the ``tmp\`` folder and its contents.

Current folder loadout at this stage:

.. code-block:: none
/rpm/$ARCH
/{BUILD,RPMS,SOURCES,SRPMS,SPECS,TMP}
/etc
/usr
/var
wazuh-indexer-min-*.rpm
wazuh-indexer.rpm.spec
# bash build-scripts/build.sh -a $ARCHITECTURE -d $PACKAGE_TYPE -n $(bash build-scripts/baptizer.sh -a $ARCHITECTURE -d $PACKAGE_TYPE -m)
.. group-tab:: DEB
After this step, a minimal package (without plugins) will be present under the ``artifacts`` directory.

For DEB packages, the ``assemble.sh`` script will perform the following operations:

#. Extract the deb package using ``ar`` and ``tar`` tools.

By default, ``ar`` and ``tar`` tools expect the package to be in ``wazuh-indexer/artifacts/tmp/deb``.
Full package assembly
=====================

The script takes care of creating the required folder structure, copying also the min package and the Makefile.

Current folder loadout at this stage:

.. code-block:: none
artifacts/
|-- dist
| |-- wazuh-indexer-min_|WAZUH_CURRENT|_amd64.deb
`-- tmp
`-- deb
|-- Makefile
|-- data.tar.gz
|-- debmake_install.sh
|-- etc
|-- usr
|-- var
`-- wazuh-indexer-min_|WAZUH_CURRENT|_amd64.deb
``usr``, ``etc`` and ``var`` folders contain ``wazuh-indexer`` files, extracted from ``wazuh-indexer-min-*.deb``.
1. Set the ``ARCHITECTURE`` and ``PACKAGE_TYPE`` environment variables replacing ``<package-type>`` with one of ``tar``, ``deb`` or ``rpm`` and ``<arch>`` with ``x64`` or ``arm64`` depending on the target system the packages are being built for.

``Makefile`` and the ``debmake_install`` are copied over from ``wazuh-indexer/distribution/packages/src/deb``.
.. code:: console
The ``wazuh-indexer-performance-analyzer.service`` file is also copied from the same folder.
# ARCHITECTURE=<arch>
# PACKAGE_TYPE=<package-type>
It is a dependency of the SPEC file.

#. Install the plugins using the ``opensearch-plugin`` CLI tool.

#. Set up configuration files.

Included in ``min-package``. Default files are overwritten.

#. Bundle a DEB file with ``debmake`` and the ``Makefile``.

``debmake`` and other dependencies can be installed using the ``provision.sh`` script.
The script is invoked by the GitHub Workflow.

Current folder loadout at this stage:

.. code-block:: none
artifacts/
|-- artifact_name.txt
|-- dist
| |-- wazuh-indexer-min_|WAZUH_CURRENT|_amd64.deb
| `-- wazuh-indexer_|WAZUH_CURRENT|_amd64.deb
`-- tmp
`-- deb
|-- Makefile
|-- data.tar.gz
|-- debmake_install.sh
|-- etc
|-- usr
|-- var
|-- wazuh-indexer-min_|WAZUH_CURRENT|_amd64.deb
`-- debian/
| -- control
| -- copyright
| -- rules
| -- preinst
| -- prerm
| -- postinst
.. group-tab:: TAR
2. Run the assembly process:

.. code:: console
# bash build-scripts/assemble.sh -a $ARCHITECTURE -d $PACKAGE_TYPE
The assembly process for tarballs consists on:

#. Extraction of the minimal package
#. Bundling of plugins
#. Addition of Wazuh configuration files and tooling
#. Compression

.. code:: console
# bash packaging_scripts/assemble.sh -a x64 -d tar -r 1
Build and assemble scripts reference
------------------------------------
####################################
.. raw:: html

<div class="accordion-section open">

The packages' generation process is guided through bash scripts.

Expand Down

0 comments on commit b7ea426

Please sign in to comment.