Development - OVA/AMI - Create provisioner module

[Enaraque]

Description

The provisioning module will be responsible for obtaining and installing all Wazuh components, including their dependencies, as well as downloading the cert-tool.sh script required for certificate creation.

Prerequisites for Execution

Credentials

This module assumes that an instance already exists where components and dependencies can be installed. Therefore, one of the main inputs should be the necessary credentials to connect to the instance.

List of URLs

Starting from version 5.0.0, the method of installing components will change. Previously, components were installed through a directory, but now they will be installed via packages.
These packages will be published and should be downloadable through a URL.
Thus, one of the main inputs should include the various URLs required for the installation process.

List of Dependencies

In addition to the URLs for installing each component, a list of all necessary dependencies must also be provided. These dependencies are required to ensure that each Wazuh component functions correctly.

This list of dependencies, along with the component URLs, provides the necessary context to determine what needs to be prepared on the remote machine.

Summary

The key requirements for the module to download components and dependencies are:

• URLs of the components.
• Required credentials to connect to the machine.
• Necessary dependencies for each component.
• (to be defined if any additional inputs are required)

Tests

Unit Tests

As the module is being developed, unit tests will need to be written continuously. These tests must validate the code to ensure that each method and/or function behaves as expected.

The tests will be written using pytest. To verify that an optimal percentage of the code is being tested, the overall test coverage should be calculated and monitored.

Deployment Tests

These tests must verify that all necessary dependencies and each Wazuh component are correctly installed.

Proposed Provisioner Structure

├── provisioner/
│   ├── __init__.py
│   ├── provisioner.py
|   ├── ...
├── tests/
    ├── unit/
    |   ├── provisioner/
    |       ├── __init__.py
    |       ├── test_provisioner.py
 	| 	    ├── ...
    ├── deployment/
        ├── provisioner/
            ├── __init__.py
            ├── test_provision_deployment.py

Related issue

• Create OVA and AMI Configuration Modules #157

DRI

@teddytpc1

[Enaraque]

Update report

The logic for obtaining and storing package URLs, the cert-tool, and the dependencies for each component has begun to be designed.

Classes have been created to retrieve and store this information, allowing flexible access to any package URL as well as its dependencies.

With this, by passing the following parameters to the provisioner module, we now have the foundation to begin downloading the components on the target machine:

• Architecture (arm64, amd64...)
• Package manager (apt, yum)
• Package type (deb, rpm)
• Component(s) to install
• Inventory
• Path to the URL file

[Enaraque]

Update report

So far, different parts of the code have been developed to allow downloading dependencies and packages for Wazuh components.

A specific module has been created for logging, which prints messages to the screen as the provisioner executes, providing context on what is happening.

The idea of using either Paramiko or Ansible for connecting to and executing commands on the remote machine was considered. After evaluating the options and the ease of implementation, Paramiko was chosen due to its better integration with Python. Additionally, if Ansible had been used, the playbooks would have only served to execute the provisioner module itself and nothing else, making its use somewhat unnecessary in this case.

With this approach, a solid logic has been established to retrieve and download packages in a clean and straightforward manner while also ensuring good execution context through the logging system.

[Enaraque]

Update Report

The code for the provisioner logic has been completed.
It has been tested to ensure that everything works correctly, from connecting to the remote machine using the inventory obtained from the allocator execution to downloading the Wazuh packages, dependencies, and the cert-tool.

Provisioner execution

Additionally, the following has been designed:

URL File Formatter

The file that will be used to obtain the URLs for the packages and the cert-tool will have a key-value format. Initially, the provisioner module was designed to have a different format that was clearer for Python code. However, since it is necessary for this file to be as simple as possible to facilitate its use in other repositories, the key-value format was retained.

This led to the idea of creating a script dedicated to formatting the input URL file into the format that was previously used in the provisioner. This approach has several advantages:

1. No modifications were needed to the existing logic—only a function call to format the file.
2. It provides a level of abstraction between this file and the core logic. Thanks to this abstraction, if the URL file format changes in the future, there will be no need to modify the core logic—only the formatting function will need to be updated.

pyproject.toml and Hatch

A pyproject.toml file has been added to the repository. Thanks to this file, we can centralize the configurations for different Python tools (such as Ruff, Pytest, and Hatch) in a single location. It defines the dependencies required to run the provisioner and its tests.

Additionally, the environments and scripts to be used with Hatch have been defined in this file.

Hatch is a tool that helps create isolated Python environments for executing code in various modules, such as the provisioner. With Hatch, we can have isolated environments where the required dependencies (specified in pyproject.toml) are automatically installed, along with additional dependencies specific to each environment.

Thanks to Hatch, we can execute the code with just Hatch installed:

hatch run provisioner:run --inventory provisioner/inventory.yaml --packages-url-path provisioner/static/raw_urls.yaml

We can also run tests with:

hatch run provisioner:test-cov

Tests

Once the logic was completed, test development began. These tests can be executed using Hatch, the provisioner environment, and the script test-cov created in pyproject.toml.

When running tests with Hatch, a coverage report is displayed, showing which lines of each provisioner file are untested and the total coverage percentage.

Tests execution

Currently, the tests are still in development, so the coverage is low.

[Enaraque]

Update report

The tests for the provisioner have continued. So far, only the tests for the main functions of the code remain, which are still being developed.

Actual tests coverage

After a meeting with @CarlosALgit , where we discussed the new OVA development and its implementation, we identified the need to add a local provisioning option to the provisioner. Now, if we provide an inventory to the provisioner, it will provision remotely based on the inventory. If no inventory is provided, it will provision the necessary packages on the local machine.
This change makes developments like the OVA easier, as it will be needed to clone the virtual-machines repo into the instance created in AWS.

[Enaraque]

Update report

All tests related to the provisioner have been completed. A total of 99% coverage has been achieved, successfully testing all lines of the provisioner with a total of 122 tests.