Refactor Dependency Management and Update Build Instructions

[vikman90]

Description

The current build process has some inconsistencies in dependency management and toolchain setup that require improvement. The following issues were identified:

1. Dependencies:
   • Several libraries (libmagic-dev, libsqlite3-dev, liblua5.4-dev, libarchive-dev, openssl) are installed directly through the system package manager but are essential for the agent. These should be managed through vcpkg, our chosen package manager for library dependencies.
2. vcpkg Submodule:
   • vcpkg is currently installed as a submodule of the repository. To streamline the repository and improve flexibility, vcpkg should instead be installed as part of the external toolchain.
3. LLVM Nightly:
   • The role of LLVM Nightly builds is unclear. Clarification is needed on whether these are essential for compiling the agent or are required solely for debugging and testing purposes.
4. Toolchain Separation:
   • The essential toolchain (required to compile the agent) should be separated from the development toolchain (used for testing and debugging).
5. Documentation:
   • The BUILD.md file needs updating to reflect these changes and provide clear instructions for setting up the environment and building the agent.

Tasks

1. Move all necessary code dependencies (libmagic, libsqlite3, liblua5.4, libarchive, and openssl) to vcpkg.
2. Remove vcpkg as a submodule from the repository and ensure it is installed externally as part of the toolchain.
3. Clarify the necessity of LLVM Nightly builds in the build process and document its role in the BUILD.md file.
4. Separate the essential toolchain (compilation) from the development toolchain (debugging and testing).
5. Update the BUILD.md file with clear and accurate instructions.

Expected Outcome

• Streamlined dependency management with vcpkg handling all required libraries.
• A cleaner repository without the vcpkg submodule.
• Clear delineation between the essential and development toolchains.
• Comprehensive and up-to-date build documentation.

[jr0me]

Update

Going through VCPKG documentation, testing which dependencies should be moved from a system-wide package installation to vcpkg.

[jr0me]

Update

Removed vcpkg submodule and configuration of the toolchain in CMake files.

At the moment I can build the project by:

1. Cloning Vcpkg at a different location and running the bootstrap script, adding the repository root to the system path so it can find the vcpkg binary.
2. In wazuh-agent repository run the cmake configuration command with the option -DCMAKE_TOOLCHAIN_FILE=</path/to/vcpkg>/scripts/buildsystems/vcpkg.cmake
3. Running cmake build command as usual.

Note: the vcpkg repository still needs to be checked out, there are no packages for vcpkg, so this adds an extra step to the build process. We will have to adjust all of github actions and workflows, currently looking into: https://github.com/marketplace/actions/run-vcpkg in this regard having it as a submodule streamlines the process by a fair amount.

[jr0me]

Update

Using FetchContent to use vcpkg seems like a good compromise between having it as a submodule and checking it out manually. This has been used and tested so far in the linked branch.

Made some modifications to the procps and librpm port overlays, trying to minimize packages installations and that the dependencies use the ones from vcpkg and not a system installation when available.

In a clean Ubuntu 24 I was able to build the agent installing only these packages via apt:

• curl zip unzip tar (vcpkg)
• cmake
• g++
• pkg-config
• autoconf
• autopoint
• libtool
• gettext
• libsystemd-dev

Trying to build packages failed since they rely on vcpkg as a submodule and have hardcoded this to enable binary caching.