Problem: Undocumented breaking change in Galaxy role installation from v2.10.31

[ShaneMcC]

Issue

Commit c76f04a makes a breaking change in how semaphore installs roles/collections.

Previously, semaphore would install roles from:

• <playbook_dir>/collections/requirements.yml or <repo_dir>/collections/requirements.yml
• <playbook_dir>/roles/requirements.yml or <repo_dir>/roles/requirements.yml

But this change now makes it only install relative to the playbook path:

• <playbook_dir>/collections/requirements.yml or <playbook_dir>/requirements.yml
• <playbook_dir>/roles/requirements.yml or <playbook_dir>/requirements.yml

Ignoring <repo_dir>/roles/requirements.yml and <repo_dir>/collections/requirements.yml files entirely that were previously used (similar to anisble tower which looks for these at the top-level)

In cases where the playbook is in the repo root, this change makes no difference, however in cases where we are running a playbook from a subdirectory this causes problems.

This was documented in the release for v2.10.31 as Handle <playbook_dir>/requirements.txt by ansible-galaxy. but no mention was made around this removing the previous behaviour.

If we want to also look at <playbook_dir>/requirements.yml first then this should be in addition to the previous behaviour not instead of.

Impact

Ansible (task execution)

Installation method

Docker

Database

Postgres

Browser

Chrome

Semaphore Version

v2.10.35-a71a3c5-1730103926

Ansible Version

No response

Logs & errors

No response

Manual installation - system information

No response

Configuration

No response

Additional information

No response

[jp1337]

Hi @ShaneMcC and @fiftin!
In addition to the original problem, we encountered another one today.

If you merge roles and collections together in a requirements.yml file and place it in the playbook directory, no roles are installed.

The following test case:

1. We had placed a requirements.yml with only one role directly next to the playbook in its directory.
2. When the task was executed, an attempt was first made to install the collections from the requirements.yml.
3. This created a file with the md5 hash of the requirements.yml.
4. During the role installation, the requirements.yml was then skipped because the md5 hash had not changed.

So currently the only way is via a roles and a collections subfolder in the playbook directory. From our point of view, no other way is possible.

Example structure:

playbooks/subdir/
├── collections
│   └── requirements.yml
├── playbook.yml
├── roles
     └── requirements.yml

In addition to this structure the logic should also work for the following example where roles and requirements are places in a single requirements.yml file.

This structure is officially supported and documented by ansible in this guide:
Installing roles and collections from the same requirements.yml file

playbooks/subdir/
├── playbook.yml
└── requirements.yml

Since we have a large number of playbooks and also a large number of folders, we had to help ourselves out with a Python script. This automatically generates the requirements.yml files and the corresponding collections and roles folders.
This script would not be needed if we could use a single requirements.yml file for roles and requirements like described in the issue.

[ShaneMcC]

@jp1337 I think that is a separate issue than this one - semaphore has always needed separate requirements.yml files for collections and roles. But previously it always at least read them at the top-level not only within the playbook directory.

(I would also prefer that it was able to use a single file for both, but it has always been separate as long as I've used it and it at least worked as-expected before once I separated them)