run RHEL rpminspect tests

[kwk]

@nikic I think you asked about this in our retrospective, right? I took the freedom to open a ticket for this here.

[jchecahi]

@kwk I thought I'd chime in and share some thoughts about this.

Some key points about rpminspect:

• rpminspect does NOT support testing builds from copr. Only koji/brew (identified by nvr), or local RPM builds.
• The existing automations and scripts only work taking koji/brew build ids as input.
• The common way to run rpminspect is in comparison mode. This is running inspections on the tested and the last good packages and compare them.

For the llvm-snapshots use case there are some things we need to take into account, as we'll probably need to script it from scratch. The following points only consider running rpminspect via some script, leaving out any considerations regarding integration with the snapshots manager, the github issues, labels, recovering testing-farm runs, etc...

• As COPR builds are not supported by rpminspect, each chroot build needs to be downloaded somewhere, then run rpminspect on it as a local build.
• If we want special reporting -- such as separate results per inspection as in gating pipelines -- we need to script it ourselves.
• To run comparison mode, we'd need to somehow map each "last good" chroot and store it over the time so we can download that one. In Fedora/RHEL gating we use this mode. For snapshots, we would need to somehow mark the "last good" build to compare, for each chroot. IMHO, analysis mode without comparison should work just fine.
• For RHEL chroots, we need to run with rpminspect-data-redhat config, which is in an internal copr repo. We need to make sure that wherever we are running the test, it's able to reach RedHat internal resources.
• Each package can have a dedicated config (like this one from compiler-rt). OSCI automations grab it from the corresponding repos and so we'd need to automate that too. For our use case it might be simpler as right now we're only building llvm and that's a well known repo url.

As for snapshots manager integration, without having deep knowledge of the code to make an informed guess, looks like a refactor would be needed:

• The logic assumes that a chroot has one and only one testing-farm request for tests. If more are found, these are assumed to be outdated test runs. If we add rpminspect testing we would need 2 testing-farm requests per chroot.
• Testing farm classes counts on hardcoded tmt plan for snapshots functional tests (the tmt tests).
• New labels would need to be added, as well as the logic to update the github issues

[kwk]

@jchecahi thank you for your thoughts and ideas.

Does rpminspect need to run on the OS and architecture that RPMs are dedicated for?

Some key points about rpminspect:

* rpminspect does NOT support testing builds from copr. Only koji/brew (identified by nvr), or local RPM builds.

As long as the RPM itself is good enough, a local download of the RPM should be fine.

* The existing automations and scripts only work taking koji/brew build ids as input.

What scripts are you talking about?

* The common way to run rpminspect is in comparison mode. This is running inspections on the tested and the last good packages and compare them.

For the llvm-snapshots use case there are some things we need to take into account, as we'll probably need to script it from scratch. The following points only consider running rpminspect via some script, leaving out any considerations regarding integration with the snapshots manager, the github issues, labels, recovering testing-farm runs, etc...

* As COPR builds are not supported by rpminspect, each chroot build needs to be downloaded somewhere, then run rpminspect on it as a local build.

* If we want special reporting -- such as separate results per inspection as in gating pipelines -- we need to script it ourselves.

* To run comparison mode, we'd need to somehow map each "last good" chroot and store it over the time so we can download that one. In Fedora/RHEL gating we use this mode. For snapshots, we would need to somehow mark the "last good" build to compare, for each chroot. IMHO, analysis mode without comparison should work just fine.

The "last good" chroot can always be found here: https://copr.fedorainfracloud.org/coprs/g/fedora-llvm-team/llvm-snapshots/ . "Good" currently does not respect good rpminspect results for now of course. But once we integrate rpminspect, it can certainly vote before a build gets promoted.

* For RHEL chroots, we need to run with rpminspect-data-redhat config, which is in an internal copr repo. We need to make sure that wherever we are running the test, it's able to reach RedHat internal resources.

Okay, this bit is tricky but if we need testing-farm to run rpminspect anyways (see open question above) we should be able to reach internal Red Hat resources.

* Each package can have a dedicated config (like [this one from compiler-rt](https://src.fedoraproject.org/rpms/compiler-rt/blob/rawhide/f/rpminspect.yaml)). OSCI automations grab it from the corresponding repos and so we'd need to automate that too. For our use case it might be simpler as right now we're only building llvm and that's a well known repo url.

We do build more than just llvm. The rpminspect.yaml files have to be added to the big-merge branch in the future anyways as compiler-rt is now build together with llvm (See this build for example).

As for snapshots manager integration, without having deep knowledge of the code to make an informed guess, looks like a refactor would be needed:

* The logic assumes that a chroot has one and only one testing-farm request for tests. If more are found, these are assumed to be outdated test runs. If we add rpminspect testing we would need 2 testing-farm requests per chroot.

This is only partially correct. Indeed, we currently associate each chroot with just one testing-farm request but we do it explicitly and it should be easy to add another one. All we do is we store the association as an invisible HTML comment inside the issue body like this:

<!--TESTING_FARM:fedora-rawhide-aarch64/e9fdbd5b-fdc1-4d7b-87b1-fb81d918bac8/7663432-->

We could easily add another request by adding a comment like this with a different prefix:

<!--TESTING_FARM_RPMINSPECT:fedora-rawhide-aarch64/a96484c5-8b9b-4fc7-aea9-168b9ede5c1a/7663432-->

Then we associate the copr build ID (7663432) of the llvm package with the uuid of the testing-farm request (a96484c5-8b9b-4fc7-aea9-168b9ede5c1a) for the chroot (fedora-rawhide-aarch64). I hope this makes sense.

All in all we don't find testing-farm requests but we remember them.

* Testing farm classes counts on hardcoded tmt plan for snapshots functional tests (the tmt tests).

Are you referring to the --plan /tests/snapshot-gating parameter here?

* New labels would need to be added, as well as the logic to update the github issues

This is something we've done in the past extensively and it should be doable analogously.

[jchecahi]

@kwk let me reply in line

Does rpminspect need to run on the OS and architecture that RPMs are dedicated for?

No, not necessarily. rpminspect uses configuration files with profiles to ensure it's testing the requirements for the distro and release the build is targetted to. As a rough example you could run rpminspect in a rawhide system but pass the config files for RHEL, specifying the profile for RHEL-9. More important, can be run in any arch, so you can test s390x builds on x86_64 host.

What scripts are you talking about?

There are existiting scripts that fedora-ci (and OSCI) use to run rpminspect on gated builds. rpminspect-runner for the script and rpminspect-pipeline for the actual CI config. As I mentioned before these work taking a koji task id. We can create our own using a similar approach but adding some extra steps to download the copr builds.

The "last good" chroot can always be found here: https://copr.fedorainfracloud.org/coprs/g/fedora-llvm-team/llvm-snapshots/ . "Good" currently does not respect good rpminspect results for now of course. But once we integrate rpminspect, it can certainly vote before a build gets promoted.

Yeah that makes sense. "last good" can be whatever we need it to be.

We do build more than just llvm. The rpminspect.yaml files have to be added to the big-merge branch in the future anyways as compiler-rt is now build together with llvm (See this build for example).

Yeah sorry I explained it very poorly. I meant, as we're building the big-merge strategy we're only building the llvm srpm, from the llvm repo. Hence, it's well known where to get the rpminspect.yaml file.

This is only partially correct. Indeed, we currently associate each chroot with just one testing-farm request but we do it explicitly and it should be easy to add another one. All we do is we store the association as an invisible HTML comment inside the issue body like this:

<!--TESTING_FARM:fedora-rawhide-aarch64/e9fdbd5b-fdc1-4d7b-87b1-fb81d918bac8/7663432-->

We could easily add another request by adding a comment like this with a different prefix:

<!--TESTING_FARM_RPMINSPECT:fedora-rawhide-aarch64/a96484c5-8b9b-4fc7-aea9-168b9ede5c1a/7663432-->

Then we associate the copr build ID (7663432) of the llvm package with the uuid of the testing-farm request (a96484c5-8b9b-4fc7-aea9-168b9ede5c1a) for the chroot (fedora-rawhide-aarch64). I hope this makes sense.

All in all we don't find testing-farm requests but we remember them.

I see thanks for the explanation! I didn't take a very deep look at the python code as I focused more on the rpminspect part. These ideas sound very reasonable and (I think) easy to implement.

Are you referring to the --plan /tests/snapshot-gating parameter here?

Yes, that one. It's probably not a big deal to either parametrize it or add another method that creates a different TF request, but I saw it and wanted to point it out :)

* New labels would need to be added, as well as the logic to update the github issues

This is something we've done in the past extensively and it should be doable analogously.

Sounds good!

I think we can create a custom plan for running rpminspect on the snapshots. There are existing containers ready to run rpminspect (public for fedora, internal for RHEL) which we can use with TF. We can also write a wrapper script that gets the copr project and chroot as we do with functional tests. This script can retrieve the build, prepares whatever rpminspect config we need, run it, and report in a nicely way.