Add nightly linux package tests

[mowies]

Description

Create a nightly version of the linux package tests that use the latest snapshots of main branch of core and contrib distributions from the respective repos.

This is a follow-up to open-telemetry/opentelemetry-collector-contrib#34372

[mowies]

i'd like to work on this, pls assign me :)

[mowies]

I started implementing this but hit some road blocks it seems. I tried building the otelcol-contrib on the spot while setting all component versions to main to do a fully up-to-date main branch build. Turns out, that comes with some complications and really really long build times (~45m to 1h).
After some more thinking I came up with a better solution: Use the already built artifacts that are attached to every main branch build in the contrib repo (e.g. artifacts from this build) and then just build the linux packages from that with the actual distribution files from the releases repo.

There are 2 options on where to do this:

• run tests nightly in releases repo and have a github token with proper permissions to download the artifacts from contrib repo
• run tests nightly/on merge in contrib repo and just pull in the linux package files from the releases repo and have access to the artifacts ootb

would be cool to get some feedback on this, maybe from @andrzej-stencel or @mx-psi or @codeboten

[mx-psi]

that comes with some complications and really really long build times (~45m to 1h)

Do we know (or can we know) why that is the case? Is it downloading dependencies that takes too long? Cloning the repo? The build process itself?

Use the already built artifacts that are attached to every main branch build in the contrib repo (e.g. artifacts from this build) and then just build the linux packages from that with the actual distribution files from the releases repo.

One downside of this is that these artifacts do not necessarily reflect the build time options we use here, but for testing it feels like they should be a net positive

run tests nightly in releases repo and have a github token with proper permissions to download the artifacts from contrib repo

I would prefer this, I'd like to keep anything related to linux package files on this repository

[mowies]

Do we know (or can we know) why that is the case? Is it downloading dependencies that takes too long? Cloning the repo? The build process itself?

From what I could gather, it's trying to fetch the latest main version from the Go proxy and waits for minutes until finally getting back a 404, then going to the github repo and downloading the whole thing from there, which is not a small download either. And I don't think lots of things are cached either in-between dependencies, so it does the whole process over and over.

I would prefer this, I'd like to keep anything related to linux package files on this repository

sounds good, then I'll continue in this direction

[mowies]

New status on this:
I'll use the prebuilt builder from goreleaser pro which lets you split a goreleaser job into build and package/archive/docker/rest.
This let's us either use the latest contrib collector binary from main branch or build it as usual from scratch.

the only downside is that maintainability will suffer a bit since the contrib goreleaser config is split into 2 files. But I think this is the only sensible approach after also speaking with goreleaser maintainers (goreleaser/goreleaser#5234).