Why we need to fetch wildcard when webhook received?

[andrejshapal]

Hello,

I am receiving webhook:

INFO[2024-07-12T18:46:40+03:00] project ref not currently exported but its configuration matches a wildcard, triggering a pull of the project  project-name=ns/grafana-dashboards ref=main ref-kind=branch

I have the following config:

wildcards:
  - owner: 
      name: ns

I am getting:

done looking up for wildcards matching the project ref  project-name=ns/grafana-dashboards ref=main ref-kind=branch
INFO[2024-07-12T18:46:49+03:00] discovered new ref                            project-name=ns/grafana-dashboards ref=436 ref-kind=merge-request
INFO[2024-07-12T18:46:49+03:00] discovered new ref                            project-name=ns/grafana-dashboards ref=275 ref-kind=merge-request
INFO[2024-07-12T18:46:49+03:00] discovered new ref                            project-name=ns/grafana-dashboards ref=438 ref-kind=merge-request
INFO[2024-07-12T18:46:49+03:00] discovered new ref                            project-name=ns/grafana-dashboards ref=529 ref-kind=merge-request
INFO[2024-07-12T18:46:49+03:00] discovered new ref                            project-name=ns/grafana-dashboards ref=187 ref-kind=merge-request

What I am not getting, why we should discover refs from wildcard? We received ref from webhook:
project-name=ns/grafana-dashboards ref=main ref-kind=branch
We checked if it matches the wildcard and we know we should track this ref. Why do we need to track anything else in wildcard except this ref?
I know we can limit refs from wildcard by using max_age_seconds and most_recent, but I think this is dirty solution in my case.

The reason why I think this is not efficient:
Let's imagine we have monorepo with 3000 stale branches and 100 engineers concurrently are pushing changes. At any moment suddenly stale branch can become not stale. At the same time, we have multiple monorepo in namespace and we can not use projects:. In such case we use wildcard. On every webhook push exporter is fetching refs from wildcard, meaning, it is getting 3000 stale branches and 1000 stale merge requests.

From the other side, when projects: is used, the behavior is correct. It just tracks the ref from webhook without fetching anything.

Solution:
Add flag to wildcard config to not fetch anything and just add track the ref from webhook.

wildcards:
 - owner:
      ...
    webhook:
      discovery: false 

Expected result:
When webhook is called, exporter checks if the arrived event matches wildcard and if it is, starts to track it (fetches pipeline and jobs status on metrics interval. According to garbage collector interval, the ref is removed in some and exporter stops to fetch pipeline data on metrics interval.
We will have less cardinality.
If exporter is restarted it will not track any ref until webhook is triggered. Inconsistent data. But this is standard behavior for prometheus metrics.