Setting scaling.enabled along with scaling.webhooks.enabled results in web hook endpoint returning 404 or 502

[knuurr]

We're experimenting with introducing at least some HA to our workflow processing, as very often we experience that main n8n pod is hanging and needs manual restart (and sometimes workers too).

This is our current, working setup:

            extraEnv:
              WEBHOOK_URL: "https://n8n.domain.com/"
               ....

            scaling:
              enabled: true
              worker:
                count: 2
                concurrency: 2
              webhook:
                enabled: false
                count: 1
              redis:
                host: <redis url>
                password: <redis pwd>

In normal condition, when we try to POST our webhook endpoint, everything is ok and status is 200

curl -I -X POST https://n8n.domain.com/webhook/someendpoint
HTTP/2 200 
date: Mon, 28 Oct 2024 15:34:30 GMT
content-type: application/json; charset=utf-8
...

As I introduce changes to values in chart, setting scaling.webhook.enabled to true, the same endpoint can no longer be accessed, with returning 404 or 502 (not sure what it depends on yet - but it doesn't work)

            scaling:
              enabled: true
              worker:
                count: 2
                concurrency: 2
              webhook:
                enabled: true # change
                count: 1
              redis:
                host: <redis url>
                password: <redis pwd>

curl -I -X POST https://n8n.domain.com/webhook/someendpoint
HTTP/2 404 
date: Mon, 28 Oct 2024 15:33:10 GMT
content-type: text/html; charset=utf-8
....

We also see the same in our endpoint monitoring tool that once change is live, monitoring fails.

I'm not entirely sure if this is a question related to chart, or general n8n question. But since we use this chart for deployment I decided that maybe asking here is a good start.

What are we doing wrong here? If any info is missing I will happily provide

n8n Version 1.40.0

[nidhalkratos]

After doing lots of research and going through this chart line by line, and reading the n8n queue mode documentation, I came to the conclusion that the webhook-test must be done by the main instance. Which means, there should not an ingress rule in the ingress.yaml file for the /webhook-test/ prefix, and so all the requests should go to the main service. Only the /webhook/ rule should point to the n8n-webhooks service. To validate my suggestion, simply manually edit the ingress rule in your cluster and remove that part and save and see if it works.
I hope that helps!
PS: I just noticed that the chart in its ingress file already have the webhook-test point to the main instance. It did not initially notice that because I was using my own ingress rules... Anyway, just leaving this in case someone else faced the same issue.