Ingress broke after Nginx Ingress Controller update 1.12.0

[sim500]

Expected Behavior

The ingresses of the console and the api server should work properly, managing HTTPS requests from the outside and proxy them to the HTTPS minio backend service.

Current Behavior

Using minio-operator and minio-tenant v6.0.4, both installed with helm.
After the latest upgrade of Nginx Ingress Controller to the application version 1.12.0, we are encountering issues with the annotations specified here https://github.com/minio/operator/blob/master/docs/nginx-ingress.md, that are the following:

  nginx.ingress.kubernetes.io/backend-protocol: HTTPS
  nginx.ingress.kubernetes.io/configuration-snippet: |
    chunked_transfer_encoding off;
  nginx.ingress.kubernetes.io/proxy-body-size: "0"
  nginx.ingress.kubernetes.io/proxy-ssl-verify: "off"
  nginx.ingress.kubernetes.io/rewrite-target: /
  nginx.ingress.kubernetes.io/server-snippet: |
        client_max_body_size 0;
        # Deny access to the cluster metrics endpoint
        location = /minio/v2/metrics/cluster {
                                  deny all;
                                  return 403;
                                }

Issuing a request towards an ingress configured in this way, now return a 404 error from nginx. The Nginx Ingress Controller doesn't log any specific error, other then the warning:
annotation group ConfigurationSnippet contains risky annotation based on ingress configuration
It's not clear which one of those annotations cause this malfunction, and I'm not sure if they're all necessary for the correct functioning of minio.

Possible Solution

After understanding which nginx annotations are causing the issue, removing or editing some of them should fix the problem

Steps to Reproduce (for bugs)

1. Remove all those annotations, the Ingress is able to contact the backend minio service, responding:
   "Client sent an HTTP request to an HTTPS server."
2. Add all the annotations again, the Ingress continues to work, still responding
   "Client sent an HTTP request to an HTTPS server."
   Which is weird since now there's the nginx.ingress.kubernetes.io/backend-protocol: HTTPS annotation there.
3. Remove all of them again, except for these two:

nginx.ingress.kubernetes.io/backend-protocol: HTTPS
nginx.ingress.kubernetes.io/proxy-ssl-verify: "off"

Now you finally receive an Access Denied from the actual minio backend.
Add again all the other annotation and it should continue to work fine.

Context

Both Minio backend and console are not accessible from the outside

Your Environment

k8s version: 1.31
minio-operator and minio-tenant versions: v6.0.4, both installed with helm
Nginx Ingress Controller version: 1.12.0

[hannesr]

I was experiencing the same error, and was able to fix it by escalating the allowed risk level of the ingress annotations in nginx-ingress. This appears as following line in the ingress controller configuration: annotations-risk-level: "Critical". After this the warning about the risky annotation is also gone.

The annotation risk levels are described here: https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations-risk/

[sim500]

Hi @hannesr, thank you very much for your help. It works!
I added the annotation you specified, and the ingresses are now accepted by the Ingress Controller.
So I can finally access the backends through them again.

Just for reference, I link this two pages that helped me set up the nginx ingress controller helm values to add the annotation:
https://ellie.wtf/notes/ingress-nginx-risky-annotations
kubernetes/ingress-nginx#12618