ArgoCD is flooding logs with "invalid session: oidc:"

[BeyerJC]

Hi everyone,

i wasnt sure if its better to post here or in Issues. My application logs for argo-server are flooded (5-10 log entries per second)
with this error

time="2022-07-28T11:31:09Z" level=info msg="finished streaming call with code Unauthenticated" error="rpc error: code = Unauthenticated desc = invalid session: oidc: token is expired (Token Expiry: 2022-07-26 08:17:25 +0000 UTC)" grpc.code=Unauthenticated grpc.method=Watch grpc.service=application.ApplicationService grpc.start_time="2022-07-28T11:31:09Z" grpc.time_ms=5.886 span.kind=server system=grpc
time="2022-07-28T11:31:09Z" level=info msg="finished streaming call with code Unauthenticated" error="rpc error: code = Unauthenticated desc = invalid session: oidc: token is expired (Token Expiry: 2022-07-26 08:17:25 +0000 UTC)" grpc.code=Unauthenticated grpc.method=Watch grpc.service=application.ApplicationService grpc.start_time="2022-07-28T11:31:09Z" grpc.time_ms=3.068 span.kind=server system=grpc
time="2022-07-28T11:31:09Z" level=info msg="finished streaming call with code Unauthenticated" error="rpc error: code = Unauthenticated desc = invalid session: oidc: token is expired (Token Expiry: 2022-07-26 08:17:25 +0000 UTC)" grpc.code=Unauthenticated grpc.method=WatchResourceTree grpc.service=application.ApplicationService grpc.start_time="2022-07-28T11:31:09Z" grpc.time_ms=3.94 span.kind=server system=grpc
time="2022-07-28T11:31:10Z" level=info msg="finished streaming call with code Unauthenticated" error="rpc error: code = Unauthenticated desc = invalid session: oidc: token is expired (Token Expiry: 2022-07-26 08:17:25 +0000 UTC)" grpc.code=Unauthenticated grpc.method=Watch grpc.service=application.ApplicationService grpc.start_time="2022-07-28T11:31:10Z" grpc.time_ms=3.61 span.kind=server system=grpc
time="2022-07-28T11:31:10Z" level=info msg="finished streaming call with code Unauthenticated" error="rpc error: code = Unauthenticated desc = invalid session: oidc: token is expired (Token Expiry: 2022-07-26 08:17:25 +0000 UTC)" grpc.code=Unauthenticated grpc.method=WatchResourceTree grpc.service=application.ApplicationService grpc.start_time="2022-07-28T11:31:10Z" grpc.time_ms=3.883 span.kind=server system=grpc

I dont now if its an internal problem or how to solve it. The only hint i can see is the "oidc", which i have enabled for login with this config

    oidc.config: |
      name: Keycloak
      issuer: https://my-keycloak.org/auth/realms/master
      clientID: argocd
      clientSecret: xxxxx
      requestedScopes: 
        - openid
        - profile
        - email

But even when i deactivate this configuration and use "local" login, the error is still logged.

Any ideas what could cause this problem or how to solve it ?

[BeyerJC]

It was caused by an internal problem not related to ArgoCD!

[krishgu]

Hi @BeyerJC - what was the issue?

We're running into the same. Thx!

[BeyerJC]

Hey @krishgu , in my case it was caused by an employee that did not close the pod logs. The Browser kept making requests with an expired token.

[crenshaw-dev]

Would one of y'all kindly submit an issue for that? We should probably have a mechanism in the API server to stop logging those failures.

[krishgu]

@BeyerJC - thank you!

I'll create a new ticket later today. I don't have any residual requests, but may be my SSO is issuing a token for too short a time.. Investigating further

[BeyerJC]

Would one of y'all kindly submit an issue for that? We should probably have a mechanism in the API server to stop logging those failures.

#10126

I opened an issue when i had that problem. @crenshaw-dev

[krishgu]

@BeyerJC @crenshaw-dev thanks. I won't open a new one. #10126 looks good

In my case too, this issue applied. Couple of other tabs accessing the same URL might have been the cause.

But I had fully restarted in this order - 1) ingress controller(nginx) 2) argocd-controller 3) argocd-repo-server 4) argocd-server. Then the session timeout seems to be throwing back the login screen, and the issue looks resolved.