[platform] list_enterprise_source_stubs error keeps happening

[khoramism]

Helm Chart Version

1.3.1

What step the error happened?

On deploy

Relevant information

Hey folks, I am trying to deploy airbyte in an on-prem environment using helm charts, I use this command to install the helm chart.

helm install airbyte airbyte/airbyte --set minio.image.repository=quay.io/minio/minio --set minio.image.tag=RELEASE.2024-01-01T16-36-33Z-cpuv1 --namespace airbyte

After the installation happens and everything in clean, i get the same error in my source and destinations, here is the error, i have extracted the error from the airbyte server pod

I have tried putting the value of ENTERPRISE_SOURCE_STUBS_URL to empty using this env var in my helm values

ENTERPRISE_SOURCE_STUBS_URL: ""

but the issue still exists, any idea what's wrong!?

Relevant log output

2025-01-13 10:21:08,705 [io-executor-thread-2]  WARN    i.a.c.i.AirbyteCompatibleConnectorVersionsProvider(getCompatibleConnectorsMatrix):53 - Disabling compatibility validation: request to fetch platform compatibility matrix failed: 403 with message: .
2025-01-13 10:21:08,876 [io-executor-thread-2]  ERROR   i.a.s.a.ApiHelper(execute):46 - Unexpected Exception
java.lang.RuntimeException: Failed to fetch connector registry entry for airbyte/source-easypost:0.0.1
        at io.airbyte.config.specs.RemoteDefinitionsProvider.getConnectorRegistryEntryJson(RemoteDefinitionsProvider.java:226)
        at io.airbyte.config.specs.RemoteDefinitionsProvider.getSourceDefinitionByVersion(RemoteDefinitionsProvider.java:115)
        at io.airbyte.config.persistence.ActorDefinitionVersionResolver.fetchRemoteActorDefinitionVersion(ActorDefinitionVersionResolver.java:78)
        at io.airbyte.config.persistence.ActorDefinitionVersionResolver.resolveVersionForTag(ActorDefinitionVersionResolver.java:58)
        at io.airbyte.commons.server.handlers.helpers.ActorDefinitionHandlerHelper.defaultDefinitionVersionFromUpdate(ActorDefinitionHandlerHelper.java:120)
        at io.airbyte.commons.server.handlers.SourceDefinitionsHandler.updateSourceDefinition(SourceDefinitionsHandler.java:319)
        at io.airbyte.server.apis.SourceDefinitionApiController.lambda$updateSourceDefinition$9(SourceDefinitionApiController.java:179)
        at io.airbyte.server.apis.ApiHelper.execute(ApiHelper.kt:31)
        at io.airbyte.server.apis.SourceDefinitionApiController.updateSourceDefinition(SourceDefinitionApiController.java:179)
        at io.airbyte.server.apis.$SourceDefinitionApiController$Definition$Exec.dispatch(Unknown Source)
        at io.micronaut.context.AbstractExecutableMethodsDefinition$DispatchedExecutableMethod.invokeUnsafe(AbstractExecutableMethodsDefinition.java:461)
        at io.micronaut.context.DefaultBeanContext$BeanContextUnsafeExecutionHandle.invokeUnsafe(DefaultBeanContext.java:4354)
        at io.micronaut.web.router.AbstractRouteMatch.execute(AbstractRouteMatch.java:272)
        at io.micronaut.web.router.DefaultUriRouteMatch.execute(DefaultUriRouteMatch.java:38)
        at io.micronaut.http.server.RouteExecutor.executeRouteAndConvertBody(RouteExecutor.java:488)
        at io.micronaut.http.server.RouteExecutor.lambda$callRoute$5(RouteExecutor.java:465)
        at io.micronaut.core.execution.ExecutionFlow.lambda$async$1(ExecutionFlow.java:87)
        at io.micronaut.core.propagation.PropagatedContext.lambda$wrap$3(PropagatedContext.java:211)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.io.IOException: getConnectorRegistryEntryJson request ran into status code error: 403 with message: 
        at io.airbyte.config.specs.RemoteDefinitionsProvider.getConnectorRegistryEntryJson(RemoteDefinitionsProvider.java:223)
        ... 20 common frames omitted

2025-01-13 10:21:08,888 [io-executor-thread-2]  ERROR   i.a.c.s.e.h.UncaughtExceptionHandler(handle):33 - Uncaught exception
java.lang.RuntimeException: Failed to fetch connector registry entry for airbyte/source-easypost:0.0.1
        at io.airbyte.config.specs.RemoteDefinitionsProvider.getConnectorRegistryEntryJson(RemoteDefinitionsProvider.java:226)
        at io.airbyte.config.specs.RemoteDefinitionsProvider.getSourceDefinitionByVersion(RemoteDefinitionsProvider.java:115)
        at io.airbyte.config.persistence.ActorDefinitionVersionResolver.fetchRemoteActorDefinitionVersion(ActorDefinitionVersionResolver.java:78)
        at io.airbyte.config.persistence.ActorDefinitionVersionResolver.resolveVersionForTag(ActorDefinitionVersionResolver.java:58)
        at io.airbyte.commons.server.handlers.helpers.ActorDefinitionHandlerHelper.defaultDefinitionVersionFromUpdate(ActorDefinitionHandlerHelper.java:120)
        at io.airbyte.commons.server.handlers.SourceDefinitionsHandler.updateSourceDefinition(SourceDefinitionsHandler.java:319)
        at io.airbyte.server.apis.SourceDefinitionApiController.lambda$updateSourceDefinition$9(SourceDefinitionApiController.java:179)
        at io.airbyte.server.apis.ApiHelper.execute(ApiHelper.kt:31)
        at io.airbyte.server.apis.SourceDefinitionApiController.updateSourceDefinition(SourceDefinitionApiController.java:179)
        at io.airbyte.server.apis.$SourceDefinitionApiController$Definition$Exec.dispatch(Unknown Source)
        at io.micronaut.context.AbstractExecutableMethodsDefinition$DispatchedExecutableMethod.invokeUnsafe(AbstractExecutableMethodsDefinition.java:461)
        at io.micronaut.context.DefaultBeanContext$BeanContextUnsafeExecutionHandle.invokeUnsafe(DefaultBeanContext.java:4354)
        at io.micronaut.web.router.AbstractRouteMatch.execute(AbstractRouteMatch.java:272)
        at io.micronaut.web.router.DefaultUriRouteMatch.execute(DefaultUriRouteMatch.java:38)
        at io.micronaut.http.server.RouteExecutor.executeRouteAndConvertBody(RouteExecutor.java:488)
        at io.micronaut.http.server.RouteExecutor.lambda$callRoute$5(RouteExecutor.java:465)
        at io.micronaut.core.execution.ExecutionFlow.lambda$async$1(ExecutionFlow.java:87)
        at io.micronaut.core.propagation.PropagatedContext.lambda$wrap$3(PropagatedContext.java:211)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.io.IOException: getConnectorRegistryEntryJson request ran into status code error: 403 with message: 
        at io.airbyte.config.specs.RemoteDefinitionsProvider.getConnectorRegistryEntryJson(RemoteDefinitionsProvider.java:223)
        ... 20 common frames omitted

2025-01-13 10:21:25,847 [io-executor-thread-10] ERROR   i.a.c.s.h.EnterpriseSourceStubsHandler(listEnterpriseSourceStubs):53 - Encountered an HTTP error fetching enterprise connectors. Message: Unexpected code 403
2025-01-13 10:21:25,853 [io-executor-thread-10] ERROR   i.a.c.s.e.h.UncaughtExceptionHandler(handle):33 - Uncaught exception
java.lang.RuntimeException: java.io.IOException: HTTP error fetching enterprise sources
        at io.airbyte.server.apis.ApiHelper.execute(ApiHelper.kt:43)
        at io.airbyte.server.apis.SourceDefinitionApiController.listEnterpriseSourceStubs(SourceDefinitionApiController.java:125)
        at io.airbyte.server.apis.$SourceDefinitionApiController$Definition$Exec.dispatch(Unknown Source)
        at io.micronaut.context.AbstractExecutableMethodsDefinition$DispatchedExecutableMethod.invokeUnsafe(AbstractExecutableMethodsDefinition.java:461)
        at io.micronaut.context.DefaultBeanContext$BeanContextUnsafeExecutionHandle.invokeUnsafe(DefaultBeanContext.java:4354)
        at io.micronaut.web.router.AbstractRouteMatch.execute(AbstractRouteMatch.java:231)
        at io.micronaut.web.router.DefaultUriRouteMatch.execute(DefaultUriRouteMatch.java:38)
        at io.micronaut.http.server.RouteExecutor.executeRouteAndConvertBody(RouteExecutor.java:488)
        at io.micronaut.http.server.RouteExecutor.lambda$callRoute$5(RouteExecutor.java:465)
        at io.micronaut.core.execution.ExecutionFlow.lambda$async$1(ExecutionFlow.java:87)
        at io.micronaut.core.propagation.PropagatedContext.lambda$wrap$3(PropagatedContext.java:211)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.io.IOException: HTTP error fetching enterprise sources
        at io.airbyte.commons.server.handlers.EnterpriseSourceStubsHandler.listEnterpriseSourceStubs(EnterpriseSourceStubsHandler.kt:57)
        at io.airbyte.server.apis.ApiHelper.execute(ApiHelper.kt:31)
        ... 13 common frames omitted
Caused by: java.io.IOException: Unexpected code 403
        at io.airbyte.commons.server.handlers.EnterpriseSourceStubsHandler.listEnterpriseSourceStubs(EnterpriseSourceStubsHandler.kt:40)
        ... 14 common frames omitted

[marcosmarxm]

It seems other API calls are failing as well. Is this a fresh deployment @khoramism?

[khoramism]

I am using the 1.3.1 official helm chart version

[marcosmarxm]

@khoramism did you upgrade from a previous version or installed 1st time?

[khoramism]

@khoramism did you upgrade from a previous version or installed 1st time?

First time

[khoramism]

Just tested version 1.1.1, such a problem doesn't exist there.