Skip to content

Commit

Permalink
Add postgres pod status check for k8s tests in CI (#1320)
Browse files Browse the repository at this point in the history
It appears we have a flaky Kubernetes test that failed in PR #1313. As
shown in the error log
[here](https://github.com/astronomer/astronomer-cosmos/actions/runs/11796817624/job/32867560902?pr=1313#step:7:473),
the PostgreSQL pod did not reach the ready state and instead entered an
error status. Since the cause of the error status is unclear, this PR
introduces a status check for the PostgreSQL pod to ensure it becomes
fully running and healthy. If the pod enters an ERROR state, we now run
a `kubectl describe` command on the pod to capture the event logs for
debugging. The test will also exit with an error code of 1 to prevent
further execution.

related: #1319
  • Loading branch information
pankajkoti authored Nov 13, 2024
1 parent 2b21f4c commit 92330f5
Showing 1 changed file with 17 additions and 0 deletions.
17 changes: 17 additions & 0 deletions scripts/test/kubernetes-setup.sh
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,23 @@ kind load docker-image dbt-jaffle-shop:1.0.0
# The output is filtered to get the first pod's name
POD_NAME=$(kubectl get pods -n default -l app=postgres -o jsonpath='{.items[0].metadata.name}')

# Wait for the PostgreSQL pod to be in the 'Running' and 'Ready' state
echo "Waiting for PostgreSQL pod to be ready..."
while true; do
POD_STATUS=$(kubectl get pod "$POD_NAME" -n default -o jsonpath='{.status.phase}')

if [ "$POD_STATUS" = "Running" ] && [ "$(kubectl get pod "$POD_NAME" -n default -o 'jsonpath={..status.conditions[?(@.type=="Ready")].status}')" = "True" ]; then
echo "PostgreSQL pod is up and running!"
break
elif [ "$POD_STATUS" = "Error" ]; then
echo "Error: PostgreSQL pod failed to start. Exiting..."
kubectl describe pod "$POD_NAME" -n default # Show details for debugging
exit 1
else
echo "Pod $POD_NAME is not ready yet (status: $POD_STATUS). Waiting..."
sleep 5
fi
done
# Print the name of the PostgreSQL pod
echo "$POD_NAME"

Expand Down

0 comments on commit 92330f5

Please sign in to comment.