test-network-k8s: chaincode deploy step fails on k3s environment

[thejjw]

Hello,

I cannot get through chaincode deploy step using test-network-k8s. I was able to do all the steps up to then: cluster init, up, channel create...but when I issues ./network chaincode deploy ... command it got stuck Launching chaincode container "asset-transfer-basic"... and eventually errored exceeding progress deadline.
I tried to diagnose further using k9s tool and as seen from the screenshot below it looks like that the pod failed trying to pull nonexistent image url "docker.io/library/asset-transfer-basic:latest".
My k8s environment is built using k3s. Should that be a problem?
Any help troubleshooting this is appreciated.

Thanks.

[1]

[2]

[3]

note: I have experienced similar error on using sample-network from fabric-operator: hyperledger-labs/fabric-operator#94

[thejjw]

relevant bits from network-debug.log:

Content

+ sleep 0.5
+ [[ 4 -lt 1 ]]
+ MODE=chaincode
+ shift
+ '[' chaincode == kind ']'
+ '[' chaincode == unkind ']'
+ [[ chaincode == \c\l\u\s\t\e\r ]]
+ [[ chaincode == \k\8\s ]]
+ [[ chaincode == \k\u\b\e ]]
+ '[' chaincode == up ']'
+ '[' chaincode == down ']'
+ '[' chaincode == channel ']'
+ [[ chaincode == \c\h\a\i\n\c\o\d\e ]]
+ chaincode_command_group deploy asset-transfer-basic ../asset-transfer-basic/chaincode-java
+ COMMAND=deploy
+ shift
+ '[' deploy == deploy ']'
+ log 'Deploying chaincode'
+ echo -e Deploying chaincode
+ deploy_chaincode asset-transfer-basic ../asset-transfer-basic/chaincode-java
+ local cc_name=asset-transfer-basic
+ local cc_label=asset-transfer-basic
++ absolute_path ../asset-transfer-basic/chaincode-java
++ local relative_path=../asset-transfer-basic/chaincode-java
+++ cd ../asset-transfer-basic/chaincode-java
+++ pwd
++ local abspath=/root/fabric-samples/asset-transfer-basic/chaincode-java
++ echo /root/fabric-samples/asset-transfer-basic/chaincode-java
+ local cc_folder=/root/fabric-samples/asset-transfer-basic/chaincode-java
++ mktemp -d
+ local temp_folder=/tmp/tmp.KGrFsMH6Dk
+ local cc_package=/tmp/tmp.KGrFsMH6Dk/asset-transfer-basic.tgz
+ prepare_chaincode_image /root/fabric-samples/asset-transfer-basic/chaincode-java asset-transfer-basic
+ local cc_folder=/root/fabric-samples/asset-transfer-basic/chaincode-java
+ local cc_name=asset-transfer-basic
+ build_chaincode_image /root/fabric-samples/asset-transfer-basic/chaincode-java asset-transfer-basic
+ local cc_folder=/root/fabric-samples/asset-transfer-basic/chaincode-java
+ local cc_name=asset-transfer-basic
+ push_fn 'Building chaincode image asset-transfer-basic'
+ echo -ne '   - Building chaincode image asset-transfer-basic ...'
+ nerdctl build --namespace k8s.io -t asset-transfer-basic /root/fabric-samples/asset-transfer-basic/chaincode-java
#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 1.02kB done
#1 DONE 0.0s

#2 [internal] load .dockerignore
#2 transferring context: 2B done
#2 DONE 0.0s

#3 [internal] load metadata for docker.io/library/gradle:jdk11-alpine
#3 DONE 1.7s

#4 [internal] load metadata for docker.io/library/openjdk:11-jre
#4 DONE 1.6s

#5 [internal] load build context
#5 transferring context: 1.20kB done
#5 DONE 0.0s

#6 [stage-1 1/6] FROM docker.io/library/openjdk:11-jre@sha256:356949c3125c4fa8104745e7ea92bd995da4567634e6599b470d2f972d13e0e2
#6 resolve docker.io/library/openjdk:11-jre@sha256:356949c3125c4fa8104745e7ea92bd995da4567634e6599b470d2f972d13e0e2 0.0s done
#6 DONE 0.0s

#7 [gradle_build 1/4] FROM docker.io/library/gradle:jdk11-alpine@sha256:235ce49c53047d8a049e39e89e694639e8fa34096621206acc7bec6c3c630903
#7 resolve docker.io/library/gradle:jdk11-alpine@sha256:235ce49c53047d8a049e39e89e694639e8fa34096621206acc7bec6c3c630903 0.0s done
#7 DONE 0.0s

#8 https://github.com/krallin/tini/releases/download/v0.19.0/tini
#8 DONE 0.7s

#9 [gradle_build 3/4] COPY build.gradle ./
#9 CACHED

#10 [stage-1 2/6] ADD https://github.com/krallin/tini/releases/download/v0.19.0/tini /tini
#10 CACHED

#11 [stage-1 5/6] COPY --chown=javauser:javauser --from=GRADLE_BUILD /home/gradle/build/libs/chaincode.jar /chaincode.jar
#11 CACHED

#12 [stage-1 3/6] RUN chmod +x /tini
#12 CACHED

#13 [stage-1 4/6] RUN addgroup --system javauser && useradd -g javauser javauser
#13 CACHED

#14 [gradle_build 2/4] COPY src/ src/
#14 CACHED

#15 [gradle_build 4/4] RUN gradle --no-daemon build shadowJar -x checkstyleMain -x checkstyleTest
#15 CACHED

#16 [stage-1 6/6] COPY --chown=javauser:javauser docker/docker-entrypoint.sh /docker-entrypoint.sh
#16 CACHED

#17 exporting to docker image format
#17 exporting layers done
#17 exporting manifest sha256:2ec7c9f00f2f325d53bb66e3db7ad1a8be643c6bcc7686c24180fc795d742cb9 done
#17 exporting config sha256:37df60b77385de7a4017634937cbdc99ca14d1385eca4b7428ef24b2eebe8238 done
#17 sending tarball
unpacking docker.io/library/asset-transfer-basic:latest (sha256:2ec7c9f00f2f325d53bb66e3db7ad1a8be643c6bcc7686c24180fc795d742cb9)...
#17 sending tarball 3.1s done
#17 DONE 3.1s
Loaded image: docker.io/library/asset-transfer-basic:latest
+ pop_fn
+ '[' 0 -eq 0 ']'
+ echo -ne '\r✅'
+ echo ''
+ return
+ '[' k3s == k3s ']'
+ export CHAINCODE_IMAGE=asset-transfer-basic
+ CHAINCODE_IMAGE=asset-transfer-basic
+ package_chaincode asset-transfer-basic asset-transfer-basic /tmp/tmp.KGrFsMH6Dk/asset-transfer-basic.tgz
+ '[' ccaas == k8s ']'
+ '[' ccaas == ccaas ']'
+ package_ccaas_chaincode asset-transfer-basic asset-transfer-basic /tmp/tmp.KGrFsMH6Dk/asset-transfer-basic.tgz
+ local cc_name=asset-transfer-basic
+ local cc_label=asset-transfer-basic
+ local cc_archive=/tmp/tmp.KGrFsMH6Dk/asset-transfer-basic.tgz
++ dirname /tmp/tmp.KGrFsMH6Dk/asset-transfer-basic.tgz
+ local cc_folder=/tmp/tmp.KGrFsMH6Dk
++ basename /tmp/tmp.KGrFsMH6Dk/asset-transfer-basic.tgz
+ local archive_name=asset-transfer-basic.tgz
+ push_fn 'Packaging ccaas chaincode asset-transfer-basic'
+ echo -ne '   - Packaging ccaas chaincode asset-transfer-basic ...'
+ mkdir -p /tmp/tmp.KGrFsMH6Dk
+ local 'cc_default_address={{.peername}}-ccaas-asset-transfer-basic:9999'
+ local 'cc_address={{.peername}}-ccaas-asset-transfer-basic:9999'
+ cat
+ cat
+ tar -C /tmp/tmp.KGrFsMH6Dk -zcf /tmp/tmp.KGrFsMH6Dk/code.tar.gz connection.json
+ tar -C /tmp/tmp.KGrFsMH6Dk -zcf /tmp/tmp.KGrFsMH6Dk/asset-transfer-basic.tgz code.tar.gz metadata.json
+ rm /tmp/tmp.KGrFsMH6Dk/code.tar.gz
+ pop_fn
+ '[' 0 -eq 0 ']'
+ echo -ne '\r✅'
+ echo ''
+ return
+ '[' ccaas == ccaas ']'
+ set_chaincode_id /tmp/tmp.KGrFsMH6Dk/asset-transfer-basic.tgz
+ local cc_package=/tmp/tmp.KGrFsMH6Dk/asset-transfer-basic.tgz
++ shasum -a 256 /tmp/tmp.KGrFsMH6Dk/asset-transfer-basic.tgz
++ tr -s ' '
++ cut -d ' ' -f 1
+ cc_sha256=353334bac2ad1c949c9ad48a62940feeebad9a6a0cb17522d23c5812e6cca99a
++ tar zxfO /tmp/tmp.KGrFsMH6Dk/asset-transfer-basic.tgz metadata.json
++ jq -r .label
+ cc_label=asset-transfer-basic
+ CHAINCODE_ID=asset-transfer-basic:353334bac2ad1c949c9ad48a62940feeebad9a6a0cb17522d23c5812e6cca99a
+ launch_chaincode asset-transfer-basic asset-transfer-basic:353334bac2ad1c949c9ad48a62940feeebad9a6a0cb17522d23c5812e6cca99a asset-transfer-basic
+ local org=org1
+ local cc_name=asset-transfer-basic
+ local cc_id=asset-transfer-basic:353334bac2ad1c949c9ad48a62940feeebad9a6a0cb17522d23c5812e6cca99a
+ local cc_image=asset-transfer-basic
+ launch_chaincode_service org1 peer1 asset-transfer-basic asset-transfer-basic:353334bac2ad1c949c9ad48a62940feeebad9a6a0cb17522d23c5812e6cca99a asset-transfer-basic
+ local org=org1
+ local peer=peer1
+ local cc_name=asset-transfer-basic
+ local cc_id=asset-transfer-basic:353334bac2ad1c949c9ad48a62940feeebad9a6a0cb17522d23c5812e6cca99a
+ local cc_image=asset-transfer-basic
+ push_fn 'Launching chaincode container "asset-transfer-basic"'
+ echo -ne '   - Launching chaincode container "asset-transfer-basic" ...'
+ cat kube/org1/org1-cc-template.yaml
+ sed 's,{{CHAINCODE_NAME}},asset-transfer-basic,g'
+ sed 's,{{CHAINCODE_ID}},asset-transfer-basic:353334bac2ad1c949c9ad48a62940feeebad9a6a0cb17522d23c5812e6cca99a,g'
+ sed 's,{{CHAINCODE_IMAGE}},asset-transfer-basic,g'
+ exec kubectl -n test-network apply -f -
+ sed 's,{{PEER_NAME}},peer1,g'
deployment.apps/org1peer1-ccaas-asset-transfer-basic created
service/org1peer1-ccaas-asset-transfer-basic created
+ kubectl -n test-network rollout status deploy/org1peer1-ccaas-asset-transfer-basic
Waiting for deployment "org1peer1-ccaas-asset-transfer-basic" rollout to finish: 0 of 1 updated replicas are available...
error: deployment "org1peer1-ccaas-asset-transfer-basic" exceeded its progress deadline
+ exit_fn
+ rc=1
+ set +x

[thejjw]

I must have been mistaken; I was able to run past chaincode deployment and query/invoke when I reinitialized environment with k3s this time specifying to use docker runtime.
I thought using the below configuration should suffice to make the script compatible with default k3s(containerd runtime), but I must have been wrong.

export TEST_NETWORK_CLUSTER_RUNTIME="k3s"
export TEST_NETWORK_CONTAINER_NAMESPACE="--namespace k8s.io"
export CONTAINER_CLI="nerdctl"

Closing the issue as I was able to get past the problem albeit with a different configuration.