Adding s390x arch support in k8s 1.28 provider #1201
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,33 +1,56 @@ | ||
| FROM quay.io/fedora/fedora:39 AS base | ||
|
|
||
| RUN dnf -y install jq iptables iproute dnsmasq qemu socat openssh-clients screen bind-utils tcpdump iputils libguestfs-tools-c && dnf clean all | ||
|
|
||
| FROM base AS imageartifactdownload | ||
|
|
||
| ARG BUILDARCH | ||
|
|
||
| ARG centos_version | ||
|
|
||
| WORKDIR / | ||
|
|
||
| RUN echo "Centos9 version $centos_version" | ||
|
|
||
| COPY scripts/download_box.sh / | ||
|
|
||
| RUN if test "$BUILDARCH" != "s390x"; then \ | ||
| /download_box.sh https://cloud.centos.org/centos/9-stream/x86_64/images/CentOS-Stream-Vagrant-9-$centos_version.x86_64.vagrant-libvirt.box && \ | ||
| curl -L -o /initramfs-amd64.img http://mirror.stream.centos.org/9-stream/BaseOS/x86_64/os/images/pxeboot/initrd.img && \ | ||
| curl -L -o /vmlinuz-amd64 http://mirror.stream.centos.org/9-stream/BaseOS/x86_64/os/images/pxeboot/vmlinuz; \ | ||
| else \ | ||
| /download_box.sh https://cloud.centos.org/centos/9-stream/s390x/images/CentOS-Stream-GenericCloud-9-$centos_version.s390x.qcow2 && \ | ||
|
There was a problem hiding this comment. we can't use the same generic cloud image for x86? There was a problem hiding this comment. @brianmcarey I think we can change, but just want to not to mix those changes for x86 here. As this is already XXL, wanted to have that as a separate PR. I will create that as a follow-up PR for this one. There was a problem hiding this comment. Added the placeholder PR #1242 so that this can be further tracked there. |
||
| # Access virtual machine disk images directly by using LIBGUESTFS_BACKEND=direct, instead of libvirt | ||
| export LIBGUESTFS_BACKEND=direct && \ | ||
| guestfish --ro --add box.qcow2 --mount /dev/sda1:/ ls /boot/ | grep -E '^vmlinuz-|^initramfs-' | xargs -I {} guestfish --ro --add box.qcow2 -i copy-out /boot/{} / ; \ | ||
| fi | ||
|
|
||
|
|
||
| FROM base as nodecontainer | ||
|
|
||
| ARG BUILDARCH | ||
|
|
||
| WORKDIR / | ||
|
|
||
| COPY vagrant.key /vagrant.key | ||
|
|
||
| RUN chmod 700 vagrant.key | ||
|
|
||
| ENV DOCKERIZE_VERSION v0.6.1 | ||
|
|
||
| RUN if test "$BUILDARCH" != "s390x"; then \ | ||
| curl -L -o dockerize-linux-$BUILDARCH.tar.gz https://github.com/jwilder/dockerize/releases/download/$DOCKERIZE_VERSION/dockerize-linux-amd64-$DOCKERIZE_VERSION.tar.gz; \ | ||
| else \ | ||
| # Temporary till s390x support is upstreamed to dockerize (https://github.com/jwilder/dockerize/pull/200) | ||
| curl -L -o dockerize-linux-$BUILDARCH.tar.gz https://github.com/ibm-jitendra/kubevirt_pkgs/raw/main/dockerize-linux-s390x.tar.gz; \ | ||
|
There was a problem hiding this comment. Is there any timeline for the upstreaming of this? I would prefer not to rely on a random tarball. There was a problem hiding this comment. @brianmcarey : No update as of now. We are tried to reach upstream owner on PR, but no luck. Now, Vamsi is trying to reach him on LinkedIn. There was a problem hiding this comment. Thanks for @vamsikrishna-siddu for the followup. Now changes are upstreamed. |
||
| fi && \ | ||
| tar -xzvf dockerize-linux-$BUILDARCH.tar.gz && \ | ||
| rm dockerize-linux-$BUILDARCH.tar.gz && \ | ||
| chmod u+x dockerize && \ | ||
| mv dockerize /usr/local/bin/ | ||
|
|
||
| COPY --from=imageartifactdownload /box.qcow2 box.qcow2 | ||
| COPY --from=imageartifactdownload /vmlinuz-* /vmlinuz | ||
| COPY --from=imageartifactdownload /initramfs-* /initrd.img | ||
|
|
||
| COPY scripts/* / | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| root=/dev/vda1 ro no_timer_check console=tty0 console=ttyS0,115200n8 net.ifnames=0 biosdevname=0 crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M | ||
|
There was a problem hiding this comment. It would be good to have a separate commit for these kernel args that explains why some of the different options are required. There was a problem hiding this comment. These were the defaults within the generic cloud image of centos9 for s390x. I've copied them to this file to mainly keep parity with x86 based kernel args, which I think exists for configuring args for kernel. Will try to come up with a readme around this. There was a problem hiding this comment. Added this as separate commit in k8s-1.30-provider-slim-s390x branch, where in commit I've explained all the details. |
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -6,6 +6,7 @@ import ( | |
| "os" | ||
| "os/signal" | ||
| "path/filepath" | ||
| "runtime" | ||
| "strconv" | ||
| "strings" | ||
|
|
||
|
|
@@ -51,6 +52,7 @@ func NewProvisionCommand() *cobra.Command { | |
|
|
||
| func provisionCluster(cmd *cobra.Command, args []string) (retErr error) { | ||
| var base string | ||
| sshUser := utils.GetSSHUserByArchitecture(runtime.GOARCH) | ||
| packagePath := args[0] | ||
| versionBytes, err := os.ReadFile(filepath.Join(packagePath, "version")) | ||
| if err != nil { | ||
|
|
@@ -228,13 +230,14 @@ func provisionCluster(cmd *cobra.Command, args []string) (retErr error) { | |
| } | ||
|
|
||
| // Wait for ssh.sh script to exist | ||
| logrus.Info("Wait for ssh.sh script to exist") | ||
| err = _cmd(cli, nodeContainer(prefix, nodeName), "while [ ! -f /ssh_ready ] ; do sleep 1; done", "checking for ssh.sh script") | ||
| if err != nil { | ||
| logrus.Info("Error: Wait for ssh.sh script to exist") | ||
| return err | ||
| } | ||
|
|
||
| err = waitForVMToBeUp(cli, prefix, nodeName) | ||
| if err != nil { | ||
| return err | ||
| } | ||
|
|
@@ -252,21 +255,21 @@ func provisionCluster(cmd *cobra.Command, args []string) (retErr error) { | |
| if err != nil { | ||
| return err | ||
| } | ||
| err = _cmd(cli, nodeContainer(prefix, nodeName), fmt.Sprintf("if [ -f /scripts/extra-pre-pull-images ]; then scp -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -i vagrant.key -P 22 /scripts/extra-pre-pull-images %s@192.168.66.101:/tmp/extra-pre-pull-images; fi", sshUser), "copying /scripts/extra-pre-pull-images if existing") | ||
|
There was a problem hiding this comment. I think some of this ssh handling has changed since #1209 was merged There was a problem hiding this comment. @brianmcarey This PR changes are already part of my branch. But will check if I need to adjust it. As PR#1209 is quite big digesting it still. There was a problem hiding this comment. We don't require to change this as in #1029 these function calls have not changed. |
||
| if err != nil { | ||
| return err | ||
| } | ||
| err = _cmd(cli, nodeContainer(prefix, nodeName), fmt.Sprintf("if [ -f /scripts/fetch-images.sh ]; then scp -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -i vagrant.key -P 22 /scripts/fetch-images.sh %s@192.168.66.101:/tmp/fetch-images.sh; fi", sshUser), "copying /scripts/fetch-images.sh if existing") | ||
| if err != nil { | ||
| return err | ||
| } | ||
|
|
||
| err = _cmd(cli, nodeContainer(prefix, nodeName), fmt.Sprintf("ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -i vagrant.key %s@192.168.66.101 'mkdir -p /tmp/ceph /tmp/cnao /tmp/nfs-csi /tmp/nodeports /tmp/prometheus /tmp/whereabouts /tmp/kwok'", sshUser), "Create required manifest directories before copy") | ||
| if err != nil { | ||
| return err | ||
| } | ||
| // Copy manifests to the VM | ||
| err = _cmd(cli, nodeContainer(prefix, nodeName), fmt.Sprintf("scp -r -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -i vagrant.key -P 22 /scripts/manifests/* %s@192.168.66.101:/tmp", sshUser), "copying manifests to the VM") | ||
| if err != nil { | ||
| return err | ||
| } | ||
|
|
||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why are you dropping the version here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@brianmcarey I just made the version consistent across this file :)
I saw in a place it is 1.27
So everywhere I changed to 1.28
Let me change all these to 1.30
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
addressed in k8s-1.30-provider-slim-s390x branch