websocat: WebSocketError/unknown port

[dhruvbalwada]

@jbusecke and I have been trying to setup jupyter-sshd-proxy on the LEAP hub.

Following the instructions in the readme, we setup this image:
Repo for new image: https://github.com/leap-stc/test-image-balwada
Image: https://quay.io/repository/leap-stc/test-image-balwada?tab=tags&tag=5b146d660ff2

However, when trying to ssh to this, we get the error:

websocat: WebSocketError: WebSocketError: Redirected (302 Found) to /user/dhruvbalwada/sshd
websocat: error running
Connection closed by UNKNOWN port 65535

To test if there were any problems with our ssh connections, we tested by sshing to hub with test image (quay.io/yuvipanda/pangeo-jupyter-sshd-proxy:latest). This worked and we were able to use terminal to navigate.

Wondering what we might be doing wrong with our own test image?

[yuvipanda]

OOOO sorry for not seeing this before, @dhruvbalwada

can you post full sshd config, ssh command line used?

[dhruvbalwada]

Here is the ssh command line :

When the hub instance was started using my test image

(base) ➜  ~ ssh LEAP-HUB   
websocat: WebSocketError: WebSocketError: Redirected (302 Found) to /user/dhruvbalwada/sshd
websocat: error running
Connection closed by UNKNOWN port 65535

When hub instance was started using your test imaeg

(base) ➜  ~ ssh LEAP-HUB
(notebook) $ 

I sent the sshd config on slack, as I was unsure if it is safe to post it here. Let me know what else you need, or if I didn't send the right things.

[yuvipanda]

I can reproduce it with this image, but I also don't see jupyter-sshd-proxy installed in the image. If I start a server with that image, and do pip list | grep ssh I see nothing. I did a pip list and also found that the sshd-proxy was not there.

Can you try rebuilding the image and see if jupyter-sshd-proxy gets actually installed?

[jbusecke]

Hey folks, so I just looked at the build log and it seems like its getting installed (see here and here) and then pushed to quay.io/leap-stc/test-image-balwada:5b146d660ff2.

I just tested pulling the image and it is not installed! I faintly remember that I had issues with pip installs following this pattern (there must have been a reason I installed them seperately e.g. here but I cant quite recall).
Is maybe CONDA_DIR not set properly?

[jbusecke]

I did the same thing here and that worked!

[jbusecke]

Or do we need to provide NB_PYTHON_PREFIX instead of CONDA_DIR? I have to admit I still do not understand this part of the pangeo docker image magic very well, which is probably at the root of our problem here?

[yuvipanda]

FROM pangeo/pangeo-notebook:2024.07.30

RUN mamba install -c conda-forge openssh
RUN pip install --no-cache jupyter-sshd-proxy

This is my dockerfile, fwiw

[jbusecke]

I think the crux here is that we are using the pangeo-notebook image (which installs into the notebook env). I have convinced myself here and below that we are installing stuff into the base conda env, but we really want it in the notebook one?

Let me test my hypothesis and report back.

[jbusecke]

Yep (i believe) I fixed it in leap-stc/test-image-balwada#4. I can now pull quay.io/leap-stc/test-image-balwada:51661c98ddc8 onto our hub and pip list |grep ssh in the terminal shows:

jupyter-sshd-proxy            0.2.0

@dhruvbalwada could you test with that again?

[dhruvbalwada]

It works now! Thank you both for sorting this out.

[jbusecke]

I have moved ahead and made a PR to the pangeo docker images to get this upstream. EDIT: I cant close. @yuvipanda

[jbusecke]

Hey @yuvipanda I added jupyter-sshd-proxy to the pangeo image and set this new image as default (pangeo/pangeo-notebook:2024.10.11). I am however still running into issues:
I followed the instructions here and this is my redacted local ssh config

Host leap-hub
    User jovyan
    ProxyCommand websocat --binary -H='Authorization: token XXX' asyncstdio: wss://%h/user/jbusecke/sshd/

When I try to run

ssh leap-hub

I get

websocat: WebSocketError: I/O failure
websocat: error running
Connection closed by UNKNOWN port 65535

This is totally out of my wheelhouse unfortunately. If you have a minute could you look into this?

[yuvipanda]

I just dug into this a little, turns out that sshd is not present in the image. I opened pangeo-data/pangeo-docker-images#588 which I think should fix this.

In addition, unless you have aliases set up otherwise, your config should be Host leap.2i2c.cloud rather than Host leap-hub, as the name there is used in the %h in ProxyCommand

[dhruvbalwada]

I just checked on leap.2i2c.cloud and am still getting the same error, but it also seems that is because leap.2i2c.cloud is not inheriting the latest docker image that was generated after pangeo-data/pangeo-docker-images#588. Is that something that will happen automatically after some period?

[jbusecke]

@dhruvbalwada can you manually check with the latest image version [2024.10.26](https://github.com/pangeo-data/pangeo-docker-images/releases/tag/2024.10.26)?

Is that something that will happen automatically after some period?

Nope its manual, happy to submit the update once we confirmed that it works.

[dhruvbalwada]

What will be image name that I put in when starting the hub? The text in gray above returns invalid image.

[jbusecke]

Oh sorry I meant to just just past the tag. You have to enter quay.io/pangeo/pangeo-notebook:2014.10.16 (as described here)

[dhruvbalwada]

The tag should be quay.io/pangeo/pangeo-notebook:2024.10.26

[dhruvbalwada]

Just checked, and this works!!

[jbusecke]

Nice! Ill get the default image changed rn.

[jbusecke]

2i2c-org/infrastructure#5011