Encountering segmentation fault (core dumped)

[AlanWu886]

Hi,

Just want to ask if any have had this issue before while running GPU version of napari-n2v:

(napari_gpu) [mwu56@nodegpu234 ~]$ napari
WARNING: QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/lustre_scratch/scratch_lsf/lsftmpdir/229994719.tmpdir/runtime-mwu56'
10:25:13 WARNING QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/lustre_scratch/scratch_lsf/lsftmpdir/229994719.tmpdir/runtime-mwu56'
WARNING: 'xrandr' output has screen dimension of 0mm, can't compute proper DPI
10:25:14 WARNING 'xrandr' output has screen dimension of 0mm, can't compute proper DPI
2024-06-12 10:25:31.182930: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-06-12 10:25:41.677343: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
/research/rgs01/home/clusterHome/mwu56/n2v_test/Training_set
WARNING: QXcbConnection: XCB error: 3 (BadWindow), sequence: 1602, resource id: 12648351, major code: 40 (TranslateCoords), minor code: 0
10:26:14 WARNING QXcbConnection: XCB error: 3 (BadWindow), sequence: 1602, resource id: 12648351, major code: 40 (TranslateCoords), minor code: 0
2024-06-12 10:26:26.553515: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 79089 MB memory:  -> device: 0, name: NVIDIA A100-SXM4-80GB, pci bus id: 0000:07:00.0, compute capability: 8.0
The input image is of type uint16 and will be casted to float32 for prediction.
Segmentation fault (core dumped)

Below is how I installed tensorflow and cuda:

conda install -c conda-forge cudatoolkit=11.8.0
python3 -m pip install nvidia-cudnn-cu11==8.6.0.163 tensorflow==2.13.*
mkdir -p $CONDA_PREFIX/etc/conda/activate.d
echo 'CUDNN_PATH=$(dirname $(python -c "import nvidia.cudnn;print(nvidia.cudnn.__file__)"))' >> $CONDA_PREFIX/etc/conda/activate.d/env_vars.sh
echo 'export LD_LIBRARY_PATH=$CUDNN_PATH/lib:$CONDA_PREFIX/lib/:$LD_LIBRARY_PATH' >> $CONDA_PREFIX/etc/conda/activate.d/env_vars.sh
source $CONDA_PREFIX/etc/conda/activate.d/env_vars.sh

And here is the list of dependencies:

Package                       Version
----------------------------- ------------
absl-py                       2.1.0
aiobotocore                   2.13.0
aiohttp                       3.9.5
aioitertools                  0.11.0
aiosignal                     1.3.1
alabaster                     0.7.16
annotated-types               0.7.0
app-model                     0.2.7
appdirs                       1.4.4
asciitree                     0.3.3
asttokens                     2.4.1
astunparse                    1.6.3
async-timeout                 4.0.3
attrs                         23.2.0
autopep8                      2.2.0
Babel                         2.15.0
bioimageio.core               0.5.11
bioimageio.spec               0.4.9.post5
bioio                         1.0.1
bioio-base                    1.0.0
botocore                      1.34.106
build                         1.2.1
cachetools                    5.3.3
cachey                        0.2.1
cellpose                      3.0.9
cellpose-napari               0.1.5
certifi                       2024.2.2
charset-normalizer            3.3.2
click                         8.1.7
cloudpickle                   3.0.0
comm                          0.2.2
contourpy                     1.2.1
csbdeep                       0.7.4
cycler                        0.12.1
dask                          2024.5.1
debugpy                       1.8.1
decorator                     5.1.1
distributed                   2024.5.1
docstring_parser              0.16
docutils                      0.21.2
exceptiongroup                1.2.1
executing                     2.0.1
fasteners                     0.19
fastremap                     1.14.1
filelock                      3.14.0
flatbuffers                   24.3.25
flexcache                     0.3
flexparser                    0.3.1
fonttools                     4.53.0
freetype-py                   2.4.0
frozenlist                    1.4.1
fsspec                        2024.5.0
gast                          0.4.0
google-auth                   2.30.0
google-auth-oauthlib          1.0.0
google-pasta                  0.2.0
grpcio                        1.64.1
h5py                          3.11.0
HeapDict                      1.0.1
hsluv                         5.0.4
idna                          3.7
imagecodecs                   2024.1.1
imageio                       2.27.0
imageio-ffmpeg                0.5.0
imagesize                     1.4.1
importlib_metadata            7.1.0
in-n-out                      0.2.1
ipykernel                     6.29.4
ipython                       8.25.0
jedi                          0.19.1
Jinja2                        3.1.4
jmespath                      1.0.1
jsonschema                    4.22.0
jsonschema-specifications     2023.12.1
jupyter_client                8.6.2
jupyter_core                  5.7.2
keras                         2.13.1
kiwisolver                    1.4.5
lazy_loader                   0.4
libclang                      18.1.1
llvmlite                      0.42.0
locket                        1.0.0
lxml                          5.2.2
lxml_html_clean               0.1.1
magicgui                      0.8.2
Markdown                      3.6
markdown-it-py                3.0.0
MarkupSafe                    2.1.5
marshmallow                   3.21.3
marshmallow-jsonschema        0.13.0
marshmallow-union             0.1.15.post1
matplotlib                    3.9.0
matplotlib-inline             0.1.7
mdurl                         0.1.2
mpmath                        1.3.0
msgpack                       1.0.8
multidict                     6.0.5
n2v                           0.3.3
napari                        0.4.19.post1
napari-console                0.0.9
napari-n2v                    0.1.1
napari-plugin-engine          0.2.0
napari-plugin-manager         0.1.0a2
napari-svg                    0.1.10
napari-time-slicer            0.5.0
napari-tools-menu             0.1.19
napari-workflows              0.2.10
natsort                       8.4.0
nest-asyncio                  1.6.0
networkx                      3.3
npe2                          0.7.5
numba                         0.59.1
numcodecs                     0.12.1
numpy                         1.26.4
numpydoc                      1.7.0
nvidia-cublas-cu11            2022.4.8
nvidia-cublas-cu117           11.10.1.25
nvidia-cublas-cu12            12.1.3.1
nvidia-cuda-cupti-cu12        12.1.105
nvidia-cuda-nvrtc-cu12        12.1.105
nvidia-cuda-runtime-cu12      12.1.105
nvidia-cudnn-cu11             8.6.0.163
nvidia-cudnn-cu12             8.9.2.26
nvidia-cufft-cu12             11.0.2.54
nvidia-curand-cu12            10.3.2.106
nvidia-cusolver-cu12          11.4.5.107
nvidia-cusparse-cu12          12.1.0.106
nvidia-nccl-cu12              2.20.5
nvidia-nvjitlink-cu12         12.5.40
nvidia-nvtx-cu12              12.1.105
nvidia-pyindex                1.0.9
oauthlib                      3.2.2
ome-types                     0.5.1.post1
ome-zarr                      0.9.0
opencv-python-headless        4.10.0.82
opt-einsum                    3.3.0
packaging                     24.0
pandas                        2.2.2
parso                         0.8.4
partd                         1.4.2
pexpect                       4.9.0
pillow                        10.3.0
Pint                          0.23
pip                           24.0
platformdirs                  4.2.2
polars                        0.20.30
pooch                         1.8.1
prompt_toolkit                3.0.45
protobuf                      4.25.3
psutil                        5.9.8
psygnal                       0.11.1
ptyprocess                    0.7.0
pure-eval                     0.2.2
pyasn1                        0.6.0
pyasn1_modules                0.4.0
pycodestyle                   2.11.1
pyconify                      0.1.6
pydantic                      2.7.2
pydantic-compat               0.1.2
pydantic_core                 2.18.3
Pygments                      2.18.0
PyOpenGL                      3.1.7
pyparsing                     3.1.2
pyproject_hooks               1.1.0
PyQt5                         5.15.10
PyQt5-Qt5                     5.15.2
PyQt5-sip                     12.13.0
pyqtgraph                     0.13.7
python-dateutil               2.9.0.post0
pytz                          2024.1
PyYAML                        6.0.1
pyzmq                         26.0.3
qtconsole                     5.5.2
QtPy                          2.4.1
readlif                       0.6.6
referencing                   0.35.1
requests                      2.32.3
requests-oauthlib             2.0.0
ric                           0.1.2
rich                          13.7.1
roifile                       2024.5.24
rpds-py                       0.18.1
rsa                           4.9
ruamel.yaml                   0.18.6
ruamel.yaml.clib              0.2.8
s3fs                          2024.5.0
scikit-image                  0.22.0
scipy                         1.13.1
semver                        3.0.2
setuptools                    70.0.0
shellingham                   1.5.4
six                           1.16.0
snowballstemmer               2.2.0
sortedcontainers              2.4.0
Sphinx                        7.3.7
sphinxcontrib-applehelp       1.0.8
sphinxcontrib-devhelp         1.0.6
sphinxcontrib-htmlhelp        2.0.5
sphinxcontrib-jsmath          1.0.1
sphinxcontrib-qthelp          1.0.7
sphinxcontrib-serializinghtml 1.1.10
stack-data                    0.6.3
superqt                       0.6.6
sympy                         1.12.1
tabulate                      0.9.0
tblib                         3.0.0
tensorboard                   2.13.0
tensorboard-data-server       0.7.2
tensorflow                    2.13.1
tensorflow-estimator          2.13.0
tensorflow-io-gcs-filesystem  0.37.0
termcolor                     2.4.0
tifffile                      2024.5.22
tomli                         2.0.1
tomli_w                       1.0.0
toolz                         0.12.1
torch                         2.3.1
tornado                       6.4
tqdm                          4.66.4
traitlets                     5.14.3
triangle                      20230923
triton                        2.3.1
typer                         0.12.3
typing_extensions             4.12.0
tzdata                        2024.1
urllib3                       2.2.1
vispy                         0.14.2
wcwidth                       0.2.13
Werkzeug                      3.0.3
wheel                         0.43.0
wrapt                         1.16.0
x2tif                         0.1.1
xarray                        2024.5.0
xsdata                        24.3.1
yarl                          1.9.4
zarr                          2.18.2
zict                          3.0.0
zipp                          3.19.1

Thanks for the help!
Alan