Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error during Inference. ValueError: Found Multiple nodes named (name). #1918

Open
1 of 4 tasks
hemanyaradadia opened this issue Aug 26, 2024 · 7 comments
Open
1 of 4 tasks
Labels
bug Something isn't working fixed in future release Fix or feature is merged into develop and will be available in future release.

Comments

@hemanyaradadia
Copy link

Bug description

Running Inference and Tracking throws up 'ValueError: Found multiple nodes named (name).' on a Linux system. While running the same command on a Windows 10 system, it works without any error.

Expected behaviour

A .slp output file with predictions/track in each frame.

Actual behaviour

The error shows up in the terminal. While .slp file is generated but without any predictions or frames. Essentially an empty .slp project file.

Your personal set up

  • OS: Proxmox VE 8.1.4
  • Version(s): SLEAP 1.3.3 Python 3.7.12
Environment packages
# packages in environment at /mnt/pve/Homes/hemanya/miniconda3/envs/sleap-env:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
absl-py                   1.0.0                    pypi_0    pypi
aiofiles                  22.1.0             pyhd8ed1ab_0    conda-forge
aiosqlite                 0.20.0             pyhd8ed1ab_0    conda-forge
alsa-lib                  1.2.3.2              h166bdaf_0    conda-forge
anyio                     3.7.1              pyhd8ed1ab_0    conda-forge
argon2-cffi               23.1.0             pyhd8ed1ab_0    conda-forge
argon2-cffi-bindings      21.2.0           py37h540881e_2    conda-forge
arrow                     1.2.3              pyhd8ed1ab_0    conda-forge
astunparse                1.6.3                    pypi_0    pypi
attrs                     21.4.0             pyhd8ed1ab_0    conda-forge
babel                     2.14.0             pyhd8ed1ab_0    conda-forge
backcall                  0.2.0              pyh9f0ad1d_0    conda-forge
backports                 1.0                pyhd8ed1ab_4    conda-forge
backports-zoneinfo        0.2.1                    pypi_0    pypi
backports.functools_lru_cache 2.0.0              pyhd8ed1ab_0    conda-forge
beautifulsoup4            4.12.3             pyha770c72_0    conda-forge
blas                      1.1                    openblas    conda-forge
bleach                    6.1.0              pyhd8ed1ab_0    conda-forge
brotli                    1.1.0                hd590300_1    conda-forge
brotli-bin                1.1.0                hd590300_1    conda-forge
brotli-python             1.0.9            py37hd23a5d3_7    conda-forge
bzip2                     1.0.8                h4bc722e_7    conda-forge
c-ares                    1.32.3               h4bc722e_0    conda-forge
ca-certificates           2024.7.4             hbcca054_0    conda-forge
cached-property           1.5.2                hd8ed1ab_1    conda-forge
cached_property           1.5.2              pyha770c72_1    conda-forge
cachetools                4.2.4                    pypi_0    pypi
cairo                     1.16.0            h6cf1ce9_1008    conda-forge
cattrs                    1.1.1              pyhd8ed1ab_0    conda-forge
certifi                   2024.7.4           pyhd8ed1ab_0    conda-forge
cffi                      1.15.1           py37h43b0acd_1    conda-forge
charset-normalizer        2.0.9                    pypi_0    pypi
cloudpickle               2.2.1              pyhd8ed1ab_0    conda-forge
cuda-nvcc                 11.3.58              h2467b9f_0    nvidia
cudatoolkit               11.3.1              hb98b00a_13    conda-forge
cudnn                     8.2.1.32             h86fa8c9_0    conda-forge
cycler                    0.11.0             pyhd8ed1ab_0    conda-forge
cytoolz                   0.12.0           py37h540881e_0    conda-forge
dask-core                 2022.2.0           pyhd8ed1ab_0    conda-forge
dbus                      1.13.6               h5008d03_3    conda-forge
debugpy                   1.6.3            py37hd23a5d3_0    conda-forge
decorator                 5.1.1              pyhd8ed1ab_0    conda-forge
defusedxml                0.7.1              pyhd8ed1ab_0    conda-forge
efficientnet              1.0.0                    pypi_0    pypi
entrypoints               0.4                pyhd8ed1ab_0    conda-forge
exceptiongroup            1.2.2              pyhd8ed1ab_0    conda-forge
expat                     2.6.2                h59595ed_0    conda-forge
ffmpeg                    4.3.2                h37c90e5_3    conda-forge
fftw                      3.3.10          nompi_hf1063bd_110    conda-forge
flatbuffers               2.0                      pypi_0    pypi
fontconfig                2.14.2               h14ed4e7_0    conda-forge
fonttools                 4.38.0           py37h540881e_0    conda-forge
fqdn                      1.5.1              pyhd8ed1ab_0    conda-forge
freetype                  2.12.1               h267a509_2    conda-forge
fsspec                    2023.1.0           pyhd8ed1ab_0    conda-forge
gast                      0.4.0                    pypi_0    pypi
geos                      3.11.0               h27087fc_0    conda-forge
gettext                   0.22.5               h59595ed_2    conda-forge
gettext-tools             0.22.5               h59595ed_2    conda-forge
gmp                       6.3.0                hac33072_2    conda-forge
gnutls                    3.6.13               h85f3911_1    conda-forge
google-auth               2.3.3                    pypi_0    pypi
google-auth-oauthlib      0.4.6                    pypi_0    pypi
google-pasta              0.2.0                    pypi_0    pypi
graphite2                 1.3.13            h59595ed_1003    conda-forge
grpcio                    1.43.0                   pypi_0    pypi
gst-plugins-base          1.18.5               hf529b03_3    conda-forge
gstreamer                 1.18.5               h9f60fe5_3    conda-forge
h5py                      3.1.0           nompi_py37h1e651dc_100    conda-forge
harfbuzz                  2.9.1                h83ec7ef_1    conda-forge
hdf5                      1.10.6          nompi_h6a2412b_1114    conda-forge
icu                       68.2                 h9c3ff4c_0    conda-forge
idna                      3.3                      pypi_0    pypi
image-classifiers         1.0.0                    pypi_0    pypi
imagecodecs-lite          2019.12.3        py37hc105733_5    conda-forge
imageio                   2.34.2             pyh12aca89_0    conda-forge
imgaug                    0.4.0              pyhd8ed1ab_1    conda-forge
imgstore                  0.2.9                    pypi_0    pypi
importlib-metadata        4.2.0                    pypi_0    pypi
importlib-resources       5.12.0                   pypi_0    pypi
importlib_metadata        4.11.4               hd8ed1ab_0    conda-forge
importlib_resources       6.0.0              pyhd8ed1ab_0    conda-forge
ipykernel                 6.16.2             pyh210e3f2_0    conda-forge
ipython                   7.33.0           py37h89c1867_0    conda-forge
ipython_genutils          0.2.0              pyhd8ed1ab_1    conda-forge
isoduration               20.11.0            pyhd8ed1ab_0    conda-forge
jasper                    1.900.1           h07fcdf6_1006    conda-forge
jedi                      0.19.1             pyhd8ed1ab_0    conda-forge
jinja2                    3.1.4              pyhd8ed1ab_0    conda-forge
joblib                    1.3.2              pyhd8ed1ab_0    conda-forge
jpeg                      9e                   h0b41bf4_3    conda-forge
jsmin                     3.0.1              pyhd8ed1ab_0    conda-forge
json5                     0.9.25             pyhd8ed1ab_0    conda-forge
jsonpickle                1.2                        py_0    conda-forge
jsonpointer               2.0                        py_0    conda-forge
jsonschema                4.17.3             pyhd8ed1ab_0    conda-forge
jsonschema-with-format-nongpl 4.17.3             pyhd8ed1ab_0    conda-forge
jupyter_client            7.4.9              pyhd8ed1ab_0    conda-forge
jupyter_core              4.11.1           py37h89c1867_0    conda-forge
jupyter_events            0.6.3              pyhd8ed1ab_1    conda-forge
jupyter_server            1.23.4             pyhd8ed1ab_0    conda-forge
jupyter_server_fileid     0.9.2              pyhd8ed1ab_0    conda-forge
jupyter_server_ydoc       0.8.0              pyhd8ed1ab_0    conda-forge
jupyter_ydoc              0.2.4              pyhd8ed1ab_0    conda-forge
jupyterlab                3.6.7              pyhd8ed1ab_0    conda-forge
jupyterlab_pygments       0.3.0              pyhd8ed1ab_0    conda-forge
jupyterlab_server         2.24.0             pyhd8ed1ab_0    conda-forge
keras                     2.7.0                    pypi_0    pypi
keras-applications        1.0.8                    pypi_0    pypi
keras-preprocessing       1.1.2                    pypi_0    pypi
keyutils                  1.6.1                h166bdaf_0    conda-forge
kiwisolver                1.4.4            py37h7cecad7_0    conda-forge
krb5                      1.19.3               h3790be6_0    conda-forge
lame                      3.100             h166bdaf_1003    conda-forge
lcms2                     2.14                 h6ed2654_0    conda-forge
ld_impl_linux-64          2.40                 hf3520f5_7    conda-forge
lerc                      4.0.0                h27087fc_0    conda-forge
libasprintf               0.22.5               h661eb56_2    conda-forge
libasprintf-devel         0.22.5               h661eb56_2    conda-forge
libblas                   3.9.0           23_linux64_openblas    conda-forge
libbrotlicommon           1.1.0                hd590300_1    conda-forge
libbrotlidec              1.1.0                hd590300_1    conda-forge
libbrotlienc              1.1.0                hd590300_1    conda-forge
libcblas                  3.9.0           23_linux64_openblas    conda-forge
libclang                  12.0.0                   pypi_0    pypi
libcurl                   7.86.0               h7bff187_1    conda-forge
libdeflate                1.14                 h166bdaf_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 hd590300_2    conda-forge
libevent                  2.1.10               h9b69904_4    conda-forge
libexpat                  2.6.2                h59595ed_0    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 14.1.0               h77fa898_0    conda-forge
libgettextpo              0.22.5               h59595ed_2    conda-forge
libgettextpo-devel        0.22.5               h59595ed_2    conda-forge
libgfortran-ng            14.1.0               h69a702a_0    conda-forge
libgfortran5              14.1.0               hc5f4f2c_0    conda-forge
libglib                   2.80.2               hf974151_0    conda-forge
libgomp                   14.1.0               h77fa898_0    conda-forge
libiconv                  1.17                 hd590300_2    conda-forge
liblapack                 3.9.0           23_linux64_openblas    conda-forge
liblapacke                3.9.0           23_linux64_openblas    conda-forge
libllvm11                 11.1.0               he0ac6c6_5    conda-forge
libnghttp2                1.51.0               hdcd2b5c_0    conda-forge
libnsl                    2.0.1                hd590300_0    conda-forge
libogg                    1.3.5                h4ab18f5_0    conda-forge
libopenblas               0.3.27          pthreads_hac2b453_1    conda-forge
libopencv                 4.5.3            py37h25009ff_1    conda-forge
libopus                   1.3.1                h7f98852_1    conda-forge
libpng                    1.6.43               h2797004_0    conda-forge
libpq                     13.8                 hd77ab85_0    conda-forge
libprotobuf               3.16.0               h780b84a_0    conda-forge
libsodium                 1.0.18               h36c2ea0_1    conda-forge
libsqlite                 3.46.0               hde9e2c9_0    conda-forge
libssh2                   1.10.0               haa6b8db_3    conda-forge
libstdcxx-ng              14.1.0               hc0a3c3a_0    conda-forge
libtiff                   4.4.0                h82bc61c_5    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libvorbis                 1.3.7                h9c3ff4c_0    conda-forge
libwebp-base              1.4.0                hd590300_0    conda-forge
libxcb                    1.13              h7f98852_1004    conda-forge
libxkbcommon              1.0.3                he3ba5ed_0    conda-forge
libxml2                   2.9.12               h72842e0_0    conda-forge
libxslt                   1.1.33               h15afd5d_2    conda-forge
libzlib                   1.2.13               h4ab18f5_6    conda-forge
locket                    1.0.0              pyhd8ed1ab_0    conda-forge
markdown                  3.3.6                    pypi_0    pypi
markdown-it-py            2.2.0              pyhd8ed1ab_0    conda-forge
markupsafe                2.1.1            py37h540881e_1    conda-forge
matplotlib-base           3.5.3            py37hf395dca_2    conda-forge
matplotlib-inline         0.1.7              pyhd8ed1ab_0    conda-forge
mdurl                     0.1.2              pyhd8ed1ab_0    conda-forge
mistune                   3.0.2              pyhd8ed1ab_0    conda-forge
munkres                   1.1.4              pyh9f0ad1d_0    conda-forge
mysql-common              8.0.32               h14678bc_0    conda-forge
mysql-libs                8.0.32               h54cf53e_0    conda-forge
nbclassic                 1.1.0              pyhd8ed1ab_0    conda-forge
nbclient                  0.7.0              pyhd8ed1ab_0    conda-forge
nbconvert-core            7.6.0              pyhd8ed1ab_0    conda-forge
nbformat                  5.8.0              pyhd8ed1ab_0    conda-forge
ncurses                   6.5                  h59595ed_0    conda-forge
ndx-pose                  0.1.1                    pypi_0    pypi
nest-asyncio              1.6.0              pyhd8ed1ab_0    conda-forge
nettle                    3.6                  he412f7d_0    conda-forge
networkx                  2.7                pyhd8ed1ab_0    conda-forge
nixio                     1.5.3                    pypi_0    pypi
notebook                  6.5.7              pyha770c72_0    conda-forge
notebook-shim             0.2.4              pyhd8ed1ab_0    conda-forge
nspr                      4.35                 h27087fc_0    conda-forge
nss                       3.100                hca3bf56_0    conda-forge
numpy                     1.19.5                   pypi_0    pypi
oauthlib                  3.1.1                    pypi_0    pypi
openblas                  0.3.27          pthreads_h9eca1d5_1    conda-forge
opencv                    4.5.3            py37h89c1867_1    conda-forge
opencv-python-headless    4.2.0.34                 pypi_0    pypi
openh264                  2.1.1                h780b84a_0    conda-forge
openjpeg                  2.5.0                h7d73246_1    conda-forge
openssl                   1.1.1w               h7f8727e_0
opt-einsum                3.3.0                    pypi_0    pypi
packaging                 21.3                     pypi_0    pypi
pandas                    1.3.5            py37he8f5f7f_0    conda-forge
pandocfilters             1.5.0              pyhd8ed1ab_0    conda-forge
parso                     0.8.4              pyhd8ed1ab_0    conda-forge
partd                     1.4.1              pyhd8ed1ab_0    conda-forge
patsy                     0.5.6              pyhd8ed1ab_0    conda-forge
pcre2                     10.43                hcad00b1_0    conda-forge
pexpect                   4.9.0              pyhd8ed1ab_0    conda-forge
pickleshare               0.7.5                   py_1003    conda-forge
pillow                    9.2.0            py37h850a105_2    conda-forge
pip                       24.0               pyhd8ed1ab_0    conda-forge
pixman                    0.43.2               h59595ed_0    conda-forge
pkgutil-resolve-name      1.3.10             pyhd8ed1ab_1    conda-forge
prometheus_client         0.17.1             pyhd8ed1ab_0    conda-forge
prompt-toolkit            3.0.47             pyha770c72_0    conda-forge
protobuf                  3.19.1                   pypi_0    pypi
psutil                    5.9.3            py37h540881e_0    conda-forge
pthread-stubs             0.4               h36c2ea0_1001    conda-forge
ptyprocess                0.7.0              pyhd3deb0d_0    conda-forge
py-opencv                 4.5.3            py37h6531663_1    conda-forge
pyasn1                    0.4.8                    pypi_0    pypi
pyasn1-modules            0.2.8                    pypi_0    pypi
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pygments                  2.17.2             pyhd8ed1ab_0    conda-forge
pykalman                  0.9.7              pyhd8ed1ab_0    conda-forge
pynwb                     2.3.3                    pypi_0    pypi
pyparsing                 3.0.6                    pypi_0    pypi
pyrsistent                0.19.3                   pypi_0    pypi
pyside2                   5.13.2           py37hfa98aef_7    conda-forge
pysocks                   1.7.1            py37h89c1867_5    conda-forge
python                    3.7.12          hb7a2778_100_cpython    conda-forge
python-dateutil           2.9.0              pyhd8ed1ab_0    conda-forge
python-fastjsonschema     2.20.0             pyhd8ed1ab_0    conda-forge
python-json-logger        2.0.7              pyhd8ed1ab_0    conda-forge
python-rapidjson          1.9              py37hd23a5d3_0    conda-forge
python_abi                3.7                     4_cp37m    conda-forge
pytz                      2024.1             pyhd8ed1ab_0    conda-forge
pywavelets                1.3.0            py37hda87dfa_1    conda-forge
pyyaml                    6.0              py37h540881e_4    conda-forge
pyzmq                     24.0.1           py37h0c0c2a8_0    conda-forge
qimage2ndarray            1.10.0                   pypi_0    pypi
qt                        5.12.9               hda022c4_4    conda-forge
qtpy                      2.4.1              pyhd8ed1ab_0    conda-forge
readline                  8.2                  h8228510_1    conda-forge
requests                  2.26.0                   pypi_0    pypi
requests-oauthlib         1.3.0                    pypi_0    pypi
rfc3339-validator         0.1.4              pyhd8ed1ab_0    conda-forge
rfc3986-validator         0.1.1              pyh9f0ad1d_0    conda-forge
rich                      13.7.1             pyhd8ed1ab_0    conda-forge
ruamel-yaml               0.17.32                  pypi_0    pypi
ruamel-yaml-clib          0.2.7                    pypi_0    pypi
scikit-image              0.19.2           py37he8f5f7f_0    conda-forge
scikit-learn              1.0              py37hf0f1638_1    conda-forge
scikit-video              1.1.11             pyh24bf2e0_0    conda-forge
scipy                     1.7.3            py37hf838250_2    anaconda
seaborn                   0.12.2               hd8ed1ab_0    conda-forge
seaborn-base              0.12.2             pyhd8ed1ab_0    conda-forge
segmentation-models       1.0.1                    pypi_0    pypi
send2trash                1.8.3              pyh0d859eb_0    conda-forge
setuptools                59.8.0           py37h89c1867_1    conda-forge
setuptools-scm            6.3.2                    pypi_0    pypi
shapely                   1.8.5            py37ha4e3bd1_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
sleap                     1.3.3                    pypi_0    pypi
sniffio                   1.3.1              pyhd8ed1ab_0    conda-forge
soupsieve                 2.3.2.post1        pyhd8ed1ab_0    conda-forge
sqlite                    3.46.0               h6d4b2fc_0    conda-forge
statsmodels               0.13.2           py37hda87dfa_0    conda-forge
tensorboard               2.7.0                    pypi_0    pypi
tensorboard-data-server   0.6.1                    pypi_0    pypi
tensorboard-plugin-wit    1.8.0                    pypi_0    pypi
tensorflow                2.7.0                    pypi_0    pypi
tensorflow-estimator      2.7.0                    pypi_0    pypi
tensorflow-hub            0.13.0             pyh56297ac_0    conda-forge
tensorflow-io-gcs-filesystem 0.23.1                   pypi_0    pypi
termcolor..................................................      
Logs CLI command and resulting output

sleap-track /mnt/pve/Homes/xxxxxx/xxxxx/NAS/xxxxx/sleap-xxx/xxxx/xxxx/20240521-v90-82836.mp4 --max_instances 1 --tracking.tracker simplemaxtracks --tracking.max_tracking true --tracking.max_tracks 1 --tracking.target_instance_count 1 --tracking.post_connect_single_breaks 1 --tracking.match hungarian --tracking.track_window 5 --tracking.similarity instance --no-empty-frames -o 240826_model_20240521-v90-82836.predictions.slp -m /mnt/pve/Homes/xxxx/xxxx/xxx-SLEAP/models/240826_120107.single_instance.n=46

Started inference at: 2024-08-26 17:59:09.095691
Args:
{
│   'data_path': '/mnt/pve/Homes/xxxxxx/xxxxx/NAS/xxxxx/sleap-xxx/xxxx/xxxx/20240521-v90-82836.mp4',
│   'models': ['/mnt/pve/Homes/xxxxx/xxxx/rishika-SLEAP/models/240826_120107.single_instance.n=46'],
│   'frames': '',
│   'only_labeled_frames': False,
│   'only_suggested_frames': False,
│   'output': '240826_model_20240521-v90-82836.predictions.slp',
│   'no_empty_frames': True,
│   'verbosity': 'rich',
│   'video.dataset': None,
│   'video.input_format': 'channels_last',
│   'video.index': '',
│   'cpu': False,
│   'first_gpu': False,
│   'last_gpu': False,
│   'gpu': 'auto',
│   'max_edge_length_ratio': 0.25,
│   'dist_penalty_weight': 1.0,
│   'batch_size': 4,
│   'open_in_gui': False,
│   'peak_threshold': 0.2,
│   'max_instances': 1,
│   'tracking.tracker': 'simplemaxtracks',
│   'tracking.max_tracking': True,
│   'tracking.max_tracks': 1,
│   'tracking.target_instance_count': 1,
│   'tracking.pre_cull_to_target': None,
│   'tracking.pre_cull_iou_threshold': None,
│   'tracking.post_connect_single_breaks': 1,
│   'tracking.clean_instance_count': None,
│   'tracking.clean_iou_threshold': None,
│   'tracking.similarity': 'instance',
│   'tracking.match': 'hungarian',
│   'tracking.robust': None,
│   'tracking.track_window': 5,
│   'tracking.min_new_track_points': None,
│   'tracking.min_match_points': None,
│   'tracking.img_scale': None,
│   'tracking.of_window_size': None,
│   'tracking.of_max_levels': None,
│   'tracking.save_shifted_instances': None,
│   'tracking.kf_node_indices': None,
│   'tracking.kf_init_frame_count': None
}

2024-08-26 17:59:09.123480: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2024-08-26 17:59:09.135030: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2024-08-26 17:59:09.135353: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
INFO:sleap.nn.inference:Auto-selected GPU 0 with 11979 MiB of free memory.
Versions:
SLEAP: 1.3.3
TensorFlow: 2.7.0
Numpy: 1.19.5
Python: 3.7.12
OS: Linux-6.5.13-1-pve-x86_64-with-debian-12.5

System:
GPUs: 1/1 available
  Device: /physical_device:GPU:0
         Available: True
        Initalized: False
     Memory growth: True

Video: /mnt/pve/Homes/xxxxxx/xxxxx/NAS/xxxxx/sleap-xxx/xxxx/xxxx/20240521-v90-82836.mp4
2024-08-26 17:59:09.241083: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-08-26 17:59:09.250593: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2024-08-26 17:59:09.250943: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2024-08-26 17:59:09.251199: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2024-08-26 17:59:09.882394: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2024-08-26 17:59:09.882727: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2024-08-26 17:59:09.882987: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2024-08-26 17:59:09.883255: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 10090 MB memory:  -> device: 0, name: NVIDIA GeForce RTX 4070 Ti, pci bus id: 0000:c2:00.0, compute capability: 8.9
Predicting... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% ETA: -:--:-- ?2024-08-26 17:59:13.940530: I tensorflow/stream_executor/cuda/cuda_dnn.cc:366] Loaded cuDNN version 8201
2024-08-26 17:59:15.104484: W tensorflow/stream_executor/gpu/asm_compiler.cc:230] Falling back to the CUDA driver for PTX compilation; ptxas does not support CC 8.9
2024-08-26 17:59:15.104544: W tensorflow/stream_executor/gpu/asm_compiler.cc:233] Used ptxas at ptxas
2024-08-26 17:59:15.104815: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] UNIMPLEMENTED: ptxas ptxas too old. Falling back to the driver to compile.
Relying on driver to perform ptx compilation.
Modify $PATH to customize ptxas location.
This message will be only logged once.
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/mnt/pve/Homes/hemanya/miniconda3/envs/sleap-env/lib/python3.7/threading.py", line 926, in _bootstrap_inner
    self.run()
  File "/mnt/pve/Homes/hemanya/miniconda3/envs/sleap-env/lib/python3.7/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/mnt/pve/Homes/hemanya/miniconda3/envs/sleap-env/lib/python3.7/site-packages/sleap/nn/inference.py", line 1564,
in _object_builder
    skeleton=skeleton,
  File "/mnt/pve/Homes/hemanya/miniconda3/envs/sleap-env/lib/python3.7/site-packages/sleap/instance.py", line 1176, in
from_numpy
    points, point_confidences, instance_score, skeleton, track=track
  File "/mnt/pve/Homes/hemanya/miniconda3/envs/sleap-env/lib/python3.7/site-packages/sleap/instance.py", line 1116, in
from_arrays
    track=track,
  File "<attrs generated init sleap.instance.PredictedInstance>", line 13, in __init__
    self.__attrs_post_init__()
  File "/mnt/pve/Homes/hemanya/miniconda3/envs/sleap-env/lib/python3.7/site-packages/sleap/instance.py", line 1011, in
__attrs_post_init__
    super(PredictedInstance, self).__attrs_post_init__()
  File "/mnt/pve/Homes/hemanya/miniconda3/envs/sleap-env/lib/python3.7/site-packages/sleap/instance.py", line 456, in
__attrs_post_init__
    Instance._points_dict_to_array(self._points, parray, self.skeleton)
  File "/mnt/pve/Homes/hemanya/miniconda3/envs/sleap-env/lib/python3.7/site-packages/sleap/instance.py", line 491, in
_points_dict_to_array
    points = {skeleton.find_node(name): point for name, point in points.items()}
  File "/mnt/pve/Homes/hemanya/miniconda3/envs/sleap-env/lib/python3.7/site-packages/sleap/instance.py", line 491, in
<dictcomp>
    points = {skeleton.find_node(name): point for name, point in points.items()}
  File "/mnt/pve/Homes/hemanya/miniconda3/envs/sleap-env/lib/python3.7/site-packages/sleap/skeleton.py", line 573, in
find_node
    raise ValueError("Found multiple nodes named ({}).".format(name))
ValueError: Found multiple nodes named (name).

Predicting... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% ETA: 0:00:00 108.6 FPS
Finished inference at: 2024-08-26 17:59:20.006508
Total runtime: 10.910846710205078 secs
Predicted frames: 0/372
Provenance:
{
│   'model_paths': [
│   │   '/mnt/pve/Homes/xxxx/xxxx/xxxxxx-SLEAP/models/240826_120107.single_instance.n=46/training_config.json'
│   ],
│   'predictor': 'SingleInstancePredictor',
│   'sleap_version': '1.3.3',
│   'platform': 'Linux-6.5.13-1-pve-x86_64-with-debian-12.5',
│   'command': '/mnt/pve/Homes/hemanya/miniconda3/envs/sleap-env/bin/sleap-track /mnt/pve/Homes/xxxxxx/xxxxx/NAS/xxxxx/sleap-xxx/xxxx/xxxx/20240521-v90-82836.mp4 --max_instances 1 --tracking.tracker simplemaxtracks --tracking.max_tracking true --tracking.max_tracks 1 --tracking.target_instance_count 1 --tracking.post_connect_single_breaks 1 --tracking.match hungarian --tracking.track_window 5 --tracking.similarity instance --no-empty-frames -o 240826_model_20240521-v90-82836.predictions.slp -m /mnt/pve/Homes/hemanya/HemData/rishika-SLEAP/models/240826_120107.single_instance.n=46',
│   'data_path': '/mnt/pve/Homes/xxxxxx/xxxxx/NAS/xxxxx/sleap-xxx/xxxx/xxxx/20240521-v90-82836.mp4',
│   'output_path': '240826_model_20240521-v90-82836.predictions.slp',
│   'total_elapsed': 10.910846710205078,
│   'start_timestamp': '2024-08-26 17:59:09.095691',
│   'finish_timestamp': '2024-08-26 17:59:20.006508'
}

Saved output: 240826_model_20240521-v90-82836.predictions.slp
Skeleton .json file used
{"directed": true, "graph": {"name": "Skeleton-3", "num_edges_inserted": 5}, "links": [{"edge_insert_idx": 0, "key": 0, "source": {"py/object": "sleap.skeleton.Node", "py/state": {"name": "nose", "weight": 1.0}}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"name": "right-ear", "weight": 1.0}}, "type": {"py/reduce": [{"py/type": "sleap.skeleton.EdgeType"}, {"py/tuple": [1]}]}}, {"edge_insert_idx": 1, "key": 0, "source": {"py/id": 1}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"name": "left-ear", "weight": 1.0}}, "type": {"py/id": 3}}, {"edge_insert_idx": 2, "key": 0, "source": {"py/id": 2}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"name": "thorax", "weight": 1.0}}, "type": {"py/id": 3}}, {"edge_insert_idx": 3, "key": 0, "source": {"py/id": 4}, "target": {"py/id": 5}, "type": {"py/id": 3}}, {"edge_insert_idx": 4, "key": 0, "source": {"py/id": 5}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"name": "tail-base", "weight": 1.0}}, "type": {"py/id": 3}}], "multigraph": true, "nodes": [{"id": {"py/id": 1}}, {"id": {"py/id": 2}}, {"id": {"py/id": 4}}, {"id": {"py/id": 5}}, {"id": {"py/id": 6}}]}

@hemanyaradadia hemanyaradadia added the bug Something isn't working label Aug 26, 2024
@roomrys
Copy link
Collaborator

roomrys commented Aug 26, 2024

Hi @hemanyaradadia,

We have run into this before, and the issue seems to stem from the skeleton.json file format containing

"py/state": {"name": "nose", "weight": 1.0}

when it should contain

"py/state": {"py/tuple": ["nose", 1.0]}

.

While I can (and will shortly) write a file converter to reformat the file into the expected format, this still does not explain the root cause of the behavior.

We use the jsonpickle library to encode/decode our Skeleton objects to/from a json file. Unfortunately, jsonpickle has released minor versions that break backwards compatibility (which is a no-no, but happens). We discovered this a long time ago and have had jsonpickle pinned to 1.2 for 5 years now. Thus, it would be a longshot, but I am wondering if perhaps a different version of jsonpickle was used to create the initial skeleton.json and now the current version of jsonpickle ==1.2 is having trouble decoding the skeleton.json.

The commit that initially pinned jsonpickle ==1.2 states that "1.2 can read anything from 1.1 but 1.1 can't read pickled objects from 1.2.".

I am testing this on my computer as well and will report back, but would you be able to share the environment information for the Windows computer that does not have this Found Multiple nodes named (name) issue and also the installation method/SLEAP-version?

Thanks,
Liezl

Related

@roomrys
Copy link
Collaborator

roomrys commented Aug 26, 2024

TLDR

The change in attrs version used by jsonpickle to write/read the skeleton.json is causing problems, namely, attrs >=22.2.0 returns the object state as a dictionary instead of a tuple. This results in incompatibilities when attempting to deserialize skeleton.json files (w/ an older attrs <22.2.0) that were serialized with attrs >=22.2.0. Based on the discussion in #1085 it sounds like attrs ==22.2.0 was unable to deserialize pickled data from attrs <22.2.0, and attrs ==23.1.0 (>=23.1.0?) at least fixes backwards compatibility. The long-term plan for attrs (discussed in #1091) is to remove support for deserializing tuples after giving everyone a few versions of attrs to convert their data over to a dictionary serialization.

Action Plan (developer)

The underlying theme in these discussions is that "pickling across versions" is an "antipattern" and we should perhaps find a different way to serialize our Skeleton objects. The short-term bandaid would be to pin our attrs version to either attrs <22.2.0 (keeping us in the past, but ensuring slp v1.3.3+ files are backwards compatible) or pin attrs >=22.2.0 (converting all slp files to safer dictionary serialization, but breaking backwards compatibility).

Action Plan (user)

Report to us that you are having this problem (including your environment info if possible), until a new release is published, use this script to convert incompatible skeleton.json files (that use a dictionary for the "py/state") to compatible skeleton.json files (that use a tuple for the "py/state").

Journaling the possibility that something might be happening inside jsonpickle

The Skeleton object we pass into jsonpickle.encode is a dictionary:

sleap/sleap/skeleton.py

Lines 1010 to 1026 in 1370782

# Encode to JSON
graph = json_graph.node_link_data(indexed_node_graph)
# SLEAP v1.3.0 added `description` and `preview_image` to `Skeleton`, but saving
# these fields breaks data format compatibility. Currently, these are only
# added in our custom template skeletons. To ensure backwards data format
# compatibilty of user data, we only save these fields if they are not None.
if self.is_template:
data = {
"nx_graph": graph,
"description": self.description,
"preview_image": self.preview_image,
}
else:
data = graph
json_str = jsonpickle.encode(data)

that looks something like this:

{'directed': True, 'multigraph': True, 'graph': {'name': 'Fly', 'num_edges_inserted': 4}, 'nodes': [{...}, {...}, {...}, {...}, {...}], 'links': [{...}, {...}, {...}, {...}, {...}, {...}]}

. Thus, jsonpickle's Pickler._get_flattener returns Pickler._flatten_dict_obj.

Then, jsonpickle calls Pickler._flatten_key_value_pair on each key-value pair in the dictionary, which basically finds a way to flatten both the key and the value.

Thus, since the formatting of the "py/state" is the problem, we need to pay attention to any changes to either jsonpickle's Pickler._getstate method (which also calls Pickler._flatten) or the code block within Pickler._flatten_obj_instance that calls Pickler._getstate. It could very likely be a change to attrs which creates a custom __getstate__ method (which returns a tuple for attrs ==21.4.0 ). We expect that the error occurs when the state is read in as a dict instead of a tuple.

Now pointing fingers at attr

In #1009 of attrs, I found that they changed the nested slots_getstate function (inside the _ClassBuilder._make_getstate_setstate method) to return a dictionary instead of a tuple as reportedly returned in attrs ==21.4.0.

In the aforementioned attrs PR, one user commented:

I feel like this may be a breaking change. Do objects serialized with the previous version can be deserialized with the new version?

to which an/the attrs' maintainer replied

They cannot. A fix has been added in #1085 that will be in 23.1, but (de)serializing classes across versions is risky (in general, not just the attrs case) so we'd encourage to look into avoiding that.

@hemanyaradadia
Copy link
Author

Hi @roomrys,

Thanks for the reply. The skeleton file I am using was created on the mentioned Windows 10 computer. I had installed SLEAP using conda from the package.

conda env info for the windows computer
sleap_env on jupiter

# packages in environment at C:\Miniconda3\envs\sleap_env:
#
# Name                    Version                   Build  Channel
absl-py                   0.15.0                   pypi_0    pypi
aiofiles                  22.1.0             pyhd8ed1ab_0    conda-forge
aiosqlite                 0.20.0             pyhd8ed1ab_0    conda-forge
anyio                     3.7.1              pyhd8ed1ab_0    conda-forge
argon2-cffi               23.1.0             pyhd8ed1ab_0    conda-forge
argon2-cffi-bindings      21.2.0           py37hcc03f2d_2    conda-forge
arrow                     1.2.3              pyhd8ed1ab_0    conda-forge
astunparse                1.6.3                    pypi_0    pypi
attrs                     21.4.0                   pypi_0    pypi
babel                     2.14.0             pyhd8ed1ab_0    conda-forge
backcall                  0.2.0              pyh9f0ad1d_0    conda-forge
backports                 1.0                pyhd8ed1ab_3    conda-forge
backports-zoneinfo        0.2.1                    pypi_0    pypi
backports.functools_lru_cache 2.0.0              pyhd8ed1ab_0    conda-forge
beautifulsoup4            4.12.3             pyha770c72_0    conda-forge
bleach                    6.1.0              pyhd8ed1ab_0    conda-forge
brotli-python             1.0.9            py37hf2a7229_7    conda-forge
ca-certificates           2024.7.4             h56e8100_0    conda-forge
cached-property           1.5.2                hd8ed1ab_1    conda-forge
cached_property           1.5.2              pyha770c72_1    conda-forge
cachetools                4.2.4                    pypi_0    pypi
cattrs                    1.1.1                    pypi_0    pypi
certifi                   2023.11.17               pypi_0    pypi
cffi                      1.15.1           py37ha95fbe2_1    conda-forge
charset-normalizer        3.3.2              pyhd8ed1ab_0    conda-forge
clang                     5.0                      pypi_0    pypi
colorama                  0.4.6              pyhd8ed1ab_0    conda-forge
comm                      0.2.2              pyhd8ed1ab_0    conda-forge
commonmark                0.9.1                    pypi_0    pypi
cudatoolkit               11.8.0               hd77b12b_0    anaconda
cudnn                     8.9.2.26               cuda11_0    anaconda
cycler                    0.11.0                   pypi_0    pypi
debugpy                   1.6.3            py37hf2a7229_0    conda-forge
decorator                 5.1.1              pyhd8ed1ab_0    conda-forge
defusedxml                0.7.1              pyhd8ed1ab_0    conda-forge
efficientnet              1.0.0                    pypi_0    pypi
entrypoints               0.4                pyhd8ed1ab_0    conda-forge
exceptiongroup            1.2.0              pyhd8ed1ab_2    conda-forge
flatbuffers               1.12                     pypi_0    pypi
fonttools                 4.38.0                   pypi_0    pypi
fqdn                      1.5.1              pyhd8ed1ab_0    conda-forge
gast                      0.4.0                    pypi_0    pypi
google-auth               1.35.0                   pypi_0    pypi
google-auth-oauthlib      0.4.6                    pypi_0    pypi
google-pasta              0.2.0                    pypi_0    pypi
grpcio                    1.60.0                   pypi_0    pypi
h5py                      3.1.0                    pypi_0    pypi
hdmf                      3.6.1                    pypi_0    pypi
idna                      3.6                      pypi_0    pypi
image-classifiers         1.0.0                    pypi_0    pypi
imageio                   2.31.2                   pypi_0    pypi
imgaug                    0.4.0                    pypi_0    pypi
imgstore                  0.2.9                    pypi_0    pypi
importlib-metadata        4.2.0                    pypi_0    pypi
importlib-resources       5.12.0                   pypi_0    pypi
importlib_metadata        4.11.4               hd8ed1ab_0    conda-forge
importlib_resources       6.0.0              pyhd8ed1ab_0    conda-forge
ipykernel                 6.16.2             pyh025b116_0    conda-forge
ipython                   7.33.0           py37h03978a9_0    conda-forge
ipython_genutils          0.2.0                      py_1    conda-forge
ipywidgets                8.1.2              pyhd8ed1ab_0    conda-forge
isoduration               20.11.0            pyhd8ed1ab_0    conda-forge
jedi                      0.19.1             pyhd8ed1ab_0    conda-forge
jinja2                    3.1.3              pyhd8ed1ab_0    conda-forge
joblib                    1.3.2                    pypi_0    pypi
jsmin                     3.0.1                    pypi_0    pypi
json5                     0.9.25             pyhd8ed1ab_0    conda-forge
jsonpickle                1.2                      pypi_0    pypi
jsonpointer               2.0                        py_0    conda-forge
jsonschema                4.17.3             pyhd8ed1ab_0    conda-forge
jsonschema-with-format-nongpl 4.17.3             pyhd8ed1ab_0    conda-forge
jupyter                   1.0.0             pyhd8ed1ab_10    conda-forge
jupyter_client            7.4.9              pyhd8ed1ab_0    conda-forge
jupyter_console           6.5.1              pyhd8ed1ab_0    conda-forge
jupyter_core              4.11.1           py37h03978a9_0    conda-forge
jupyter_events            0.6.3              pyhd8ed1ab_1    conda-forge
jupyter_server            1.23.4             pyhd8ed1ab_0    conda-forge
jupyter_server_fileid     0.9.2              pyhd8ed1ab_0    conda-forge
jupyter_server_ydoc       0.8.0              pyhd8ed1ab_0    conda-forge
jupyter_ydoc              0.2.4              pyhd8ed1ab_0    conda-forge
jupyterlab                3.6.7              pyhd8ed1ab_0    conda-forge
jupyterlab_pygments       0.3.0              pyhd8ed1ab_0    conda-forge
jupyterlab_server         2.24.0             pyhd8ed1ab_0    conda-forge
jupyterlab_widgets        3.0.10             pyhd8ed1ab_0    conda-forge
keras                     2.6.0                    pypi_0    pypi
keras-applications        1.0.8                    pypi_0    pypi
keras-preprocessing       1.1.2                    pypi_0    pypi
kiwisolver                1.4.5                    pypi_0    pypi
libclang                  16.0.6                   pypi_0    pypi
libsodium                 1.0.18               h8d14728_1    conda-forge
libsqlite                 3.44.2               hcfcfb64_0    conda-forge
libzlib                   1.2.13               hcfcfb64_5    conda-forge
markdown                  3.3.4                    pypi_0    pypi
markupsafe                2.1.3                    pypi_0    pypi
matplotlib                3.5.3                    pypi_0    pypi
matplotlib-inline         0.1.7              pyhd8ed1ab_0    conda-forge
mistune                   3.0.2              pyhd8ed1ab_0    conda-forge
nbclassic                 1.0.0              pyhb4ecaf3_1    conda-forge
nbclient                  0.7.0              pyhd8ed1ab_0    conda-forge
nbconvert                 7.6.0              pyhd8ed1ab_0    conda-forge
nbconvert-core            7.6.0              pyhd8ed1ab_0    conda-forge
nbconvert-pandoc          7.6.0              pyhd8ed1ab_0    conda-forge
nbformat                  5.8.0              pyhd8ed1ab_0    conda-forge
ndx-pose                  0.1.1                    pypi_0    pypi
nest-asyncio              1.6.0              pyhd8ed1ab_0    conda-forge
networkx                  2.6.3                    pypi_0    pypi
nixio                     1.5.3                    pypi_0    pypi
notebook                  6.5.6              pyha770c72_0    conda-forge
notebook-shim             0.2.4              pyhd8ed1ab_0    conda-forge
numpy                     1.19.5                   pypi_0    pypi
oauthlib                  3.2.2                    pypi_0    pypi
opencv-python             4.5.5.64                 pypi_0    pypi
openssl                   3.3.1                h2466b09_3    conda-forge
opt-einsum                3.3.0                    pypi_0    pypi
packaging                 23.2               pyhd8ed1ab_0    conda-forge
pandas                    1.3.5                    pypi_0    pypi
pandoc                    3.1.13               h57928b3_0    conda-forge
pandocfilters             1.5.0              pyhd8ed1ab_0    conda-forge
parso                     0.8.4              pyhd8ed1ab_0    conda-forge
pickleshare               0.7.5                   py_1003    conda-forge
pillow                    8.4.0                    pypi_0    pypi
pip                       23.3.1             pyhd8ed1ab_0    conda-forge
pkgutil-resolve-name      1.3.10             pyhd8ed1ab_1    conda-forge
prometheus_client         0.17.1             pyhd8ed1ab_0    conda-forge
prompt-toolkit            3.0.42             pyha770c72_0    conda-forge
prompt_toolkit            3.0.42               hd8ed1ab_0    conda-forge
protobuf                  3.19.6                   pypi_0    pypi
psutil                    5.9.6                    pypi_0    pypi
pyasn1                    0.5.1                    pypi_0    pypi
pyasn1-modules            0.3.0                    pypi_0    pypi
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pygments                  2.17.2             pyhd8ed1ab_0    conda-forge
pykalman                  0.9.5                    pypi_0    pypi
pynwb                     2.3.3                    pypi_0    pypi
pyparsing                 3.1.1                    pypi_0    pypi
pyrsistent                0.19.3                   pypi_0    pypi
pyside2                   5.14.1                   pypi_0    pypi
pysocks                   1.7.1            py37h03978a9_5    conda-forge
python                    3.7.12          h900ac77_100_cpython    conda-forge
python-dateutil           2.8.2                    pypi_0    pypi
python-fastjsonschema     2.19.1             pyhd8ed1ab_0    conda-forge
python-json-logger        2.0.7              pyhd8ed1ab_0    conda-forge
python-rapidjson          1.10                     pypi_0    pypi
python_abi                3.7                     4_cp37m    conda-forge
pytz                      2023.3.post1             pypi_0    pypi
pywavelets                1.3.0                    pypi_0    pypi
pywin32                   303              py37hcc03f2d_0    conda-forge
pywinpty                  2.0.8            py37h7f67f24_0    conda-forge
pyyaml                    6.0.1                    pypi_0    pypi
pyzmq                     25.1.2                   pypi_0    pypi
qimage2ndarray            1.10.0                   pypi_0    pypi
qtconsole-base            5.4.4              pyha770c72_0    conda-forge
qtpy                      2.4.1              pyhd8ed1ab_0    conda-forge
requests                  2.31.0                   pypi_0    pypi
requests-oauthlib         1.3.1                    pypi_0    pypi
rfc3339-validator         0.1.4              pyhd8ed1ab_0    conda-forge
rfc3986-validator         0.1.1              pyh9f0ad1d_0    conda-forge
rich                      10.16.1                  pypi_0    pypi
rsa                       4.9                      pypi_0    pypi
ruamel-yaml               0.18.5                   pypi_0    pypi
ruamel-yaml-clib          0.2.8                    pypi_0    pypi
scikit-image              0.19.3                   pypi_0    pypi
scikit-learn              1.0.2                    pypi_0    pypi
scikit-video              1.1.11                   pypi_0    pypi
scipy                     1.7.3                    pypi_0    pypi
seaborn                   0.12.2                   pypi_0    pypi
segmentation-models       1.0.1                    pypi_0    pypi
send2trash                1.8.3              pyh5737063_0    conda-forge
setuptools                68.2.2             pyhd8ed1ab_0    conda-forge
shapely                   2.0.2                    pypi_0    pypi
shiboken2                 5.14.1                   pypi_0    pypi
six                       1.15.0                   pypi_0    pypi
sleap                     1.3.3                    pypi_0    pypi
sniffio                   1.3.1              pyhd8ed1ab_0    conda-forge
soupsieve                 2.3.2.post1        pyhd8ed1ab_0    conda-forge
sqlite                    3.44.2               hcfcfb64_0    conda-forge
tensorboard               2.6.0                    pypi_0    pypi
tensorboard-data-server   0.6.1                    pypi_0    pypi
tensorboard-plugin-wit    1.8.1                    pypi_0    pypi
tensorflow                2.6.3                    pypi_0    pypi
tensorflow-estimator      2.6.0                    pypi_0    pypi
tensorflow-hub            0.14.0                   pypi_0    pypi
tensorflow-io-gcs-filesystem 0.31.0                   pypi_0    pypi
termcolor                 1.1.0                    pypi_0    pypi
terminado                 0.17.0             pyh08f2357_0    conda-forge
threadpoolctl             3.1.0                    pypi_0    pypi
tifffile                  2021.11.2                pypi_0    pypi
tinycss2                  1.3.0              pyhd8ed1ab_0    conda-forge
tomli                     2.0.1              pyhd8ed1ab_0    conda-forge
tornado                   6.2              py37hcc03f2d_0    conda-forge
tqdm                      4.66.5             pyhd8ed1ab_0    conda-forge
traitlets                 5.9.0              pyhd8ed1ab_0    conda-forge
typing-extensions         3.10.0.2                 pypi_0    pypi
typing_extensions         4.7.1              pyha770c72_0    conda-forge
tzdata                    2023.3                   pypi_0    pypi
tzlocal                   5.1                      pypi_0    pypi
ucrt                      10.0.22621.0         h57928b3_0    conda-forge
uri-template              1.3.0              pyhd8ed1ab_0    conda-forge
urllib3                   1.26.18                  pypi_0    pypi
vc                        14.3                h64f974e_17    conda-forge
vc14_runtime              14.36.32532         hdcecf7f_17    conda-forge
vs2015_runtime            14.36.32532         h05e6639_17    conda-forge
wcwidth                   0.2.10             pyhd8ed1ab_0    conda-forge
webcolors                 1.13               pyhd8ed1ab_0    conda-forge
webencodings              0.5.1              pyhd8ed1ab_2    conda-forge
websocket-client          1.6.1              pyhd8ed1ab_0    conda-forge
werkzeug                  2.2.3                    pypi_0    pypi
wheel                     0.42.0             pyhd8ed1ab_0    conda-forge
widgetsnbextension        4.0.10             pyhd8ed1ab_0    conda-forge
win_inet_pton             1.1.0              pyhd8ed1ab_6    conda-forge
winpty                    0.4.3                         4    conda-forge
wrapt                     1.12.1                   pypi_0    pypi
y-py                      0.5.4            py37had2a638_0    conda-forge
yaml                      0.2.5                h8ffe710_2    conda-forge
ypy-websocket             0.8.2              pyhd8ed1ab_0    conda-forge
zeromq                    4.3.4                h0e60522_1    conda-forge
zipp                      3.15.0             pyhd8ed1ab_0    conda-forge
zlib                      1.2.13               hcfcfb64_5    conda-forge

Interestingly, this environment has the same version of jsonpickle as Linux enviornment.

@roomrys
Copy link
Collaborator

roomrys commented Aug 27, 2024

Hi @hemanyaradadia,

This is a file format compatibility issue that will be officially addressed in our future release (by constraining the version of attr used and/or moving away from jsonpickle as our method for saving Skeletons).

A (temporary) peace offering

Apologies you have run into this, but thank you for reporting! Until this issue is officially addressed in a release, you can run this script to convert the incompatible skeleton.json that uses a dict for the "py/state" to the expected tuple for the "py/state".

To run the script:

  1. download the script
  2. in the if __name__ == "__main__" block edit the input_json to point to the path of your skeleton.json
  3. (optional) edit the output_json to point to where you want to save the new skeleton to
  4. open a terminal wherever you downloaded the script
  5. activate your sleap environment
  6. run the command
    python sleap_skeleton_convert_dict_state_to_tuple_state.py

Results: Converting your skeleton

I set your current skeleton.json as the input_json:

{"directed": true, "graph": {"name": "Skeleton-3", "num_edges_inserted": 5}, "links": [{"edge_insert_idx": 0, "key": 0, "source": {"py/object": "sleap.skeleton.Node", "py/state": {"name": "nose", "weight": 1.0}}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"name": "right-ear", "weight": 1.0}}, "type": {"py/reduce": [{"py/type": "sleap.skeleton.EdgeType"}, {"py/tuple": [1]}]}}, {"edge_insert_idx": 1, "key": 0, "source": {"py/id": 1}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"name": "left-ear", "weight": 1.0}}, "type": {"py/id": 3}}, {"edge_insert_idx": 2, "key": 0, "source": {"py/id": 2}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"name": "thorax", "weight": 1.0}}, "type": {"py/id": 3}}, {"edge_insert_idx": 3, "key": 0, "source": {"py/id": 4}, "target": {"py/id": 5}, "type": {"py/id": 3}}, {"edge_insert_idx": 4, "key": 0, "source": {"py/id": 5}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"name": "tail-base", "weight": 1.0}}, "type": {"py/id": 3}}], "multigraph": true, "nodes": [{"id": {"py/id": 1}}, {"id": {"py/id": 2}}, {"id": {"py/id": 4}}, {"id": {"py/id": 5}}, {"id": {"py/id": 6}}]}

and this is the output skeleton.json that was saved as output_json:

{"directed": true, "graph": {"name": "Skeleton-3", "num_edges_inserted": 5}, "links": [{"edge_insert_idx": 0, "key": 0, "source": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["nose", 1.0]}}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["right-ear", 1.0]}}, "type": {"py/reduce": [{"py/type": "sleap.skeleton.EdgeType"}, {"py/tuple": [1]}]}}, {"edge_insert_idx": 1, "key": 0, "source": {"py/id": 1}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["left-ear", 1.0]}}, "type": {"py/id": 3}}, {"edge_insert_idx": 2, "key": 0, "source": {"py/id": 2}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["thorax", 1.0]}}, "type": {"py/id": 3}}, {"edge_insert_idx": 3, "key": 0, "source": {"py/id": 4}, "target": {"py/id": 5}, "type": {"py/id": 3}}, {"edge_insert_idx": 4, "key": 0, "source": {"py/id": 5}, "target": {"py/object": "sleap.skeleton.Node", "py/state": {"py/tuple": ["tail-base", 1.0]}}, "type": {"py/id": 3}}], "multigraph": true, "nodes": [{"id": {"py/id": 1}}, {"id": {"py/id": 2}}, {"id": {"py/id": 4}}, {"id": {"py/id": 5}}, {"id": {"py/id": 6}}]}

(notice the differences in the py/states).

Then when loading each skeleton.json as a Skeleton, we see that the reformatted skeleton.json is compatible while the original skeleton.json is being read in incorrectly:

In [1]: from sleap import Skeleton

In [2]: input_json = "dict_state.json"

In [3]: output_json = "tuple_state.json"

In [4]: skeleton_input = Skeleton.load_json(input_json)

In [5]: skeleton_input
Out[5]: Skeleton(name='Skeleton-3', description='None', nodes=['name', 'name', 'name', 'name', 'name'], edges=[('name', 'name'), ('name', 'name'), ('name', 'name'), ('name', 'name'), ('name', 'name')], 
symmetries=[])

In [6]: skeleton_output = Skeleton.load_json(output_json)

In [7]: skeleton_output
Out[7]: Skeleton(name='Skeleton-3', description='None', nodes=['nose', 'right-ear', 'left-ear', 'thorax', 'tail-base'], edges=[('nose', 'right-ear'), ('nose', 'left-ear'), ('right-ear', 'thorax'), ('left-ear', 'thorax'), ('thorax', 'tail-base')], symmetries=[])

Let us know if you run into further trouble.

Thanks,
Liezl

@roomrys
Copy link
Collaborator

roomrys commented Aug 27, 2024

Hi @roomrys,

Thanks for the reply. The skeleton file I am using was created on the mentioned Windows 10 computer. I had installed SLEAP using conda from the package.

conda env info for the windows computer
Interestingly, this environment has the same version of jsonpickle as Linux enviornment.

Yes, I also originally thought it might be a jsonpickle issue (and it sort-of is), but it seems the breaking change came from the version of attr being used underneath (indirectly called by jsonpickle when calling Node.__getstate__ - which is a method generated by attr).

I see that your attr version SEEMs to be the same, but there is just too much evidence that it is the attr version causing this. I suspect that your installation has both an attr >=22.2.0 installed from conda and and attr ==21.4.0 (as shown in your environment) installed from PyPI. If you pip uninstall attr and then run conda list again, we should be able to see the underlying conda version of attr - which is the expected verion used.

Although, this is still strange because, while we do not have a pin on the PyPI installation of attr - that comes from the "dev" requirements myst-nb and myst-parser (which would only be installed via "conda from source") - we do have a pin on the attr version in the meta.yml of our 1.3.3 package:

- conda-forge::attrs ==21.4.0

I am wondering if perhaps the Windows computer might have a "conda from source" installation which uses the development branch in which attr currently has no upper limit:
https://github.com/search?q=repo%3Atalmolab%2Fsleap+attrs+path%3A*.y*ml&type=code
The fact that the Windows computer has a PyPI install of attr points to a "conda from source" installation.

Again, this is our fault as we list dependency constraints, I am working on building a 1.3.4 mac conda packages as those have no upper bound on the attr version.
https://github.com/search?q=repo%3Atalmolab%2Fsleap+attrs+path%3A.conda_mac&type=code
This will need to be handled properly in our future release.

Thanks,
Liezl

@hemanyaradadia
Copy link
Author

Hi @roomrys,

Thanks for the script to convert the skeleton .json into the correct format. I changed the skeleton object in training_config.json of the trained model manually. The updated model works perfectly on the Linux system.

@roomrys
Copy link
Collaborator

roomrys commented Sep 3, 2024

We have released SLEAP 1.3.4 (off #1927) which adds additional dependency constraints to prevent this from happening, but we are also switching to manually handling the Skeleton (de)serialization #1934 (including a conversion script incase the users have already created Skeleton files in the new (currently unsupported) format).

I will leave this issue open until our next release (where we include a conversion script as well).

@talmo talmo pinned this issue Sep 12, 2024
@roomrys roomrys added the fixed in future release Fix or feature is merged into develop and will be available in future release. label Sep 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working fixed in future release Fix or feature is merged into develop and will be available in future release.
Projects
None yet
Development

No branches or pull requests

8 participants
@roomrys @hemanyaradadia and others