Crashing during inference on Apple M2 Max

[uklibaite]

sleap crashes during inference of entire video (even short video) but runs inference on a small number of suggested frames. Samples during training and these small inference batches look good.
Inference starts and the count of inferred frames increases (then sometimes the speed of inference appears to double shortly before a crash). The popup says there is probably more info in the command line (posted below).

Expected behaviour

Inference running normally (oddly it did run once all the way through a movie that was 180000 frames).
If relevant - this happens both when tracking a simple movie with a single animal or when tracking about 5 individuals. Training runs normally.

Actual behaviour

Command line outputs for training and inference below.

Your personal set up

I installed using the mamba instructions for apple silicon (M2 Max). I use conda activate sleap to load the environment.

• OS:
  macOS Sonoma 14.2.1

• Version(s):

• SLEAP installation method (listed here):
  • Conda from package
  • Conda from source
  • pip package
  • [ X] Apple Silicon Macs

Environment packages

conda list
# packages in environment at /Users/ugne/mambaforge3/envs/sleap:
#
# Name                    Version                   Build  Channel
abseil-cpp                20211102.0           he4e09e4_3    conda-forge
absl-py                   1.4.0                    pypi_0    pypi
aiohttp                   3.9.1            py39h17cfd9d_0    conda-forge
aiosignal                 1.3.1              pyhd8ed1ab_0    conda-forge
aom                       3.5.0                h7ea286d_0    conda-forge
astunparse                1.6.3              pyhd8ed1ab_0    conda-forge
async-timeout             4.0.3              pyhd8ed1ab_0    conda-forge
attrs                     23.1.0             pyh71513ae_1    conda-forge
blinker                   1.7.0              pyhd8ed1ab_0    conda-forge
blosc                     1.21.5               hc338f07_0    conda-forge
brotli                    1.0.9                h1a8c8d9_9    conda-forge
brotli-bin                1.0.9                h1a8c8d9_9    conda-forge
brotli-python             1.0.9            py39h23fbdae_9    conda-forge
brunsli                   0.1                  h9f76cd9_0    conda-forge
bzip2                     1.0.8                h93a5062_5    conda-forge
c-ares                    1.24.0               h93a5062_0    conda-forge
c-blosc2                  2.11.3               h8eb3132_0    conda-forge
ca-certificates           2023.11.17           hf0a4a13_0    conda-forge
cached-property           1.5.2                hd8ed1ab_1    conda-forge
cached_property           1.5.2              pyha770c72_1    conda-forge
cachetools                5.3.1                    pypi_0    pypi
cairo                     1.16.0            h73a0509_1014    conda-forge
cattrs                    1.1.1              pyhd8ed1ab_0    conda-forge
certifi                   2023.11.17         pyhd8ed1ab_0    conda-forge
cffi                      1.16.0           py39he153c15_0    conda-forge
cfitsio                   4.2.0                h2f961c4_0    conda-forge
charls                    2.3.4                hbdafb3b_0    conda-forge
charset-normalizer        3.2.0                    pypi_0    pypi
click                     8.1.7           unix_pyh707e725_0    conda-forge
contourpy                 1.2.0            py39he9de807_0    conda-forge
cryptography              39.0.0           py39haa0b8cc_0    conda-forge
cycler                    0.12.1             pyhd8ed1ab_0    conda-forge
dav1d                     1.2.1                hb547adb_0    conda-forge
efficientnet              1.0.0                    pypi_0    pypi
expat                     2.5.0                hb7217d7_1    conda-forge
ffmpeg                    4.4.2           gpl_hf318d42_112    conda-forge
font-ttf-dejavu-sans-mono 2.37                 hab24e00_0    conda-forge
font-ttf-inconsolata      3.000                h77eed37_0    conda-forge
font-ttf-source-code-pro  2.038                h77eed37_0    conda-forge
font-ttf-ubuntu           0.83                 h77eed37_1    conda-forge
fontconfig                2.14.2               h82840c6_0    conda-forge
fonts-conda-ecosystem     1                             0    conda-forge
fonts-conda-forge         1                             0    conda-forge
fonttools                 4.47.0           py39h17cfd9d_0    conda-forge
freetype                  2.12.1               hadb7bae_2    conda-forge
frozenlist                1.4.1            py39h17cfd9d_0    conda-forge
gast                      0.4.0              pyh9f0ad1d_0    conda-forge
geos                      3.12.1               h965bd2d_0    conda-forge
gettext                   0.21.1               h0186832_0    conda-forge
giflib                    5.2.1                h1a8c8d9_3    conda-forge
glib                      2.78.3               h9e231a4_0    conda-forge
glib-tools                2.78.3               h9e231a4_0    conda-forge
gmp                       6.3.0                h965bd2d_0    conda-forge
gnutls                    3.7.9                hd26332c_0    conda-forge
google-auth               2.23.0                   pypi_0    pypi
google-auth-oauthlib      0.4.6              pyhd8ed1ab_0    conda-forge
google-pasta              0.2.0              pyh8c360ce_0    conda-forge
graphite2                 1.3.13            h9f76cd9_1001    conda-forge
grpc-cpp                  1.46.4               hcaf9be7_3    conda-forge
grpcio                    1.58.0                   pypi_0    pypi
gst-plugins-base          1.22.8               h09b4b5e_0    conda-forge
gstreamer                 1.22.8               h551c6ff_0    conda-forge
h5py                      3.8.0           nompi_py39hc9149d8_100    conda-forge
harfbuzz                  5.3.0                hddbc195_0    conda-forge
hdf5                      1.12.2          nompi_h55deafc_101    conda-forge
hdmf                      3.9.0                    pypi_0    pypi
icu                       70.1                 h6b3803e_0    conda-forge
idna                      3.4                      pypi_0    pypi
image-classifiers         1.0.0                    pypi_0    pypi
imagecodecs               2022.9.26        py39hd7f743f_4    conda-forge
imageio                   2.33.1             pyh8c1a49c_0    conda-forge
imgaug                    0.4.0              pyhd8ed1ab_1    conda-forge
imgstore                  0.2.9                    pypi_0    pypi
importlib-metadata        7.0.0              pyha770c72_0    conda-forge
importlib-resources       6.1.1              pyhd8ed1ab_0    conda-forge
importlib_resources       6.1.1              pyhd8ed1ab_0    conda-forge
jasper                    2.0.33               hc3cd1e9_1    conda-forge
joblib                    1.3.2              pyhd8ed1ab_0    conda-forge
jpeg                      9e                   h1a8c8d9_3    conda-forge
jsmin                     3.0.1              pyhd8ed1ab_0    conda-forge
jsonpickle                1.2                        py_0    conda-forge
jsonschema                4.19.0                   pypi_0    pypi
jsonschema-specifications 2023.7.1                 pypi_0    pypi
jxrlib                    1.1                  h27ca646_2    conda-forge
keras                     2.9.0              pyhd8ed1ab_0    conda-forge
keras-applications        1.0.8                    pypi_0    pypi
keras-preprocessing       1.1.2              pyhd8ed1ab_0    conda-forge
kiwisolver                1.4.5            py39hbd775c9_1    conda-forge
krb5                      1.20.1               h127bd45_0    conda-forge
lame                      3.100             h1a8c8d9_1003    conda-forge
lazy_loader               0.3                pyhd8ed1ab_0    conda-forge
lcms2                     2.14                 h8193b64_0    conda-forge
lerc                      4.0.0                h9a09cb3_0    conda-forge
libabseil                 20211102.0      cxx17_h28b99d4_3    conda-forge
libaec                    1.1.2                h13dd4ca_1    conda-forge
libavif                   0.11.1               h9f83d30_2    conda-forge
libblas                   3.9.0           20_osxarm64_openblas    conda-forge
libbrotlicommon           1.0.9                h1a8c8d9_9    conda-forge
libbrotlidec              1.0.9                h1a8c8d9_9    conda-forge
libbrotlienc              1.0.9                h1a8c8d9_9    conda-forge
libcblas                  3.9.0           20_osxarm64_openblas    conda-forge
libclang                  16.0.6                   pypi_0    pypi
libclang13                14.0.6          default_hc7183e1_1    conda-forge
libcurl                   7.87.0               hbe9bab4_0    conda-forge
libcxx                    16.0.6               h4653b0c_0    conda-forge
libdeflate                1.14                 h1a8c8d9_0    conda-forge
libedit                   3.1.20191231         hc8eb9b7_2    conda-forge
libev                     4.33                 h93a5062_2    conda-forge
libexpat                  2.5.0                hb7217d7_1    conda-forge
libffi                    3.4.2                h3422bc3_5    conda-forge
libgfortran               5.0.0           13_2_0_hd922786_1    conda-forge
libgfortran5              13.2.0               hf226fd6_1    conda-forge
libglib                   2.78.3               hb438215_0    conda-forge
libiconv                  1.17                 h0d3ecfb_2    conda-forge
libidn2                   2.3.4                h1a8c8d9_0    conda-forge
liblapack                 3.9.0           20_osxarm64_openblas    conda-forge
liblapacke                3.9.0           20_osxarm64_openblas    conda-forge
libllvm14                 14.0.6               hd1a9a77_4    conda-forge
libnghttp2                1.51.0               hd184df1_0    conda-forge
libogg                    1.3.4                h27ca646_1    conda-forge
libopenblas               0.3.25          openmp_h6c19121_0    conda-forge
libopencv                 4.6.0            py39he1c1adf_3    conda-forge
libopus                   1.3.1                h27ca646_1    conda-forge
libpng                    1.6.39               h76d750c_0    conda-forge
libpq                     15.1                 hbce9e56_3    conda-forge
libprotobuf               3.20.3               hb5ab8b9_0    conda-forge
libsodium                 1.0.18               h27ca646_1    conda-forge
libsqlite                 3.44.2               h091b4b1_0    conda-forge
libssh2                   1.10.0               hb80f160_3    conda-forge
libtasn1                  4.19.0               h1a8c8d9_0    conda-forge
libtiff                   4.4.0                heb92581_5    conda-forge
libunistring              0.9.10               h3422bc3_0    conda-forge
libvorbis                 1.3.7                h9f76cd9_0    conda-forge
libvpx                    1.11.0               hc470f4d_3    conda-forge
libwebp-base              1.3.2                hb547adb_0    conda-forge
libxcb                    1.13              h9b22ae9_1004    conda-forge
libxml2                   2.10.3               h67585b2_4    conda-forge
libxslt                   1.1.37               h1bd8bc4_0    conda-forge
libzlib                   1.2.13               h53f4e23_5    conda-forge
libzopfli                 1.0.3                h9f76cd9_0    conda-forge
llvm-openmp               17.0.6               hcd81f8e_0    conda-forge
lz4-c                     1.9.4                hb7217d7_0    conda-forge
markdown                  3.4.4                    pypi_0    pypi
markdown-it-py            3.0.0              pyhd8ed1ab_0    conda-forge
markupsafe                2.1.3            py39h0f82c59_1    conda-forge
matplotlib-base           3.8.2            py39h1a09f3e_0    conda-forge
mdurl                     0.1.0              pyhd8ed1ab_0    conda-forge
multidict                 6.0.4            py39h02fc5c5_1    conda-forge
munkres                   1.1.4              pyh9f0ad1d_0    conda-forge
mysql-common              8.0.32               hab468bb_0    conda-forge
mysql-libs                8.0.32               hea58576_0    conda-forge
ncurses                   6.4                  h463b476_2    conda-forge
ndx-pose                  0.1.1                    pypi_0    pypi
nettle                    3.9.1                h40ed0f5_0    conda-forge
networkx                  3.2.1              pyhd8ed1ab_0    conda-forge
nixio                     1.5.3                    pypi_0    pypi
nspr                      4.35                 hb7217d7_0    conda-forge
nss                       3.96                 h5ce2875_0    conda-forge
numpy                     1.22.4           py39h7df2422_0    conda-forge
oauthlib                  3.2.2              pyhd8ed1ab_0    conda-forge
opencv                    4.6.0            py39hdf13c20_3    conda-forge
openh264                  2.3.1                hb7217d7_2    conda-forge
openjpeg                  2.5.0                h5d4e404_1    conda-forge
openssl                   1.1.1w               h53f4e23_0    conda-forge
opt_einsum                3.3.0              pyhc1e730c_2    conda-forge
p11-kit                   0.24.1               h29577a5_0    conda-forge
packaging                 23.2               pyhd8ed1ab_0    conda-forge
pandas                    2.1.4            py39hf8cecc8_0    conda-forge
patsy                     0.5.4              pyhd8ed1ab_0    conda-forge
pcre2                     10.42                h26f9a81_0    conda-forge
pillow                    9.2.0            py39h139752e_3    conda-forge
pip                       23.3.2             pyhd8ed1ab_0    conda-forge
pixman                    0.42.2               h13dd4ca_0    conda-forge
protobuf                  3.19.6                   pypi_0    pypi
psutil                    5.9.7            py39h17cfd9d_0    conda-forge
pthread-stubs             0.4               h27ca646_1001    conda-forge
py-opencv                 4.6.0            py39hfa6204d_3    conda-forge
pyasn1                    0.5.0                    pypi_0    pypi
pyasn1-modules            0.3.0              pyhd8ed1ab_0    conda-forge
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pygments                  2.17.2             pyhd8ed1ab_0    conda-forge
pyjwt                     2.8.0              pyhd8ed1ab_0    conda-forge
pykalman                  0.9.5                      py_1    conda-forge
pynwb                     2.5.0                    pypi_0    pypi
pyopenssl                 23.2.0             pyhd8ed1ab_1    conda-forge
pyparsing                 3.1.1              pyhd8ed1ab_0    conda-forge
pyside2                   5.15.8           py39h0adaba8_2    conda-forge
pysocks                   1.7.1              pyha2e5f31_6    conda-forge
python                    3.9.15          h2d96c93_0_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python-flatbuffers        1.12               pyhd8ed1ab_1    conda-forge
python-rapidjson          1.14             py39hf3050f2_0    conda-forge
python-tzdata             2023.3             pyhd8ed1ab_0    conda-forge
python_abi                3.9                      4_cp39    conda-forge
pytz                      2023.3.post1       pyhd8ed1ab_0    conda-forge
pyu2f                     0.1.5              pyhd8ed1ab_0    conda-forge
pywavelets                1.4.1            py39hf4a74a7_1    conda-forge
pyyaml                    6.0.1            py39h0f82c59_1    conda-forge
pyzmq                     25.1.2           py39he1e2164_0    conda-forge
qimage2ndarray            1.10.0                   pypi_0    pypi
qt-main                   5.15.8               hfe8d25c_6    conda-forge
qtpy                      2.4.1              pyhd8ed1ab_0    conda-forge
re2                       2022.06.01           h9a09cb3_1    conda-forge
readline                  8.2                  h92ec313_1    conda-forge
referencing               0.30.2                   pypi_0    pypi
requests                  2.31.0             pyhd8ed1ab_0    conda-forge
requests-oauthlib         1.3.1              pyhd8ed1ab_0    conda-forge
rich                      13.7.0             pyhd8ed1ab_0    conda-forge
rpds-py                   0.10.3                   pypi_0    pypi
rsa                       4.9                pyhd8ed1ab_0    conda-forge
ruamel-yaml               0.17.32                  pypi_0    pypi
ruamel-yaml-clib          0.2.7                    pypi_0    pypi
scikit-image              0.22.0           py39hf8cecc8_2    conda-forge
scikit-learn              1.0              py39h12ba089_1    conda-forge
scikit-video              1.1.11             pyh24bf2e0_0    conda-forge
scipy                     1.9.0            py39h14896cb_0    conda-forge
seaborn                   0.13.0               hd8ed1ab_0    conda-forge
seaborn-base              0.13.0             pyhd8ed1ab_0    conda-forge
segmentation-models       1.0.1                    pypi_0    pypi
setuptools                68.2.2             pyhd8ed1ab_0    conda-forge
shapely                   2.0.2            py39ha70ab96_1    conda-forge
six                       1.15.0                   pypi_0    pypi
sleap                     1.3.3                    pypi_0    pypi
snappy                    1.1.10               h17c5cce_0    conda-forge
sqlite                    3.44.2               hf2abe2d_0    conda-forge
statsmodels               0.14.1           py39h373d45f_0    conda-forge
svt-av1                   1.4.1                h7ea286d_0    conda-forge
tensorboard               2.9.1                    pypi_0    pypi
tensorboard-data-server   0.6.1            py39haa0b8cc_4    conda-forge
tensorboard-plugin-wit    1.8.1              pyhd8ed1ab_0    conda-forge
tensorflow                2.9.1           cpu_py39h2839aeb_0    conda-forge
tensorflow-base           2.9.1           cpu_py39ha1ad4ae_0    conda-forge
tensorflow-estimator      2.9.1           cpu_py39h7b621ec_0    conda-forge
tensorflow-hub            0.12.0             pyhca92ed8_0    conda-forge
tensorflow-macos          2.9.2                    pypi_0    pypi
tensorflow-metal          0.5.0                    pypi_0    pypi
termcolor                 2.3.0              pyhd8ed1ab_0    conda-forge
threadpoolctl             3.2.0              pyha21a80b_0    conda-forge
tifffile                  2022.10.10         pyhd8ed1ab_0    conda-forge
tk                        8.6.13               h5083fa2_1    conda-forge
typing-extensions         4.9.0                hd8ed1ab_0    conda-forge
typing_extensions         4.9.0              pyha770c72_0    conda-forge
tzdata                    2023c                h71feb2d_0    conda-forge
tzlocal                   5.0.1                    pypi_0    pypi
unicodedata2              15.1.0           py39h0f82c59_0    conda-forge
urllib3                   1.26.16                  pypi_0    pypi
werkzeug                  2.3.7                    pypi_0    pypi
wheel                     0.42.0             pyhd8ed1ab_0    conda-forge
wrapt                     1.15.0                   pypi_0    pypi
x264                      1!164.3095           h57fd34a_2    conda-forge
x265                      3.5                  hbc6ce65_3    conda-forge
xorg-libxau               1.0.11               hb547adb_0    conda-forge
xorg-libxdmcp             1.1.3                h27ca646_0    conda-forge
xz                        5.2.6                h57fd34a_0    conda-forge
yaml                      0.2.5                h3422bc3_2    conda-forge
yarl                      1.9.3            py39h17cfd9d_0    conda-forge
zeromq                    4.3.5                h965bd2d_0    conda-forge
zfp                       1.0.1                ha8f4885_0    conda-forge
zipp                      3.17.0             pyhd8ed1ab_0    conda-forge
zlib                      1.2.13               h53f4e23_5    conda-forge
zlib-ng                   2.0.7                h1a8c8d9_0    conda-forge
zstd                      1.5.5                h4f39d0f_0    conda-forge

Logs

Command line call:
sleap-track /Users/ugne/Dropbox/pregnant/labels.v001_pupx1.slp --only-suggested-frames -m /Users/ugne/Dropbox/pregnant/models/231223_213850.centroid.n=8 -m /Users/ugne/Dropbox/pregnant/models/231223_214910.centered_instance.n=8 --max_instances 1 -o /Users/ugne/Dropbox/pregnant/predictions/labels.v001_pupx1.slp.231223_220359.predictions.slp --verbosity json --no-empty-frames

Started inference at: 2023-12-23 22:04:03.074597
Args:
{
│   'data_path': '/Users/ugne/Dropbox/pregnant/labels.v001_pupx1.slp',
│   'models': [
│   │   '/Users/ugne/Dropbox/pregnant/models/231223_213850.centroid.n=8',
│   │   '/Users/ugne/Dropbox/pregnant/models/231223_214910.centered_instance.n=8'
│   ],
│   'frames': '',
│   'only_labeled_frames': False,
│   'only_suggested_frames': True,
│   'output': '/Users/ugne/Dropbox/pregnant/predictions/labels.v001_pupx1.slp.231223_220359.predictions.slp',
2023-12-23 22:04:03.684157: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:305] Could not identify NUMA node of platform GPU ID 0, defaulting to 0. Your kernel may not have been built with NUMA support.
2023-12-23 22:04:03.684311: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:271] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 0 MB memory) -> physical PluggableDevice (device: 0, name: METAL, pci bus id: <undefined>)
│   'no_empty_frames': True,
│   'verbosity': 'json',
│   'video.dataset': None,
│   'video.input_format': 'channels_last',
│   'video.index': '',
│   'cpu': False,
│   'first_gpu': False,
│   'last_gpu': False,
│   'gpu': 'auto',
│   'max_edge_length_ratio': 0.25,
│   'dist_penalty_weight': 1.0,
│   'batch_size': 4,
│   'open_in_gui': False,
│   'peak_threshold': 0.2,
│   'max_instances': 1,
│   'tracking.tracker': None,
│   'tracking.max_tracking': None,
│   'tracking.max_tracks': None,
2023-12-23 22:04:04.592476: W tensorflow/core/platform/profile_utils/cpu_utils.cc:128] Failed to get CPU frequency: 0 Hz
│   'tracking.target_instance_count': None,
│   'tracking.pre_cull_to_target': None,
│   'tracking.pre_cull_iou_threshold': None,
│   'tracking.post_connect_single_breaks': None,
│   'tracking.clean_instance_count': None,
│   'tracking.clean_iou_threshold': None,
│   'tracking.similarity': None,
│   'tracking.match': None,
│   'tracking.robust': None,
│   'tracking.track_window': None,
│   'tracking.min_new_track_points': None,
│   'tracking.min_match_points': None,
│   'tracking.img_scale': None,
│   'tracking.of_window_size': None,
│   'tracking.of_max_levels': None,
│   'tracking.save_shifted_instances': None,
│   'tracking.kf_node_indices': None,
│   'tracking.kf_init_frame_count': None
}

INFO:sleap.nn.inference:Failed to query GPU memory from nvidia-smi. Defaulting to first GPU.
Metal device set to: Apple M2 Max
2023-12-23 22:04:06.796696: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled.
2023-12-23 22:04:06.874764: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: -47 } dim { size: -48 } dim { size: -49 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -15 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -15 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 12 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -15 } dim { size: -50 } dim { size: -51 } dim { size: 1 } } }
2023-12-23 22:04:06.875581: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_UINT8 } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_UINT8 shape { dim { size: 4 } dim { size: 1200 } dim { size: 1920 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -16 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -16 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 12 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -16 } dim { size: -63 } dim { size: -64 } dim { size: 3 } } }
2023-12-23 22:04:06.878681: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: -131 } dim { size: -132 } dim { size: -133 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -22 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -22 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 12 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -22 } dim { size: -135 } dim { size: -136 } dim { size: 1 } } }
loc("mps_select"("(mpsFileLoc): /AppleInternal/Library/BuildRoots/0032d1ee-80fd-11ee-8227-6aecfccc70fe/Library/Caches/com.apple.xbs/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Core/Files/MPSGraphUtilities.mm":294:0)): error: 'anec.gain_offset_control' op result #0 must be 4D/5D memref of 16-bit float or 8-bit signed integer or 8-bit unsigned integer values, but got 'memref<4x3x1x1xi1>'
Versions:
SLEAP: 1.3.3
TensorFlow: 2.9.2
Numpy: 1.22.4
Python: 3.9.15
OS: macOS-14.2.1-arm64-arm-64bit

System:
GPUs: 1/1 available
  Device: /physical_device:GPU:0
         Available: True
2023-12-23 22:04:08.017111: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled.
        Initalized: False
     Memory growth: True
2023-12-23 22:04:08.096753: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: -58 } dim { size: -59 } dim { size: -60 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -15 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -15 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 12 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -15 } dim { size: -61 } dim { size: -62 } dim { size: 1 } } }
2023-12-23 22:04:08.097681: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_UINT8 } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_UINT8 shape { dim { size: -25 } dim { size: 1200 } dim { size: 1920 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -16 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -16 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 12 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -16 } dim { size: -74 } dim { size: -75 } dim { size: 3 } } }
2023-12-23 22:04:08.100799: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: -142 } dim { size: -143 } dim { size: -144 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -22 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -22 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 12 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -22 } dim { size: -146 } dim { size: -147 } dim { size: 1 } } }

loc("mps_select"("(mpsFileLoc): /AppleInternal/Library/BuildRoots/0032d1ee-80fd-11ee-8227-6aecfccc70fe/Library/Caches/com.apple.xbs/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Core/Files/MPSGraphUtilities.mm":294:0)): error: 'anec.gain_offset_control' op result #0 must be 4D/5D memref of 16-bit float or 8-bit signed integer or 8-bit unsigned integer values, but got 'memref<1x3x1x1xi1>'
Finished inference at: 2023-12-23 22:04:08.754709
Total runtime: 5.680121898651123 secs
Predicted frames: 13/13
Provenance:
{
│   'model_paths': [
│   │   '/Users/ugne/Dropbox/pregnant/models/231223_213850.centroid.n=8/training_config.json',
Process return code: 0
Using already trained model for centroid: /Users/ugne/Dropbox/pregnant/models/231223_213850.centroid.n=8/training_config.json
Using already trained model for centered_instance: /Users/ugne/Dropbox/pregnant/models/231223_214910.centered_instance.n=8/training_config.json
Command line call:
sleap-track /Users/ugne/Dropbox/pregnant/labels.v001_pupx1.slp --video.index 0 --frames 0,-14999 -m /Users/ugne/Dropbox/pregnant/models/231223_213850.centroid.n=8/training_config.json -m /Users/ugne/Dropbox/pregnant/models/231223_214910.centered_instance.n=8/training_config.json --tracking.tracker none --max_instances 1 -o /Users/ugne/Dropbox/pregnant/predictions/labels.v001_pupx1.slp.231223_220450.predictions.slp --verbosity json --no-empty-frames

Started inference at: 2023-12-23 22:04:54.576367
Args:
{
│   'data_path': '/Users/ugne/Dropbox/pregnant/labels.v001_pupx1.slp',
│   'models': [
│   │   '/Users/ugne/Dropbox/pregnant/models/231223_213850.centroid.n=8/training_config.json',
│   │   '/Users/ugne/Dropbox/pregnant/models/231223_214910.centered_instance.n=8/training_config.json'
│   ],
│   'frames': '0,-14999',
│   'only_labeled_frames': False,
│   'only_suggested_frames': False,
│   'output': '/Users/ugne/Dropbox/pregnant/predictions/labels.v001_pupx1.slp.231223_220450.predictions.slp',
2023-12-23 22:04:55.185517: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:305] Could not identify NUMA node of platform GPU ID 0, defaulting to 0. Your kernel may not have been built with NUMA support.
2023-12-23 22:04:55.185676: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:271] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 0 MB memory) -> physical PluggableDevice (device: 0, name: METAL, pci bus id: <undefined>)
│   'no_empty_frames': True,
│   'verbosity': 'json',
│   'video.dataset': None,
│   'video.input_format': 'channels_last',
│   'video.index': '0',
│   'cpu': False,
│   'first_gpu': False,
│   'last_gpu': False,
│   'gpu': 'auto',
│   'max_edge_length_ratio': 0.25,
│   'dist_penalty_weight': 1.0,
│   'batch_size': 4,
│   'open_in_gui': False,
│   'peak_threshold': 0.2,
│   'max_instances': 1,
│   'tracking.tracker': 'none',
│   'tracking.max_tracking': None,
│   'tracking.max_tracks': None,
2023-12-23 22:04:56.094568: W tensorflow/core/platform/profile_utils/cpu_utils.cc:128] Failed to get CPU frequency: 0 Hz
│   'tracking.target_instance_count': None,
│   'tracking.pre_cull_to_target': None,
│   'tracking.pre_cull_iou_threshold': None,
│   'tracking.post_connect_single_breaks': None,
│   'tracking.clean_instance_count': None,
│   'tracking.clean_iou_threshold': None,
│   'tracking.similarity': None,
│   'tracking.match': None,
│   'tracking.robust': None,
│   'tracking.track_window': None,
│   'tracking.min_new_track_points': None,
│   'tracking.min_match_points': None,
│   'tracking.img_scale': None,
│   'tracking.of_window_size': None,
│   'tracking.of_max_levels': None,
│   'tracking.save_shifted_instances': None,
│   'tracking.kf_node_indices': None,
│   'tracking.kf_init_frame_count': None
}

INFO:sleap.nn.inference:Failed to query GPU memory from nvidia-smi. Defaulting to first GPU.
Metal device set to: Apple M2 Max
2023-12-23 22:04:57.975830: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled.
2023-12-23 22:04:58.054578: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: -47 } dim { size: -48 } dim { size: -49 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -15 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -15 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 12 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -15 } dim { size: -50 } dim { size: -51 } dim { size: 1 } } }
2023-12-23 22:04:58.055451: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_UINT8 } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_UINT8 shape { dim { size: 4 } dim { size: 1200 } dim { size: 1920 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -16 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -16 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 12 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -16 } dim { size: -63 } dim { size: -64 } dim { size: 3 } } }
2023-12-23 22:04:58.058697: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: -131 } dim { size: -132 } dim { size: -133 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -22 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -22 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 12 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -22 } dim { size: -135 } dim { size: -136 } dim { size: 1 } } }
loc("mps_select"("(mpsFileLoc): /AppleInternal/Library/BuildRoots/0032d1ee-80fd-11ee-8227-6aecfccc70fe/Library/Caches/com.apple.xbs/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Core/Files/MPSGraphUtilities.mm":294:0)): error: 'anec.gain_offset_control' op result #0 must be 4D/5D memref of 16-bit float or 8-bit signed integer or 8-bit unsigned integer values, but got 'memref<4x3x1x1xi1>'
Versions:
SLEAP: 1.3.3
TensorFlow: 2.9.2
Numpy: 1.22.4
Python: 3.9.15
OS: macOS-14.2.1-arm64-arm-64bit

System:
GPUs: 1/1 available
  Device: /physical_device:GPU:0
         Available: True
        Initalized: False
     Memory growth: True

Traceback (most recent call last):
  File "/Users/ugne/mambaforge3/envs/sleap/bin/sleap-track", line 33, in <module>

    sys.exit(load_entry_point('sleap==1.3.3', 'console_scripts', 'sleap-track')())
  File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/sleap/nn/inference.py", line 5424, in main
    labels_pr = predictor.predict(provider)
  File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/sleap/nn/inference.py", line 526, in predict
    self._make_labeled_frames_from_generator(generator, data)
  File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/sleap/nn/inference.py", line 2633, in _make_labeled_frames_from_generator
    for ex in generator:
  File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/sleap/nn/inference.py", line 457, in _predict_generator
    ex = process_batch(ex)
  File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/sleap/nn/inference.py", line 399, in process_batch
    preds = self.inference_model.predict_on_batch(ex, numpy=True)
  File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/sleap/nn/inference.py", line 1069, in predict_on_batch
    outs = super().predict_on_batch(data, **kwargs)
  File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/engine/training.py", line 2230, in predict_on_batch
    outputs = self.predict_function(iterator)
  File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/tensorflow/python/util/traceback_utils.py", line 153, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/tensorflow/python/eager/execute.py", line 54, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.InternalError: Graph execution error:

Detected at node 'top_down_inference_model/find_instance_peaks/model/stack0_dec0_s16_to_s8_interp_bilinear/resize/ResizeBilinear' defined at (most recent call last):
    File "/Users/ugne/mambaforge3/envs/sleap/bin/sleap-track", line 33, in <module>
      sys.exit(load_entry_point('sleap==1.3.3', 'console_scripts', 'sleap-track')())
    File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/sleap/nn/inference.py", line 5424, in main
      labels_pr = predictor.predict(provider)
    File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/sleap/nn/inference.py", line 526, in predict
      self._make_labeled_frames_from_generator(generator, data)
    File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/sleap/nn/inference.py", line 2633, in _make_labeled_frames_from_generator
      for ex in generator:
    File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/sleap/nn/inference.py", line 457, in _predict_generator
      ex = process_batch(ex)
    File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/sleap/nn/inference.py", line 399, in process_batch
      preds = self.inference_model.predict_on_batch(ex, numpy=True)
    File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/sleap/nn/inference.py", line 1069, in predict_on_batch
      outs = super().predict_on_batch(data, **kwargs)
    File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/engine/training.py", line 2230, in predict_on_batch
      outputs = self.predict_function(iterator)
    File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/engine/training.py", line 1845, in predict_function
      return step_function(self, iterator)
    File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/engine/training.py", line 1834, in step_function
      outputs = model.distribute_strategy.run(run_step, args=(data,))
    File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/engine/training.py", line 1823, in run_step
      outputs = model.predict_step(data)
    File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/engine/training.py", line 1791, in predict_step
      return self(x, training=False)
    File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
      return fn(*args, **kwargs)
    File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/engine/training.py", line 490, in __call__
      return super().__call__(*args, **kwargs)
    File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
      return fn(*args, **kwargs)
    File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/engine/base_layer.py", line 1014, in __call__
      outputs = call_fn(inputs, *args, **kwargs)
    File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
      return fn(*args, **kwargs)
    File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/sleap/nn/inference.py", line 2256, in call
      if isinstance(self.instance_peaks, FindInstancePeaksGroundTruth):
    File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/sleap/nn/inference.py", line 2265, in call
      peaks_output = self.instance_peaks(crop_output)
    File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
      return fn(*args, **kwargs)
    File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/engine/base_layer.py", line 1014, in __call__
      outputs = call_fn(inputs, *args, **kwargs)
    File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
      return fn(*args, **kwargs)
    File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/sleap/nn/inference.py", line 2088, in call
      out = self.keras_model(crops)
    File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
      return fn(*args, **kwargs)
    File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/engine/training.py", line 490, in __call__
      return super().__call__(*args, **kwargs)
    File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
      return fn(*args, **kwargs)
    File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/engine/base_layer.py", line 1014, in __call__
      outputs = call_fn(inputs, *args, **kwargs)
    File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
      return fn(*args, **kwargs)
    File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/engine/functional.py", line 458, in call
      return self._run_internal_graph(
    File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/engine/functional.py", line 596, in _run_internal_graph
      outputs = node.layer(*args, **kwargs)
    File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
      return fn(*args, **kwargs)
    File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/engine/base_layer.py", line 1014, in __call__
      outputs = call_fn(inputs, *args, **kwargs)
    File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
      return fn(*args, **kwargs)
    File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/layers/reshaping/up_sampling2d.py", line 129, in call
      return backend.resize_images(
    File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/backend.py", line 3432, in resize_images
      x = tf.image.resize(x, new_shape, method=interpolations[interpolation])
Node: 'top_down_inference_model/find_instance_peaks/model/stack0_dec0_s16_to_s8_interp_bilinear/resize/ResizeBilinear'
Detected at node 'top_down_inference_model/find_instance_peaks/model/stack0_dec0_s16_to_s8_interp_bilinear/resize/ResizeBilinear' defined at (most recent call last):
    File "/Users/ugne/mambaforge3/envs/sleap/bin/sleap-track", line 33, in <module>
      sys.exit(load_entry_point('sleap==1.3.3', 'console_scripts', 'sleap-track')())
    File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/sleap/nn/inference.py", line 5424, in main
      labels_pr = predictor.predict(provider)
    File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/sleap/nn/inference.py", line 526, in predict
      self._make_labeled_frames_from_generator(generator, data)
    File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/sleap/nn/inference.py", line 2633, in _make_labeled_frames_from_generator
      for ex in generator:
    File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/sleap/nn/inference.py", line 457, in _predict_generator
      ex = process_batch(ex)
    File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/sleap/nn/inference.py", line 399, in process_batch
      preds = self.inference_model.predict_on_batch(ex, numpy=True)
    File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/sleap/nn/inference.py", line 1069, in predict_on_batch
      outs = super().predict_on_batch(data, **kwargs)
    File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/engine/training.py", line 2230, in predict_on_batch
      outputs = self.predict_function(iterator)
    File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/engine/training.py", line 1845, in predict_function
      return step_function(self, iterator)
    File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/engine/training.py", line 1834, in step_function
      outputs = model.distribute_strategy.run(run_step, args=(data,))
    File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/engine/training.py", line 1823, in run_step
      outputs = model.predict_step(data)
    File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/engine/training.py", line 1791, in predict_step
      return self(x, training=False)
    File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
      return fn(*args, **kwargs)
    File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/engine/training.py", line 490, in __call__
      return super().__call__(*args, **kwargs)
    File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
      return fn(*args, **kwargs)
    File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/engine/base_layer.py", line 1014, in __call__
      outputs = call_fn(inputs, *args, **kwargs)
    File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
      return fn(*args, **kwargs)
    File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/sleap/nn/inference.py", line 2256, in call
      if isinstance(self.instance_peaks, FindInstancePeaksGroundTruth):
    File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/sleap/nn/inference.py", line 2265, in call
      peaks_output = self.instance_peaks(crop_output)
    File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
      return fn(*args, **kwargs)
    File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/engine/base_layer.py", line 1014, in __call__
      outputs = call_fn(inputs, *args, **kwargs)
    File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
      return fn(*args, **kwargs)
    File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/sleap/nn/inference.py", line 2088, in call
      out = self.keras_model(crops)
    File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
      return fn(*args, **kwargs)
    File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/engine/training.py", line 490, in __call__
      return super().__call__(*args, **kwargs)
    File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
      return fn(*args, **kwargs)
    File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/engine/base_layer.py", line 1014, in __call__
      outputs = call_fn(inputs, *args, **kwargs)
    File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
      return fn(*args, **kwargs)
    File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/engine/functional.py", line 458, in call
      return self._run_internal_graph(
    File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/engine/functional.py", line 596, in _run_internal_graph
      outputs = node.layer(*args, **kwargs)
    File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
      return fn(*args, **kwargs)
    File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/engine/base_layer.py", line 1014, in __call__
      outputs = call_fn(inputs, *args, **kwargs)
    File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
      return fn(*args, **kwargs)
    File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/layers/reshaping/up_sampling2d.py", line 129, in call
      return backend.resize_images(
    File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/backend.py", line 3432, in resize_images
      x = tf.image.resize(x, new_shape, method=interpolations[interpolation])
Node: 'top_down_inference_model/find_instance_peaks/model/stack0_dec0_s16_to_s8_interp_bilinear/resize/ResizeBilinear'
2 root error(s) found.
  (0) INTERNAL:  Missing 0-th output from {{node top_down_inference_model/find_instance_peaks/model/stack0_dec0_s16_to_s8_interp_bilinear/resize/ResizeBilinear}}
	 [[top_down_inference_model/find_instance_peaks/PartitionedCall/cond/else/_82/cond/add_1/_402]]
  (1) INTERNAL:  Missing 0-th output from {{node top_down_inference_model/find_instance_peaks/model/stack0_dec0_s16_to_s8_interp_bilinear/resize/ResizeBilinear}}
0 successful operations.
0 derived errors ignored. [Op:__inference_predict_function_4953]
systemMemory: 64.00 GB
maxCacheSize: 24.00 GB


Process return code: 1

Screenshots

How to reproduce

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

[talmo]

Hi @uklibaite,

It looks like this is the main error:

Node: 'top_down_inference_model/find_instance_peaks/model/stack0_dec0_s16_to_s8_interp_bilinear/resize/ResizeBilinear'
2 root error(s) found.
  (0) INTERNAL:  Missing 0-th output from {{node top_down_inference_model/find_instance_peaks/model/stack0_dec0_s16_to_s8_interp_bilinear/resize/ResizeBilinear}}
	 [[top_down_inference_model/find_instance_peaks/PartitionedCall/cond/else/_82/cond/add_1/_402]]
  (1) INTERNAL:  Missing 0-th output from {{node top_down_inference_model/find_instance_peaks/model/stack0_dec0_s16_to_s8_interp_bilinear/resize/ResizeBilinear}}

This is currently a bug in the Mac TensorFlow working differently than Windows/Linux in how it handles empty tensors.

We have an issue tracking this (#1100 (comment)) and an in-progress PR to fix it (#1547).

We'll keep you updated once the fix is ready for testing!

Talmo

[shiladityaneuron]

Hi Following the post.

[toniig]

Hi! I am having the same error using a Appl M2 pro! Is there any solution? Thank you!!!

[talmo]

Hi folks,

Just to update with a quick comment: we will revisit this after the v1.4 release (one PR left!) which will also update TensorFlow versions and change the underlying behavior.

We'll take another pass at this after v1.4, but if the fix requires a big redesign of the TensorFlow backend (which is what present evidence points to), it might need to wait until we fully switch over to our new PyTorch backend in the coming months.

We'll keep everyone updated here and this issue pinned until either path is clear.

Sorry about that!

Talmo

[calebvogt]

Just chiming in here to say that I recently acquired an M3 pro and I ran into the same error with a top down model. Prediction runs for maybe ~20 minutes or so before throwing the error.