WARNING:tensorflow:5 out of the last 5 calls to <function Model.make_predict_function.<locals>.predict_function at 0x7f83d8166950> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.

[mikegerber]

calamari-predict issues lots of TensorFlow warnings like these:

WARNING:tensorflow:5 out of the last 5 calls to <function Model.make_predict_function.<locals>.predict_function at 0x7f83d8166950> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.

To reproduce use the dta19 1828-platen_gedichte files from GT4HistOCR and our GT4HistOCR model trained for Calamari 1.0:

% pwd
/var/tmp/dta19/1828-platen_gedichte
% calamari-predict --files *.png  --checkpoint /home/mike/devel/ocrd_calamari/gt4histocr-calamari/*.json
[ .... more output ... ]
Total params: 1,597,915
Trainable params: 1,597,915
Non-trainable params: 0
__________________________________________________________________________________________________
None
Prediction:   0%|                                                                                                         | 4/5103 [00:16<5:55:52,  4.19s/it]WARNING:tensorflow:5 out of the last 5 calls to <function Model.make_predict_function.<locals>.predict_function at 0x7f2014526950> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.
WARNING:tensorflow:5 out of the last 5 calls to <function Model.make_predict_function.<locals>.predict_function at 0x7f20a5b14320> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.
WARNING:tensorflow:5 out of the last 5 calls to <function Model.make_predict_function.<locals>.predict_function at 0x7f20a53c8ef0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.
[ .... more output ... ]

% pip list | egrep -i 'tensorflow|keras|calamari'
calamari-ocr           1.0.5
Keras-Preprocessing    1.1.2
ocrd-calamari          0.0.6     /home/mike/devel/ocrd_calamari
tensorflow             2.2.0
tensorflow-estimator   2.2.0
tensorflow-gpu         2.2.0

See also this issue why this might prove to be a major performance problem. My experiments with ocrd_calamari using Calamari 1.0 are currently 5 times slower compared to Calamari 0.3.5, 2 times with optimization.

Also note that I am testing this on the CPU, which should be fine for prediction.

[mikegerber]

Installing TensorFlow 2.3.0rc2 reduces the warnings to ~ 10. (See also: tensorflow/tensorflow#41347)

[mikegerber]

@andbue has a possible patch that needs investigating: OCR-D/ocrd_calamari#8 (comment)

[andbue]

I am not sure it helps, unfortunately... I did some testing today with tensorflow 2.3.0rc2 and it seems to be working without any warnings (tested with ~500 lines on CPU and GPU) with current master. Maybe it really is just a bug in 2.2?

% pip list | egrep -i 'tensorflow|keras|calamari'
calamari-ocr           1.0.5
Keras-Preprocessing    1.1.2
tensorflow             2.3.0rc2
tensorflow-estimator   2.3.0

TF nightly/2.4 produces an enormous amount of warnings again:
Executor start aborting: Invalid argument: You must feed a value for placeholder tensor 'Placeholder/_2' with dtype string and shape [1,1] (same on CPU and GPU). Let's hope it's another bug that will be fixed in 2.5...

[mikegerber]

I am not sure it helps, unfortunately... I did some testing today with tensorflow 2.3.0rc2 and it seems to be working without any warnings (tested with ~500 lines on CPU and GPU) with current master. Maybe it really is just a bug in 2.2?

I believe it is connected to the varying dimensions of the input data, so depending on your test data the problem may not show. The data above does trigger it, even on TF 2.3 but to a smaller extent.

[andbue]

The warning is only triggered when using more than one model and even then only at the start of the prediction. It is unclear to me why tensorflow would complain if an operation, but if I understand tensorflow/python/eager/def_function.py correctly, it builds up a counter using some key as index and counts all calls to that function, even if we have several instances of that function.
When I predict using two models and lower the warning threshold from 5 to 1, I get:

1 out of the last 1 calls to <function Model.make_predict_function.<locals>.predict_function at 0x14681c9d4b80> triggered tf.function retracing. 
2 out of the last 2 calls to <function Model.make_predict_function.<locals>.predict_function at 0x14681c91d160> triggered tf.function retracing. 
3 out of the last 3 calls to <function Model.make_predict_function.<locals>.predict_function at 0x14681c9d4b80> triggered tf.function retracing.
4 out of the last 4 calls to <function Model.make_predict_function.<locals>.predict_function at 0x14681c91d160> triggered tf.function retracing.
5 out of the last 5 calls to <function Model.make_predict_function.<locals>.predict_function at 0x14681c9d4b80> triggered tf.function retracing.
6 out of the last 6 calls to <function Model.make_predict_function.<locals>.predict_function at 0x14681c91d160> triggered tf.function retracing.

As you can see, the warnings are alternately triggered the two instances 0x14681c9d4b80 and 0x14681c91d160. I guess this means that just the first three calls to each graph function need tracing. The warnings are a bit annoying, but I don't think they are meaningful in our case.

[mikegerber]

Alright, I did some CPU-only measurements and I don' see a performance problem with 1.0:

calamari-predict v0.3.5:

Prediction of 5 models took 1206.796199798584s
Average sentence confidence: 99.64%
Prediction of 5 models took 1194.413318157196s
Average sentence confidence: 99.64%

calamari-predict v1.0.5:

Prediction of 5 models took 739.4088222980499s
Average sentence confidence: 99.59%
Prediction of 5 models took 770.8834066390991s
Average sentence confidence: 99.59%

Calamari 1.0.5 (it's actually master) uses TF 2.3, here installed from PyPI. Those TF packages have optimizations for AVX2, so I expect some of the improvement(!) to come from there.

• Reproducer: https://github.com/mikegerber/2020-10-calamari1-performance
• Full log https://raw.githubusercontent.com/mikegerber/2020-10-calamari1-performance/master/full-log.txt

Comparing the first half of the full log (Calamari 0.3.5) with the second half of the log (Calamari 1.0) the TF warning messages are less annoying now (old warnings gone, but less obnoxious new retracing warnings).

TL&DR: The retracing warnings are only a cosmetic problem.

[ChWick]

This could/should be resolved with the new Calamari 2.0 release. Feel free to reopen it if the warning still persists.