Stuck: Detection From One Camera on ALL

[gniknalu]

Back to an issue that caused all the trouble from the other tickets I opened. (My apologies!) This issue happens with the Dev branch and with the Latest Branch but not when using the CPU (instead of GPU).

Let's use the names, for simplicity sake, Camera 1, Camera 2, Camera 3. When I start viseron, Camera 1 detects a car. That detection appears on Camera 1 as well as on Camera 2 and Camera 3 and no other detection takes place. (See images from Camera 2 and Camera 3)

Debugging is turned on for everything. I do not see the actual "detection" that took place for Camera 1 in the log (so is it from cache??). There are no errors and everything looks "normal". But when I walk outside, none of the cameras 'detect' me and there is nothing in the logs that there was anything detected.

As you can see, the system is not under any significant load:

 nvidia-smi
Fri Sep 10 09:42:22 2021
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.91.03    Driver Version: 460.91.03    CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce GTX 107...  Off  | 00000000:01:00.0 Off |                  N/A |
| 34%   62C    P2    48W / 180W |   1734MiB /  8118MiB |     49%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

CONTAINER ID   NAME      CPU %     MEM USAGE / LIMIT     MEM %     NET I/O          BLOCK I/O       PIDS
13dcb36b246a   viseron   136.07%   1.888GiB / 29.34GiB   6.43%     993MB / 3.81GB   127kB / 410kB   92

And here is the start-up - which is similar in the "latest" branch as with the "dev" branch:

docker logs -f viseron
[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] 10-adduser: executing...
************************ UID/GID *************************
User uid:    0
User gid:    0
************************** Done **************************
[cont-init.d] 10-adduser: exited 0.
[cont-init.d] 20-gid-video-device: executing...
[cont-init.d] 20-gid-video-device: exited 0.
[cont-init.d] 30-edgetpu-permission: executing...
************** Setting EdgeTPU permissions ***************
Coral Vendor IDs:
"1a6e"
"18d1"
No EdgeTPU USB device was found
************************** Done **************************
[cont-init.d] 30-edgetpu-permission: exited 0.
[cont-init.d] 40-set-env-vars: executing...
****** Checking for hardware acceleration platforms ******
OpenCL is available!
VA-API cannot be used
CUDA is available!
*********************** Done *****************************
[cont-init.d] 40-set-env-vars: exited 0.
[cont-init.d] 50-check-if-rpi: executing...
********** Checking if we are running on an RPi **********
Not running on any supported RPi
*********************** Done *****************************
[cont-init.d] 50-check-if-rpi: exited 0.
[cont-init.d] 55-check-if-jetson: executing...
****** Checking if we are running on a Jetson Board ******
Not running on any supported Jetson board
*********************** Done *****************************
[cont-init.d] 55-check-if-jetson: exited 0.
[cont-init.d] 60-ffmpeg-path: executing...
****************** Getting FFmpeg path *******************
FFmpeg path: /usr/local/bin/ffmpeg
*********************** Done *****************************
[cont-init.d] 60-ffmpeg-path: exited 0.
[cont-init.d] done.
[services.d] starting services
[services.d] done.
[2021-09-10 09:51:07] [INFO    ] [viseron                 ] - -------------------------------------------
[2021-09-10 09:51:07] [INFO    ] [viseron                 ] - Initializing...
[2021-09-10 09:51:07] [DEBUG   ] [viseron.watchdog        ] - Starting scheduler
[2021-09-10 09:51:07] [DEBUG   ] [viseron                 ] - Starting cleanup scheduler
[2021-09-10 09:51:07] [DEBUG   ] [viseron                 ] - Running initial cleanup
[2021-09-10 09:51:07] [DEBUG   ] [viseron.cleanup         ] - Running cleanup

Is there a way to 'clear' a detection? Or to clear the cache? Or to find out why no further detections are taking place? What other debugging can I provide?

Reference #194

[roflcoopter]

There is something strange goin on in the background that's causing this.
It happened before and I solved that by implementing a lock that is acquired when running object detection, because spawning processes for the recorder for example would make the detections duplicate.

Since then a lot has changed and there might be other places that need this locking mechanism. I'll check on it soon and let you know!

[gniknalu]

I'll check on it soon and let you know!

Thank you!

[roflcoopter]

I have been looking at this for a while and i cant for the life of me reproduce it.
Can you post your config?

[gniknalu]

I have been looking at this for a while and i cant for the life of me reproduce it.
Can you post your config?

It's the same config as was posted earlier.

It's most likely a GPU issue. (My NEWER GPU!!) I just migrated to another Ubuntu node with an older GPU and it works there. Can't figure out why it, all of a sudden, started doing this. Going to reload all the Nvidia drivers. Please close this one.

[roflcoopter]

Actually as we speak it just happened for me, i setup 10 cameras to run simultaneously and then it eventually occured.

Gonna do some tweaking and see if i can find a fix

[roflcoopter]

This is fixed now in dev

The issue happens when subprocess.Popen (which is done a lot when a camera is starting up) is called at the same time as object detection is running (I have no idea why, must be some low level bug in OpenCV)
I had previously added a lock which is acquired before each Popen call aswell as the inference call.
When doing some refactoring i missed adding this to a few new places.

I could consistently reproduce it before but now i cannot anymore.
Would live if you could try but i guess that OpenCL error is stopping you from doing that? 😢

[gniknalu]

This is fixed now in dev
I could consistently reproduce it before but now i cannot anymore.
Would live if you could try but i guess that OpenCL error is stopping you from doing that? 😢

I'll give it a try - if I can get past the OpenCL error.
I'll rebuild one of the boxes if the error persists. Right now both are running the "latest"
Thank you for all the work you put into this!

[gniknalu]

This is fixed now in dev
I could consistently reproduce it before but now i cannot anymore.
Would live if you could try but i guess that OpenCL error is stopping you from doing that? 😢

Three observations after updating the 'bad' node to dev:

1. Yes I still get the OpenCL error
2. But the OpenCL error is no longer fatal (viseron is running even with the error!)
3. And No longer are there detections from the one camera on all the others!

So now I'll be returning the higher performing node, using the dev version, into service!

Thank you!

[gniknalu]

Fixed!