Phoebus screen and network issues when accessing PVAccess PV field that does not exist

[Monarda]

As part of our transition from our previous control system to an EPICS PVaccess based control system we have many control screens that have been automatically converted to Phoebus HMIs. Recently we came across strange behaviour on such a HMI. We believe we have isolated the cause to a single automatically generated Text Entry widget which referenced a PV that exists but a field within the PV which does not.

In this case we were attempting to access pva://ISEPTUM::COMMON:SET_CURRENT/CPARAM10

The PV pva://ISEPTUM::COMMON:SET_CURRENT exists but does not have a CPARAM10 field. (It's equivalent in the old control system does.)

This HMI had a number of other Text Entry, Text Update, and LED (Multi State) widgets. When the HMI was loaded these other widgets would intermittently connect, display correct values, and then disconnect. The overall effect was that the HMI would "flicker" between good values and disconnected PVs, with about 50% of the screen working as a maximum.

These PVs are accessed through a p4p PVA Gateway. Inspecting the logs there showed the following messages:

Jan 31 11:17:24 athos.isis.rl.ac.uk bash[1954472]: 2025-01-31T11:17:24.630983991 CRIT pvxs.tcp.io Client Error while processing cmd 0x0d: pvRequest must select at least one field
Jan 31 11:17:24 athos.isis.rl.ac.uk bash[1954472]: Dumping a stack trace of thread 'PVXTCP':
Jan 31 11:17:24 athos.isis.rl.ac.uk bash[1954472]: [    0x7f4db83f5e9b]: /usr/local/lib64/python3.6/site-packages/p4p/../epicscorelibs/lib/libCom.so.7.0.7.99.1(epicsStackTrace+0x4b)
Jan 31 11:17:24 athos.isis.rl.ac.uk bash[1954472]: [    0x7f4db86c0605]: /usr/local/lib64/python3.6/site-packages/p4p/../pvxslibs/lib/libpvxs.so.1.3(_ZN4pvxs6detail11_log_printfEjPKcz+0xa5)
Jan 31 11:17:24 athos.isis.rl.ac.uk bash[1954472]: [    0x7f4db86a4f96]: /usr/local/lib64/python3.6/site-packages/p4p/../pvxslibs/lib/libpvxs.so.1.3(_ZN4pvxs4impl8ConnBase7bevReadEv+0x696)
Jan 31 11:17:24 athos.isis.rl.ac.uk bash[1954472]: [    0x7f4db871baf9]: /usr/local/lib64/python3.6/site-packages/p4p/../pvxslibs/lib/libpvxs.so.1.3(_ZN4pvxs4impl10ServerConn7bevReadEv+0x9)
Jan 31 11:17:24 athos.isis.rl.ac.uk bash[1954472]: [    0x7f4db86a52f1]: /usr/local/lib64/python3.6/site-packages/p4p/../pvxslibs/lib/libpvxs.so.1.3(_ZN4pvxs4impl8ConnBase8bevReadSEP11buffereventPv+0x21)
Jan 31 11:17:24 athos.isis.rl.ac.uk bash[1954472]: [    0x7f4db7c04982]: /usr/local/lib64/python3.6/site-packages/p4p/../pvxslibs/lib/./libevent_core.so.2.2.0(bufferevent_run_deferred_callbacks_locked+0x82)
Jan 31 11:17:24 athos.isis.rl.ac.uk bash[1954472]: [    0x7f4db7c0fa26]: /usr/local/lib64/python3.6/site-packages/p4p/../pvxslibs/lib/./libevent_core.so.2.2.0(event_process_active_single_queue.isra.33+0x446)
Jan 31 11:17:24 athos.isis.rl.ac.uk bash[1954472]: [    0x7f4db7c103cf]: /usr/local/lib64/python3.6/site-packages/p4p/../pvxslibs/lib/./libevent_core.so.2.2.0(event_base_loop+0x49f)
Jan 31 11:17:24 athos.isis.rl.ac.uk bash[1954472]: [    0x7f4db86dc4c9]: /usr/local/lib64/python3.6/site-packages/p4p/../pvxslibs/lib/libpvxs.so.1.3(_ZN4pvxs4impl6evbase3Pvt3runEv+0x179)
Jan 31 11:17:24 athos.isis.rl.ac.uk bash[1954472]: [    0x7f4db841e009]: /usr/local/lib64/python3.6/site-packages/p4p/../epicscorelibs/lib/libCom.so.7.0.7.99.1(epicsThreadCallEntryPoint+0x69)
Jan 31 11:17:24 athos.isis.rl.ac.uk bash[1954472]: [    0x7f4db83f8f8a]: /usr/local/lib64/python3.6/site-packages/p4p/../epicscorelibs/lib/libCom.so.7.0.7.99.1(start_routine+0xda)
Jan 31 11:17:24 athos.isis.rl.ac.uk bash[1954472]: [    0x7f4dbe5de1ca]: /lib64/libpthread.so.0(start_thread+0xea)
Jan 31 11:17:24 athos.isis.rl.ac.uk bash[1954472]: [    0x7f4dbdaaf8d3]: /lib64/libc.so.6(clone+0x43)

In addition, we had an instance of Phoebus that produced a huge network traffic spike. It was displaying the HMI described above when the relevant PVA Gateway was restarted; it appears to have been the PVA Gateway restart that triggered the traffic spike. Once the HMI in Phoebus was closed network traffic returned to normal. The excess network traffic was UDP broadcast traffic so we suspect it was PVAccess search requests. Unfortunately we don't have any useful Phoebus logs to contribute.

[Monarda]

Apologies if this is actually a p4p / pvxs bug! We had to pick a project to file an issue against first and the network traffic issue we observed meant we inclined towards Phoebus first.

[georgweiss]

This seems akin to a similar issue we've seen when Phoebus is attempting to address the MLIS field, which I understand should not be queried by clients.