Frontend becomes unusable while QC jobs are running

[ch-koehler]

We have a deployment of the CLMS QC Tool Release 2.0.6 for running QC on many deliveries. We have noticed that the UI becomes completely unusable when we start multiple (possibly hundreds of) new QC jobs, probably because update_job_statuses is refreshing the status of jobs at a high rate, causing high CPU load and unresponsive UI elements. Here is a section of the browser log, showing 31 refreshes within 0.5 seconds:

17:03:48.016 refreshing table row in UI with id: 1917 deliveries.js:287:37
17:03:48.029 refreshing table row in UI with id: 1949 deliveries.js:287:37
17:03:48.049 refreshing table row in UI with id: 1901 deliveries.js:287:37
17:03:48.062 refreshing table row in UI with id: 2001 deliveries.js:287:37
17:03:48.075 refreshing table row in UI with id: 1898 deliveries.js:287:37
17:03:48.098 refreshing table row in UI with id: 1962 deliveries.js:287:37
17:03:48.113 refreshing table row in UI with id: 1991 deliveries.js:287:37
17:03:48.131 refreshing table row in UI with id: 1975 deliveries.js:287:37
17:03:48.152 refreshing table row in UI with id: 1951 deliveries.js:287:37
17:03:48.168 refreshing table row in UI with id: 1950 deliveries.js:287:37
17:03:48.182 refreshing table row in UI with id: 1955 deliveries.js:287:37
17:03:48.194 refreshing table row in UI with id: 1911 deliveries.js:287:37
17:03:48.215 refreshing table row in UI with id: 1915 deliveries.js:287:37
17:03:48.229 refreshing table row in UI with id: 1987 deliveries.js:287:37
17:03:48.242 refreshing table row in UI with id: 1923 deliveries.js:287:37
17:03:48.260 refreshing table row in UI with id: 1973 deliveries.js:287:37
17:03:48.280 refreshing table row in UI with id: 1965 deliveries.js:287:37
17:03:48.295 refreshing table row in UI with id: 1892 deliveries.js:287:37
17:03:48.314 refreshing table row in UI with id: 1913 deliveries.js:287:37
17:03:48.328 refreshing table row in UI with id: 1960 deliveries.js:287:37
17:03:48.350 refreshing table row in UI with id: 1919 deliveries.js:287:37
17:03:48.365 refreshing table row in UI with id: 1976 deliveries.js:287:37
17:03:48.379 refreshing table row in UI with id: 1988 deliveries.js:287:37
17:03:48.392 refreshing table row in UI with id: 1905 deliveries.js:287:37
17:03:48.418 refreshing table row in UI with id: 1908 deliveries.js:287:37
17:03:48.431 refreshing table row in UI with id: 1974 deliveries.js:287:37
17:03:48.444 refreshing table row in UI with id: 1989 deliveries.js:287:37
17:03:48.456 refreshing table row in UI with id: 1953 deliveries.js:287:37
17:03:48.468 refreshing table row in UI with id: 1947 deliveries.js:287:37
17:03:48.491 refreshing table row in UI with id: 2008 deliveries.js:287:37
17:03:48.507 refreshing table row in UI with id: 1910 deliveries.js:287:37

The refreshes continue nonstop until all QC jobs are finished, possibly taking several hours.

[tomicekj]

I will discuss this issue with a colleague who has more experience in this and let you know as soon as possible.

[jirikadlec2]

@ch-koehler In QC tool release 2.0.9, the default rate of auto-refreshing job statuses has been changed to every 30 seconds. It can be adjusted by changing the UPDATE_JOB_STATUSES_INTERVAL environment variable.
In addition there is an environment variable UPDATE_JOB_STATUSES. If needed you can set UPDATE_JOB_STATUSES=no to completely switch off auto-refreshing of running job statuses from the frontend page. See https://github.com/eea/copernicus_quality_tools/blob/master/docker/NOTES.environ.txt#L37