Shared Public Resource Considerations

[slominskir]

TLDR: It's really easy to knock myquery over

myquery doesn't scale and not much thought has gone into providing a robust and performant query service for public (authenticated JLab account holders) consumption.

In some ways this is good that myquery is sorta the "crash crumple zone" that easily folds over before a denial of service occurs on MYA. But handling large volumes of data could be useful and is presumably supported using the command line tools. There is nothing stopping users from issuing high volume queries directly from accelerator and hall workstations using the command line. MariaDB probably does a good job handling this.

With myquery there currently aren't any resource limits imposed and monitoring is lite. There is no clustering / scaling.

The jmyapi lib could monitor the MariaDB health to limit overload if the database doesn't already protect itself.

The main purpose of myquery is to make it easier to access archiver data, especially from web apps such as WAVE. Increasingly users are asking to use myquery in a more programmatic fashion. The trend to use big data, machine learning, and AI may be pushing users to do more with archiver data as well.

Related:

• One obstacle to programmatic access from offsite is authentication. We probably need app-scoped authentication tokens in lieu of the current username and password approach, to prevent users from embedding plaintext credentials. This applies generally to all apps on epicsweb and is documented here: Auth for epicsweb.jlab.org should use Keycloak wave#23
• One obstacle to large data is our JSON synchronous responses which consume a huge amount of memory and don't scale. This is documented here: Stream Data #1