API Needs a Subscribe Interface

[jwise-mfg]

Multiple requests to include a Subscribe interface, including #3

There is nothing in here around subscriptions? It would be nice to have an interface that says "subscribe to elementID X" and get streamed data changes

[jwise-mfg]

Copying in comments from Randy Armstrong on #3

Subscribing to a LiveValue is a complex problem when accessing a system where the underlying data is not already being monitored. Questions like sampling rates, deadbands, queue lengths need to be addressed, Also the technology available for 'streaming' varies widely. MQTT has intrinsic QoS capabilities that can handle network interruptions. Others like WebSockets or long polling with HTTPS need and API that can manage the QoS features.

The working group met on this topic, and agreed we'd like to invite more outside input, marking as Help Wanted. A SME has been invited for early next year.

[jwise-mfg]

Working Group Discussion Points:

• Subscribe should be a different end point
• Would be nice to define "smart" subscriptions -- eg: clients want to subscribe to a particular attribute of an Element
• Other requests:
  • if you subscribe to an object, do you get only the change?
  • filter on the values (deadband/threshold)
  • also subscribe to metadata
  • also subscribe to relationships
  • also subscribe to events
  • subscription may need to have data rate (parameter and max)

[randy-armstrong]

subscribing to an 'object' makes it too easy for clients to do stupid things that result in a DoS attack.
suggest a restricted variation:

• subscribe to properties.
• subscribe to immediate variables.

Otherwise, each element needs to be explicitly selected. This covers metadata and data.

subscribe to events requires an entire event model to be defined which is not simple if a general purpose model is desired.
suggest a variation on the UA event model where values are are returned as nested json objects instead of flat list.

Propose event field selection:

{
   "EventId": {}, 
   "EventType":{},
   "Time":{},
   "Message": {},
   "Severity": {},
   "EnabledState": {
       "Id": {},
       "TransitionTime": {}
  }
}

And event notification:

{
   "EventId": "eYDHKvZHQdSjbIj7jjMqpw==",
   "EventType":"nsu=http://acme.com/wilyecoyote;s=Grape",
   "Time": "2024-07-15T18:45:01Z",
   "Message": {
      "Text": "Orange",
      "Locale": "en-CA"
    },
    "Severity": 500,
    "EnabledState": {
       "Id": true,
       "TransitionTime": "2024-07-15T18:45:01Z"
    }
}

[barnstee]

I'm for publishing subscribed data to an MQTT or Kafka topic to an external broker. This is the safest approach.

[jwise-mfg]

At this stage (RFC) we're trying to avoid specifying mechanisms -- this is closer to a BRD than a FuncSpec.

[jwise-mfg]

Notes, feedback, and summarization resulting from 2025-01-28 Working Group meeting with SME:

Consensus

Query and Subscription Are Individual Interfaces

• A Query and a Subscription can be two separate calls. A Query may be complex, may need to accept wildcards or regular expressions, and may support depth traversal; the result of the Query includes a collection of ElementIds that satisfy the query. The subsequent Subscription call can pass back an array of ElementIds to be subscribed to.
• As an example: an Pump health application may query for Types to discover if the underlying platform has a definition for Pumps. If Pump Type is present, the app may query for ObjectsByType(Pump) to discover Pump object instances. If a Pump instance is present, the underlying platform will return the structure of the Pump instance object, including ElementIds for the Pump instance and all its properties. The app may then call subscribe with the array of ElementIds belonging to the Pump instance discovered in the previous queries.
• We should NOT allow subscription to Objects or subscriptions to traversals through Object relationships (eg: structural subscriptions). The client should use the Query-then-Subscribe pattern to first find the resultset of interest, then Subscribe to the relevant properties. The response payload to a Subscribe call should be explicit about successful/failed/disallowed subscriptions.

Subscribed Items May Change

• A mechanism needs to be defined to inform a client/app if the subscribed elements have had structural/toplogical/metadata changes, or otherwise expires. For example, if a subscribed property no longer exists or has been re-parented to a different object.
• Some existing commercial platforms have a mechanism for subscribing to structural changes, but it was agreed this can be resolved with a (polled) query -- at least in v1.0

Open Questions

Optimizing Subscriptions

The query-then-subscribe pattern provides some optimization, by allowing the client to refine the elements of interest before the subscription, but there are still some open considerations...

• Subscriptions often use deadbands, to avoid over-sending. This may need to be either an implementation-specific setting, or a subscription parameter.
• Filtering, matching, or wildcards as Subscribe interface parameters may still be necessary. We need to explore some use-cases in-depth to better establish the parameters for the Subscribe interface.

[griemenschneider]

Subscription capabilities are absolutely required. However, taking a step back I think the goal is to move the data to a real time display, third party application, or BI tool. The mainstream capabilities of modern BI tools or software must then first be considered before defining the requirements of the API. Most BI tools today simply do not support GraphQL and I have not heard of any supporting subscription APIs. I see the note above about not wanting to name specific mechanisms, however, understanding that the goal is to make the data consumable and easy to access in real-time I would have to agree with the above comment that the most logical way would be to require smart manufacturing platforms to push or expose the data in the form of a broker, not an API

[aronsemle]

"Some existing commercial platforms have a mechanism for subscribing to structural changes, but it was agreed this can be resolved with a (polled) query -- at least in v1.0"

Just including this for reference. Agree we can add it later.

This is how Aveva does it with Asset Framework. I'm not saying we need all these options, but as a dev this worked well for triggering a refresh of subscriptions/clear cache etc.

Aveva - FindChangedItems API

[jwise-mfg]

Document subscription as a capability of a Value interface (as summarized above), and review with WG for next meeting.

[jwise-mfg]

Proposed language here: 98a14e1

[jwise-mfg]

WG discussion points:

• Clarify how "wildcards" can be handled in query-the-subscribe pattern
• Subscribing to whole Objects is still attractive, is there an approach where I can ensure that a root ElementID is just an object (and only a discrete object) and not a deep traversal? Might be better for the system to decline the request if it can't tell the difference.

[aronsemle]

How would this relate to example for subscribing to the root of an OPC UA server or a branch through the API? How about # in an MQTT broker? Assume I'm implementing the API and I expose the root as an ElementID. Clients can subscribe to it, and I provide data changes up as an object of the tags that have changed.

Do I have the flexibility to allow this as an implementer?

OPC UA requires you to browse the full branch, subscribe to all the individual tags, and then pull it all back together. It would be nice if this were an option in our API for implementers, but not a requirement. I.e. there was flexibility.

For example, maybe my UA server is pretty small and this isn't an issue. Maybe it's huge and I want some control over how I divide it up.

---

WG discussion points:

• Clarify how "wildcards" can be handled in query-the-subscribe pattern
  *** Subscribing to whole Objects is still attractive, is there an approach where I can ensure that a root ElementID is just an object (and only a discrete object) and not a deep traversal? Might be better for the system to decline the request if it can't tell the difference.**