Find the best time to set properties.metadata_id as required

[antje-s]

In the long term, the metadata should be required so that no data is disseminated for which there is no metadata.
The GB monitoring metric
wmo_wis2_gb_messages_no_metadata_total
could be used to determine a good point in time.
(It should be ensured that no fake metadata is accepted but only metadata that exists in the GDC.)

Open questions:

1. How to deal with differences in GDCs?
   (e.g. if the answer is negative (not existing), at least one other GDC should be requested)
2. Which threshold value should be used? Neither the data flow should be stopped nor should the limit value not be reached in real life (cf. WIS1)

[kaiwirt]

In my opinion this should be required before the start of the operational phase. Otherwise, setting this property might break an (then) operational data exchange.

[tomkralidis]

TT-WISMD 2024-10-25:

• consider changing language in Manual/Guide or WNM
• add grace period for implementation, schema requirement
• monitor over time for uptake and report SC-IMT

[amilan17]

A change to the specification that requires all notification messages to include a metadata_id and then enforcing this validation by not accepting invalid notification messages would probably be a breaking change. Therefore, this will probably need to be introduced at a session with a very clear strategy for implementation.

[josusky]

I recently went through a metadata creation exercise with a small WIS2 node and it was pretty straight forward. The smaller is the center, the less metadata records it needs. The larger data providers have (need) more metadata records but also have more resources. All in all, making metadata_id required would not be as painful as it might seem. But a gradual approach is OK, we can start kindly asking data providers to add the metadata_id and make it a strong requirement later on.

[tomkralidis]

From ET-W2IT 2024-11-11 meeting:

• 2025-01-01: enable strict WTH checking
• 2025-04-01: enable WNM schema validation
• 2025-09-01: enable requirement of properties.metadata_id

[josusky]

@tomkralidis could you please elaborate? What do you mean by "enable ..."? Who will enable and where?

[tomkralidis]

@josusky the above "enablements" are based on GB checks, so:

From ET-W2IT 2024-11-11 meeting:

• 2025-01-01: GBs to enable strict WTH checking
• 2025-04-01: GBs to enable WNM schema validation
• 2025-09-01: GBs to enable requirement of properties.metadata_id

[amilan17]

@tomkralidis, I'm confused. These are the decisions in the notes from the 11-11-ET-W2IT meeting ... ?

Timeline for GB to check:
- to enable the Topic Hierarchy control from 1 January 2025
- Schema validation: 1 April 2025
- Metadata validation: 1 September 2025

[amilan17]

When we introduce the requirement for metadata-id in the WNM, it would be useful to include requirements or guidance on how this field supports the requirement for discovery metadata as stated in the Manual on WIS

for example:
3.3.3 Describe data with discovery metadata 3.3.3.1 An NC shall create discovery metadata about the data it manages and ensure that these discovery metadata remain up to date.

@maaikelimper

[tomkralidis]

TT-WISMD 2025-02-19

• ACTION: @tomkralidis to validate w/ ET-W2IT on what "metadata validation" means

[6a6d74]

Added to the agenda for ET-W2IT on Monday 23-Feb.

[tomkralidis]

To clarify:

2025-09-01:
As per https://wmo-im.github.io/wis2-guide/guide/wis2-guide-APPROVED.html#_2_7_3_1_technical_considerations, GBs to implement the following:

A Global Broker will check if a discovery metadata record exists corresponding to the topic on which a message has been published. If there is no corresponding discovery metadata record, the Global Broker will discard non-compliant messages and will raise an alert. The metric wmo_wis2_gb_messages_no_metadata_total will be increased by 1. The Global Broker should not request information from a Global Discovery Catalogue for each notification message but should keep a cache of all valid topics for every centre-id.

INFCOM-4:

• introduce breaking change in WNM (properties.metadata_id required for validation)

[tomkralidis]

2025-09-01 workaround support in wmo-im/wis2-guide#185