Should sending timestamped data in an observation request response be a supported use case?

[mjviljan]

Question

Hi,

I'm trying Leshan out with a water meter installed on an actual water pipe. The meter's manufacturer has specified that the device management server should register as an observer on the object instance resource /10262/0/2 (the object being titled "Interval Data Delivery") which includes several instances of meter readings, all of those timestamped.

Now, when I send an observation request from the server to the device, the device responds by sending a success response and the latest object value — which in this case is an array of several instances. The response is sent in SENML_CBOR format. The server fails to parse the response, giving an error

Unable to decode response payload of request [ObserveRequest [path=/10262/0/2 format=null]] from client […]: Unable to decode node[path:/10262/0/2] : value should not be timestamped

This seems to come from LwM2mNodeSenMLDecoder which has a specific check that there is no timestamp in the resolved record.

The actual request is accepted and when the device starts notifying the server of the resource's changes, the messages are parsed totally fine. Based on my investigation, this seems to happen because the notification messages are decoded using the LwM2MDecoder's method decodeTimestampedData, whereas the payload of the observation request response is using the decode method.

So, I get the notifications fine, but I miss the first values the device is sending, because those are sent already in the observation request response and decoding that fails. This is a problem especially if the connection between the device and the server has been down for some reason for a while, as then the device will send a bigger buffer of stored values in one go in the response, and those are now lost.

I've found these issues which I believe are related, as well as the PR and its comments (esp. about ObserveResponse):

• LwM2mNodeSenMLDecoder : Support for timestamped values #1479
• READ RESPONSE with timestamped SenML #1553
• Read Response with timestamped SenML #1610

I've tried to investigate this further on my own but I'm unsure about the following:

• If the PR mentioned above was to include ObserveResponse also, would that fix my case also? (If so, you definitely have my vote on including that also. 😄)
• Is the device following the standard correctly in the first place when it sends the object instance resource value already in the observation request response? I've tried to re-read this part of the LwM2M spec and it seems the response is usually depicted in the pictures with the value, but it's always just an integer or something, so I'm not quite sure. (I know you're not writing the spec and have trouble getting answers from OMA also, so might not know this either.) If it seems the device shouldn't work like this, I could always try to convince the manufacturer about this also...

[sbernard31]

If the #1610 was to include ObserveResponse also, would that fix my case also?

I didn't get exactly what does your device. So Maybe #1610 will helps but this is not sure.
There is case which will not be supported. (See #1553 (comment))

So to answer more precisely, I should know what contains the first observe response which cause issue in your case.
You will also be able to test the PR when ready (it should be soon).
I will let you know, so you can valid it works before next milestones release which should happen in June. (#1612)

Is the device following the standard correctly in the first place when it sends the object instance resource value already in the observation request response?

If I would know exactly what does your device (which kind of response is sent), I can let your know (from my point of view) if :

• it follows the specfication
• it doesn't
• OR it is in the "grey" zone.

[sbernard31]

@mjviljan #1610 should contains support of timestamp for observe. (with limitation defined above ☝️)

[mjviljan]

I tested the PR but unfortunately the data I get doesn't work after those changes either.

I'll see if I can share some sensible data from the device here. If not, I'll close this issue soon.

[sbernard31]

I'll see if I can share some sensible data from the device here.

This could be great because :

1. I can give you my opinion about your device compliancy,
2. Maybe this doesn't work because you find a bug in Leshan and so it would be good to know that to fix it.

[mjviljan]

I've taken an example of a SENML_CBOR message that the device sends. As I mentioned in the opening message, parsing this payload fails if the device sends it as a response to an observe request but succeeds if it's sent as a notification of a changed value later.

The raw payload in hex:

89 a4 21 6a 2f 31 30 32 36 32 2f 30 2f 32 00 62 2f 30 22 fb 41 d9 93 5e b1 00 00 00 08 52 83 19 28 1a 00 83 1a 66 4d 27 70 1a 00 01 51 80 81 00 a2 00 62 2f 31 08 58 40 83 19 28 1a 01 83 1a 66 4b dc f8 19 07 08 98 30 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 a2 00 62 2f 32 08 52 83 19 28 1a 02 83 1a 66 4d 27 70 1a 00 01 51 80 81 00 a2 00 62 2f 33 08 54 83 19 28 1b 00 83 1a 66 4d 27 70 1a 00 01 51 80 81 82 00 00 a2 00 62 2f 34 08 55 83 19 28 1c 00 83 1a 66 4c 0e 30 19 38 40 86 00 00 00 00 00 00 a2 00 62 2f 35 08 58 21 83 19 28 1c 01 83 1a 66 4c 0e 30 19 38 40 86 82 00 00 82 00 00 82 00 00 82 00 00 82 00 00 82 00 00 a2 00 62 2f 36 08 56 83 19 28 1e 00 83 1a 66 4d 27 70 1a 00 01 51 80 81 82 18 25 18 61 a2 00 62 2f 37 08 56 83 19 28 1e 01 83 1a 66 4e 78 f0 1a 00 01 51 80 81 1a 7a fd 00 f0 a2 00 62 2f 38 08 41 80

Which is then parsed to CBOR:

[
  {0: "/0", 8: h'8319281A00831A664D27701A000151808100', -2: "/10262/0/2", -3: 1716353732},
  {0: "/1", 8: h'8319281A01831A664BDCF81907089830000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000'},
  {0: "/2", 8: h'8319281A02831A664D27701A000151808100'},
  {0: "/3", 8: h'8319281B00831A664D27701A0001518081820000'},
  {0: "/4", 8: h'8319281C00831A664C0E3019384086000000000000'},
  {0: "/5", 8: h'8319281C01831A664C0E3019384086820000820000820000820000820000820000'},
  {0: "/6", 8: h'8319281E00831A664D27701A00015180818218251861'},
  {0: "/7", 8: h'8319281E01831A664E78F01A00015180811A7AFD00F0'},
  {0: "/8", 8: h'80'}
]

And when the parsing succeeds, the actual objects inside are:

[10266, 0, [1716332400, 86400, [0]]]
[10266, 1, [1716247800, 1800, [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]]
[10266, 2, [1716332400, 86400, [0]]]
[10267, 0, [1716332400, 86400, [[0, 0]]]]
[10268, 0, [1716260400, 14400, [0, 0, 0, 0, 0, 0]]]
[10268, 1, [1716260400, 14400, [[0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0]]]]
[10270, 0, [1716332400, 86400, [[37, 97]]]]
[10270, 1, [1716418800, 86400, [2063401200]]]
[]

If I debug the device server, LwM2mNodeSenMLDecoder fails the timestamp check for each item.

Does this make any sense, or would you need more info?

[sbernard31]

🤔 at first sight I think this should work ...

I will try to investigate/test that more.

[sbernard31]

I tried to decode the hex value you shared ☝️ with DefaultLwM2mDecoder using SenML_CBOR and I get :

[
  TimestampedLwM2mNode [
      timestamp=2024-05-22T04:55:32Z,
      node=LwM2mMultipleResource [
            id=2,
            values={
                 0=LwM2mResourceInstance [id=0, value=18Bytes, type=OPAQUE],
                1=LwM2mResourceInstance [id=1, value=64Bytes, type=OPAQUE],
                2=LwM2mResourceInstance [id=2, value=18Bytes, type=OPAQUE],
                3=LwM2mResourceInstance [id=3, value=20Bytes, type=OPAQUE],
                4=LwM2mResourceInstance [id=4, value=21Bytes, type=OPAQUE],
                5=LwM2mResourceInstance [id=5, value=33Bytes, type=OPAQUE],
                6=LwM2mResourceInstance [id=6, value=22Bytes, type=OPAQUE],
                7=LwM2mResourceInstance [id=7, value=22Bytes, type=OPAQUE],
                8=LwM2mResourceInstance [id=8, value=1Bytes, type=OPAQUE]},
            type=OPAQUE]
    ]
]

I didn't face issue and works as expected.

I also tried to adapt an integration test and I'm able to decode this payload when I do an ObserveRequest.

So to be clear expected results are :

• before Read Response with timestamped SenML #1610, if a device send that hex value on ObserveRequest, you should get : "value should not be timestamped"
• after Read Response with timestamped SenML #1610, this should work for ReadRequest and ObserveRequest.

After testing, it seems to me that code works as expected.
Is there any chance that you didn't test the right version ?

[mjviljan]

Well, this is embarrassing... Apparently I didn't checkout the PR branch after all before building the test version. 🙄 I thought I did, but it seems I concentrated so much on updating the version numbers in the POM files to make sure I'd get the correct version of Leshan libraries that I forgot to switch branches first. (Edit: Well, I seem to have checked out Jaroslaw's fork, but then used the master branch there...)

I re-tested this now after the PR merge, and it indeed seems to work fine! 🎉

Thank you for the effort in digging into this! 🙇 And sorry for the extra trouble, caused by my mistake!

I believe this will work great when M15 gets released, so I'm closing the issue. Thank you again!

[sbernard31]

No problem 😉
Glad to see that M15 will help you.
Do not hesitate to come back to discuss again about any problem you face with Leshan.

[sbernard31]

@mjviljan there is discussion about #1610 PR.
And we are not sure anymore that this new API is a right way and so we are considering to rollback it 🤔

Discussion are :

• Plan for 2.0.0-M15 🎯 #1612 (comment)
• READ RESPONSE with timestamped SenML #1553 (comment)
• Fix some behavior about Timestamped Node in SenML and Old JSON content Format #1621

Sorry this is maybe not so easy to follow. (but do not hesitate to share your opinion there ☝️)

Anyway, if we revert it and this could impact you because your use case will not work anyway.
Will it be OK for you to just be able to provide a custom encoder to just ignore timestamp validation. (see #1621 (comment)) (and so lost timestamp data ?)

[mjviljan]

Hi,

Thanks for the ping! I've been following the discussion in Plan for 2.0.0-M15 🎯 but not the others.

I was happy about the change as that would fix my use case nicely, but of course you must think of what works best for Leshan in general and go with that.

I can definitely write a custom decoder for my case. And as I believe the payloads I receive will include the measurement timestamps in the actual data, losing the timestamp from the root node should not be an issue. 🤔

[sbernard31]

I forgot to thank you for your quick feedback.

Just in case I try to clarify my mind about timestamp usage in LWM2M : #1623

For now, there is no decision about what we will do about "time fields" usage issue... 😬

[sbernard31]

(You probably already notice that but finally decide to integrate the timestamp feature, this is now available in 2.0.0-M15)

[mjviljan]

I did, and immediately took that version into use. Works nicely. :-)

Thank you again!