049 pod fault with consecutive temp basal pod commands due to logic error

[itsmojo]

Describe the bug

A user incurred an 049 pod fault due to a zero Temp Basal command to the pod being issued one second after a zero Temp Basal command has been sent and confirmed without cancelling the currently in effect TB.

Attach a Log

Here's a parsed Omnipod log showing the failure with consecutive set TB commands being issued to the pod without a cancel TB before the 2nd attempt which results in an 0x31 (049) pod fault.

COMMAND: 2025-01-16 15:46:51 [SetInsulinScheduleCommand(nonce:494e532e, tempBasal(secondsRemaining: 1800, firstSegmentPulses: 0, table: BasalDeliveryTable([InsulinTableEntry(segments:2, pulses:0, alternateSegmentPulse:false)]))), TempBasalExtraCommand(completionBeep:false, programReminderInterval:0s, remainingPulses:0.2, delayUntilFirstPulse:5h, rateEntries:[RateEntry(rate:0.01, duration:1h)])]
RESPONSE: 2025-01-16 15:46:51 [StatusResponse(deliveryStatus:Temp basal running, progressStatus:Normal, timeActive:40h7m, reservoirLevel:50+, insulinDelivered:47.05, bolusNotDelivered:0.00, lastProgrammingMessageSeqNum:3, alerts:No alerts)]
COMMAND: 2025-01-16 15:46:52 [SetInsulinScheduleCommand(nonce:494e532e, tempBasal(secondsRemaining: 1800, firstSegmentPulses: 0, table: BasalDeliveryTable([InsulinTableEntry(segments:2, pulses:0, alternateSegmentPulse:false)]))), TempBasalExtraCommand(completionBeep:false, programReminderInterval:0s, remainingPulses:0.2, delayUntilFirstPulse:5h, rateEntries:[RateEntry(rate:0.01, duration:1h)])]
RESPONSE: 2025-01-16 15:46:52 [PodInfoResponse(podInfoResponse, detailedStatus, ## DetailedStatus

• rawHex: 020d0000000303ad31096703ff096700000800001537
• podProgressStatus: Fault event occurred
• deliveryStatus: Suspended
• bolusNotDelivered: 0.00 U
• lastProgrammingMessageSeqNum: 3
• totalInsulinDelivered: 47.05 U
• reservoirLevel: 50+ U
• timeActive: 40h7m
• unacknowledgedAlerts: No alerts
• faultEventCode: Fault Event Code 0x31: Incorrect pod state for command or error during insulin command setup
• faultAccessingTables: false
• faultEventTimeSinceActivation: 40h7m
• errorEventInfo: rawValue: 0x08, insulinStateTableCorruption: false, occlusionType: 0, immediateBolusInProgress: false, podProgressStatus: Normal
• previousPodProgressStatus: Normal
• possibleFaultCallingAddress: 0x1537
  ]

To Reproduce

By inspection it appears you'd need to have at least 2 different threads attempting to set a TB approximately concurrently to induce a reproduction of the 049 pod fault situation.

Expected behavior

No preventable 0x31 pod crashes. If the higher levels of the app requests concurrent TB's using more than one threads, each thread needs to handle this gracefully and issue a cancel TB command anytime there might be an active TB before issuing its TB command to the pod.

Screenshots

NA

Setup Information (please complete the following information):

Smartphone:

All

Pump:

• Manufacturer: Insulet
• Model: either Omnipod Dash or Eros

CGM:

NA

Trio Version:

• Version Number: anything Trio 0.2.2 or later?

Technical Details

The underlying cause is a logic error caused that was enabled by the "guard ..., let podState = self.state.podState else {" statement at the very beginning of runTemporaryBasalProgram(). This creates a local copy of the podState variable which will be stale and thus useless for any pod state related checking using this "podState" variable within the self.podComms.runSession closure (when the self.state.podState will actually be locked by the current thread) whenever the current thread has to block before executing its runSession closure. Specifically in the current dev version of OmniBLE (commit e2933d5), the testing done at line 2010 (to decided whether or not a cancel TB command needs to be executed) as well as at line 2002 (to fail if still bolusing when not simply cancelling the current TB) are using this stale copy of self.state.podState and thus these tests will be incorrect if the pump manager's podState has changed since the podState variable was created at the beginning of runTemporaryBasalProgram() (that should have never been in there in the first place) allowing for a TB command to be sent without cancelling the current TB command if one was started in between the time that this function was first entered and when the checking is actually being done.

By changing the statement at line 1992 which is checking for a possible suspended pod situation from "if let podState = self.state.podState, A || B {" to "guard let podState = self.state.podState, !A && !B else {", then the podState variable can be correctly and safely used as intended for the tests using the podState variable at lines 2002 and 2010.

enactBolus() had the exact same if compound statement checking for a possible suspended pod. While enactBolus() doesn't any subsequent pod state testing using podState (and a "guard let podState = ..." outside the runSession closure) thus didn't have the same logic error as enactTempBasal(), I changed this statement to match the change made for this test runTemporaryBasalProgram() for better clarity and for cleaner code if any additional podState related checking is ever added to enactBolus().

Additional context

This logic error was apparently introduced to OmniBLE back in February 21 2024 in PR titled "Improved verification and checking to prevent possible 0x31 pod faults (#110)".

[itsmojo]

The fixes for this issue can be found here:

• Logic fix for 049 pod fault with concurrent temp basal commands LoopKit/OmniBLE#140
• Logic fix for 049 pod fault with concurrent temp basal commands LoopKit/OmniKit#48