A ChannelManager is stale compared to the current ChannelMonitor!

[philipp1992]

Hi,

we are experiencing this issue, what could be the cause and is there any way to mitigate it?

kind regards

2025-01-30T10:33:55.371Z hydra_bitcoin::node::logger:22: A ChannelManager is stale compared to the current ChannelMonitor!
imports.wbg.__wbg_error_80de38b3f7cc3c3c @ hydra_node.js:9025
hydra_node.js:9025 ERROR 2025-01-30T10:33:55.372Z hydra_bitcoin::node::logger:22:  The channel will be force-closed and the latest commitment transaction from the ChannelMonitor broadcast.
imports.wbg.__wbg_error_80de38b3f7cc3c3c @ hydra_node.js:9025
hydra_node.js:9025 ERROR 2025-01-30T10:33:55.372Z hydra_bitcoin::node::logger:22:  The ChannelMonitor for channel b37b4660dec0340c000ab33a9c877d671a7e4fe3f7f9945cd9704668e760430f is at update_id 27 but the ChannelManager is at update_id 26.
imports.wbg.__wbg_error_80de38b3f7cc3c3c @ hydra_node.js:9025
hydra_node.js:9025 ERROR 2025-01-30T10:33:55.372Z hydra_bitcoin::node::logger:22:  The ChannelMonitor for channel b37b4660dec0340c000ab33a9c877d671a7e4fe3f7f9945cd9704668e760430f is at holder commitment number 281474976710644 but the ChannelManager is at holder commitment number 281474976710645.
imports.wbg.__wbg_error_80de38b3f7cc3c3c @ hydra_node.js:9025
hydra_node.js:9025 ERROR 2025-01-30T10:33:55.372Z hydra_bitcoin::node::logger:22: A ChannelManager is stale compared to the current ChannelMonitor!
imports.wbg.__wbg_error_80de38b3f7cc3c3c @ hydra_node.js:9025
hydra_node.js:9025 ERROR 2025-01-30T10:33:55.372Z hydra_bitcoin::node::logger:22:  The channel will be force-closed and the latest commitment transaction from the ChannelMonitor broadcast.
imports.wbg.__wbg_error_80de38b3f7cc3c3c @ hydra_node.js:9025
hydra_node.js:9025 ERROR 2025-01-30T10:33:55.372Z hydra_bitcoin::node::logger:22:  The ChannelMonitor for channel d91933328927a0fcd20613150cbb852bf8d9d5873b716ab44ce50d504975b8a4 is at update_id 15 but the ChannelManager is at update_id 13.
imports.wbg.__wbg_error_80de38b3f7cc3c3c @ hydra_node.js:9025
hydra_node.js:9025 ERROR 2025-01-30T10:33:55.372Z hydra_bitcoin::node::logger:22:  The ChannelMonitor for channel d91933328927a0fcd20613150cbb852bf8d9d5873b716ab44ce50d504975b8a4 is at holder commitment number 281474976710649 but the ChannelManager is at holder commitment number 281474976710650.
imports.wbg.__wbg_error_80de38b3f7cc3c3c @ hydra_node.js:9025
hydra_node.js:9025 ERROR 2025-01-30T10:33:55.372Z hydra_bitcoin::node::logger:22:  The ChannelMonitor for channel d91933328927a0fcd20613150cbb852bf8d9d5873b716ab44ce50d504975b8a4 is at revoked counterparty transaction number 281474976710650 but the ChannelManager is at revoked counterparty transaction number 281474976710651.
imports.wbg.__wbg_error_80de38b3f7cc3c3c @ hydra_node.js:9025

[tnull]

Could you provide some more information regarding the context in which you're seeing this? What persistence backend are you using? Are you somehow killing the node during runtime or are you making sure to always shut it down properly, eg., before restarting?

[TheBlueMatt]

Do you have logs for the run before this one? Also did the previous run crash?

[philipp1992]

Do you have logs for the run before this one? Also did the previous run crash?

Yes it did crash but unfortunately I don't have the logs from that. Is there any way to recover the channel instead of closing it?

[tnull]

Yes it did crash but unfortunately I don't have the logs from that. Is there any way to recover the channel instead of closing it?

Unfortunately I don't think there is a way to discover without closing in this scenario, as essential updates that need to be persisted didn't make it to disk. Note that we're working on a larger refactor that would avoid this being possible at all, but for now you'll need to make sure that your node doesn't randomly crash or is shutdown.

Just to confirm: are you using LDK Node, or your own LDK-based implementation?

[Fedeparma74]

Just to confirm: are you using LDK Node, or your own LDK-based implementation?

We are running our own LDK-based implementation

[tnull]

We are running our own LDK-based implementation

Alright, thanks, transferred the issue to the general rust-lightning repository.

[TheBlueMatt]

You should ensure that you're fully complying with the persistence APIs. Specifically that if you indicate something was durably stored to disk that there's no way for it to not be used on reload (rather than a previous version). On most platforms this means fsync'ing the file and the directory that contains it.