mssmt: add new batched ms-smt leaf insertion with recursive merge update

[Roasbeef]

Implement Batched Insertion for MS-SMT to Optimize Database Operations (Fix #451)

Aider Usage

All modifications in these files were produced using aider (including this PR description). The changes were generated with aider v0.73.0 using:

• Main model: o3-mini (architect edit format)
• Editor model: gpt-4o (editor-diff edit format)
• Weak model: gpt-4o-mini

The command executed was similar to:

aider --model o3-mini --reasoning-effort high --architect

This ensured that all modifications to the batched insertion algorithm, merge logic, and recursive update functionality were handled consistently across the codebase.

Overview

This PR introduces a new batched insertion routine for the Merkle-Sum Sparse Merkle Tree (MS-SMT), significantly reducing the number of database transactions by updating the root and internal nodes only once per batch. The new algorithm recursively partitions a sorted batch of leaf insertions by
examining individual bits of the insertion keys at each tree level. When a subtree is empty and a single leaf is inserted, a compacted leaf is created using NewCompactedLeafNode.

In collision cases—when an existing compacted leaf is encountered with a different key—the algorithm calls the
merge function to combine the new and existing leaves into a newly reconstructed subtree. For multiple inbound insertions, the algorithm recursively processes left and right partitions and finally creates a branch node to combine the updated children.

Recursive Merge & Time Complexity

• Recursive Partitioning:
  The routine processes the tree level by level (from height 0 to last bit index) and partitions the sorted batch based on the bit value (0 or 1) at that level. This recursive process ensures that the insertion position is determined in O(log U) time per leaf (where U is the number of bits in th
  key, ~256).

• Merge Optimization:
  Instead of performing N separate insertions (each incurring O(log U) database operations), the new batched algorithm processes the entire batch within one transaction. Although it still performs O(log U) work per leaf internally, the overall database overhead is reduced from O(N log U) to O(log U) + O(1) per batch, greatly improving efficiency for large batches.

Conclusion

This change improves the efficiency of bulk insertions in the MS-SMT by reducing database write transactions and by applying a robust recursive merge/update algorithm. The new implementation replaces multiple single insert calls with one batched update, leading to significant performance gains in
high-load scenarios.

This PR fixes issue #451.

[coveralls]

Pull Request Test Coverage Report for Build 13108154176

Details

• 70 of 142 (49.3%) changed or added relevant lines in 2 files are covered.
• 21 unchanged lines in 5 files lost coverage.
• Overall coverage increased (+0.05%) to 40.748%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
mssmt/compacted_tree.go	69	141	48.94%

Files with Coverage Reduction	New Missed Lines	%
tapdb/addrs.go	2	75.29%
asset/asset.go	2	76.88%
tapchannel/aux_leaf_signer.go	3	43.43%
tapgarden/caretaker.go	4	68.11%
universe/interface.go	10	52.81%

Totals
Change from base Build 13075453647:	0.05%
Covered Lines:	26840
Relevant Lines:	65868

---

💛 - Coveralls