Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Interleaved view support, version 5 #150

Draft
wants to merge 6 commits into
base: main
Choose a base branch
from

Conversation

martinling
Copy link
Member

@martinling martinling commented Jul 30, 2024

This PR contains my work in progress on my fifth attempt at merging the interleaved view. This has been an ongoing project since the very beginning of Packetry's development. It's difficult, but we're getting there.

What is an interleaved view?

The current view in Packetry is hierarchical, but not fully chronological.

If you expand a transfer, then all its transactions appear beneath it, before any other transfers -- even if the transfers actually overlap in time. The timestamp column therefore becomes out-of-order.

This hierarchical approach can be useful when trying to look at the content of a specific transfer, but also misleading, when trying to understand how different transfers are related in time.

In an interleaved view, expanding parent items always maintains chronological ordering. This means that the transactions of a transfer must appear mixed in amongst other items below. It's also possible to expand multiple transfers, and see how their transactions are interleaved in time. Here's a simple example:

image

Why is it hard?

In principle, it's not hard. Conceptually, it just requires sorting all rows in the tree model by timestamp and tree depth.

However, it's hard to do efficiently.

Packetry is designed from the ground up to support captures of unlimited size. It achieves this by not loading everything in advance. The tree list view you see is never fully populated. When you scroll through the view, the items visible at each position are loaded and decoded on demand. And everything is indexed and organised such that this lookup is very fast.

This is much harder for an interleaved view. Let's say you have two overlapping transfers, each with a couple million transactions, and you expand them both. And then you scroll to row 1,528,913 somewhere in the middle of the overlap. Quick, what should be there? Well, that depends on the ordering of all the transactions between the two expanded transfers.

The naive way to find the answer would be to load those several million transactions, sort them, and then take the one at that index. But that's going to slow the UI to a crawl. We need a solution that gets us there near-immediately.

And that's what this PR does, in essence. It uses an algorithm to find what should be at any given row of an interleaved view, in O(m log(n)) time, where m is the number of expanded transfers and n is the number of transactions involved.

But to reduce the problem to the point where we can apply that algorithm, we also need to do a lot of work maintaining the region map of our custom TreeListModel implementation, such that we always know what form of lookup we need to make for any given row.

What's new in this version?

Rather than trying to make the interleaving the standard view, I'm making it one of multiple selectable view modes, building on the work in #145.

What's the status of this?

It's usable in a lot of cases. Give it a try!

It still has some bugs though, which can result in the UI hanging. I need to bring across some further tests and debug work from earlier versions of the branch, which may help track down the remaining issues.

In particular, the interaction between live update and interleaving is complex, and the implementation where the two overlap is known to have some long-standing, hard-to-find bugs. So I'm planning to initially make interleaving available only when capture/loading is stopped, in the hope of getting it merged sooner.

Additionally, the connecting lines between items are still using the same implementation as for the hierarchical view, so they don't always show all the relationships between items correctly.

This commit adds support for interleaved regions and nodes to the tree
model, but does not use it yet. This commit can be tested to verify
that the changes do not impact the non-interleaved case.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant