fix: always end sync, even when there is missing data #96
Conversation
|
As a future reference for what a more complete fix might look like:
|
okdistribute
changed the title
lib/db-sync-progress.js
Outdated
| var listeners = [] | ||
|
|
||
| multifeed.ready(function () { | ||
| multifeed.feeds().forEach(onFeed) | ||
| multifeed.on('feed', onFeed) | ||
| stream.on('remote-feeds', () => { |
There was a problem hiding this comment.
Could you remove this listener on eos below also?
| progress[feed.key.toString('hex')] = feed.downloaded(0, feed.length) | ||
| var total = feeds.reduce(function (acc, feed) { return acc + feed.length }, 0) | ||
| var sofar = feeds.reduce(function (acc, feed) { return acc + feed.downloaded(0, feed.length) }, 0) | ||
| var all = Array.from(feeds.values()) |
There was a problem hiding this comment.
I didn't know about this trick!
lib/db-sync-progress.js
Outdated
| return acc + feed.length | ||
| }, 0) | ||
| var sofar = all.reduce(function (acc, feed) { | ||
| return acc + feed.stats.totals.downloadedBlocks + feed.stats.totals.uploadedBlocks |
There was a problem hiding this comment.
I think this might give misleading results?
My understanding of how stats.totals works is that it records every upload and download, across multiple peer sessions. We share the same multifeed and thus the same hypercore references across all syncs, so if someone syncs with on peer they'd maybe see their uploadedBlocks go to 100, but on a second sync (/wo restarting Mapeo), they'll see uploadedBlocks start at 100.
In the v9 branch, I include the totals in the handshake, and count 'upload' and 'download' events. Maybe we just continue to track downloads only for now, but do use the 'upload' events for detection of inactivity?
Co-authored-by: Kira Oakley <[email protected]>
Co-authored-by: Kira Oakley <[email protected]>
I added a temporary hack, based on conversations with @noffle, who said that making hypercore sparse might be more to to chew than we can at the moment... a proper fix can come in v9
remote-feedsevent from multifeed v5 into multifeed v4.