-
-
Notifications
You must be signed in to change notification settings - Fork 293
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add support for combined iterator lineage #2949
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good idea. I like that you basically treat all iterator source nodes as a single iterator source node.
A few questions in that regard:
- Since there's only one iteration loop, which node's "Stop at first error" option does it use?
- How do you group iterators by length? We explicitly don't know the length of some iterables for certain (e.g. Load Video), so what do you group by?
- As I see it, you completely disabled the lineage system. And I mean: completely. Yes, the code is still there, but it doesn't do anything anymore AFAICT. Does it still have any effect right now?
iterators (renamed to generators in this PR to match Python's naming scheme)
Iterators and generator are not synonymous. Why did you rename everything to generator?
Whichever is enabled and gets triggered first. Or at least, that's how it should work. That might not be what this code is set up to do. I will double check later when I can.
I don't. They're grouped by connectedness and then checked for validity by expected length afterwards. If the length is unknown, then it will be assumed to be the same. Perhaps I should just make the loop stop as soon as we get a stopiteration exception, and just use the shortest connected length in the case where it can't be determined beforehand? For most videos though, they do have an expected length (which is what gets checked against)
There are still some valid checks that take place in that function, but yes the existing iterator lineage checks have been removed as they are now unnecessary. However, I expect that lineage will make a return once we have types for sequence length.
Because they generate iterable values from non-iterable values. What would you expect the naming system to look like when we have these four cases:
My proposal is for 3 to be generators and 4 to be iterators, as that makes sense to me. |
Just double checked. I did account for this. It saves the last |
typing.Iterator also guarantees next (which makes sense given next is an iterator thing), so i used that instead. |
Added some unit tests to confirm that the set combining function works properly under a few different test cases. |
With the added unit tests, I'm pretty confident in that function. It actually does do almost the same thing that I was trying to do, just more efficiently (and fully correct). |
I'm good with this the way it is, personally. If you have anything that you want changed, feel free to submit follow up PRs. |
This PR allows iterators (renamed to generators in this PR to match Python's naming scheme) across different lineages to validly connect with each other. As there is no Navi types for sequence size, there is instead a runtime requirement that enforces that iterators must be of the same expected length in order to work properly together. I expect that in the future the UX for this will be better as we add type information to sequences.
The implementation of this was pretty simple, all things considered. I made it so it first splits up the chain into groups, so any connected generators, collectors, and nodes in between in the same chain of nodes get put into their own group. These groups then each get processed one by one. This was necessary to prevent all generators from being run at the same time, even if they had a completely separate lineage. Then, it just collects each of the generators in a group and manually calls
next()
on them, and caches their outputs the same way that is done now. It also takes all the connected collectors and output nodes from the group and merges them together into a single array for each, which is then run like a single chain. It is then set up to stop iterating once each iterator has reached its StopIteration exception.From my testing, this all works pretty well, and you can even do things like connect a Load Images node's directory output to another Load Images node, and it just works because it treats them as separate groups.
I'm sure there's edge cases with this that I haven't thought of, but I think it is at least better than our current restricted system. Unfortunately this iteration system still doesn't allow nodes that take in a sequence and return a differently sized sequence, which is what I initially had planned to try to implement in this. It was only at a certain point that I realized I could pretty easily just do what I did here to get this result, which is still a win.
Total things done in this PR:
Also, apologies for doing a bit too much at once with this PR, renaming to generator probably should have been its own PR first, but I had already gone too deep into this by the time I thought about that.