Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a setting to cancel live jobs upon change #1248

Closed

Conversation

modelflat
Copy link
Contributor

Hi, thanks for this amazing plugin!

I've noticed an inconvenience with the way the jobs are scheduled in the live mode, and thought to contribute a fix I implemented for myself.

Current implementation of live mode will try to schedule jobs "sequentially":

  • the first change causes a job to be scheduled
  • we stop listening for changes until the job is finished
  • if the job wasn't for the latest change, a new job is scheduled, etc.

This often results in having to wait for the intermediate results to complete, and is not always convenient.

This change adds a setting, which, if toggled on, instructs Live mode to clear the queue and interrupt any running jobs upon receiving a new change, thus avoiding having to wait for intermediate results.

Current implementation of live mode will try to schedule jobs
"sequentially":

- the first change causes a job to be scheduled
- we stop listening for changes until the job is finished
- if the job wasn't for the latest change, a new job is scheduled, etc.

This often results in having to wait for the intermediate results to
complete, and is not always convenient.

This change adds a setting, which, if toggled on, instructs Live mode to
clear the queue and interrupt any running jobs upon receiving a new change,
thus avoiding having to wait for intermediate results.
# likely to be an out-of-order progress from cancelled/interrupted job so don't log these
pass
else:
log.error(f"Received message {msg} but there is no active job")
Copy link
Contributor Author

@modelflat modelflat Oct 8, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure why are we logging these at all, especially at the Error level

@Acly
Copy link
Owner

Acly commented Oct 21, 2024

There is some discussion about this in #628

The gist is that I'm not convinced cancel is a good idea, mainly because it means you have to actively restrain yourself from making any kind of change to avoid accidentally interrupting image generation when you don't want to. The basic idea of Live mode is that your are making small changes constantly and get (reasonably fast) feedback.

If you find yourself waiting for results before continuing to work, the queued workflow or manual trigger via hotkey might be a better fit?

I also have some issues with the implementation, usually there is a flow like this:

  • cancel active job
  • generation is interrupted on the server
  • server send message that job has been cancelled
  • client receives message and forwards to model / jobs
  • job state is set to cancelled (and may be removed from list)
  • live mode can continue by scheduling the next image as a result of previous job being completed/cancelled

_cancel_everything is taking a kind of shortcut, and maybe getting away with it because all jobs should be nuked afterwards on all sides... but it shouldn't be necessary

@modelflat
Copy link
Contributor Author

modelflat commented Nov 8, 2024

@Acly Thanks for the feedback!

it means you have to actively restrain yourself from making any kind of change to avoid accidentally interrupting image generation when you don't want to

After having worked with the plugin some more and getting used to it, I realize that this is a very important point. Even if the live generation takes a while (up to 4 seconds on my setup), it is still more meaningful for it to run sort of "in the background" while you are working uninterrupted.

Regarding the suggestion to use a hotkey to generate: while I believe this might be a good workaround for very slow setups, I still consider it a very significant UX hit as these are additional actions you have to actively take, as opposed to simply keeping an eye on the preview panel.

So, there are two potential improvements I see for the live generation process still:

  1. Debounce the change events (as already suggested in Update live preview only after a deliberate change rather than mid-change #628). This should be of help at least in the case of rapid modification and/or on weaker setups. I believe buffering the events by 100-200ms (or configurable) window should reduce unwanted generations, while preserving the desired properties of live updates
  2. Have a backlog of the last N images generated in live mode. Again, this applies more to slower setups like mine. At the sketching stage, the model can sometimes produce interesting results which you might wanna build from, but it's hard to go back to it if you've been making changes rapidly

@Acly
Copy link
Owner

Acly commented Nov 8, 2024

  1. Debounce the change events

No objections there. Easiest option would be just a configurable grace period. Possible extensions:

  • Also add an upper bound. When there's a change, generation starts after there hasn't been any changes for 100~200ms or fixed 1s after the initial change.
  • Automatically configure it based on how long images take. If they take <1s on average, schedule instantly for smallest latency, if it takes longer ramp up the grace perior (or use a fixed value).

Might be nice even for people with very powerful setups to get the behavior when resolution is large, without losing responsiveness at low res.

modelflat added a commit to modelflat/krita-ai-diffusion that referenced this pull request Nov 8, 2024
…on in live preview

* This will prevent some of the "useless" generations, e.g. from the very start of
  the brush stroke
* Period is configurable in settings; setting the default to 0 to
  preserve the existing behaviour

This at least partially addresses/follows the discussions in Acly#628 and Acly#1248
@modelflat
Copy link
Contributor Author

Continued in #1412

@modelflat modelflat closed this Nov 17, 2024
modelflat added a commit to modelflat/krita-ai-diffusion that referenced this pull request Nov 26, 2024
…on in live preview

* This will prevent some of the "useless" generations, e.g. from the very start of
  the brush stroke
* Period is configurable in settings; setting the default to 0 to
  preserve the existing behaviour

This at least partially addresses/follows the discussions in Acly#628 and Acly#1248
modelflat added a commit to modelflat/krita-ai-diffusion that referenced this pull request Nov 26, 2024
…on in live preview

* This will prevent some of the "useless" generations, e.g. from the very start of
  the brush stroke
* Period is configurable in settings; setting the default to 0 to
  preserve the existing behaviour

This at least partially addresses/follows the discussions in Acly#628 and Acly#1248
Acly pushed a commit that referenced this pull request Nov 29, 2024
…on in live preview (#1412)

* Add a grace period between detecting a change and triggering generation in live preview
* This will prevent some of the "useless" generations, e.g. from the very start of
  the brush stroke
* Period is configurable in settings; setting the default to 0 to
  preserve the existing behaviour
This at least partially addresses/follows the discussions in #628 and #1248
* Don't exit live generation loop when switching documents
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants