Skip samples on training start #640
Conversation
|
do we want a switch to just skip the first sample... ...or an interval, same as for checkpoint saves: [just a question - a switch is already useful on its own already] |
modules/trainer/GenericTrainer.py
Outdated
| @@ -495,6 +495,8 @@ def save(self, train_progress: TrainProgress, print_msg: bool = True, print_cb: | |||
| torch_gc() | |||
|
|
|||
| def __needs_sample(self, train_progress: TrainProgress): | |||
| if self.config.skip_sample_on_train_start and train_progress.epoch == 0 and train_progress.epoch_step == 0: | |||
There was a problem hiding this comment.
I think the same can be accomplished easier by passing start_at_zero==False to repeating_action_needed
There was a problem hiding this comment.
I will check this out then
hmm, probably the |
|
Yes. I also think it would be more useful. |
|
I really like this idea! And agree that "Skip first" with a custom field is more useful. Edit: You can see here how the |
|
@sangoi-exe Do you still plan to implement the more useful "skip first" option, instead? |
fuck, totally forgot, it's been crazy times at work, sorry 😫 |
sangoi-exe
force-pushed
the
skip-samples
branch
from
21e4804 to
f09c0ae
Compare
|
@Nerogar sorry for the excessive delay 😬 but I think it's done now 😁 |
| return self.single_action_elapsed( | ||
| "sample_skip_first", self.config.sample_skip_first, self.config.sample_after_unit, train_progress | ||
| ) and self.repeating_action_needed( | ||
| "sample", self.config.sample_after, self.config.sample_after_unit, train_progress, start_at_zero=False |
There was a problem hiding this comment.
setting start_at_zero=False means that it will never sample before training starts. If you remove that again, it can be configured by setting sample_skip_first=1 instead.
| @@ -400,6 +400,7 @@ class TrainConfig(BaseConfig): | |||
| sample_audio_format: AudioFormat | |||
| samples_to_tensorboard: bool | |||
| non_ema_sampling: bool | |||
| sample_skip_first: int | |||
There was a problem hiding this comment.
Can you move this line up just below the other two sampler_after lines? Same for the initialization of the default values further down.
Just a QoL option that makes OT skip sample generation when training starts.
I’ve been annoyed by this for a long time, always having to enable and disable samples at the start of each training session. 😉