Vits2 / Yourtts2 doesnt work with accelerate

[Marioando]

I'm trying to run multigpu training for the vits2 prototype but it doesnt seem to work with accelerate. Training with trainer distribute work. The error backtrace shows some issue computing grad_norm function in trainer (some parametr gradient are none). Please have a look at the train_step of the prototype. Thank you

[eginhard]

Please discuss issues with a PR on that PR itself to have the context and share the full error message, otherwise nobody can help. Also add the relevant cross-link (#137) to help finding it.