-
|
Is the learning rate during warmup steps 0? Or is it a % of the base LR? Or is it being gradually increased from 0 to the base LR? I understand all three are methods used in ML. |
Beta Was this translation helpful? Give feedback.
Answered by
laksjdjf
Apr 19, 2023
Replies: 1 comment
-
|
its gradually increased from 0 to the base LR. |
Beta Was this translation helpful? Give feedback.
0 replies
Answer selected by
qwertea
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
its gradually increased from 0 to the base LR.
https://huggingface.co/docs/transformers/main_classes/optimizer_schedules#schedules