Skip to content

Commit

Permalink
Update docs for OneCycle on main page
Browse files Browse the repository at this point in the history
  • Loading branch information
@darsnack
darsnack committed Mar 5, 2024
1 parent bd4f4b7 commit 20495b4
Show file tree
Hide file tree
Showing 4 changed files with 45 additions and 40 deletions.
21 changes: 21 additions & 0 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,27 @@ lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false) # hi

<tr><td>

[`OneCycle(nsteps, maxval)`](https://fluxml.ai/ParameterSchedulers.jl/api/complex.html#ParameterSchedulers.OneCycle)

</td>
<td>

[One cycle cosine](https://arxiv.org/abs/1708.07120)

</td>
<td> Complex </td>
<td style="text-align:center">

```@example
using UnicodePlots, ParameterSchedulers # hide
t = 1:10 |> collect # hide
s = OneCycle(10, 1.0) # hide
lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false) # hide
```
</td></tr>

<tr><td>

[`Triangle(l0, l1, period)`](https://fluxml.ai/ParameterSchedulers.jl/api/cyclic.html#ParameterSchedulers.Triangle)

</td>
Expand Down
44 changes: 22 additions & 22 deletions docs/src/cheatsheet.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,29 +6,29 @@ If you are coming from PyTorch or Tensorflow, the following table should help yo
PyTorch typically wraps an optimizer as the first argument, but we ignore that functionality in the table. To wrap a Flux.jl optimizer with a schedule from the rightmost column, use [`ParameterSchedulers.Scheduler`](@ref).
The variable `lr` in the middle/rightmost column refers to the initial learning rate of the optimizer.

| PyTorch | Tensorflow | ParameterSchedulers.jl |
|:-------------------------------------------------------------------------------|:------------------------------------------------------|:------------------------------------------------------|
| `LambdaLR(_, lr_lambda)` | N/A | `lr_lambda` |
| `MultiplicativeLR(_, lr_lambda)` | N/A | N/A |
| `StepLR(_, step_size, gamma)` | `ExponentialDecay(lr, step_size, gamma, True)` | `Step(lr, gamma, step_size)` |
| `MultiStepLR(_, milestones, gamma)` | N/A | `Step(lr, gamma, milestones)` |
| `ConstantLR(_, factor, total_iters)` | N/A | `Sequence(lr * factor => total_iters, lr => nepochs)` |
| PyTorch | Tensorflow | ParameterSchedulers.jl |
|:-------------------------------------------------------------------------------|:------------------------------------------------------|:--------------------------------------------------------|
| `LambdaLR(_, lr_lambda)` | N/A | `lr_lambda` |
| `MultiplicativeLR(_, lr_lambda)` | N/A | N/A |
| `StepLR(_, step_size, gamma)` | `ExponentialDecay(lr, step_size, gamma, True)` | `Step(lr, gamma, step_size)` |
| `MultiStepLR(_, milestones, gamma)` | N/A | `Step(lr, gamma, milestones)` |
| `ConstantLR(_, factor, total_iters)` | N/A | `Sequence(lr * factor => total_iters, lr => nepochs)` |
| `LinearLR(_, start_factor, end_factor, total_iters)` | N/A | `Sequence(Triangle(lr * start_factor, lr * end_factor, 2 * total_iters) => total_iters, lr => nepochs)` |
| `ExponentialLR(_, gamma)` | `ExponentialDecay(lr, 1, gamma, False)` | `Exp(lr, gamma)` |
| N/A | `ExponentialDecay(lr, steps, gamma, False)` | `Interpolator(Exp(lr, gamma), steps)` |
| `CosineAnnealingLR(_, T_max, eta_min)` | `CosineDecay(lr, T_max, eta_min)` | `CosAnneal(lr, eta_min, T_0, false)` |
| `CosineAnnealingRestarts(_, T_0, 1, eta_min)` | `CosineDecayRestarts(lr, T_0, 1, 1, eta_min)` | `CosAnneal(lr, eta_min, T_0)` |
| `CosineAnnealingRestarts(_, T_0, T_mult, eta_min)` | `CosineDecayRestarts(lr, T_0, T_mult, 1, alpha)` | See [below](@ref "Cosine annealing variants") |
| N/A | `CosineDecayRestarts(lr, T_0, T_mult, m_mul, alpha)` | See [below](@ref "Cosine annealing variants") |
| `SequentialLR(_, schedulers, milestones)` | N/A | `Sequence(schedulers, milestones)` |
| `ReduceLROnPlateau(_, mode, factor, patience, threshold, 'abs', 0)` | N/A | See [below](@ref "`ReduceLROnPlateau` style schedules") |
| `CyclicLR(_, base_lr, max_lr, step_size, step_size, 'triangular', _, None)` | N/A | `Triangle(base_lr, max_lr, step_size)` |
| `CyclicLR(_, base_lr, max_lr, step_size, step_size, 'triangular2', _, None)` | N/A | `TriangleDecay2(base_lr, max_lr, step_size)` |
| `CyclicLR(_, base_lr, max_lr, step_size, step_size, 'exp_range', gamma, None)` | N/A | `TriangleExp(base_lr, max_lr, step_size, gamma)` |
| `CyclicLR(_, base_lr, max_lr, step_size, step_size, _, _, scale_fn)` | N/A | See [Arbitrary looping schedules](@ref) |
| N/A | `InverseTimeDecay(lr, 1, decay_rate, False)` | `Inv(lr, decay_rate, 1)` |
| N/A | `InverseTimeDecay(lr, decay_step, decay_rate, False)` | `Interpolator(Inv(lr, decay_rate, 1), decay_step)` |
| N/A | `PolynomialDecay(lr, decay_steps, 0, power, False)` | `Poly(lr, power, decay_steps)` |
| `ExponentialLR(_, gamma)` | `ExponentialDecay(lr, 1, gamma, False)` | `Exp(lr, gamma)` |
| N/A | `ExponentialDecay(lr, steps, gamma, False)` | `Interpolator(Exp(lr, gamma), steps)` |
| `CosineAnnealingLR(_, T_max, eta_min)` | `CosineDecay(lr, T_max, eta_min)` | `CosAnneal(lr, eta_min, T_0, false)` |
| `CosineAnnealingRestarts(_, T_0, 1, eta_min)` | `CosineDecayRestarts(lr, T_0, 1, 1, eta_min)` | `CosAnneal(lr, eta_min, T_0)` |
| `CosineAnnealingRestarts(_, T_0, T_mult, eta_min)` | `CosineDecayRestarts(lr, T_0, T_mult, 1, alpha)` | See [below](@ref "Cosine annealing variants") |
| N/A | `CosineDecayRestarts(lr, T_0, T_mult, m_mul, alpha)` | See [below](@ref "Cosine annealing variants") |
| `SequentialLR(_, schedulers, milestones)` | N/A | `Sequence(schedulers, milestones)` |
| `ReduceLROnPlateau(_, mode, factor, patience, threshold, 'abs', 0)` | N/A | See [below](@ref "`ReduceLROnPlateau` style schedules") |
| `CyclicLR(_, base_lr, max_lr, step_size, step_size, 'triangular', _, None)` | N/A | `Triangle(base_lr, max_lr, step_size)` |
| `CyclicLR(_, base_lr, max_lr, step_size, step_size, 'triangular2', _, None)` | N/A | `TriangleDecay2(base_lr, max_lr, step_size)` |
| `CyclicLR(_, base_lr, max_lr, step_size, step_size, 'exp_range', gamma, None)` | N/A | `TriangleExp(base_lr, max_lr, step_size, gamma)` |
| `CyclicLR(_, base_lr, max_lr, step_size, step_size, _, _, scale_fn)` | N/A | See [Arbitrary looping schedules](@ref) |
| N/A | `InverseTimeDecay(lr, 1, decay_rate, False)` | `Inv(lr, decay_rate, 1)` |
| N/A | `InverseTimeDecay(lr, decay_step, decay_rate, False)` | `Interpolator(Inv(lr, decay_rate, 1), decay_step)` |
| N/A | `PolynomialDecay(lr, decay_steps, 0, power, False)` | `Poly(lr, power, decay_steps)` |

## Cosine annealing variants

Expand Down
3 changes: 2 additions & 1 deletion src/cyclic.jl
View file
Original file line number Diff line number Diff line change
Expand Up @@ -208,8 +208,9 @@ CosAnneal(range, offset, period) = CosAnneal(range, offset, period, true)
function CosAnneal(; kwargs...)
kwargs = depkwargs(:CosAnneal, kwargs, :λ0 => :l0, :λ1 => :l1)
l0, l1 = kwargs.l0, kwargs.l1
restart = get(kwargs, :restart, true)

return CosAnneal(abs(l0 - l1), min(l0, l1), kwargs.period, kwargs.restart)
return CosAnneal(abs(l0 - l1), min(l0, l1), kwargs.period, restart)
end

Base.eltype(::Type{<:CosAnneal{T}}) where T = T
Expand Down
17 changes: 0 additions & 17 deletions toc.md
View file

This file was deleted.

0 comments on commit 20495b4

Please sign in to comment.