LR Scheduler¶

class lightning_asr.optim.lr_scheduler.lr_scheduler.LearningRateScheduler(optimizer, init_lr)[source]¶: Provides inteface of learning rate scheduler.

Note

Do not use this class directly, use one of the sub classes.

Transformer LR Scheduler¶

class lightning_asr.optim.lr_scheduler.transformer_lr_scheduler.TransformerLRScheduler(optimizer, peak_lr, final_lr, final_lr_scale, warmup_steps, decay_steps)[source]¶: Implement the learning rate scheduler in https://arxiv.org/abs/1706.03762

Tri-Stage LR Scheduler¶

class lightning_asr.optim.lr_scheduler.tri_stage_lr_scheduler.TriStageLRScheduler(optimizer, init_lr, peak_lr, final_lr, init_lr_scale, final_lr_scale, warmup_steps, total_steps)[source]¶

Implement the learning rate scheduler in https://arxiv.org/pdf/1904.08779.pdf Similar to inverse_squre_root scheduler, but tri_stage learning rate employs three stages LR scheduling:

warmup stage, starting from lr * init_lr_scale, linearly increased to lr in warmup_steps iterations

hold stage, after warmup_steps, keep the LR as lr for hold_steps iterations

decay stage, after hold stage, decay LR exponetially to lr * final_lr_scale in decay_steps; after that LR is keep as final_lr_scale * lr