DeepDetect v0.9.3 - Training optimizers update

This release mainly adds new optimizers to the Caffe backend, along with an important bug fix to the optimizer selection when training of models from scratch (not affecting transfer learning).

Main changes:

• Lookahead Optimizer: https://arxiv.org/abs/1907.08610, see #621
• Rectified Adam: https://arxiv.org/abs/1908.03265v1, see #628
• Decoupled Weight Decay Regularization: https://arxiv.org/abs/1711.05101, see #627
• Training learning rate warmup, see #623

Other changes:

• Improved dede server command line model start list behavior, see #620
• Learning rate value now returned on training status call and plotted on platform, see #624

Bug fixes:

• Fixed training optimizer selection when training models from scratch, see #626

API changes:

The new optimizers include improvements from some papers released this summer 2019. The main new training API parameters in the solver object are:

• parameters.solver.lookahead: true/false, triggers lookahead optimizer
• parameters.solver.lookahead_steps: default to 6
• parameters.solver.lookahead_alpha : default to 0.5
• parameters.mllib.solver.warmup_lr: initial learning rate linearly increased to base_lr over parameters.mllib.solver.warmup_iter steps
• parameters.mllib.solver.warmup_iter: number of warmup steps
• ADAMW, SGDW and AMSGRADW optimizers implemented decoupled weight decay regularization
• parameters.mllib.solver.rectified: activates the rectified optimization scheme