Develop a tuning toolkit

[PonteIneptique]

General idea :

• tuning.py based on pandora.cli.tuning_cmd
• Runs a series of configs directly from the command line
• Loads a list of configs from a folder through --configs_dir configs/*.txt
• Keep the last evaluation score in a csv using the name of the config as a key
• Produce a CSV with learning curve to adapt where necessary the learning rate and other params

[PonteIneptique]

Other idea :

The actual tuning.py could have a second option to generate such configs using a csv file :

name,include_lemma,include_pos,nb_left_tokens...
config1,True,True,2
config2,True,True,1

[emanjavacas]

Given a dev set there is nothing stopping us from doing hyperparameter optimization. There are a couple of non-bayesian methods worth trying and offering them for people with GPUs and time to invest in squeezing the most out of their data. Things like hyperband or successive halving.

…

On Mon 23. Oct 2017 at 12:45, Thibault Clérice ***@***.***> wrote: Other idea : The actual tuning.py could have a second option to generate such configs using a csv file : name,include_lemma,include_pos,nb_left_tokens... config1,True,True,2 config2,True,True,1 — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#81 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AF6Ho2XoE6EKKatPIoCE-ml3JhTVZD6tks5svG42gaJpZM4QCmIH> .

[PonteIneptique]

I am all for method-based hyperparameter optimization. The only thing is that it might take more time that just what I described. But please, if you find the time to do this one, I'd be glad to test it :)

[PonteIneptique]

Note : we could also simply use existing libraries such as https://github.com/hyperopt/hyperopt