[RFC] Optimize SF parameters for matecheck performance. #121
Conversation
An additional python script to optimize tunable parameters in SF for matetrack performance.
One example:
matetrack at 1M nodes:
Total FENs: 6554
branch:
Found mates: 4275
Best mates: 3308
master:
Found mates: 3271
Best mates: 2345
For a branch based on XInTheDark/Stockfish@d83d6a4
Using a budget of 4000 evaluations, invoked as:
```
mpirun -np N python3 -u -m mpi4py.futures ng4mt.py --ngBudget 4000 --engine $dir/stockfish --epdFile $dir/matetrack.epd --nodes 1000000 --concurrency $concurrency
```
Not sure this script belongs in the matetrack repo, the nevergrad and mpi4py dependencies are a bit heavy.
|
Impressive performance. What we should be careful about is to not train in the same set we test, to avoid overfitting to the test set. So we may need a (much?) larger set of positions from which to choose the training set(s). These new positions could maybe be obtained by looking at the first mate announcement per game in pgns from fishtest? Then update the bm for those if and when we find shorter ones? I'd be happy to keep these new scripts in the repo. |
| for line in output.stdout.split("\n"): | ||
| # we minimize, so return minus the number of best mates | ||
| if "Best mates: " in line: | ||
| return -int(line.split()[2]) |
There was a problem hiding this comment.
Could also subtract a small multiple of mates found?
There was a problem hiding this comment.
In any case I would also add "Better mates:", in case the bm value from the epd file is not (yet) optimal. In that case could also log some message with relevant FEN.
|
Just adding this here as a reference as well |
|
Note that such an optimization reduces Elo by roughly 50 points: |
|
Yeah, to be expected. But it offers some nice possibilities for the future. |
|
A set with 1M mates has been added here: robertnurnberg/matetools#83 Could be used to have seperate training and testing data in future. |
An additional python script to optimize tunable parameters in SF for matetrack performance.
One example:
For a branch based on XInTheDark/Stockfish@d83d6a4
Using a budget of 4000 evaluations, invoked as:
Not sure this script belongs in the matetrack repo, the nevergrad and mpi4py dependencies are a bit heavy.