Why TrainMnist so slow on gtx 1060?

[blaze79]

I ran TrainMnist example both on GPU 1.8.0-cu102mkl-win-x86_64 (cuda arch 61) and on CPU 1.8.0-mkl-win-x86_64 and the speed was the same. In first case Windows showed that 1GB of video memory was used and 40% Cuda used, java consumed CPU 2 cores.
If second - java consumed 4 cpu cores but speed was the same "speed: 1548,56 items/sec".

Is it normal or I had some problems with GPU configuration?

[lanking520]

You may end up using GPU all the time. Can you check if GPU is on for your use cases? You can do that through looking into the logs that mentioning CPU/GPU, or get the current device from NDManager, or simply look into nvidia-smi to see if there is any Java process occupying it.

[blaze79]

"Can you check if GPU is on for your use cases?" - When training process is started with cu102mkl GPU memory consumption increased from 0Gb to 1Gb and CUDA used 40% in Windows Task Manager. So with cu102mkl GPU definitl used.

When I start with 1.8.0-mkl-win-x86_64 - there was no any NVIDIA cuda toolkit installed and no mxnet.dll donwloaded. So 1060GTX coud not be used

[lanking520]

I would suggest for you to track the performance end2end, to see how long it was consumed on time for the total training time. MNIST is a failrly small dataset. The GPU performance gain should be very small. I would suggest you to try with trainCifar10 example to see if there is any difference.

[blaze79]

I started to use GPU Temp to evaluate GPU consumption, it shows 5% for MNIST and ~70% for trainCifar10. Seems huge difference. Thanks