masterworkers
This graph describes distributed neural network iteration algorithm.
In each iteration:
- Data are split by workers.
- Each worker computes its own neural network gradients and sends them to master.
- Master accumulates all gradients and updates global neural network weights and then sends global weights to all workers for next iteration.