-
Notifications
You must be signed in to change notification settings - Fork 40
nettyrpc
The old implementation is based on ZooKeeper which makes ZooKeeper is a blocker when in thousands of iterations especially for more workers. To solve such issue, message passing should be removed from ZooKeeper.
When master is started, a new Netty based TCP server is started there and the new host:port will be written into zookeeper for worker notification. Workers when started, are waiting for master RPC server and then connect to master RPC server through Netty TCP client.
In message passing of each iteration, worker will send message to master directly through these TCP channels, while master result is still stored in ZooKeeper for fault tolerance. As only one master, master result won't be a big issue for ZooKeeper now.
Such new RPC implementation is transparent for users.