Skip to content
pengzhang edited this page Sep 4, 2017 · 2 revisions

title: Netty Based RPC for Message Passing

Guagua

Why New RPC?

The old implementation is based on ZooKeeper which makes ZooKeeper is a blocker when in thousands of iterations especially for more workers. To solve such issue, message passing should be removed from ZooKeeper.

New RPC Implementation

Netty-RPC

When master is started, a new Netty based TCP server is started there and the new host:port will be written into zookeeper for worker notification. Workers when started, are waiting for master RPC server and then connect to master RPC server through Netty TCP client.

In message passing of each iteration, worker will send message to master directly through these TCP channels, while master result is still stored in ZooKeeper for fault tolerance. As only one master, master result won't be a big issue for ZooKeeper now.

Such new RPC implementation is transparent for users.

Clone this wiki locally