Nerlnet Design

Components:

Overview

Nerlnet is implemented using 3 different languages and 3 main layers as follows:

API Server:

An API that communicates with Nerlnet main server.
It gives the user control over the Nerlnet cluster and the experiment flow of it.
The API Server is a python module that can be used also from a Jupyter notebook.
The script of NerlnetJupyterLaunch.sh opens an environment to use API Server to conduct an Erlang experiment.
See the help menu from an instance of an APIServer by calling the help method:

   from apiServer import *
   api_server_inst = ApiServer()
   api_server_inst.help()

Main Server:

The Main-Server is a Cowboy HTTP server. The Main-Server is the orchestrator of Nerlnet. It generates Nerlnet cluster from a distributed configuration (DC json).

Main-Server is controlled by the API server, receiving inputs from user and gathers results of phases (Training/Predictions).
The main server must be connected with all entities of Nerlnet through its router.

Router:

Router is an entity in Nerlnet that connects between entities: Clients, Sensors, other routers, the main-server and API-server. The router uses a static routing table that is generated based on graph topology given by user (communication map json) on NerlnetApp initialization stage. The router communicates with other entities through HTTP protocol. 
It is a Cowboy HTTP server.

Source:

Source streams data to its target workers. It is a state machine with two states: Idle and Casting. It streams batches in a predefined policy and frequency and batch size (# of samples). Policies are: 
Broadcast, Round-Robin, Random. Source types are:
-CSV.
-USB Camera (not implemented yet).

Client:

The client is an envelope of workers that allows them communication with other entities or workers.

When worker action requires sending data, it is forwarded to the 
client which sends it to the target Nerlnet entity through routers path. The client is an HTTP Cowboy server and a state machine. Its state machine syncs workers jobs in each phase and 
validates that workers behavior is normal.

Workers State Machine:

Worker is an entity in Nerlnet that manages the ML model on the edge. The worker has NIF functions that bridges Erlang with Cpp libraries for efficient NN implementation.
The worker behaving as a state machine with 4 states: Init -> Idle <-> Train <-> Idle <-> Predict.
The communication of worker with other entities and vice versa is managed by its host client. Train/Predict states execute learning/prediction computation and return results to FSM on wait state.
During wait (train or predict) states new messages are rejected.

Controller Function

TODO

Streams

TODO

OpenNN Bridge (NIF):

TODO Contains: NerlNIF.cpp - The NIF itself. Converts Erlang commands to C++. Opens new threads for train/predict functions. Creates new modules and maintains the bridgeController singleton. bridgeController.h - Singleton. Maintains all the modules and their MID in a map. nifpp.h - External library for that wraps erlang nif with C++ abilities. erlModule.erl - Erland side of the NerlNIF. Contains train/predict/create functions calls from the Erlang side.