In this project a modular neuron and synapse are designed and implemented to be part of a feed-forward neural network on an field-programmable gate array (FPGA) where the information is encoded stochastically in binary. The design makes use of no floating point operations that are otherwise common in feed-forward neural networks. The implemented synapse could function as a constant weighted connection between two neurons operating on stochastic binary data. The implemented neuron demonstrates a transfer function with characteristics similar to classical activation functions, but operating on stochastic binary encoded inputs. The modular design coupled with the simplicity of the neural and synaptic operations potentially maps itself well onto the architecture of an FPGA where the logic blocks are modular and can be networked like synaptic connections between neurons.