2019-09-13, 10:30–12:00, Room C
Let's write a (tiny but working) Neural Network library from scratch!
Well, almost from scratch, we will still use
And we will try to do it in less than 1000 lines of code.
We see many talks that explain neural networks (NNs), that describe the
different architectures and the problems for which each architecture is best.
We are not going to talk about that here. Instead, we will talk about the
implementation of NNs, about how we make NNs work in code - not by using a
library but by writing it ourselves.
Have you ever experimented with
Have you built and trained a NN with one of these libraries? The process of
doing so is simple: You describe the overall NN architecture, feed the data,
and then execute a training procedure which runs for a while. This training
process "magically" produces a trained NN (be it well trained or badly
trained), which can then be used for predictions.
The process of getting a trained NN begs a question: what is the magic that
trains the NN into a predictive tool? Several resources explain NNs as a bunch
of circles connected by arrows, that is a good conceptual representation of a
NN but has no similarity with the actual implementation of the NN or of its
training procedure. Below the hood the implementation of a NN and its training
makes no circles or arrows, instead it is a clever sequence of matrix
We will go through the conceptual representation of a NN and then expand it and
describe the implementation. We will look at some mathematical concepts and
then we will see how to implement those concepts in
NumPy. We will use
NumPy but we will see that we can substitute the
NumPy parts of our code
with GPU or distributed processing.
The objective is to understand how NN libraries are programmed, and understand
where the GPU (
pytorch) or the distributed processing
PySpark) is used within a NN library. We want to demystify NN libraries, to
prove that it is possible to write a simple NN library in less than 1000 lines
NumPy matrix multiplication (
np.dot) knowledge is required.