In this lesson, we'll learn how to train two kinds of artificial neural networks to detect handwritten digits in an image. By the end of this lesson, students will be able to:
- Identify neural network model parameters and hyperparameters.
- Determine the number of weights and biases in a multilayer perceptron and convolutional neural network.
- Explain how the layers of a convolutional neural network extract information from an input image.
First, let's watch the beginning of 3Blue1Brown's introduction to neural networks while we wait for the imports and dataset to load. We'll also later explore the [**TensorFlow Playground**](https://playground.tensorflow.org/#activation=linear&batchSize=10&dataset=circle®Dataset=reg-plane&learningRate=0.03®ularizationRate=0&noise=0&networkShape=&seed=0.67725&showTestData=false&discretize=false&percTrainData=50&x=true&y=true&xTimesY=false&xSquared=false&ySquared=false&cosX=false&sinX=false&cosY=false&sinY=false&collectStats=false&problem=classification&initZero=false&hideText=false) to learn more about neural networks from the perspective of linear models.
For this lesson, we'll re-examine machine learning algorithms from scikit-learn. We'll also later use `keras`, a machine learning library designed specifically for building complex neural networks.
We'll be working with the [MNIST dataset](https://en.wikipedia.org/wiki/MNIST_database), which is composed of 70,000 images of handwritten digits, of which 60,000 were drawn from employees at the U.S. Census Bureau and 10,000 drawn from U.S. high school students.
In the video clip above, we saw how to transform an image from a 28-by-28 square to a 784-length vector that takes each of the 28 rows of 28-wide pixels and arranges them side-by-side in a line. This process **flattens** the image from 2 dimensions to 1 dimension.
To create a neural network, scikit-learn provides an `MLPClassifier`, or **multilayer perceptron** classifier, that can be used to match the video example with two hidden layers of 16 neurons each. While we wait for the training to complete, let's watch the rest of the video.
Neural networks are highly sensistive to hyperparameter values such as the width and depth of hidden layers. Other hyperparameter values like the initial learning rate for gradient descent can also affect training efficacy. Early stopping is used to evaluate performance on a validation set accuracy (rather than training set loss) in order to determine when to stop training.
We can also [visualize MLP weights (coefficients) on MNIST](https://scikit-learn.org/stable/auto_examples/neural_networks/plot_mnist_filters.html). These 28-by-28 images represent each of the 40 neurons in this single-layer neural network.
In the 3Blue1Brown video, we examined how a single neuron could serve as an edge detector. But in a plain multilayer perceptron, neurons are linked directly to specific inputs (or preceding hidden layers), so they are location-sensitive. The MNIST dataset was constructed by centering each digit individually in the middle of the box. In the real-world, we might not have such perfectly-arranged image data, particularly when we want to identify real-world objects in a complex scene (which is probably harder than identifying handwritten digits centered on a black background).
**Convolutional neural networks** take the idea of a neural network and applies it to learn the weights in a convolution kernel.
The [following example](https://keras.io/examples/vision/mnist_convnet/), courtesy of François Chollet (the original author of Keras), shows how to load in the MNIST dataset using Keras.
The Keras `Sequential` model allows us to specify the specific sequence of layers and operations to pass from one step to the next.
- The `Input` layer handles inputs of the given shape.
- The `Conv2D` layer learns a convolution kernel with `kernel_size` number of weights plus a bias. It outputs the given number of `filters`, such as 32 or 64 used in the example below.
- The `MaxPooling2D` layer to downsample the output from a `Conv2D` layer. The maximum value in each 2-by-2 window is passed to the next layer.
- The `Flatten` layer flattens the given data into a single dimension.
- The `Dropout` layer randomly sets input values to 0 at the given frequency during training to help prevent overfitting. (Bypassed during **inference**: evaluation or use of the model.)
- The `Dense` layer is a regular densely-connected neural network layer like what we learned before.
Whereas `MLPClassifier` counted an entire round through the training data as an iteration, Keras uses the term **epoch** to refer to the same idea of iterating through the entire training dataset and performing gradient descent updates accordingly. Here, each gradient descent update step examines 200 images each time, so there are a total of 270 update steps for the 54000 images in the training set.
Write Keras code to recreate the two-hidden-layer multilayer perceptron model that we built using scikit-learn with the expression `MLPClassifier(hidden_layer_sizes=(16, 16))`. For the hidden layers, specify `activation="relu"` to match scikit-learn.
To visualize a convolutional layer, we can apply a similar technique to plot the weights for each layer. Below are the 32 convolutional kernels learned by the first `Conv2D` layer.
The remaining `Conv2D` and `Dense` layers become much harder to visualize because they have so many weights to examine. So let's instead [visualize how the network activates in response to a sample image](https://github.com/fchollet/deep-learning-with-python-notebooks/blob/master/first_edition/5.4-visualizing-what-convnets-learn.ipynb). The first plot below shows the result of convolving each of the above kernels on a sample image. The kernels above act as edge detectors.
Let's compare this result to another kernel by examining the table of filters above and changing the last indexing digit to a different value between 0 and 31.
The activations from this first layer are passed as inputs to the `MaxPooling2D` second layer, and so forth. We can visualize this whole process by creating a plot that shows how the inputs flow through the model.
