Transfer Learning: Training a Convolutional Neural Network to Classify the CIFAR 10 Dataset

4 min readJul 4, 2020

Abstract

To learn more about Transfer Learning, we were tasked with training a convolutional neural network to classify the CIFAR 10 Dataset, using one of the pre-trained applications listed in Keras Applications, and achieve a validation accuracy of 88% or above. I went with the VGG Model, which earned top performance in the ILSVRC 2014 competition. Below we will discuss what is transfer learning, CIFAR 10 Dataset and my findings through out this project.

Introduction

What is transfer learning?

Humans use transfer learning to pass on knowledge through generations. A parent teaching their child how to speak is an example of transfer learning. As individuals we use transfer learning as well. We are able to transfer what we know in a field such as mathematics and use it to better understand and learn machine learning. In machine learning, transfer learning can be used when you do not have enough labeled data to train a neural network, so you use a already trained model and tweak it to build a new model for your problem. Some data scientists and researchers even believe that it can get us closer to true artificial general intelligence. (More on AGI here)

Materials and Methods

CIFAR 10 Dataset

The CIFAR 10 is a collection of images computer scientist and researchers can use to train their machine learning algorithms. It contains 60,000 32x32 images in 10 different classes, 50,000 of the images for training and 10,000 for testing. The classes include airplanes, automobiles, birds, cats, deer, dogs, frogs, horses, ships, and trucks. These are very small images, intended for computer vision research. Fig 1 shows examples of the photos, as you see, they are very small and low render, making classification an extra challenge.

The Keras Applications to choose from were trained with a much larger dataset called ImageNet. They used 1.2 million images and categorized them into 1,000 different classes.

Loading and Scaling

I first started by loading and scaling the images as shown below. Dividing it by 255 made it so that the data would be scaled between 0 and 1.

Regularization

I also used Dropout Regularization to slow down rate of convergence. Choosing a dropout rate of 50%, meaning it retains 50% of the nodes.

Data Augmentation

Data augmentation allowed me to make copies of the images with small random changes, expanding my dataset. Since the images are very small we do not want to use anything that may distort them. I ended up choosing to flip the images horizontally and shift the height and width by 10%. Code sample at the bottom of the article.

Results

At the end the model reached the target validation accuracy of over 88% after 20 epochs. It continued hovering between 88% and 89% all the way to the 50th epoch.

Discussion

In the future I would like to mess around more with different dropout rates, and freezing certain parts of the model. The model took 2 days to run all 50 epochs and crashed a few times on Google Colab.

Acknowledgments

Thanks to my classmates at Holberton, Santiago Velez, Jorge Zafra, Paulo Morillo, John Cook, Christian Williams, and Mia Morton, for sharing their insights and resources.