Summary of Krizhevsky’s ImmageNet Classification with Deep Convolutional Neural Networks

ImageNet Classification with Deep Convolutional Neural Networks

Introduction

Alex Krizhevsky, Ilya Sutskeyer, and Geoffrey E. Hinton from the University of Toronto created a neural network architecture, using convolutional layers, and called it ‘AlexNet’. They competed and won the ImageNet Large-Scale Visual Recognition Challenge (ILSVRC). Here we will summarize the paper in which they describe the architecture and training of their deep convolutional network. They pioneered object detection and image classification as we know it today.

Procedures

ImageNet has a dataset of over 15 million high-resolution labeled images, belonging to 22,000 different classes. They were able to do this by collecting images from the web and using Amazon’s Mechanical Turk crowd-sourcing tool (More info on Amazon’s Mechanical Turk crowd-sourcing tool here).

Architecture

They tested many different architectures, removing and adding layers until they got to, two GPU’s running 8 learned layers, consisting of 5 convolutional layers and three fully connected layers. Below is an image of their architecture depicting the two GPU’s, where one runs the layer-part at the top of the figure while the other runs the layer-part at the bottom, each communicating in certain layers.

Key features of their neural network architecture

ReLU Nonlinearity: To prevent overfiting and have faster learning they decided to not use sigmoid or tanh activation functions which were more common at the time.

Results

They were able to win and although their results may not look impresive in today’s standard they were able to significantly improve what had been seen before and even claimed that “Our results can be improved simply by waiting for faster GPU’s and bigger datasets to become available”. Below are the results of the ILSVRC 2010 and 2012 competitions.

Conclusion:

With the quick advances in GPU speed and bigger datasets, not to mention overall software and hardware upgrades, their research and advancement in the field has proven to be invaluable.

Personal Notes

I find it exciting what they were able to accomplish and humbled by how technology has advanced in the last decade. Only time will tell what the next decade has in store, and I hope I can be part of its progress.