深度卷积神经网络在ImageNet数据库上的分类

admin管理员组

文章数量:1531243

2024年7月17日发(作者：)

深度卷积神经网络在ImageNet数据库上的分类

Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton

多伦多大学

摘要

我们训练了一个大型的深度卷积神经网络，来将在ImageNet LSVRC-2010大赛中的120

万张高清图像分为1000个不同的类别。对测试数据，我们得到了top-1误差率37.5%，

以及top-5误差率17.0%，这个效果比之前最顶尖的都要好得多。该神经网络有6000万

个参数和650,000个神经元，由五个卷积层，以及某些卷积层后跟着的max-pooling层，

和三个全连接层，还有排在最后的1000-way的softmax层组成。为了使训练速度更快，

我们使用了非饱和的神经元和一个非常高效的GPU关于卷积运算的工具。为了减少全连

接层的过拟合，我们采用了最新开发的正则化方法，称为“dropout”，它已被证明是非常

有效的。在ILSVRC-2012大赛中，我们又输入了该模型的一个变体，并依靠top-5测试

误差率15.3%取得了胜利，相比较下，次优项的错误率是26.2%。

1 引言

当前物体识别的方法基本上都使用了机器学习方法。为了改善这些方法的性能，我们可

以收集更大的数据集，学习更强有力的模型，并使用更好的技术，以防止过拟合。直到

最近，标记图像的数据集都相当小——大约数万张图像（例如，NORB [16]，Caltech-101/256

[8, 9]，以及CIFAR-10/100 [12]）。简单的识别任务可以用这种规模的数据集解决得相当好，

特别是当它们用标签-保留转换增强了的时候。例如，在MNIST数字识别任务中当前最

好的误差率（<0.3%）接近于人类的表现[4]。但是现实环境中的物体表现出相当大的变化，

因此要学习它们以对它们进行识别就必须使用更大的训练集。事实上，小规模图像数据

集的缺陷已被广泛认同（例如，Pinto等人[21]），但是直到最近，收集有着上百万张图像

的带标签数据集才成为可能。更大型的新数据集包括LabelMe [23]，它由几十万张完全分

割图组成，还有ImageNet [6]，它由多于22,000个种类中超过1500万张带标签的高分辨

率图像组成。

为了从几百万张图像中学习数以千计的物体，我们需要一个学习能力更强的模型。然而，

物体识别任务的极大复杂性意味着这个问题不能被指定，即使是通过与ImageNet一样大

的数据集，所以我们的模型中也应该有大量的先验知识，以补偿我们所没有的全部数据。

卷积神经网络（CNN）构成了一个这种类型的模型[16, 11, 13, 18, 15, 22, 26]。它们的能力

可以通过改变其深度与广度得到控制，它们也可作出有关图像性质的强壮且多半正确的

假设（即，统计数据的稳定性和像素依赖关系的局部性）。因此，与层次规模相同的标准

前馈神经网络相比，CNN的连接关系和参数更少，所以更易于训练，而其理论上的最佳

性能可能只略差一点。

不论CNN的性质多有吸引力，也不论它们局部结构的相对效率有多高，将它们大规模地

应用到高分辨率图像中仍然是极其昂贵的。幸运的是，目前的GPU搭配了一个高度优化

的2D卷积工具，强大到足以促进大规模CNN的训练，而且最近的数据集像ImageNet

包含足够的带标签的样例来训练这样的模型，还不会有严重的过拟合。

本文的具体贡献如下：我们在ILSVRC-2010和ILSVRC-2012大赛中使用过的ImageNet

的子集上[2]，训练了迄今为止最大型的卷积神经网络之一，并取得了迄今为止在这些数

据集上报告过的最好结果。我们写了一个高度优化的GPU二维卷积工具以及训练卷积神

经网络过程中的所有其他操作，这些我们都提供了公开地址。我们的网络中包含一些既

新鲜而又不同寻常的特征，它们提高了网络的性能，并减少了网络的训练时间，这些详

见第3节。我们的网络中甚至有120万个带标签的训练样本，这么大的规模使得过拟合

成为一个显著的问题，所以我们使用了几种有效的方法来防止过拟合，这些在第4节中

给以描述。我们最终的网络包含五个卷积层和三个全连接层，且这种层次深度似乎是重

要的：我们发现，移去任何卷积层（其中每一个包含的模型参数都不超过1%）都会导致

性能变差。

最后，网络的规模主要受限于当前GPU的可用内存和我们愿意容忍的训练时间。我们的

网络在两块GTX 580 3GB GPU上训练需要五到六天。我们所有的实验表明，等更快的

GPU和更大的数据集可用以后，我们的结果就可以轻而易举地得到改进。

2 数据集

ImageNet是一个拥有超过1500万张带标签的高分辨率图像的数据集，这些图像分属于大

概22,000个类别。这些图像是从网上收集，并使用Amazon Mechanical Turk群众外包工

具来人工贴标签的。作为PASCAL视觉目标挑战赛的一部分，一年一度的ImageNet大型

视觉识别挑战赛（ILSVRC）从2010年开始就已经在举办了。ILSVRC使用ImageNet的

一个子集，分为1000种类别，每种类别中都有大约1000张图像。总之，大约有120万

张训练图像，50,000张验证图像和150,000张测试图像。

ILSVRC-2010是ILSVRC中能获得测试集标签的唯一版本，因此这也就是我们完成大部

分实验的版本。由于我们也在ILSVRC-2012上输入了模型，在第6节中我们也会报告这

个数据集版本上的结果，该版本上的测试集标签难以获取。在ImageNet上，习惯性地报

告两个误差率：top-1和top-5，其中top-5误差率是指测试图像上正确标签不属于被模型

认为是最有可能的五个标签的百分比。

ImageNet由各种分辨率的图像组成，而我们的系统需要一个恒定的输入维数。因此，我

们下采样这些图像到固定的分辨率256×256。给定一张矩形图像，我们首先重新缩放图

像，使得短边长度为256，然后从得到的图像中裁剪出中央256×256的一片。除了遍历

训练集从每个像素中减去平均活跃度外，我们没有以任何其他方式预处理图像。所以我

们用这些像素（中央那一片的）原始RGB值训练网络。

3 体系结构

图2总结了我们网络的体系结构。它包含八个学习层——五个卷积层和三个全连接层。

下面，我们将介绍该网络体系结构的一些新颖独特的功能。3.1-3.4是根据我们对于其重

要性的估计来排序的，最重要的排在最前面。

2

3.1 ReLU非线性

将神经元的输出f，作为其输入x的函数，对其建模的标准方法是用

f(x)tanh(x)

或者

f(x)



1e

x



。就梯度下降的训练时间而言，这些饱和非线性函数比不饱和非线性函

1

数

f



x



max



0,x



要慢的多。我们跟随Nair和Hinton[20]称这种不饱和非线性的神经

元为修正线性单元（ReLU）。训练带ReLUs的深度卷积神经网络比带tanh单元的同等网

络要快好几倍。如图1所示，它显示出对于特定的四层卷积网络，在CIFAR-10数据集上

达到25%的训练误差所需的迭代次数。此图显示，如果我们使用了传统的饱和神经元模

型，就不能用如此大的神经网络来对该工作完成实验。

图1：带ReLU的四层卷积神经网络（实线）在CIFAR-10数据集上达到25%训练误差率

要比带tanh神经元的同等网络（虚线）快六倍。每个网络的学习速率是独立选取的，以

使得训练尽可能快。没有使用任何形式的正则化。这里演示的效果因网络结构的不同而

不同，但带ReLU的网络学习始终比带饱和神经元的同等网络快好几倍。

我们不是第一个在CNN中考虑传统神经元模型的替代品的。例如，Jarrett等人[11]声称，

非线性函数

f



x



tanh



x



由于其后跟随局部average pooling的对比度归一化的类型，它

在Caltech-101数据集上工作得特别好。然而，在该数据集上的主要关注点是防止过拟合，

所以他们正在观察的效果不同于我们报告的为拟合训练集使用ReLU时的加速能力。更

快的学习对大型数据集上训练的大型模型的性能有很大影响。

3.2 在多个GPU上训练

单个GTX 580 GPU只有3GB内存，这限制了可以在其上训练的网络的最大规模。事实

证明，120万个训练样本才足以训练网络，这网络太大了，不适合在一个GPU上训练。

因此我们将网络分布在两个GPU上。目前的GPU特别适合跨GPU并行化，因为它们能

够直接从另一个GPU的内存中读出和写入，不需要通过主机内存。我们采用的并行方案

基本上是在每个GPU中放置一半核（或神经元），还有一个额外的技巧：GPU间的通讯

只在某些层进行。这就是说，例如，第3层的核需要从第2层中所有核映射输入。然而，

第4层的核只需要从第3层中位于同一GPU的那些核映射输入。选择连接模式是一个交

叉验证的问题，但是这让我们可以精确地调整通信量，直到它的计算量在可接受的部分。

3

由此产生的体系结构有点类似于Ciresan等人提出的“柱状”CNN的体系结构[5]，不同

之处在于我们的纵列不是独立的（见图2）。与在一个GPU上训练的每个卷积层有一半核

的网络比较，该方案将我们的top-1与top-5误差率分别减少了1.7%与1.2%。训练双GPU

网络比训练单GPU网络花费的时间略少一些 （实际上单GPU网络与双GPU网络在最

后的卷积层有着相同数量的核。这是因为大多数网络的参数在第一个全连接层，这需要

上一个卷积层作为输入。所以，为了使两个网络有数目大致相同的参数，我们不把最后

一个卷积层大小减半（也不把它后面跟随的全连接层减半）。因此，这种比较关系更偏向

有利于单GPU网络，因为它比双GPU网络的“一半大小”要大）。

3.3 局部响应归一化

ReLU具有所希望的特性，它们不需要输入归一化来防止它们达到饱和。如果至少有一些

训练样例对ReLU产生了正输入，学习就将发生在那个神经元。可是，我们仍然发现下

i

列局部归一化方案有助于一般化。用

a

x,y

表示点 （x,y）处通过应用核计算出的神经元

i

激活度，然后应用ReLU非线性，响应归一化活性

b

x,y

由下式给出

min



N1,in/2





2



i

k







a

x,y









jmax



0,in/2





b

i

x,y

a

i

x,y

其中求和覆盖了n个“相邻的”位于相同空间位置的核映射，N是该层中的核总数。核

映射的顺序当然是任意的，且在训练开始前就确定。受到在真实神经元中发现的类型启

发，这种响应归一化实现了一种侧向抑制，在使用不同核计算神经元输出的过程中创造

对大激活度的竞争。常数k，n，α和β是超参数，它们的值要用验证集来确定；我们使

用

k2,n5,



10

4

,



=0.75

。我们在某些层应用ReLU归一化后再应用这种归一化（见

3.5节）。

该方案与Jarrett等人的局部对比度归一化方案具有一些相似之处[11]，但我们的方案更正

确的命名为“亮度归一化”，因为我们不减去平均活跃度。响应归一化将我们的top-1与

top-5误差率分别减少了1.4%与1.2%。我们也验证了该方案在CIFAR-10数据集上的有效

性：四层CNN不带归一化时的测试误差率是13%，带归一化时是11%。

3.4 重叠Pooling

CNN中的Pooling层总结了同一核映射中邻近神经元组的输出。传统上，通过邻接pooling

单元总结的邻近关系不重叠（例如，[17,11,4]）。更准确地说，一个pooling层可以被认为

是由间隔s像素的pooling单元网格组成，每个网格总结出一个z×z大小的邻近关系，都

位于pooling单元的中心位置。若设s=z，我们得到传统的局部pooling，正如常用于CNN

中的那样。若设s

法，s=2，z=3。与无重叠的s=z=2相比，这一模式在产生相同维度的输出时分别将Top-1

和Top-5降低了0.4%和0.3%。我们还观察到，采用有重叠的池化能稍稍让模型更难过拟

合。

4

3.5 总体结构

现在，我们已经准备好描述CNN的总体结构。如图2所示，该网络包括八个带权层；前

五层是卷积层，剩下三层是全连接层。最后一个全连接层的输出被送到一个1000-way的

softmax层，其产生一个覆盖1000类标签的分布。我们的网络使得多分类的Logistic回归

目标最大化，这相当于最大化了预测分布下训练样本中正确标签的对数概率平均值。

第二、第四和第五个卷 积层的核只连接到前一个卷积层也位于同一GPU中的那些核

映射上（见图2）。第三个卷积层的核被连接到第二个卷积层中的所有核映射上。全连接

层中的神经元被连接到前一层中所有的神经元上。响应归一化层跟在第一、第二个卷积

层后面。3.4节中描述的那种最大Pooling层，跟在响应归一化层以及第五个卷积层之后。

ReLU非线性应用于每个卷积层及全连接层的输出。

第一个卷积层利用96个大小为11×11×3、步长为4个像素（这是同一核映射中邻近神

经元的感受野中心之间的距离）的核，来对大小为224×224×3的输入图像进行滤波。

第二个卷积层需要将第一个卷积层的（响应归一化及池化的）输出作为自己的输入，且

利用256个大小为5×5×48的核对其进行滤波。第三、第四和第五个卷积层彼此相连，

没有任何介于中间的pooling层与归一化层。第三个卷积层有384个大小为3×3×256的

核被连接到第二个卷积层的（归一化的、池化的）输出。第四个卷积层拥有384个大小

为3×3×192的核，第五个卷积层拥有256个大小为3×3×192的核。全连接层都各有

4096个神经元。

图2：CNN体系结构示意图，明确显示了两个GPU之间的职责划分。一个GPU运行图

中顶部的层次部分，而另一个GPU运行图中底部的层次部分。GPU之间仅在某些层互相

通信。该网络的输入是150,528维的，且该网络剩下各层的神经元数分别为

253,440-186,624-64,896-64,896-43,264-4096-4096–1000。

4 减少过拟合

我们的神经网络结构有6000万个参数。虽然ILSVRC的1000个类别使得每个训练样本

强加10比特约束到从图像到标签的映射上，这显示出要学习如此多的参数而不带相当大

的过拟合，这些类别是不够的。下面，我们描述减少过拟合的两种主要方法。

4.1 数据增强

5

减少图像数据过拟合最简单最常用的方法，是使用标签-保留转换，人为地扩大数据集（例

如，[25,4,5]）。我们使用数据增强的两种不同形式，这两种形式都允许转换图像用很少的

计算量从原始图像中产生，所以转换图像不需要存储在磁盘上。在我们的实现中，转换

图像是由CPU上的Python代码生成的，而GPU是在之前那一批图像上训练的。所以这

些数据增强方案实际上是计算自由。

数据增强的第一种形式由生成图像转化和水平反射组成。为此，我们从256×256的图像

中提取随机的224×224的碎片（还有它们的水平反射），并在这些提取的碎片上训练我

们的网络（这就是图2中输入图像是224×224×3维的原因）。这使得我们的训练集规模

扩大了2048倍，但是由此产生的训练样例一定高度地相互依赖。如果没有这个方案，我

们的网络会有大量的过拟合，这将迫使我们使用小得多的网络。在测试时，该网络通过

提取五个224×224的碎片（四个边角碎片和中心碎片）连同它们的水平反射（因此总共

是十个碎片）做出了预测，并在这十个碎片上来平均该网络的softmax层做出的预测。

数据增强的第二种形式包含改变训练图像中RGB通道的强度。具体来说，我们在遍及整

个ImageNet训练集的RGB像素值集合中执行PCA。对于每个训练图像，我们成倍增加

已有主成分，比例大小为对应特征值乘以一个从均值为0，标准差为0.1的高斯分布中提

RGB



取的随机变量。这样一来，对于每个RGB图像像素

I

xy





I,I,I

xyxyxy



，我们增加下面这

项：



p

1

,p

2

,p

3





1



1

,



2



2

,



3



3



T

其中

p

i

和



i

分别是RGB像素值的3×3协方差矩阵的第i个特征向量与特征值，



i

是

前面提到的随机变量。每个



i

对于特定训练图像的全部像素只提取一次，直到那个图像

再次被用于训练，在那时它被重新提取。这个方案大致抓住了自然图像的一个重要属性，

即，光照强度与颜色是变化的，而对象识别是不变的。该方案将top-1误差率减少了1%

以上。

4.2 Dropout

结合许多不同模型的预测是一种非常成功的减少测试误差的方式[1,3]，但它先前训练花

了好几天时间，似乎对于大型神经网络来说太过昂贵。然而，有一个非常有效的模型组

合版本，它在训练中只花费两倍于单模型的时间。最近推出的叫做“dropout”的技术[10]，

它做的就是以0.5的概率将每个隐层神经元的输出设置为零。以这种方式“dropped out”

的神经元既不利于前向传播，也不参与反向传播。所以每次提出一个输入，该神经网络

就尝试一个不同的结构，但是所有这些结构之间共享权重。因为神经元不能依赖于其他

特定神经元而存在，所以这种技术降低了神经元复杂的互适应关系。正因如此，要被迫

学习更为鲁棒的特征，这些特征在结合其他神经元的一些不同随机子集时有用。在测试

时，我们将所有神经元的输出都仅仅只乘以0.5，对于获取指数级dropout网络产生的预

测分布的几何平均值，这是一个合理的近似方法。

我们在图2中前两个全连接层使用dropout。如果没有dropout，我们的网络会表现出大量

6

的过拟合。dropout使收敛所需的迭代次数大致增加了一倍。

5 学习的详细过程

我们使用随机梯度下降法和一批大小为128、动力为0.9、权重衰减为0.0005的样例来训

练我们的网络。我们发现，这少量的权重衰减对于模型学习是重要的。换句话说，这里

的权重衰减不仅仅是一个正则化矩阵：它减少了模型的训练误差。对于权重w的更新规

则为

v

i1

:0.9v

i

0.0005



w

i







w

i1

:w

i

v

i1

L

w

w

i

D

i

其中i是迭代指数，v是动力变量，ε是学习率，

第i批样例

D

i

的平均值。

L

w

w

i

D

i

是目标关于w对

w

i

的导数在

我们用一个均值为0、标准差为0.01的高斯分布初始化了每一层的权重。我们用常数1

初始化了第二、第四和第五个卷积层以及全连接隐层的神经元偏差。该初始化通过提供

带正输入的ReLU来加速学习的初级阶段。我们在其余层用常数0初始化神经元偏差。

我们对于所有层都使用了相等的学习率，这是在整个训练过程中手动调整的。我们遵循

的启发式是，当验证误差率在当前学习率下不再提高时，就将学习率除以10。学习率初

始化为0.01，在终止前降低三次。我们训练该网络时大致将这120万张图像的训练集循

环了90次，在两个NVIDIA GTX 580 3GB GPU上花了五到六天。

6 结果

我们在ILSVRC-2010测试集上的结果总结于表1中。我们的网络实现了top-1测试集误

差率37.5%，top-5测试集误差率17.0% （若没有如4.1节所述的在十个碎片上平均预测，

误差率是39.0%与18.3%）。ILSVRC-2010大赛中取得的最好表现是47.1%与28.2%，它

的方法是用不同特征训练六个sparse-coding模型，对这些模型产生的预测求平均值[2]，

自那以后公布的最好结果是45.7%与25.7%，它的方法是从两类密集采样的特征中计算出

费舍尔向量（FV），用费舍尔向量训练两个分类器，再对这两个分类器的预测求平均值

[24]。

表1：ILSVRC-2010测试集上的结果比较。斜体字是他人取得的最好结果。

我们也在ILSVRC-2012大赛中输入了我们的模型，并在表2中报告结果。由于

ILSVRC-2012测试集标签是不公开的，我们不能对试过的所有模型都报告测试误差率。

7

在本段的其余部分，我们将验证误差率与测试误差率互换，因为根据我们的经验，它们

之间相差不超过0.1%（见表2）。本文所描述的CNN实现了18.2%的top-5误差率。对五

个相似CNN的预测求平均值得出了16.4%的误差率。训练一个在最末pooling层之后还

有一个额外的第六个卷积层的CNN，用以对整个ImageNet 2011年秋季发布的图像（15M

张图像，22K种类别）进行分类，然后在ILSVRC-2012上“微调”它，这种方法得出了

16.6%的误差率。用在整个2011年秋季发布的图像上预训练的两个CNN，结合先前提到

的五个CNN，再对这七个CNN作出的预测求平均值，这种方法得出了15.3%的误差率。

比赛中的第二名实现了26.2%的误差率，用的方法是从不同类密集采样的特征中计算FV，

用FV训练几个分类器，再对这几个分类器的预测求平均值[7]。

表2：在ILSVRC-2012验证集与测试集上的误差率比较。斜体字是由他人取得的最好结

果。带星号的模型是经过“预训练”以对整个ImageNet 2011年秋季发布的图像集进行分

类的。详见第6节。

6.1 定性评价

图3显示了通过该网络的两个数据连接层学习到的卷积核。该网络已经学习到各种各样

的频率与方向选择核，以及各种颜色的斑点。注意两个GPU显现出的特性，3.5节中描

述了一个结果是限制连接。GPU1上的核大多数颜色不明确，而GPU2上的核大多数颜色

明确。这种特性在每一次运行中都会出现，且独立于所有特定的随机权重初始化（以GPU

的重新编数为模）。

图3：通过 的输入图像上第一个卷积层学习到的96个大小为11*11*3 的卷积核。顶部

的48个核是从GPU1上学到的，底部的48个核是从GPU2上学到的。详见6.1节。

在图4左边面板上，通过计算该网络在八个测试图像上的top-5预测，我们定性地判断它

学到了什么。注意到即使是偏离中心的物体，比如左上角的一小块，也可以被网络识别。

大多数的top-5标签似乎合情合理。例如，只有其他类型的猫科动物被认为是对豹貌似合

理的标签。在某些情况下（铁栅、樱桃），对于图片意图的焦点存在歧义。

探测网络的视觉知识有另一种方法，就是考虑由位于最后的4096维隐层上的图像引起的

特征激活。如果两个图像用小欧氏分离产生了特征激活向量，我们可以说，在神经网络

的更高级别上认为它们是相似的。图4显示了测试集中的五个图像，以及训练集中根据

这一标准与其中每一个最相似的六个图像。注意，在像素级别，检索到的训练图像一般

8

不会接近第一列中的查询图像。例如，检索到的狗和大象表现出各种各样的姿势。我们

会在补充材料里给出更多测试图像的结果。

通过使用两个4096维实值向量之间的欧氏距离来计算相似性是低效的，但它可以通过训

练一个自动编码器将这些向量压缩为短的二进制代码来变得高效。这应该会产生一个比

应用自动编码器到原始像素要好得多的图像检索方法[14]，它不利用图像标签，此后还有

一种用相似边缘图案来检索图像的倾向，而不论它们在语义上是否相似。

图4：（左图）八个ILSVRC-2010测试图像，以及被我们的模型认为最有可能的五个标签。

正确的标签写在每个图像下面，正确标签的概率也以红色条予以显示（若它在前5之内）。

（右图）第一列是五个ILSVRC-2010测试图像。其余列显示了六个训练图像，它们在最

后的隐层产生的特征向量与测试图像的特征向量有最小的欧氏距离。

7 讨论

我们的研究结果表明，大型深度卷积神经网络在一个非常具有挑战性的数据集上使用纯

粹的监督学习，能够达到破纪录的结果。值得注意的是，如果有一个卷积层被移除，我

们的网络性能就会降低。例如，除去任何中间层都将导致该网络的top-1性能有2%的损

失。所以该层次深度对于达到我们的结果确实是重要的。

为了简化实验，我们没有使用任何无监督的预训练，即使我们预计它将带来帮助，特别

是我们可以获得足够的计算能力来显著地扩大网络规模，而不带来标记数据量的相应增

加。到目前为止，我们的结果有所改善，因为我们已经让网络更大，训练时间更久，但

是为了匹配人类视觉系统的infero-temporal路径，我们仍然有更高的数量级要去达到。最

终我们想要在视频序列上使用非常大型的深度卷积网络，其中的瞬时结构会提供非常有

用的信息，这些信息在静态图像中丢失了或极不明显。

参考文献

[1] R.M. Bell and Y. Koren. Lessons from the netflix prize challenge. ACM SIGKDD

Explorations Newsletter, 9(2):75-79, 2007.

[2] A. Berg, J. Deng, and L. Fei-Fei. Large scale visual recognition challenge 2010.

/challenges. 2010.

9

[3] L. Breiman. Random forests. Machine learning, 45(1):5-2, 2001.

[4] D. Cirsan, U. Meier, and J. Schmidhuber. Multi-column deep neural networks for image

preprint arXiv:1202.2745, 2012.

[5] D.C. Ciresan, U. Meier, J. Masci, L.M. Gambardella, and J. Schmidhuber.

High-performance neural networks for visual object classification. Arxiv preprint

arXiv:1102.0183, 2011.

[6] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale

Hierarchical Image Database. In CVPR09, 2009.

[7] J. Deng, A. Berg, S. Satheesh, H. Su, A. Khosla, and L. Fei-Fei. ILSVRC-2012, 2012. URL

/challenges/LSVRC/2012/.

[8] L. Fei-Fei, R. Fergus, and P. Perona. Learning generative visual models from few training

examples: An incremental bayesian approach tested on 101 object categories. Computer Vision

and Image Understanding, 106(1):59-70, 2007.

[9] G. Griffin, A. Holub, and P. Perona. Caltech-256 object category dataset. Technical Report

7694, California Institute of Technology, 2007. URL /7694.

[10] G.E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and R.R. Salakhutdinov.

Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint

arXiv:1207.0580, 2012.

[11] K. Jarrett, K. Kavukcuoglu, M. A. Ranzato, and Y. LeCun. What is the best multi-stage

architecture for

object recognition? In International Conference on Computer Vision, pages 2146?153. IEEE,

2009.

[12] A. Krizhevsky. Learning multiple layers of features from tiny images. Master’s thesis,

Department of Computer Science, University of Toronto, 2009.

[13] A. Krizhevsky. Convolutional deep belief networks on cifar-10. Unpublished manuscript,

2010.

[14] A. Krizhevsky and G.E. Hinton. Using very deep autoencoders for content-based image

retrieval. In ESANN, 2011.

[15] Y. Le Cun, B. Boser, J.S. Denker, D. Henderson, R.E. Howard, W. Hubbard, L.D. Jackel,

et al. Handwritten digit recognition with a back-propagation network. In Advances in neural

information processing systems, 1990.

[16] Y. LeCun, F.J. Huang, and L. Bottou. Learning methods for generic object recognition

with invariance to

pose and lighting. In Computer Vision and Pattern Recognition, 2004. CVPR 2004.

Proceedings of the 2004 IEEE Computer Society Conference on, volume 2, pages II-97. IEEE,

2004.

[17] Y. LeCun, K. Kavukcuoglu, and C. Farabet. Convolutional networks and applications in

vision. In Circuits and Systems (ISCAS), Proceedings of 2010 IEEE International Symposium

on, pages 253-256. IEEE, 2010.

[18] H. Lee, R. Grosse, R. Ranganath, and A.Y. Ng. Convolutional deep belief networks for

scalable unsupervised learning of hierarchical representations. In Proceedings of the 26th

Annual International Conference on Machine Learning, pages 609-616. ACM, 2009.

[19] T. Mensink, J. Verbeek, F. Perronnin, and G. Csurka. Metric Learning for Large Scale

Image Classification: Generalizing to New Classes at Near-Zero Cost. In ECCV - European

10

Conference on Computer Vision, Florence, Italy, October 2012.

[20] V. Nair and G. E. Hinton. Rectified linear units improve restricted boltzmann machines. In

Proc. 27

th

International Conference on Machine Learning, 2010.

[21] N. Pinto, D.D. Cox, and J.J. DiCarlo. Why is real-world visual object recognition hard?

PLoS computational biology, 4(1):e27, 2008.

[22] N. Pinto, D. Doukhan, J.J. DiCarlo, and D.D. Cox. A high-throughput screening approach

to discovering

good forms of biologically inspired visual representation. PLoS computational biology,

5(11):e1000579, 2009.

[23] B.C. Russell, A. Torralba, K.P. Murphy, and W.T. Freeman. Labelme: a database and

web-based tool for image annotation. International journal of computer vision, 77(1):157-173,

2008.

[24] J. Sachez and F. Perronnin. High-dimensional signature compression for large-scale image

classification. In Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference

on, pages 1665-1672. IEEE, 2011.

[25] P.Y. Simard, D. Steinkraus, and J.C. Platt. Best practices for convolutional neural networks

applied to visual document analysis. In Proceedings of the Seventh International Conference on

Document Analysis and Recognition, volume 2, pages 958?62, 2003.

[26] S.C. Turaga, J.F. Murray, V. Jain, F. Roth, M. Helmstaedter, K. Briggman,W. Denk, and

H.S. Seung. Convolutional networks can learn to generate affinity graphs for image

segmentation. Neural Computation,

22(2):511-538, 2010.

11

ImageNet Classification with Deep Convolutional

Neural Networks

Alex Krizhevsky Ilya Sutskever Geoffrey E. Hinton

University of Toronto University of Toronto University of Toronto

ilya@ kriz@ hinton@

Abstract

We trained a large, deep convolutional neural network to classify the 1.2 million

high-resolution images in the ImageNet LSVRC-2010 contest into the 1000 different classes.

On the test data, we achieved top-1 and top-5 error rates of 37.5% and 17.0% which is

considerably better than the previous state-of-the-art. The neural network, which has 60 million

parameters and 650,000 neurons, consists

of five convolutional layers, some of which are followed by max-pooling layers,and three

fully-connected layers with a final 1000-way softmax. To make training faster, we used

non-saturating neurons and a very efficient GPU implementation of the convolution operation.

To reduce overfitting in the fully-connected layers we employed a recently-developed

regularization method called “dropout”

that proved to be very effective. We also entered a variant of this model in the ILSVRC-2012

competition and achieved a winning top-5 test error rate of 15.3%,compared to 26.2% achieved

by the second-best entry.

1 Introduction

Current approaches to object recognition make essential use of machine learning methods. To

improve

their performance, we can collect larger datasets, learn more powerful models, and use better

techniques for preventing overfitting. Until recently, datasets of labeled images were relatively

small — on the order of tens of thousands of images (e.g., NORB [16], Caltech-101/256 [8, 9],

and CIFAR-10/100 [12]). Simple recognition tasks can be solved quite well with datasets of

this size, especially if they are augmented with label-preserving transformations. For example,

the current best error rate on the MNIST digit-recognition task (<0.3%) approaches human

performance [4].But objects in realistic settings exhibit considerable variability, so to learn to

recognize them it is necessary to use much larger training sets. And indeed, the shortcomings of

small image datasets have been widely recognized (e.g., Pinto et al. [21]), but it has only

recently become possible to collect labeled datasets with millions of images. The new larger

datasets include LabelMe [23], which consists of hundreds of thousands of fully-segmented

images, and ImageNet [6], which consists of over 15 million labeled high-resolution images in

over 22,000 categories.

To learn about thousands of objects from millions of images, we need a model with a large

learning capacity. However, the immense complexity of the object recognition task means that

12

this problem cannot be specified even by a dataset as large as ImageNet, so our model should

also have lots of prior knowledge to compensate for all the data we don’t have. Convolutional

neural networks (CNNs) constitute one such class of models [16, 11, 13, 18, 15, 22, 26]. Their

capacity can be controlled by varying their depth and breadth, and they also make strong and

mostly correct assumptions about the nature of images (namely, stationarity of statistics and

locality of pixel dependencies).Thus, compared to standard feedforward neural networks with

similarly-sized layers, CNNs have much fewer connections and parameters and so they are

easier to train, while their theoretically-best performance is likely to be only slightly worse.

Despite the attractive qualities of CNNs, and despite the relative efficiency of their local

architecture, they have still been prohibitively expensive to apply in large scale to

high-resolution images. Luckily, current GPUs, paired with a highly-optimized implementation

of 2D convolution, are powerful enough to facilitate the training of interestingly-large CNNs,

and recent datasets such as ImageNet contain enough labeled examples to train such models

without severe overfitting.

The specific contributions of this paper are as follows: we trained one of the largest

convolutional neural networks to date on the subsets of ImageNet used in the ILSVRC-2010

and ILSVRC-2012 competitions [2] and achieved by far the best results ever reported on these

datasets. We wrote a highly-optimized GPU implementation of 2D convolution and all the

other operations inherent in training convolutional neural networks, which we make available

publicly1. Our network contains a number of new and unusual features which improve its

performance and reduce its training time, which are detailed in Section 3. The size of our

network made overfitting a significant problem, even with 1.2 million labeled training

examples, so we used several effective techniques for preventing overfitting, which are

described in Section 4. Our final network contains five convolutional and three fully-connected

layers, and this depth seems to be important: we found that removing any convolutional layer

(each of which contains no more than 1% of the model’s parameters) resulted in inferior

performance.

In the end, the network’s size is limited mainly by the amount of memory available on current

GPUs and by the amount of training time that we are willing to tolerate. Our network takes

between five and six days to train on two GTX 580 3GB GPUs. All of our experiments suggest

that our results can be improved simply by waiting for faster GPUs and bigger datasets to

become available.

2 The Dataset

ImageNet is a dataset of over 15 million labeled high-resolution images belonging to roughly

22,000 categories. The images were collected from the web and labeled by human labelers

using Amazon’s Mechanical Turk crowd-sourcing tool. Starting in 2010, as part of the Pascal

Visual Object Challenge, an annual competition called the ImageNet Large-Scale Visual

Recognition Challenge (ILSVRC) has been held. ILSVRC uses a subset of ImageNet with

roughly 1000 images in each of 1000 categories. In all, there are roughly 1.2 million training

13

images, 50,000 validation images, and 150,000 testing images.

ILSVRC-2010 is the only version of ILSVRC for which the test set labels are available, so this

is the version on which we performed most of our experiments. Since we also entered our

model in the ILSVRC-2012 competition, in Section 6 we report our results on this version of

the dataset as well, for which test set labels are unavailable. On ImageNet, it is customary to

report two error rates: top-1 and top-5, where the top-5 error rate is the fraction of test images

for which the correct label is not among the five labels considered most probable by the model.

ImageNet consists of variable-resolution images, while our system requires a constant input

dimensionality. Therefore, we down-sampled the images to a fixed resolution of 256



256.

Given a rectangular image, we first rescaled the image such that the shorter side was of length

256, and then cropped out the central 256



256 patch from the resulting image. We did not

pre-process the images in any other way, except for subtracting the mean activity over the

training set from each pixel. So we trained our network on the (centered) raw RGB values of

the pixels.

3 The Architecture

The architecture of our network is summarized in Figure 2. It contains eight learned layers

—five convolutional and three fully-connected. Below, we describe some of the novel or

unusual features of our network’s architecture. Sections 3.1-3.4 are sorted according to our

estimation of their importance, with the most important first.

3.1 ReLU Nonlinearity

The standard way to model a neuron’s output f as a function of its input x is

with

f(x)tanh(x)

or

f(x)



1e

x



. In terms of training time with gradient descent, these

1

saturating nonlinearities are much slower than the non-saturating nonlinearity

f



x



max



0,x



. Following Nair and Hinton[20]

,we refer to neurons with this nonlinearity as Rectified Linear Units (ReLUs). Deep

convolutional neural networks with ReLUs train several times faster than their equivalents with

tanh units. This is demonstrated in Figure 1, which shows the number of iterations required to

reach 25% training error on the CIFAR-10 dataset for a particular four-layer convolutional

network. This plot shows that we would not have been able to experiment with such large

neural networks for this work if we had used traditional saturating neuron models.

14

Figure 1: A four-layer convolutional neural network with ReLUs (solid line) reaches a 25%

training error rate on CIFAR-10 six times faster than an equivalent network with tanh neurons

(dashed line). The learning rates for each network were chosen independently to make training

as fast as possible. No regularization of any kind was employed. The magnitude of the effect

demonstrated here varies with network architecture, but networks with ReLUs consistently

learn several times faster than equivalents

with saturating neurons.

We are not the first to consider alternatives to traditional neuron models in CNNs. For example,

Jarrett

et al. [11] claim that the nonlinearity

f



x



tanh



x



works particularly well with their type of

contrast normalization followed by local average pooling on the Caltech-101 dataset. However,

on this dataset the primary concern is preventing overfitting, so the effect they are observing is

different from the accelerated ability to fit the training set which we report when using ReLUs.

Faster learning has a great influence on the performance of large models trained on large

datasets.

3.2 Training on Multiple GPUs

A single GTX 580 GPU has only 3GB of memory, which limits the maximum size of the

networks that can be trained on it. It turns out that 1.2 million training examples are enough to

train networks which are too big to fit on one GPU. Therefore we spread the net across two

GPUs. Current GPUs are particularly well-suited to cross-GPU parallelization, as they are able

to read from and write to one another’s memory directly, without going through host machine

memory. The parallelization scheme that we employ essentially puts half of the kernels (or

neurons) on each GPU, with one additional trick: the GPUs communicate only in certain layers.

This means that, for example, the kernels of layer 3 take input from all kernel maps in layer 2.

However, kernels in layer 4 take input only from those kernel maps in layer 3 which reside on

the same GPU. Choosing the pattern of connectivity is a problem for cross-validation, but this

allows us to precisely tune the amount of communication until it is an acceptable fraction of the

amount of computation.

The resultant architecture is somewhat similar to that of the “columnar” CNN employed by

Cire¸san et al. [5], except that our columns are not independent (see Figure 2). This scheme

15

reduces our top-1 and top-5 error rates by 1.7% and 1.2%, respectively, as compared with a net

with half as many kernels in each convolutional layer trained on one GPU. The two-GPU net

takes slightly less time to train than the one-GPU net2.

3.3 Local Response Normalization

ReLUs have the desirable property that they do not require input normalization to prevent them

from saturating. If at least some training examples produce a positive input to a ReLU, learning

will happen in that neuron. However, we still find that the following local normalization

i

scheme aids generalization. Denoting by

a

x,y

the activity of a neuron computed by applying

kernel

i

at position (x; y) and then applying the ReLU nonlinearity, the response-normalized

i

activity

b

x,y

is given by the expression

min



N1,in/2





2



i

k



a





x,y









jmax



0,in/2





b

i

x,y

a

i

x,y

where the sum runs over n “adjacent” kernel maps at the same spatial position, and N is the

total number of kernels in the layer. The ordering of the kernel maps is of course arbitrary and

determined before training begins. This sort of response normalization implements a form of

lateral inhibition inspired by the type found in real neurons, creating competition for big

activities amongst neuron outputs computed using different kernels. The constants

k,n,



, and



are hyper-parameters whose values are determined using a validation set; we

used

k2,n5,



10

4

, and



=0.75

. We applied this normalization after applying the ReLU

nonlinearity in certain layers (see Section 3.5).

This scheme bears some resemblance to the local contrast normalization scheme of Jarrett et al.

[11], but ours would be more correctly termed “brightness normalization”, since we do not

subtract the mean activity. Response normalization reduces our top-1 and top-5 error rates by

1.4% and 1.2%, respectively. We also verified the effectiveness of this scheme on the

CIFAR-10 dataset: a four-layer CNN achieved a 13% test error rate without normalization and

11% with normalization.

3.4 Overlapping Pooling

Pooling layers in CNNs summarize the outputs of neighboring groups of neurons in the same

kernel map. Traditionally, the neighborhoods summarized by adjacent pooling units do not

overlap (e.g., [17, 11, 4]). To be more precise, a pooling layer can be thought of as consisting of

a grid of pooling units spaced s pixels apart, each summarizing a neighborhood of size

zz

centered at the location of the pooling unit. If we set s = z, we obtain traditional local pooling

as commonly employed in CNNs. If we set s < z, we obtain overlapping pooling. This is what

16

we use throughout our network, with s = 2 and z = 3. This scheme reduces the top-1 and top-5

error rates by 0.4% and 0.3%, respectively, as compared with the non-overlapping scheme s =

2; z = 2, which produces output of equivalent dimensions. We generally observe during training

that models with overlapping pooling find it slightly more difficult to overfit.

3.5 Overall Architecture

Now we are ready to describe the overall architecture of our CNN. As depicted in Figure 2, the

net contains eight layers with weights; the first five are convolutional and the remaining three

are fully connected. The output of the last fully-connected layer is fed to a 1000-way softmax

which produces a distribution over the 1000 class labels. Our network maximizes the

multinomial logistic regression objective, which is equivalent to maximizing the average across

training cases of the log-probability of the correct label under the prediction distribution.

The kernels of the second, fourth, and fifth convolutional layers are connected only to those

kernel maps in the previous layer which reside on the same GPU (see Figure 2). The kernels of

the third convolutional layer are connected to all kernel maps in the second layer. The neurons

in the fully connected layers are connected to all neurons in the previous layer.

Response-normalization layers follow the first and second convolutional layers. Max-pooling

layers, of the kind described in Section 3.4, follow both response-normalization layers as well

as the fifth convolutional layer. The ReLU non-linearity is applied to the output of every

convolutional and fully-connected layer.

The first convolutional layer filters the

2242243

input image with 96 kernels of size

11113

with a stride of 4 pixels (this is the distance between the receptive field centers of

Figure 2: An illustration of the architecture of our CNN, explicitly showing the delineation of

responsibilities between the two GPUs. One GPU runs the layer-parts at the top of the figure

while the other runs the layer-parts at the bottom. The GPUs communicate only at certain

layers. The network’s input is 150,528-dimensional, and the number of neurons in the

network’s remaining layers is given by 253,440–186,624–64,896–64,896–43,264–4096–4096

–1000.

17

neighboring neurons in a kernel map). The second convolutional layer takes as input the

(response-normalized and pooled) output of the first convolutional layer and filters it with 256

kernels of size

5548

. The third, fourth, and fifth convolutional layers are connected to one

another without any intervening pooling or normalization layers. The third convolutional layer

has 384 kernels of size

33256

connected to the (normalized, pooled) outputs of the second

convolutional layer. The fourth convolutional layer has 384 kernels of size

33192

, and the

fifth convolutional layer has 256 kernels of size

33192

. The fully-connected layers have

4096 neurons each.

4 Reducing Overfitting

Our neural network architecture has 60 million parameters. Although the 1000 classes of

ILSVRC make each training example impose 10 bits of constraint on the mapping from image

to label, this turns out to be insufficient to learn so many parameters without considerable

overfitting. Below, we describe the two primary ways in which we combat overfitting.

4.1 Data Augmentation

The easiest and most common method to reduce overfitting on image data is to artificially

enlarge the dataset using label-preserving transformations (e.g., [25, 4, 5]). We employ two

distinct forms of data augmentation, both of which allow transformed images to be produced

from the original images with very little computation, so the transformed images do not need to

be stored on disk. In our implementation, the transformed images are generated in Python code

on the CPU while the GPU is training on the previous batch of images. So these data

augmentation schemes are, in effect, computationally free.

The first form of data augmentation consists of generating image translations and horizontal

reflections.

We do this by extracting random

224224

patches (and their horizontal reflections) from

the

256256

images and training our network on these extracted patches4. This increases

the size of our training set by a factor of 2048, though the resulting training examples are, of

course, highly interdependent. Without this scheme, our network suffers from substantial

overfitting, which would have forced us to use much smaller networks. At test time, the

network makes a prediction by extracting five

224224

patches (the four corner patches and

the center patch) as well as their horizontal reflections (hence ten patches in all), and averaging

the predictions made by the network’s softmax layer on the ten patches.

The second form of data augmentation consists of altering the intensities of the RGB channels

in training images. Specifically, we perform PCA on the set of RGB pixel values throughout the

ImageNet training set. To each training image, we add multiples of the found principal

components, with magnitudes proportional to the corresponding eigenvalues times a random

variable drawn from a Gaussian with mean zero and standard deviation 0.1. Therefore to each

RGB

RGB image pixel

I

xy





I,I,I



xyxyxy





we add the following quantity:

18



p

1

,p

2

,p

3





1



1

,



2



2

,



3



3



T

where

p

i

and



i

are ith eigenvector and eigenvalue of the

33

covariance matrix of RGB

pixel values, respectively, and i is the aforementioned random variable. Each i is drawn only

once for all the pixels of a particular training image until that image is used for training again,

at which point it is re-drawn. This scheme approximately captures an important property of

natural images, namely, that object identity is invariant to changes in the intensity and color of

the illumination. This scheme reduces the top-1 error rate by over 1%.

4.2 Dropout

Combining the predictions of many different models is a very successful way to reduce test

errors [1, 3], but it appears to be too expensive for big neural networks that already take several

days to train. There is, however, a very efficient version of model combination that only costs

about a factor of two during training. The recently-introduced technique, called “dropout” [10],

consists of setting to zero the output of each hidden neuron with probability 0.5. The neurons

which are“dropped out” in this way do not contribute to the forward pass and do not participate

in backpropagation. So every time an input is presented, the neural network samples a different

architecture, but all these architectures share weights. This technique reduces complex

co-adaptations of neurons, since a neuron cannot rely on the presence of particular other

neurons. It is, therefore, forced to learn more robust features that are useful in conjunction with

many different random subsets of the other neurons. At test time, we use all the neurons but

multiply their outputs by 0.5, which is a reasonable approximation to taking the geometric

mean of the predictive distributions produced by the exponentially-many dropout networks.

We use dropout in the first two fully-connected layers of Figure 2. Without dropout, our

network exhibits

substantial overfitting. Dropout roughly doubles the number of iterations required to converge.

5 Details of learning

We trained our models using stochastic gradient descent with a batch size of 128 examples,

momentum of 0.9, and weight decay of 0.0005. We found that this small amount of weight

decay was important for the model to learn. In other words, weight decay here is not merely a

regularizer: it reduces the model’s training error. The update rule for weight w was

v

i1

:0.9v

i

0.0005



w

i







w

i1

:w

i

v

i1

L

w

w

i

D

i

where i is the iteration index, v is the momentum variable,



is the learning rate, and

L

w

w

i

D

i

19

is the average over the ith batch

D

i

of the derivative of the objective with respect to w,

evaluated at

w

i

.

We initialized the weights in each layer from a zero-mean Gaussian distribution with standard

deviation 0.01. We initialized the neuron biases in the second, fourth, and fifth convolutional

layers, as well as in the fully-connected hidden layers, with the constant 1. This initialization

accelerates the early stages of learning by providing the ReLUs with positive inputs. We

initialized the neuron biases in the remaining layers with the constant 0.

We used an equal learning rate for all layers, which we adjusted manually throughout training.

The heuristic which we followed was to divide the learning rate by 10 when the validation error

rate stopped improving with the current learning rate. The learning rate was initialized at 0.01

and reduced three times prior to termination. We trained the network for roughly 90 cycles

through the training set of 1.2 million images, which took five to six days on two NVIDIA

GTX 580 3GB GPUs.

6 Results

Our results on ILSVRC-2010 are summarized in Table 1. Our network achieves top-1 and top-5

test set error rates of 37.5% and 17.0%. The best performance achieved during the

ILSVRC-2010 competition was 47.1% and 28.2% with an approach that averages the

predictions produced from six sparse-coding models trained on different features [2], and since

then the best published results are 45.7% and 25.7% with an approach that averages the

predictions of two classifiers trained on Fisher Vectors (FVs) computed from two types of

densely-sampled features [24].

Table 1: Comparison of results on ILSVRC-2010 test set. In italics are best results achieved by

others.

We also entered our model in the ILSVRC-2012 competition and report our results in Table 2.

Since the

ILSVRC-2012 test set labels are not publicly available, we cannot report test error rates for all

the models that we tried. In the remainder of this paragraph, we use validation and test error

rates interchangeably because in our experience they do not differ by more than 0.1% (see

Table 2). The CNN described in this paper achieves a top-5 error rate of 18.2%. Averaging the

predictions of five similar CNNs gives an error rate of 16.4%. Training one CNN, with an extra

sixth convolutional layer over the last pooling layer, to classify the entire ImageNet Fall 2011

release (15M images, 22K categories), and then “fine-tuning” it on ILSVRC-2012 gives an

error rate of 16.6%. Averaging the predictions of two CNNs that were pre-trained on the entire

Fall 2011 release with the aforementioned five CNNs gives an error rate of 15.3%. The

20

second-best contest entry achieved an error rate of 26.2% with an approach that averages the

predictions of several classifiers trained on FVs computed from different types of

densely-sampled features [7].

Table 2: Comparison of error rates on ILSVRC-2012 validation and test sets. In italics are best

results achieved by others. Models with an asterisk* were “pre-trained” to classify the entire

ImageNet 2011 Fall

release. See Section 6 for details.

Finally, we also report our error rates on the Fall 2009 version of ImageNet with 10,184

categories and 8.9 million images. On this dataset we follow the convention in the literature of

using half of the images for training and half for testing. Since there is no established test set,

our split necessarily differs from the splits used by previous authors, but this does not affect the

results appreciably. Our top-1 and top-5 error rates on this dataset are 67.4% and 40.9%,

attained by the net described above but with an additional, sixth convolutional layer over the

last pooling layer. The best published results on this dataset are 78.1% and 60.9% [19].

6.1 Qualitative Evaluations

Figure 3 shows the convolutional kernels learned by the network’s two data-connected layers.

The network has learned a variety of frequency- and orientation-selective kernels, as well as

various colored blobs. Notice the specialization exhibited by the two GPUs, a result of the

restricted connectivity described in Section 3.5. The kernels on GPU 1 are largely

color-agnostic, while the kernels on on GPU 2 are largely color-specific. This kind of

specialization occurs during every run and is independent of any particular random weight

initialization (modulo a renumbering of the GPUs).

Figure 3: 96 convolutional kernels of size

11113

learned by the first convolutional layer on

the

2242243

input images. The top 48 kernels were learned on GPU 1 while the bottom 48

kernels were learned on GPU 2. See Section 6.1 for details.

In the left panel of Figure 4 we qualitatively assess what the network has learned by computing

its top-5 predictions on eight test images. Notice that even off-center objects, such as the mite

in the top-left, can be recognized by the net. Most of the top-5 labels appear reasonable. For

21

example, only other types of cat are considered plausible labels for the leopard. In some cases

(grille, cherry) there is genuine ambiguity about the intended focus of the photograph.

Figure 4: (Left) Eight ILSVRC-2010 test images and the five labels considered most probable

by our model.

The correct label is written under each image, and the probability assigned to the correct label

is also shown

with a red bar (if it happens to be in the top 5). (Right) Five ILSVRC-2010 test images in the

first column. The remaining columns show the six training images that produce feature vectors

in the last hidden layer with the smallest Euclidean distance from the feature vector for the test

image.

Another way to probe the network’s visual knowledge is to consider the feature activations

induced by an image at the last, 4096-dimensional hidden layer. If two images produce feature

activation vectors with a small Euclidean separation, we can say that the higher levels of the

neural network consider them to be similar. Figure 4 shows five images from the test set and

the six images from the training set that are most similar to each of them according to this

measure. Notice that at the pixel level, the retrieved training images are generally not close in

L2 to the query images in the first column. For example, the retrieved dogs and elephants

appear in a variety of poses. We present the results for many more test images in the

supplementary material.

Computing similarity by using Euclidean distance between two 4096-dimensional, real-valued

vectors is inefficient, but it could be made efficient by training an auto-encoder to compress

these vectors to short binary codes. This should produce a much better image retrieval method

than applying autoencoders to the raw pixels [14], which does not make use of image labels

and hence has a tendency to retrieve images with similar patterns of edges, whether or not they

are semantically similar.

7 Discussion

Our results show that a large, deep convolutional neural network is capable of achieving record

breaking results on a highly challenging dataset using purely supervised learning. It is notable

22

that our network’s performance degrades if a single convolutional layer is removed. For

example, removing any of the middle layers results in a loss of about 2% for the top-1

performance of the network. So the depth really is important for achieving our results.

To simplify our experiments, we did not use any unsupervised pre-training even though we

expect that it will help, especially if we obtain enough computational power to significantly

increase the size of the network without obtaining a corresponding increase in the amount of

labeled data. Thus far, our results have improved as we have made our network larger and

trained it longer but we still have many orders of magnitude to go in order to match the

infero-temporal pathway of the human visual system. Ultimately we would like to use very

large and deep convolutional nets on video sequences where the temporal structure provides

very helpful information that is missing or far less obvious in static images.

References

[1] R.M. Bell and Y. Koren. Lessons from the netflix prize challenge. ACM SIGKDD

Explorations Newsletter, 9(2):75-79, 2007.

[2] A. Berg, J. Deng, and L. Fei-Fei. Large scale visual recognition challenge 2010.

/challenges. 2010.

[3] L. Breiman. Random forests. Machine learning, 45(1):5-2, 2001.

[4] D. Cirsan, U. Meier, and J. Schmidhuber. Multi-column deep neural networks for image

preprint arXiv:1202.2745, 2012.

[5] D.C. Ciresan, U. Meier, J. Masci, L.M. Gambardella, and J. Schmidhuber.

High-performance neural networks for visual object classification. Arxiv preprint

arXiv:1102.0183, 2011.

[6] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale

Hierarchical Image Database. In CVPR09, 2009.

[7] J. Deng, A. Berg, S. Satheesh, H. Su, A. Khosla, and L. Fei-Fei. ILSVRC-2012, 2012. URL

/challenges/LSVRC/2012/.

[8] L. Fei-Fei, R. Fergus, and P. Perona. Learning generative visual models from few training

examples: An incremental bayesian approach tested on 101 object categories. Computer Vision

and Image Understanding, 106(1):59-70, 2007.

[9] G. Griffin, A. Holub, and P. Perona. Caltech-256 object category dataset. Technical Report

7694, California Institute of Technology, 2007. URL /7694.

[10] G.E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and R.R. Salakhutdinov.

Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint

arXiv:1207.0580, 2012.

[11] K. Jarrett, K. Kavukcuoglu, M. A. Ranzato, and Y. LeCun. What is the best multi-stage

architecture for

object recognition? In International Conference on Computer Vision, pages 2146?153. IEEE,

2009.

[12] A. Krizhevsky. Learning multiple layers of features from tiny images. Master’s thesis,

Department of Computer Science, University of Toronto, 2009.

[13] A. Krizhevsky. Convolutional deep belief networks on cifar-10. Unpublished manuscript,

23

2010.

[14] A. Krizhevsky and G.E. Hinton. Using very deep autoencoders for content-based image

retrieval. In ESANN, 2011.

[15] Y. Le Cun, B. Boser, J.S. Denker, D. Henderson, R.E. Howard, W. Hubbard, L.D. Jackel,

et al. Handwritten digit recognition with a back-propagation network. In Advances in neural

information processing systems, 1990.

[16] Y. LeCun, F.J. Huang, and L. Bottou. Learning methods for generic object recognition

with invariance to

pose and lighting. In Computer Vision and Pattern Recognition, 2004. CVPR 2004.

Proceedings of the 2004 IEEE Computer Society Conference on, volume 2, pages II-97. IEEE,

2004.

[17] Y. LeCun, K. Kavukcuoglu, and C. Farabet. Convolutional networks and applications in

vision. In Circuits and Systems (ISCAS), Proceedings of 2010 IEEE International Symposium

on, pages 253-256. IEEE, 2010.

[18] H. Lee, R. Grosse, R. Ranganath, and A.Y. Ng. Convolutional deep belief networks for

scalable unsupervised learning of hierarchical representations. In Proceedings of the 26th

Annual International Conference on Machine Learning, pages 609-616. ACM, 2009.

[19] T. Mensink, J. Verbeek, F. Perronnin, and G. Csurka. Metric Learning for Large Scale

Image Classification: Generalizing to New Classes at Near-Zero Cost. In ECCV - European

Conference on Computer Vision, Florence, Italy, October 2012.

[20] V. Nair and G. E. Hinton. Rectified linear units improve restricted boltzmann machines. In

Proc. 27

th

International Conference on Machine Learning, 2010.

[21] N. Pinto, D.D. Cox, and J.J. DiCarlo. Why is real-world visual object recognition hard?

PLoS computational biology, 4(1):e27, 2008.

[22] N. Pinto, D. Doukhan, J.J. DiCarlo, and D.D. Cox. A high-throughput screening approach

to discovering

good forms of biologically inspired visual representation. PLoS computational biology,

5(11):e1000579, 2009.

[23] B.C. Russell, A. Torralba, K.P. Murphy, and W.T. Freeman. Labelme: a database and

web-based tool for image annotation. International journal of computer vision, 77(1):157-173,

2008.

[24] J. Sachez and F. Perronnin. High-dimensional signature compression for large-scale image

classification. In Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference

on, pages 1665-1672. IEEE, 2011.

[25] P.Y. Simard, D. Steinkraus, and J.C. Platt. Best practices for convolutional neural networks

applied to visual document analysis. In Proceedings of the Seventh International Conference on

Document Analysis and Recognition, volume 2, pages 958?62, 2003.

[26] S.C. Turaga, J.F. Murray, V. Jain, F. Roth, M. Helmstaedter, K. Briggman,W. Denk, and

H.S. Seung. Convolutional networks can learn to generate affinity graphs for image

segmentation. Neural Computation,

22(2):511-538, 2010.

24

本文标签： 图像训练网络