Position, Padding and Predictions: A Deeper Look at Position Information in CNNs

Abstract

In contrast to fully connected networks, Convolutional Neural Networks (CNNs) achieve efficiency by learning weights associated with local filters with a finite spatial extent. Theoretically, an implication of this fact is that a filter may know what it is looking at, but not where it is positioned in the image. In this paper, we first test this hypothesis and reveal that a surprising degree of absolute position information is encoded in commonly used CNNs. We show that zero padding drives CNNs to encode position information in their internal representations, while a lack of padding precludes position encoding. This observation gives rise to deeper questions about the role of position information in CNNs: (i) What boundary heuristics enable optimal position encoding for downstream tasks? (ii) Does position encoding affect the learning of semantic representations? (iii) Does position encoding always improve performance? To provide answers, we perform the largest case study to date on the role that padding and border heuristics play in CNNs. We design novel tasks that allow us to quantify boundary effects as a function of the distance to the border. Numerous semantic objectives reveal the effect of the border on semantic representations. Finally, we demonstrate the implications of these findings on multiple real-world tasks to show that position information can both help or hurt performance.

Appendix

1.1 Experimental Details of Absolute Position Encoding Experiments

Datasets: We use the DUT-S dataset (Wang et al., 2017) as our training set, which contains 10, 533 images for training. Following the common training protocol used in Zhang et al. (2017); Liu et al. (2018), we train the model on the training set of DUT-S and evaluate the existence of position information on the natural images of the PASCAL-S (Li et al., 2014) dataset. The synthetic images (white, black and Gaussian noise) are also used as described in Sect. 3.4 of the main manuscript. Note that we follow the common setting used in saliency detection just to make sure that there is no overlap between the training and test sets. However, any images can be used in our experiments given that the position information is relatively content independent.

Evaluation Metrics: As position encoding measurement is a new direction, there is no universal metric. We use two different natural choices for metrics (Spearmen Correlation (SPC) and Mean Absoute Error (MAE)) to measure the position encoding performance. The SPC is defined as the Spearman’s correlation between the ground-truth and the predicted position map. For ease of interpretation, we keep the SPC score within range [-1 1]. MAE is the average pixel-wise difference between the predicted position map and the ground-truth gradient position map.

Implementation Details We initialize the architecture with a network pretrained for the ImageNet classification task. The new layers in the position encoding branch are initialized with xavier initialization (Glorot and Bengio, 2010). We train the networks using stochastic gradient descent for 15 epochs with momentum of 0.9, and weight decay of $1e-4$. We resize each image to a fixed size of $224 \times 224$ during training and inference. Since the spatial extent of multi-level features are different, we align all the feature maps to a size of $28 \times 28$.

1.2 Implementation Details of VGG-5 Network for Position Information

We use a simplified VGG network (VGG-5) for the position encoding experiments in Sect. 3.4 of the main manuscript and texture recognition experiments in Sec. 7 of the main manuscript. The details of the VGG-5 architecture are shown in Table 11 (in this table we show the VGG-5 network trained on the tiny ImageNet dataset, the VGG-5 network trained on texture recognition has a different input size: $224 \times 224$). Note that the network is trained from scratch. The tiny ImageNet dataset contains 200 classes and each class has 500 images for training and 50 for validation. The size of the input image is $64 \times 64$, a random crop of $56 \times 56$ is used for training and a center crop is applied for validation. The total training epochs is set to 100 with an initial learning rate of 0.01. The learning rate was decayed at the 60th and 80th epochs by multiplying the learning rate by a factor of 0.1. A momentum of 0.9 and a weight decay of $1e-4$ are applied with the the stochastic gradient descent optimizer. After the pre-training process, a simple read-out module is applied on the pre-trained frozen backbone for position evaluation, following the training protocol as used in Islam et al. (2020). Note that the type of padding strategy is consistent between the pre-training and position evaluation procedures.

Table 11 Configuration of VGG-5 architecture trained on tiny ImageNet

Fig. 14

Classification accuracy on CIFAR-10 for different constant padding values

Fig. 15

Classification accuracy on CIFAR-100 (Krizhevsky et al., 2014) for different constant padding values

1.3 Is Zero the Optimal Constant Padding Value?

From the experiments in Sect. 3.6 of the main manuscript, as zero delivers maximal position information, it is natural to ask if zero padding also outperforms other padding types for a semantic objective. To this end, we train separate ResNet-18s (He et al., 2016) for image classification on the CIFAR-10 (Krizhevsky et al., 2014) dataset and vary the padding value in the range $[-10, 10]$. We run this experiment under two settings: with data normalization (i.e., subtract the dataset mean and divide by the standard deviation) and without (i.e., pixel values are scaled to $[-1,1]$). We illustrate the padding value results in Fig. 14. The pattern is immediate and clear. For all constant padding values (excluding any dynamic padding strategies, such as reflection padding), zero gives the best performance, with what appears to be an exponential error increase as the absolute value increases (e.g., $-10$ and 10). Additionally, we also train a ResNet-18 on CIFAR-10 with reflection, replicate, and circular padding which achieve accuracies of 89.6%, 89.7%, and 87.1%, respectively, all underperforming zero padding.

We further experiment on the CIFAR-100 dataset (Krizhevsky et al., 2014) for both normalized (i.e., subtract mean and divide by variance) and unnormalized (i.e., rescaled to [0, 1]) image values. The results (see Fig. 15) are consistent with those for CIFAR-10 (see Fig. 3 in the main paper) and suggest that zero is the optimal constant padding value for image classification, irrespective of the dataset used or whether the inputs are normalized. For consistency, in addition to zero padding (76.38%), we also used reflection, replicate, and circular padding which achieved performances of 76.38%, 76.19%, and 75.59%, respectively (using normalized data).

Table 12 IoU comparison of DeepLabv3 for semantic segmentation task with three different padding (Zero, Reflect, and No pad) settings

1.4 No Padding Implementation Discussion

We include no padding comparisons for completeness and to contrast the difference in the border effects between networks trained with padding and without padding. For networks without residual connections (e.g., VGG) one can implement a no padding version by simply discarding the padding. However, controlling for consistent spatial resolution is crucial when comparing padding types since an inconsistent spatial resolution between padding and no padding would result in a significant performance drop due to the reduced dimensionality of the feature representations. Another solution is to remove all the padding from a VGG network and then padding the input image by a sufficient amount to keep the spatial resolution. However, this is not applicable to the ResNet backbone as there will be spatial misalignment between the features of layers due to the residual connections. Alternatively, one can interpolate the output feature map to the same size as the input, which is also the method used in a recent study (Xu et al., 2020). In the end, we choose the interpolation implementation because we believe the visual information near the border is better retained while working for networks with and without residual connections.

One concern of using interpolation is how to align the feature maps during the interpolation. If the features maps are aligned in the center, interpolating the feature map will move the contents of feature map slightly towards the edges. The composite will thus not have the features from the two branches perfectly line up with each other anymore. This shifting effect is largest near the edges and smallest near the center, which matches with the observed performance characteristics. The subsequent convolution layers may be able to undo some of this shifting, but only at the cost of location-dependent kernels that are tailored to fit the offset caused at different parts of the image. The other option is to align the feature map based on the corners with the interpolation mainly occurring at the center. In this scenario, the shifting effect will be reversed, with the corners being in alignment but the center of the feature map slightly misaligned.

1.5 Relationship Between Position Information and Network Depth

We further investigate the relationship between position information and depth of the network. We interestingly found that the encoding of position information correlates with the depth of the network. Since the network with higher depth has bigger receptive field, it is likely to encode more position information. We conduct positional encoding experiments with ResNet backbone of different depth and report the results in Table 13. It it clear that network with higher depth encode more position information.

Table 13 Positional encoding comparison of networks with different depth in terms of SPC and MAE across DUT-OMRON (Yang et al., 2013) and ECSSD (Shi et al., 2015) dataset

1.6 Extended Boundary Effect Analysis on Cityscapes Dataset

We continue to investigate the impact that zero padding has on the ability of a strong and deep CNN to segment objects near the image boundary. Results shown use the same network and training settings as in Sect. 7 of the main manuscript, on the Cityscapes (Cordts et al., 2016) dataset (Fig. 16).

We present additional results (see Table 12 and Fig. 17) of the analysis presented in Sect. 6 (semantic segmentation) in the main paper. Figure 16 shows sample evaluation regions used for this analysis. The no padding case has a steeper drop-off in performance as regions of evaluation get closer to the image boundary. Note how, in all cases, the performance increases from the border to the inner $25\%$, at which point the performance is somewhat stagnant until it reaches the innermost $80\%$.

Fig. 16

An illustration of the evaluation regions used for the analysis in Table 12 and Fig. 17

Fig. 17

Performance comparison of DeepLabv3 network with respect to various image regions and padding settings used in Table 12

Surprisingly, we also observe a steeper drop off in the middle of the image for the no padding case, supporting our hypothesis that boundary effects play a role at all regions of the image without the use of padding. We believe the drop in performance at the center regions is due to Cityscapes being an automotive-centric dataset, where pixels at the center of the image are often at large distances away from the camera, unless the vehicle collecting the data has an object directly in front of it.

1.7 Canvas Analysis: Cutout and Adversarial Robustness

Fig. 18 shows two training examples of Cutout strategy. Following Cutout, we simply place a rectangular mask (zero and max) over a random region during the training. Note that we evaluate on the standard PASCAL VOC 2012 validation images.

Fig. 18

Sample training images generated using Cutout (DeVries and Taylor, 2017) under two different canvases