This paper presents a benchmark data set for evaluating ball detection algorithms in the RoboCup Soccer Standard Platform League. We cr eated a la- belled data set of images with and without ball derived from vision log files rec- orded by multiple NAO robots in various lighting conditions. The data set con- tains 5209 labelled ball image regions and 10924 non - ball regions . Non - ball im- age region s all contain features that had been classified as a potential ball candi- date by an existing ball detector. The data set was used to train and evaluate 25 2 different Deep Convolutional Neural Network (CNN) architectures for ball de- tection. In order to control computational requirements , this evaluation focused on networks with 2 – 5 layers that could feasibly run in the vision and cognition cycle of a NAO robot using two cameras at full frame rate (2×30 Hz). The results show that the classification perfo rmance of the networks is quite insensitive to the details of the network design including input image size, number of layers and number of outputs at each layer . In an effort to reduce the computational requirements of CNNs we evaluated XNOR - Net architect ure s which quantize the weigh ts and ac tivations of a neural network to binary values . We examined XNOR - Nets corresponding to the real - valued CNNs we had already tested in or- der to quantify the effect on classification metrics. The results indicate that bal l classification performance degrad es by 12% on average when changing from real - valued CNN to corresponding XNOR - Net .