Article Preview
TopIntroduction
Convolutional neural networks (CNN) are becoming more and more popular in various learning tasks, especially in visual computing applications. The biggest advantage of it is that it has the ability to represent learning, thus, it can learn the characteristics of the data itself, and avoid the trouble of traditional manual feature extraction. Therefore, the application of convolutional neural networks and the study of the model structure have been hot spots for scholars to explore. In the era of big data, the amount and variety of data in the visual field are increasing. On the one hand, it increases the calculation time of the model. On the other hand, the simple neural network model is difficult to capture the essential characteristics of the data. Therefore, in order to reduce the calculation time of the model, researchers can use parallel software framework or programming model for reference to make the model parallel processing (Jing, et al, 2017; Song, Wang, Li & Gao, 2016) and shorten the calculation time. To ensure that the model can better learn the characteristics of the data, the scale of the model has also evolved toward complex aspects and the training overhead has accordingly increased. With the increase of the depth and complexity of the model, a series of problems are caused, which make it difficult to train the model, e.g. the gradient disappears (Glorot & Bengio, 2010; Bengio, Simard & Frascon, 1994). In order to solve the gradient problem of the model, the researchers proposed the Highway network (Srivastava, Greff & Schmidhubr, 2015), adding a bypass connection in the CNN architecture to enhance the information flow between layers. But this method is controlled by a gating function, which depends on data. So it is more difficult to train. ResNet (He, Zhang, Ren & Sun, 2016) uses identity connection or skip connection to solve the gating parameter problem of Highway network. However, the connection mode used in this network excessively follows the modular structure, and there is a lot of redundancy in deep ResNet. Subsequently researchers have improved their structure based on ResNet (Targ, Almeida & Lyman, 2016; Xie, Girshick, Dollar, Tu & He, 2017) to further improve the accuracy of image classification. Szegedy C. et al. introduced the inception structure into short connections (Szegedy, Ioffe, Vanhoucke & Alexander, 2017), which improved the model's convergence speed. Therefore, the introduction of short connections in the model will greatly benefit the performance of the model.
Convolutional neural networks have a more significant effect in simulating the structural characteristics of biological neural networks in terms of network depth, but their performance is far from the advanced features of real biological neural networks. Biological neurological research shows that the neural network of the brain has inherent random characteristics to a certain extent (Hu, Liao & Mao, 2003). The connection of the brain's network structure (Achard, 2006) belongs to a new type of network proposed by Watts (Watts & Strogatz, 1998) named the small world network, which between the regular network and the random network. The small-world network will improve the propagation efficiency of its network structure. However, Convolutional neural network is a kind of feedforward neural network with deep structure, and its connection architecture is approximately regular. Therefore, how to make the network structure has random features and improve network efficiency in a convolutional neural network is a key issue.