In visual tracking, the efficient and robust feature representation is the key factor to solve the problem of tracking drift in complex environments. Therefore, to solve the problems of the complex and time-consuming of the pre-training process of deep neural network and the drift of the single network tracking, an online tracking method based on an adaptive deep sparse network is proposed under the tracking structure of particle filter. A deep sparse neural network architecture, which can be adaptively selected according to different types of targets, is constructed with the implementation of the Rectifier Linear Unit (ReLU) activation function. The robustness of deep tracking network can be easily achieved only through the online training of limited labeled samples. The results of experiments show that, compared with the state-of-the-art tracking algorithm, the average success ratio and precision of the proposed algorithm are both the highest, and they are raised by 20.64% and 17.72% respectively contrasted with the Deep Learning Tracker (DLT) algorithm based on deep learning. The proposed method can solve the problems of tracking drift efficiently, and shows better robustness, especially for the complex environment such as illumination changes, background clutter and so on.