In order to solve the problems of appearance change, local occlusion and background distraction in the visual tracking, a local patch tracking algorithm based on Fast Fourier Transform(FFT)is proposed. The tracking precision can be improved by establishing object’s patch kernel ridge regression model and using patch exhaustive search based on circular structure matrix, and the efficiency can be improved by transforming time domains operation into frequency domains based on FFT. Firstly, patch kernel ridge regression model is constructed according to the initialized tracking area. Secondly, a patch exhaustive search method based on circular structure matrix is proposed, then the position model is constructed in adjoining frame. Finally, the position of the object is estimated accurately using the position model and the local patch model is updated. Experimental results indicate that the proposed algorithm not only can obtain a distinct improvement in coping with appearance change, local occlusion and background distraction, but also have high tracking efficiency.