In order to solve the problems of appearance change, background distraction and occlusion in the object tracking, an efficient algorithm for visual tracking based on the local patch model and model update is proposed. This paper combines rough-search and precise-search to enhance the tracking precision. Firstly, it constructs the local patch model according to the initialized tracking area which includes some background areas. Secondly, the target is preliminarily located through the local exhaustive search algorithm based on the integral histogram, then the final position of the target is calculated through the local patches learning. Finally, the local patch model is updated with the retained sequence during the tracking process. This paper mainly studies the search strategy, background restraining and model update, and the experimental results show that the proposed method obtains a distinct improvement in coping with appearance change, background distraction and occlusion.