Tracking effects of algorithms using color information are easily interfered by background clustering, illumination and scale changes, which can result in tracking failure. To solve these problems, an efficient model is proposed to project original RGB color space to a more robust color spaceColor Names (CN) feature space. Furthermore, objects are represented by background weighted color names histogram, and thus the similar background patches around the target are suppressed. Moreover, a two-step tuning way is adapted to estimate the scale by coarse tuning with gradient ascent and fine tuning with constrained items. Back-forward scale check is also used to ensure the precision of scale estimation. 5 representative videos are chosen to examine the proposed algorithms with four others. The results show that the proposed approach is robust to illumination variation, shadows, background clustering, and scale changes. The central distance error and tracking accuracy of the proposed approach also outperform the contrast algorithms.