To overcome the limitations of traditional image quality assessment methods, which are not well consistent with subjective human evaluation, a quality assessment algorithm of Weighting Sparse Fidelity (WSF) based on wavelet analysis is proposed. The arithmetic simulates nerve network of Human Vision System (HVS) as research point, the image is decomposed with wavelet into four-sub band images, which are divided into blocks at size of , then using Fast Independent Component Analysis training (FastICA) method to train the image blocks. Then, each image block sparse character matrix is extracted to calculate the sparse feature fidelity of the image and build the sparse fidelity quality evaluation model. On this basis, the image is divided into a plurality of interval according to the different details of the visual image information and a visual weight is set in each section, which can be consistent with subjective human evaluation. The experiment results on LIVE database show that the proposed method has a good evaluation of all kinds of distortion types and is highly consistent with human subjective evaluations. The proposed algorithm can effectively simulate the weighted visual cortex of the human visual system perception mechanisms, which compensates for deficiencies of existing image quality assessment methods.