Abstract The separability and separating conditions for mixed signals are analyzed in this paper. The limitation of nonlinear blind source separation methods is proposed. A new nonlinear BSS approach is developed by applying Edgeworth expansion and adaptive cumulant estimation into information back propagation algorithm. Computer simulation shows the validity of the proposed algorithm in some ad hoc model. A comparison with the Gram-Charlier expansion method is given.