In order to solve the problem of unclear quantification of motor displacement tracking error caused by nonlinear friction and the difficulty of compensation design, an iterative feedforward tuning strategy based on updating the error correlation coefficient is proposed. Firstly, the nonlinear friction model of “viscous friction + Coulomb friction” is used to analyse the mechanism of motor displacement tracking error in linear servo system, and the specific expression of motor displacement tracking error is obtained. Secondly, the motor displacement tracking error caused by nonlinear friction is linked with the motor displacement tracking error caused by reference trajectory, and the mapping relationship between the motor displacement tracking error and velocity, acceleration, Coulomb friction and error correlation coefficients are established. By updating and judging the magnitude of the error correlation coefficients and iterative feedforward tuning is performed to effectively compensate the motor displacement tracking error. Finally, in simulation and experiment, the size of the error correlation coefficients under different feedforward parameters proves the correctness of the motor displacement tracking error analysis, and the obvious reduction of the motor tracking error proves the effectiveness of iterative feedforward tuning based on the error correlation coefficients.