Abstract: To address the issues of mesh clogging and low screening efficiency during the screening process of fine coal below 6 mm, this study investigates compound vibration screening based on the Discrete Element Method (DEM). By constructing a mathematical model for compound vibration and determining simulation parameters, a DEM simulation model was established to compare the screening performance of compound vibration against traditional linear vibration for 6 mm fine coal. The results indicate that a compound vibration of 3 mm/16 Hz + 1 mm/50 Hz achieves maximum vibration intensity at a phase difference angle of 1.45π, identifying it as the optimal phase parameter for this screening method. Compound vibration effectively enhances the looseness and internal mobility of the particle group, significantly improving the stratification effect of fine coal and increasing the penetration rate of fine particles. Furthermore, the abrupt excitation force generated by this method prevents sieve hole blockages. Under identical screen length conditions, compound vibration demonstrates superior performance; it achieves a screening efficiency of 87% for 6 mm fine coal, an 11-percentage-point improvement over the 76% efficiency of traditional linear vibration. The superposition of dual vibration modes significantly enhances the screening performance of 6 mm fine coal, providing a vital theoretical basis for the structural design and parameter optimization of compound vibrating screens.