A test system was developed to understand the sliding mechanism of coal-rock structure. The test system was composed by a double-shear testing model and an acousto-optic monitoring system in association with a digital camera and an acoustic emission (AE) instrument. The tests can simulate the movement of activated faults and the sliding in coal-rock structure. In this regard, instable sliding conditions of coal-rock samples, sliding types under different conditions, displacement evolution law, and AE characteristics during sliding process were investigated. Several sliding types were monitored in the tests, including unstable continuous sliding, unstable discontinuous sliding, and stable sliding. The sliding types have close relation with the axial loads and loading rates. Larger axial load and smaller loading rate mean that unstable sliding is less likely to occur. The peak shear stress was positively correlated with the axial load when sliding occurred, whereas the displacement induced by unstable sliding was uncorrelated with the axial load. A large number of AE events occurred before sliding, and the AE rate decreased after stable sliding. The results show that the tests can well simulate the process of structural instability in a coal bump, and are helpful in the understanding of fault activation and the physical processes during squeezing process of roof and floor.