Cement asphalt mortar (CA mortar) plays an important role in supporting, load transferring, vibration damping and isolating in the ballast track of high-speed railroad. But it is often damaged or destroyed prematurely, which seriously affects the operation safety and efficiency of high-speed railroad. Therefore, it is of practical significance to study the mechanical properties of CA mortar under cyclic loading and unloading, as well as its damage evolution and life prediction under coupled creep fatigue loading. In this paper, the energy dissipation characteristics of CA mortar under cyclic loading and unloading are investigated experimentally. A damage evolution equation based on dissipated energy accumulation is also proposed. On this basis, the coupled damage and life prediction of CA mortar under self-weight creep load and train impulse fatigue load of ballast track are investigated. The experimental results and analyses show that the hysteretic and plastic strain accumulation characteristics of CA mortar under cyclic loading and unloading have typical Mansing and ratchet effects. Using the energy dissipation of CA mortar in cyclic loading and unloading to define the damage can better describe the damage evolution behavior of CA mortar. The life of CA mortar under the long-term action of creep and impulse fatigue coupling damage is greatly reduced to one-tenth of that under the self-weight creep loading of ballast track.