The radiation magnetohydrodynamics (RMHD) system couples the ideal magnetohydrodynamics equations with a gray radiation transfer equation. The main challenge is that the radiation travels at the speed of light while the magnetohydrodynamics changes with fluid. The time scales of these two processes can vary dramatically. In order to use mesh sizes and time steps that are independent of the speed of light, asymptotic preserving (AP) schemes in both space and time are desired. In this paper, we develop an AP scheme in both space and time for the RMHD system. Two different scalings are considered, one results in an equilibrium diffusion limit system, while the other results in a non-equilibrium system. The main idea is to decompose the radiative intensity into three parts, each part is treated differently. The performances of the semi-implicit method are presented, for both optically thin and thick regions, as well as for the radiative shock problem. Comparisons with the semi-analytic solution are given to verify the accuracy and asymptotic properties of the method.