Fetal congenital heart disease (CHD) is a prevalent and highly complicated fetal deformity. Furthermore, the number of infants with CHD accounts for as high as 6‰–8‰ among all the living newborns in China. For the early diagnosis and screening of the fetal CHD, echocardiography technology plays a vital role in assessing anatomical structures and functions of the fetal heart. However, the prenatal detection rate of CHD is still meager due to the particularity of the fetal cardiac structures and the diversity of the fetal CHD. Hence, we propose a simple yet effective residual learning diagnosis system (RLDS) for diagnosing fetal CHD to improve diagnostic accuracy, which adopts convolutional neural networks to extract discriminative features of the fetal cardiac anatomical structures. To enhance the credibility of the RLDS, we provide a global visualizing explanation for the diagnosis process of our RLDS. Moreover, we visually explain the local feature map of the residual feature learning, making the residual learning more transparent. Extensive experiments demonstrate that our proposed RLDS is very effective in diagnosing fetal CHD. Furthermore, the proposed RLDS achieves a precision of 93% and a recall of 93% on the test set, which significantly improves the prenatal detection rate for the fetal CHD.