Carbon emission reduction in the transportation sector is essential in the global mitigation effort, and a large-scale public transport system has the potential to be an effective instrument. High-speed rail (HSR) is one such example, yet it is unclear how much reduction in road traffic results from new rail routes. Using the difference-in-differences method, we show that new HSR routes in China lead to a 20.5 log-point reduction in the number of passenger vehicles and a 15.7 log-point reduction in freight vehicles running on parallel highways. These reductions were not seen on ordinary national roads. These effects translate into an annual reduction of 14.76 million tons of CO2 equivalent of GHG emissions or 1.75% of GHG emissions in China’s transport sector. This mitigation effect mainly comes from the substitution of highway goods transport with the conventional railway instead of the direct replacement of highway passenger transport with HSRs. The environmental benefit of HSR in China has not been fully realized because of the thermal-dominated electricity supply. Our further projections suggest that in greener electricity conditions, the HSR in China can substantially contribute more to the reduction in GHG emissions from the transport sector.