Recently, hazardous cargo accidents such as those at Beirut and Tianjin ports have raised serious concerns over how high-risk cargoes should be handled. While the role of human error in such accidents is widely known, knowledge of how the related factors interact is less developed. To that end, the present study analyzed 352 hazardous cargo accidents occurring at ports between 1960 and 2018. A novel research framework, the Human Factors Analysis and Classification System for Port Environment Hazardous Cargo Accidents (HFACS-PEHCA) was developed for the analysis. The findings revealed that violations, limited intellect, inappropriate supervision, and an inadequate safety culture were the most prominent factors involved in hazardous cargo accidents. Correlations between these factors were established and the probabilities of five accident causation paths were calculated. The most likely way for a hazardous cargo accident to occur was via the path “Deficient safety culture → Inappropriate supervision → Limited Intellect → Violations.” The study draws on these insights to propose safety procedures to mitigate the risk of accidents at ports dealing with hazardous cargo.