The rational design of drug delivery systems has aroused significant concern toward improving therapeutics in recent years. Material innovation has promoted more reasonable strategies for the drug encapsulation and delivery. Among the various nanocarriers, the crystalline porous framework material metal-organic frameworks (MOFs) and covalent organic frameworks (COFs) have been developed as an attractive kind of upstarts owing to their predefined structure and function by virtue of the charm of reticular chemistry. Moreover, compared with other amorphous materials, their precise framework structures and defined pores by molecular-level control can improve the efficacy of drug administration, and provide novel strategies for precision of targeting delivery. In this mini review, we summarize and illustrate the most recent progress of MOF- or COF-based carriers for drug delivery, focusing on their structure design, function regulation, and fascinating application. Furthermore, we also discuss the major challenges and future development prospects in this field.