Drainage network extraction is essential for different research and applications. However, traditional methods have low efficiency, low accuracy for flat regions, and difficulties in detecting channel heads. Although deep learning techniques have been used to solve these problems, different challenges remain unsolved. Therefore, we introduced distributed representations of aspect features to facilitate the deep learning model calculating the flow direction; adopted a semantic segmentation model, U-Net, to improve the accuracy and efficiency in predicting flow directions and in pixel classifications; and used postprocessing to delineate the flowlines. Our proposed framework achieved state-of-the-art results compared with the traditional methods and the published deep-learning-based methods. Further, case study results demonstrated that our framework can extract drainage networks with high accuracy for rivers of different widths flowing through terrains of different characteristics. This framework, requiring no parameters provided by users, can also produce waterbody polygons and allow cyclic graphs in the drainage network.