Polymer-based dielectrics (PDs) with improved permittivity (k) have considerable applications including capacitors, actuator devices and electrical power systems due to their flexibility, easy processability and low weight, etc. However, the permittivity values of commonly used polymers (usually k < 3) fails to meet the requirements of the advanced electrical components. Enormous research, including numerical and experimental works has shown that developing polymer based multi-phase materials represents the most promising avenue. This review will clarify the concepts and relationships based on the polarization mechanisms and meanwhile some key factors of improving permittivity will also be discussed including polymer chains/segments, morphology, the interface of nanofillers-matrix, processing, and multilayer structures. Combining with the recent specific advances in PDs, the latest developments in this field will be summarized which are the introduction of polar functional groups/copolymer segments, the application of core-shell nanofillers and the technology of multi-layers structures. We hope to help readers to comprehensively understand the fabrication of PDs from the selection of molecular units to the design of composites in order to provide guidance on achieving polymer based materials with high permittivity.