Smooth filtering, a common blurring and denoising operator, has often been utilized postoperatively to diminish the traces left by malicious manipulations. Most of the existing forensic methods only focus on one specific filtering artifact such as median filtering, which is insufficient to reveal the manipulation history of digital images. Unlike traditional convolutional neural network (CNN)-based networks, which normally introduce handcrafted features, including frequency domain features and median filtering residuals, into the preprocessing layer, this paper proposes an end-to-end deep learning model for robust smooth filtering identification. First, a distinctive network structure named the Squeeze-and-Excitation (SE) block is introduced to select discriminative features adaptively and suppress the irrelevant features to the smooth filtering effect. Then, as the network depth increases, multiple inception-residual blocks are stacked to extract discriminative features and reduce the information loss. Finally, different smooth filtering operations can be classified through learning hierarchical features. The experimental results on a composite database show that the proposed model outperforms the state-of-the-art methods, especially in small size and JPEG compression scenarios.