The human annulus fibrosus (HAF) is the major component in response to external forces for the intervertebral disk (IVD), which maintains the stability and flexibility of human spine. It can be assumed to be an anisotropic nearly incompressible hyperelastic composite consisting of collagen fibers and matrix in the numerical simulations of biomechanics. However, due to the geometric complexity and material nonlinearity of HAF, the conventional Finite Element Method (FEM) often gets into difficulties in mesh generation and uncertainty of accuracy control. In this paper, a modified composite-based constitutive model, which considers the slight compressibility of ground substance and the shear interaction between collagen fibers and matrix, is developed to describe the mechanical behavior of HAF. In addition, based on the gradient smoothing techniques, the selective 3D-edge-based and node-based smoothed finite element method (Selective 3D-ES/NS-FEM) is developed to alleviate volume locking and improve the accuracy of linear four-node tetrahedral (TET4) elements. Combined with the modified constitutive model, the Selective 3D-ES/NS-FEM is applied into the explicit dynamic analysis of HAF undergoing large deformation. By comparing with the experiment data in the literatures and the numerical results produced by conventional FEM, the presented approach is proved to possess excellent accuracy and efficiency in predicting the nonlinear mechanical behavior of HAF, as well as the orientation change of the collagen fibers. Moreover, the Selective 3D-ES/NS-FEM is demonstrated to have robust capability in handling element distortion, even with the simplest TET4 mesh. This study is significant to the biomechanical research of HAF, and has potential value for guiding the prevention and treatment of low back pain.