The Clifford hierarchy is a nested sequence of sets of quantum gates critical to achieving fault-tolerant quantum computation. Diagonal gates of the Clifford hierarchy and ‘nearly diagonal’ semi-Clifford gates are particularly important: they admit efficient gate teleportation protocols that implement these gates with fewer ancillary quantum resources such as magic states. Despite the practical importance of these sets of gates, many questions about their structure remain open; this is especially true in the higher-dimensional qudit setting. Our contribution is to leverage the discrete Stone–von Neumann theorem and the symplectic formalism of qudit stabilizer theory towards extending the results of Zeng et al. (2008) and Beigi & Shor (2010) to higher dimensions in a uniform manner. We further give a simple algorithm for recursively enumerating all gates of the Clifford hierarchy, a simple algorithm for recognizing and diagonalizing semi-Clifford gates, and a concise proof of the classification of the diagonal Clifford hierarchy gates due to Cui et al. (2016) for the single-qudit case. We generalize the efficient gate teleportation protocols of semi-Clifford gates to the qudit setting and prove that every third-level gate of one qudit (of any prime dimension) and of two qutrits can be implemented efficiently. Numerical evidence gathered via the aforementioned algorithms supports the conjecture that higher-level gates can be implemented efficiently.