Despite mounting evidence that materials imperfections are a major obstacle to practical applications of superconducting qubits, connections between microscopic material properties and qubit coherence are poorly understood. Here, we combine measurements of transmon qubit relaxation times (T1) with spectroscopy and microscopy of the polycrystalline niobium films used in qubit fabrication. By comparing films deposited using three different techniques, we reveal correlations between T1 and intrinsic film properties such as grain size, enhanced oxygen diffusion along grain boundaries, and the concentration of suboxides near the surface. Qubit and resonator measurements show signatures of two-level system defects, which we propose to be hosted in the grain boundaries and surface oxides. We also show that the residual resistance ratio of the polycrystalline niobium films can be used as a figure of merit for qubit lifetime. This comprehensive approach to understanding qubit decoherence charts a pathway for materials-driven improvements of superconducting qubit performance.