Externalizing psychopathology is a strong risk factor for substance use, whereas the role of internalizing manifestations of distress, and anxiety in particular, in predicting substance use remains unclear. Studies have suggested that anxiety may be either a protective or risk factor for substance use. The present study aimed to clarify evidence for anxiety-specific associations with substance use, examining sex and developmental period (adolescence vs. adulthood) as potential moderators that may help explain conflicting results in the literature. In a longitudinal twin sample, cross-sectional associations of anxiety with substance use differed in adolescents and adults and in girls/women and boys/men. Controlling for externalizing psychopathology and depression, anxiety was associated with reduced substance use in adolescent girls and increased substance use in adult women. In contrast, anxiety-specific associations with substance use were not significant in boys and men. Possible explanations for these contrasting results across development and sex are discussed.