The hypothalamo-pituitary-adrenal (HPA) axis plays a key role in the neuroendocrine response to stress and in maintaining physiological homoeostasis. However, stress that is chronic in nature can lead to HPA axis dysfunction and increase the risk for developing affective disorders, particularly if the stress is experienced during vulnerable periods in life. Sex differences in how the HPA axis responds to stress are well established, with females typically displaying heightened responses. The underlying cause of these sex differences is important to understand, as many neuropsychiatric disorders disproportionately affect females. Much research has provided evidence for gonadal sex steroids in underpinning sex differences in HPA axis responsivity; however, we suggest that neuroactive metabolites of these steroids also play a key role in the brain in mediating sex differences in HPA axis responses to stress. The relationship between neuroactive steroids and stress is complex. Acute stress rapidly increases neuroactive steroid production, which can in turn modulate activity of the HPA axis. However, under chronic stress conditions, stress can impact the brain's capacity to generate steroids, and this in turn has corollary effects on HPA axis function that may increase the propensity for psychopathology, given both HPA axis dysfunction and deficits in neuroactive steroids are implicated in affective disorders. Hence, here we review the evidence from animal and human studies for sex differences in the interactions between neuroactive steroids and the stress axis at various stages of life, under physiological and pathophysiological stress conditions and consider the implications for health and disease.