The seminal vesicles (SVs), like much of the male reproductive tract, depend on androgen-driven stromal-epithelial interactions for normal development, structure, and function. The primary function of the SVs is to synthesize proteins that contribute to the seminal plasma and this is androgen dependent. However, the cell-specific role for androgen action in adult SVs remains unclear. This study analyzed the SV in mice with targeted ablation of androgen receptors specifically in smooth muscle cells (PTM-ARKO) to determine in vivo whether it is androgen action in a subset of the SV stroma, the smooth muscle cells, that drives epithelial function and identity. These mice have significantly smaller SVs in adulthood with less smooth muscle and reduced epithelial cell height. Less epithelial cell proliferation was observed in adult PTM-ARKO SVs, compared with controls, and production of seminal proteins was reduced, indicating global impairment of epithelial cell function in PTM-ARKO SVs. None of these changes could be explained by altered serum testosterone or estradiol concentrations. We also demonstrate altered SV responsiveness to exogenous testosterone and estradiol in PTM-ARKO mice, indicating that smooth muscle androgen receptors may limit the SV epithelial proliferative response to exogenous estrogens. These results therefore demonstrate that the smooth muscle cells play a vital role in androgen-driven stromal-epithelial interactions in the SV, determining epithelial cell structure and function as well as limiting the SV epithelial proliferative response to exogenous estrogens.