Half of estrogen receptor-positive breast cancers contain a subpopulation of cytokeratin 5 (CK5)-expressing cells that are therapy resistant and exhibit increased cancer stem cell (CSC) properties. We and others have demonstrated that progesterone (P4) increases CK5+ breast cancer cells. We previously discovered that retinoids block P4 induction of CK5+ cells. Here we investigated the mechanisms by which progesterone receptors (PR) and retinoic acid receptors (RAR) regulate CK5 expression and breast CSC activity. After P4 treatment, sorted CK5+ compared to CK5- cells were more tumorigenic in vivo. In vitro, P4-treated breast cancer cells formed larger mammospheres and silencing of CK5 using small hairpin RNA abolished this P4-dependent increase in mammosphere size. Retinoic acid (RA) treatment blocked the P4 increase in CK5+ cells and prevented the P4 increase in mammosphere size. Dual small interfering RNA (siRNA) silencing of RARα and RARγ reversed RA blockade of P4-induced CK5. Using promoter deletion analysis, we identified a region 1.1 kb upstream of the CK5 transcriptional start site that is necessary for P4 activation and contains a putative progesterone response element (PRE). We confirmed by chromatin immunoprecipitation that P4 recruits PR to the CK5 promoter near the -1.1 kb essential PRE, and also to a proximal region near -130 bp that contains PRE half-sites and a RA response element (RARE). RA induced loss of PR binding only at the proximal site. Interestingly, RARα was recruited to the -1.1 kb PRE and the -130 bp PRE/RARE regions with P4, but not RA alone or RA plus P4. Treatment of breast cancer xenografts in vivo with the retinoid fenretinide reduced the accumulation of CK5+ cells during estrogen depletion. This reduction, together with the inhibition of CK5+ cell expansion through RAR/PR cross talk, may explain the efficacy of retinoids in prevention of some breast cancer recurrences.