The rapid formation of numerous tissues during development is highly dependent on the swift activation of key developmental regulators. Recent studies indicate that many key regulatory genes are repressed in embryonic stem cells (ESCs), yet poised for rapid activation due to the presence of both activating (H3K4 trimethylation) and repressive (H3K27 trimethylation) histone modifications (bivalent genes). However, little is known about bivalent gene regulation. In this study, we investigated the regulation of the bivalent gene Sox21, which is activated rapidly when ESCs differentiate in response to increases in Sox2. Chromatin immunoprecipitation demonstrated that prior to differentiation, the Sox21 gene is bound by a complex array of repressive and activating transcriptional machinery. Upon activation, all identified repressive machinery and histone modifications associated with the gene are lost, but the activating modifications and transcriptional machinery are retained. Notably, these changes do not occur when ESCs differentiate in response to retinoic acid. Moreover, ESCs lacking a functional PRC2 complex fail to activate this gene, apparently due to its association with other repressive complexes. Together, these findings suggest that bivalent genes, such as Sox21, are silenced by a complex set of redundant repressive machinery, which exit rapidly in response to appropriate differentiation signals.