Mutual influences between anxiety and working memory (WM) have been extensively studied, and their curvilinear relationship resembles the classic Yerkes-Dodson law of arousal and performance. Given the genetic bases of both anxiety and WM, it is likely that the individual differences in the Yerkes-Dodson law of anxiety and WM may have genetic correlates. The current genome wide association study (GWAS) enrolled 1115 healthy subjects to search for genes that are potential moderators of the association between anxiety and WM. Results showed that CPNE3 rs10102229 had the strongest effect, p = 3.38E-6 at SNP level and p = 2.68E-06 at gene level. Anxiety and WM had a significant negative correlation (i.e., more anxious individuals performed worse on the WM tasks) for the TT genotype of rs10102229 (resulting in lower expression of CPNE3), whereas the correlation was positive (i.e., more anxious individuals performed better on the WM tasks) for the CC carriers. The same pattern of results was found at the gene level using gene score analysis. These effects were replicated in an independent sample (N = 330). The current study is the first to report a gene that moderates the relation between anxiety and WM and potentially provides a genetic explanation for the classic Yerkes-Dodson law.