Male infertility is a multifactorial syndrome encompassing a wide variety of disorders. In recent years, several genome-wide single-nucleotide polymorphism (SNP) association studies (GWAS) have been performed on azoospermia and/or oligozoospermia in different populations including two GWAS on nonobstructive azoospermia in China; however, the association of SNPs with idiopathic male infertility, especially asthenozoospermia and oligozoospermia, and their correlation with semen parameters are still not clear. To investigate genetic variants associated with idiopathic male infertility (asthenozoospermia, oligozoospermia, and oligoasthenozoospermia) in Chinese Han people, 20 candidate SNPs were selected from GWAS results and genotyped using the Sequenom MassARRAY assay. A total of 136 subfertile men and 456 healthy fertile men were recruited. rs6476866 in SLC1A1 (P = 1.919E-4, OR = 0.5905, 95% CI: 0.447-0.78) and rs10129954 in DPF3 (P = 0.0023, OR = 2.199, 95% CI: 1.311-3.689) were strongly associated with idiopathic male infertility. In addition, positive associations were observed between asthenozoospermia and rs215702 in LSM5 (P = 0.0016, OR = 1.479, 95% CI: 1.075-2.033) and between oligoasthenozoospermia and rs2477686 in PEX10 (P = 0.0011, OR = 2.935, 95% CI: 1.492-5.775). In addition, six SNPs (rs215702 in LSM5, rs6476866 in SLC1A1, rs10129954 in DPF3, rs1801133 in MTHFR, rs2477686 in PEX10, and rs10841496 in PED3A) were significantly correlated with semen quality alterations. Our results suggest that idiopathic male infertility in different ethnic groups may share the same mechanism or pathway. Cohort expansion and further mechanistic studies on the role of genetic factors that influence spermatogenesis and sperm progressive motility are suggested.