Rectal neuroendocrine tumors (NETs) are often found as small lesions, which can be treated by endoscopic resection. However, high risk cases with lymph node (LN) metastasis are indication of radical surgery. Furthermore, rectal NETs are often associated with late recurrences and/or multiple cancer development. Therefore, proper surgical indication and patients' management are required. We investigated the clinicopathological features of 79 rectal NET cases in order to elucidate risk factors for synchronous LN metastasis, recurrence, and multiple cancers. Recently, we reported that in pancreatic NET patients, a loss of heterozygosity (LOH) in PHLDA3 was associated with poorer prognosis, and that LOH of both PHLDA3 and MEN1 was frequently observed. Therefore, PHLDA3 and MEN1 LOH were also assessed in rectal NET patients for their association with clinicopathological features. Of the 79 patients, LN metastases were found in 12.7%, recurrences in 3.8%, and multiple cancers in 30.4% of the subjects. PHLDA3 and MEN1 LOH were found in 60.0% and 66.7% of the subjects, respectively. Lymphatic invasion and WHO classification 2010 were found to be independent risks for LN metastasis. There were three cases of recurrence, all of which occurred more than 3 years after resection and two of which exhibited LN metastasis. Older age and LOH in PHLDA3 were associated with the presence of multiple cancers. Long-term and systemic management of patients with rectal NETs is therefore recommended in accordance with these risk factors.