Management of localized well-differentiated pancreatic neuroendocrine tumors (panNETs) is controversial and primarily dependent on tumor size. Upfront surgery is usually recommended for tumors larger than 2 cm in diameter since they frequently show metastatic potential, whereas smaller panNETs are generally characterized by an indolent clinical course, with a rate of relapse or metastasis below 15%. To explore whether increased tumor size is paralleled by genomic variations, we compared the rate and the mutational patterns of putative driver genes that are recurrently altered in these tumors by investigating differential cohorts of panNET surgical specimens smaller (n = 27) or larger than 2 cm (n = 29). We found that the cumulative number of mutations detected in panNETs >2 cm was significantly higher (p = 0.03) relative to smaller tumors, while mutations of DAXX were significantly more frequent in the cohort of larger tumors (p = 0.05). Moreover, mutations of DAXX were associated with features of malignancy including increased grade, nodal involvement and lymphovascular invasion, and independently predicted both relapse after surgery (p = 0.05) and reduced DFS in multivariable analysis (p = 0.02). Our data suggest that alterations of the DAXX/ATRX molecular machinery increase the malignant potential of panNETs, and that identification of mutations of DAXX/ATRX in small, nonfunctioning tumors can predict the malignant progression observed in a minority of them.