Retromer is a membrane coat complex that is recruited to endosomes by the small GTPase Rab7 and sorting nexin 3. The timing of this interaction and consequent endosomal dynamics are thought to be regulated by the guanine nucleotide cycle of Rab7. Here we demonstrate that TBC1d5, a GTPase-activating protein (GAP) for Rab7, is a high-affinity ligand of the retromer cargo selective complex VPS26/VPS29/VPS35. The crystal structure of the TBC1d5 GAP domain bound to VPS29 and complementary biochemical and cellular data show that a loop from TBC1d5 binds to a conserved hydrophobic pocket on VPS29 opposite the VPS29-VPS35 interface. Additional data suggest that a distinct loop of the GAP domain may contact VPS35. Loss of TBC1d5 causes defective retromer-dependent trafficking of receptors. Our findings illustrate how retromer recruits a GAP, which is likely to be involved in the timing of Rab7 inactivation leading to membrane uncoating, with important consequences for receptor trafficking.