Rab GTPases control vesicle movement and tethering membrane events in membrane trafficking. We used the 38 human Rab GTPase activating proteins (GAPs) to identify which of the 60 Rabs encoded in the human genome function at the Golgi complex. Surprisingly, this screen identified only two GAPs, RN-tre and TBC1D20, disrupting both Golgi organization and protein transport. RN-tre is the GAP for Rab43, and controls retrograde transport into the Golgi from the endocytic pathway. TBC1D20 is the ER-localized GAP for Rab1, and is the only GAP blocking the delivery of secretory cargo from the ER to the cell surface. Strikingly, its expression causes the loss of the Golgi complex, highlighting the importance of Rab1 for Golgi biogenesis. These effects can be antagonized by reticulon, a binding partner for TBC1D20 in the ER. Together, these findings indicate that Rab1 and Rab43 are key Rabs required for the biogenesis and maintenance of a functional Golgi structure, and suggest that other Rabs acting at the Golgi complex are likely to be functionally redundant.