Lysosomal distribution is linked to the role of lysosomes in many cellular functions, including autophagosome degradation, cholesterol homeostasis, antigen presentation, and cell invasion. Alterations in lysosomal positioning contribute to different human pathologies, such as cancer, neurodegeneration, and lysosomal storage diseases. Here we report the identification of a novel mechanism of lysosomal trafficking regulation. We found that the lysosomal transmembrane protein TMEM55B recruits JIP4 to the lysosomal surface, inducing dynein-dependent transport of lysosomes toward the microtubules minus-end. TMEM55B overexpression causes lysosomes to collapse into the cell center, whereas depletion of either TMEM55B or JIP4 results in dispersion toward the cell periphery. TMEM55B levels are transcriptionally upregulated following TFEB and TFE3 activation by starvation or cholesterol-induced lysosomal stress. TMEM55B or JIP4 depletion abolishes starvation-induced retrograde lysosomal transport and prevents autophagosome-lysosome fusion. Overall our data suggest that the TFEB/TMEM55B/JIP4 pathway coordinates lysosome movement in response to a variety of stress conditions.