Perilipin 1 is a lipid droplet coating protein known to regulate lipid metabolism in adipocytes by serving as a physical barrier as well as a recruitment site for lipases to the lipid droplet. Phosphorylation of perilipin 1 by protein kinase A rapidly initiates lipolysis, but the detailed mechanism on how perilipin 1 controls lipolysis is unknown. Here, we identify specific lipid binding properties of perilipin 1 that regulate the dynamics of lipolysis in human primary adipocytes. Cellular imaging combined with biochemical and biophysical analyses demonstrate that perilipin 1 specifically binds to cholesteryl esters, and that their dynamic properties direct segregation of perilipin 1 into topologically distinct micro domains on the lipid droplet. Together, our data points to a simple unifying mechanism that lipid assembly and segregation control lipolysis in human primary adipocytes.