Grazing exclosures have been promoted as an effective and low-cost land management strategy to recover vegetation and associated functions in degraded landscapes in the tropics. While grazing exclosures can be important reservoirs of biodiversity and carbon, their potential in playing a dual role of conservation of biodiversity and mitigation of climate change effects is not yet established. To address this gap, we assessed the effect of diversity on aboveground carbon (AGC) and the relative importance of the driving biotic (functional diversity, functional composition and structural diversity) and abiotic (climate, topography and soil) mechanisms. We used a dataset from 133 inventory plots across three altitudinal zones, i.e., highland, midland and lowland, in northern Ethiopia, which allowed local- (within altitudinal zone) and broad- (across altitudinal zones) environmental scale analysis of diversity-AGC relationships. We found that species richness-AGC relationship shifted from neutral in highlands to positive in mid- and lowlands as well as across the altitudinal zones. Structural diversity was consistently the strongest mediator of the positive effects of species richness on AGC within and across altitudinal zones, whereas functional composition linked species richness to AGC at the broad environmental scale only. Abiotic factors had direct and indirect effects via biotic factors on AGC, but their relative importance varied with altitudinal zones. Our results indicate that the effect of species diversity on AGC was altitude-dependent and operated more strongly through structural diversity (representing niche complementarity effect) than functional composition (representing selection effect). Our study suggests that maintaining high structural diversity and managing functionally important species while promoting favourable climatic and soil conditions can enhance carbon storage in grazing exclosures.