Many theoretical models predict when genetic evolution and phenotypic plasticity allow adaptation to changing environmental conditions. These models generally assume stabilizing selection around some optimal phenotype. We however often ignore how optimal phenotypes change with the environment, which limit our understanding of the adaptive value of phenotypic plasticity. Here, we propose an approach based on our knowledge of the causal relationships between climate, adaptive traits, and fitness to further these questions. This approach relies on a sensitivity analysis of the process-based model PHENOFIT, which mathematically formalizes these causal relationships, to predict fitness landscapes and optimal budburst dates along elevation gradients in three major European tree species. Variation in the overall shape of the fitness landscape and resulting directional selection gradients were found to be mainly driven by temperature variation. The optimal budburst date was delayed with elevation, while the range of dates allowing high fitness narrowed and the maximal fitness at the optimum decreased. We also found that the plasticity of the budburst date should allow tracking the spatial variation in the optimal date, but with variable mismatch depending on the species, ranging from negligible mismatch in fir, moderate in beech, to large in oak. Phenotypic plasticity would therefore be more adaptive in fir and beech than in oak. In all species, we predicted stronger directional selection for earlier budburst date at higher elevation. The weak selection on budburst date in fir should result in the evolution of negligible genetic divergence, while beech and oak would evolve counter-gradient variation, where genetic and environmental effects are in opposite directions. Our study suggests that theoretical models should consider how whole fitness landscapes change with the environment. The approach introduced here has the potential to be developed for other traits and species to explore how populations will adapt to climate change.