Drought stress modifies the physiology of individual trees and, repeated droughts eventually provoke an evolutionary change. We studied the effect of drought in silver fir (Abies alba) and common beech (Fagus sylvatica) that cohabit along an altitudinal gradient on Mont Ventoux, southern France. First, we performed a dendroclimatic time series analysis for the period of 1960–2006. We show that silver fir’s growth rate is principally sensitive to the given year’s drought conditions, while common beech responds to the previous year’s climate. The maximal growth rate was observed at low-intermediate elevations for common beech and at intermediate elevations for silver fir. Recent warming has already caused slight upward shift of these growth optima. To understand the physiological mechanisms underlying the inter-annual and -individual variation in growth rate, we developed a physio-demographic model that couples CASTANEA, an existing physiological model that simulates individual’s daily response to climate with a demographic model that converts carbohydrates into growth and seed production and models dispersion and mortality. Using dynamically downscaled climate predictions for the period 2001–2100, we predict significant shifts in the physiological optimum of these two species, which is expected to modify the population dynamics and species composition of the forests of Mont Ventoux.