During aging, stromal functions are thought to be impaired, but little is known whether this stems from changes of fibroblasts. Using population- and single-cell transcriptomics, as well as long-term lineage tracing, we studied whether murine dermal fibroblasts are altered during physiological aging under different dietary regimes that affect longevity. We show that the identity of old fibroblasts becomes undefined, with the fibroblast states present in young skin no longer clearly demarcated. In addition, old fibroblasts not only reduce the expression of genes involved in the formation of the extracellular matrix, but also gain adipogenic traits, paradoxically becoming more similar to neonatal pro-adipogenic fibroblasts. These alterations are sensitive to systemic metabolic changes: long-term caloric restriction reversibly prevents them, whereas a high-fat diet potentiates them. Our results therefore highlight loss of cell identity and the acquisition of adipogenic traits as a mechanism underlying cellular aging, which is influenced by systemic metabolism.