Obese individuals exhibit an increase in pancreatic β cell mass; conversely, scarce nutrition during pregnancy has been linked to β cell insufficiency in the offspring [reviewed in 1, 2]. These phenomena are thought to be mediated mainly through effects on β cell proliferation, given that a nutrient-sensitive β cell progenitor population in the pancreas has not been identified. Here, we employed the fluorescent ubiquitination-based cell-cycle indicator system to investigate β cell replication in real time and found that high nutrient concentrations induce rapid β cell proliferation. Importantly, we found that high nutrient concentrations also stimulate β cell differentiation from progenitors in the intrapancreatic duct (IPD). Furthermore, using a new zebrafish line where β cells are constitutively ablated, we show that β cell loss and high nutrient intake synergistically activate these progenitors. At the cellular level, this activation process causes ductal cell reorganization as it stimulates their proliferation and differentiation. Notably, we link the nutrient-dependent activation of these progenitors to a downregulation of Notch signaling specifically within the IPD. Furthermore, we show that the nutrient sensor mechanistic target of rapamycin (mTOR) is required for endocrine differentiation from the IPD under physiological conditions as well as in the diabetic state. Thus, this study reveals critical insights into how cells modulate their plasticity in response to metabolic cues and identifies nutrient-sensitive progenitors in the mature pancreas.