Aberrant fetal growth is associated with morbidities and mortality during childhood and adult life. Although genetic and environmental factors are known to influence in utero growth, their relative contributions over pregnancy is unknown. We estimated, across gestation, the genetic heritability, contribution of shared environment, and genetic correlations of fetal growth measures (abdominal circumference (AC), humerus length (HL), femur length (FL), and estimated fetal weight (EFW)) in a prospective cohort of dichorionic twin gestations recruited through the NICHD Fetal Growth Studies. Structural equation models were fit at the end of first trimester, during mid-gestation, late second trimester, and third trimester of pregnancy. The contribution of fetal genetics on fetal size increased with gestational age, peaking in late second trimester (AC = 53%, HL = 57%, FL = 72%, EFW = 71%; p < 0.05). In contrast, shared environment explained most of phenotypic variations in fetal growth in the first trimester (AC = 50%, HL = 54%, FL = 47%, EFW = 54%; p < 0.05), suggesting that the first trimester presents an intervention opportunity for a more optimal early fetal growth. Genetic correlations between growth traits (range 0.34–1.00; p < 0.05) were strongest at the end of first trimester and declined with gestation, suggesting that different fetal growth measures are more likely to be influenced by the same genes in early pregnancy.