Memory B cells (MBCs) can respond to heterologous antigens either by molding new specificities through secondary germinal centers (GCs) or by selecting preexisting clones without further affinity maturation. To distinguish these mechanisms in flavivirus infections and immunizations, we studied recall responses to envelope protein domain III (DIII). Conditional deletion of activation-induced cytidine deaminase (AID) between heterologous challenges of West Nile, Japanese encephalitis, Zika, and dengue viruses did not affect recall responses. DIII-specific MBCs were contained mostly within the plasma-cell-biased CD80⁺ subset, and few GCs arose following heterologous boosters, demonstrating that recall responses are confined by preexisting clonal diversity. Measurement of monoclonal antibody (mAb) binding affinity to DIII proteins, timed AID deletion, single-cell RNA sequencing, and lineage tracing experiments point to selection of relatively low-affinity MBCs as a mechanism to promote diversity. Engineering immunogens to avoid this MBC diversity may facilitate flavivirus-type-specific vaccines with minimized potential for infection enhancement.