Dissecting the evolution of memory B cells (MBCs) against SARS-CoV-2 is critical for understanding antibody recall upon secondary exposure. Here, we used single-cell sequencing to profile SARS-CoV-2-reactive B cells in 38 COVID-19 patients. Using oligo-tagged antigen baits, we isolated B cells specific to the SARS-CoV-2 spike, nucleoprotein (NP), open reading frame 8 (ORF8), and endemic human coronavirus (HCoV) spike proteins. SARS-CoV-2 spike-specific cells were enriched in the memory compartment of acutely infected and convalescent patients several months post symptom onset. With severe acute infection, substantial populations of endemic HCoV-reactive antibody-secreting cells were identified and possessed highly mutated variable genes, signifying preexisting immunity. Finally, MBCs exhibited pronounced maturation to NP and ORF8 over time, especially in older patients. Monoclonal antibodies against these targets were non-neutralizing and non-protective in vivo. These findings reveal antibody adaptation to non-neutralizing intracellular antigens during infection, emphasizing the importance of vaccination for inducing neutralizing spike-specific MBCs.