In addition to serum immunoglobulins, memory B cell (MBC) generation against severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) is another layer of immune protection, but the quality of MBC responses in naive and coronavirus disease 2019 (COVID-19)-recovered individuals after vaccination remains ill defined. We studied longitudinal cohorts of naive and disease-recovered individuals for up to 2 months after SARS-CoV-2 mRNA vaccination. We assessed the quality of the memory response by analysis of antibody repertoires, affinity, and neutralization against variants of concern (VOCs) using unbiased cultures of 2,452 MBCs. Upon boosting, the MBC pool of recovered individuals expanded selectively, matured further, and harbored potent neutralizers against VOCs. Although naive individuals had weaker neutralizing serum responses, half of their RBD-specific MBCs displayed high affinity toward multiple VOCs, including delta (B.1.617.2), and one-third retained neutralizing potency against beta (B.1.351). Our data suggest that an additional challenge in naive vaccinees could recall such affinity-matured MBCs and allow them to respond efficiently to VOCs.