A comprehensive understanding of the kinetics and evolution of the human B cell response to SARS-CoV-2 infection will facilitate the development of next-generation vaccines and therapies. Here, we longitudinally profiled this response in patients with mild and severe COVID-19 patients over a period of 5 months. Serum neutralizing antibody (nAb) responses waned rapidly but spike (S)–specific IgG⁺ memory B cells (MBCs) remained stable or increased over time. Analysis of 1213 monoclonal antibodies (mAbs) isolated from S-specific MBCs revealed a primarily de novo response that displayed increased somatic hypermutation, binding affinity, and neutralization potency over time, providing evidence for prolonged antibody affinity maturation. B cell immunodominance hierarchies were similar across donor repertoires and remained relatively stable as the immune response progressed. Cross-reactive B cell populations, likely re-called from prior endemic β-coronavirus exposures, comprised a small but stable fraction of the repertoires and did not contribute to the neutralizing response. The nAb response was dominated by public clonotypes that displayed significantly reduced activity against SARS-CoV-2 variants emerging in Brazil and South Africa that harbor mutations at positions 501, 484, and 417 in the S protein. Overall, the results provide insight into the dynamics, durability, and functional properties of the human B cell response to SARS-CoV-2 infection and have implications for the design of immunogens that preferentially stimulate protective B cell responses.