Emerging SARS-CoV-2 variants of concern (VOCs) pose a threat to human immunity induced by natural infection and vaccination. We assessed the recognition of three VOCs (B.1.1.7, B.1.351, and P.1) in cohorts of COVID-19 convalescent patients (n = 69) and Pfizer-BioNTech vaccine recipients (n = 50). Spike binding and neutralization against all three VOCs were substantially reduced in most individuals, with the largest four- to sevenfold reduction in neutralization being observed against B.1.351. While hospitalized patients with COVID-19 and vaccinees maintained sufficient neutralizing titers against all three VOCs, 39% of nonhospitalized patients exhibited no detectable neutralization against B.1.351. Moreover, monoclonal neutralizing antibodies show sharp reductions in their binding kinetics and neutralizing potential to B.1.351 and P.1 but not to B.1.1.7. These data have implications for the degree to which pre-existing immunity can protect against subsequent infection with VOCs and informs policy makers of susceptibility to globally circulating SARS-CoV-2 VOCs.