The evolution of SARS-CoV-2 could impair recognition of the virus by human antibody-mediated immunity. To facilitate prospective surveillance for such evolution, we map how convalescent plasma antibodies are impacted by all mutations to the spike's receptor-binding domain (RBD), the main target of plasma neutralizing activity. Binding by polyclonal plasma antibodies is affected by mutations in three main epitopes in the RBD, but longitudinal samples reveal that the impact of these mutations on antibody binding varies substantially both among individuals and within the same individual over time. Despite this inter- and intra-person heterogeneity, the mutations that most reduce antibody binding usually occur at just a few sites in the RBD's receptor-binding motif. The most important site is E484, where neutralization by some plasma is reduced >10-fold by several mutations, including one in the emerging 20H/501Y.V2 and 20J/501Y.V3 SARS-CoV-2 lineages. Going forward, these plasma escape maps can inform surveillance of SARS-CoV-2 evolution.