SARS-CoV-2 is one of three coronaviruses that have crossed the animal-to-human barrier and caused widespread disease in the past two decades. The development of a universal human coronavirus vaccine could prevent future pandemics. We characterize 198 antibodies isolated from four COVID-19+ subjects and identify 14 SARS-CoV-2 neutralizing antibodies. One targets the N-terminal domain (NTD), one recognizes an epitope in S2, and 11 bind the receptor-binding domain (RBD). Three anti-RBD neutralizing antibodies cross-neutralize SARS-CoV-1 by effectively blocking binding of both the SARS-CoV-1 and SARS-CoV-2 RBDs to the ACE2 receptor. Using the K18-hACE transgenic mouse model, we demonstrate that the neutralization potency and antibody epitope specificity regulates the in vivo protective potential of anti-SARS-CoV-2 antibodies. All four cross-neutralizing antibodies neutralize the B.1.351 mutant strain. Thus, our study reveals that epitopes in S2 can serve as blueprints for the design of immunogens capable of eliciting cross-neutralizing coronavirus antibodies.