The fate of protective immunity following mild severe acute respiratory syndrome-coronavirus-2 (SARS-CoV-2) infection remains ill defined. Here, we characterize antibody responses in a cohort of participants recovered from mild SARS-CoV-2 infection with follow-up to 6 months. We measure immunoglobulin A (IgA), IgM, and IgG binding and avidity to viral antigens and assess neutralizing antibody responses over time. Furthermore, we correlate the effect of fever, gender, age, and time since symptom onset with antibody responses. We observe that total anti-S trimer, anti-receptor-binding domain (RBD), and anti-nucleocapsid protein (NP) IgG are relatively stable over 6 months of follow-up, that anti-S and anti-RBD avidity increases over time, and that fever is associated with higher levels of antibodies. However, neutralizing antibody responses rapidly decay and are strongly associated with declines in IgM levels. Thus, while total antibody against SARS-CoV-2 may persist, functional antibody, particularly IgM, is rapidly lost. These observations have implications for the duration of protective immunity following mild SARS-CoV-2 infection.