There are conflicting reports on the effects of HIV on COVID-19. Here, we analyzed disease severity and immune cell changes during and after SARS-CoV-2 infection in 236 participants from South Africa, of which 39% were people living with HIV (PLWH), during the first and second (Beta dominated) infection waves. The second wave had more PLWH requiring supplemental oxygen relative to HIV-negative participants. Higher disease severity was associated with low CD4 T cell counts and higher neutrophil to lymphocyte ratios (NLR). Yet, CD4 counts recovered and NLR stabilized after SARS-CoV-2 clearance in wave 2 infected PLWH, arguing for an interaction between SARS-CoV-2 and HIV infection leading to low CD4 and high NLR. The first infection wave, where severity in HIV negative and PLWH was similar, still showed some HIV modulation of SARS-CoV-2 immune responses. Therefore, HIV infection can synergize with the SARS-CoV-2 variant to change COVID-19 outcomes.