The SARS-CoV-2 pandemic has resulted in millions of infections, yet the role of host immune responses in early COVID-19 pathogenesis remains unclear. By investigating 17 acute and 24 convalescent patients, we found that acute SARS-CoV-2 infection resulted in broad immune cell reduction including T, natural killer, monocyte, and dendritic cells (DCs). DCs were significantly reduced with functional impairment, and ratios of conventional DCs to plasmacytoid DCs were increased among acute severe patients. Besides lymphocytopenia, although neutralizing antibodies were rapidly and abundantly generated in patients, there were delayed receptor binding domain (RBD)- and nucleocapsid protein (NP)-specific T cell responses during the first 3 weeks after symptoms onset. Moreover, acute RBD- and NP-specific T cell responses included relatively more CD4 T cells than CD8 T cells. Our findings provided evidence that impaired DCs, together with timely inverted strong antibody but weak CD8 T cell responses, could contribute to acute COVID-19 pathogenesis and have implications for vaccine development.