There is a pressing need for an in-depth understanding of immunity to SARS-CoV-2. In this study, we investigated human T cell recall responses to fully glycosylated spike trimer, recombinant N protein, as well as to S, N, M, and E peptide pools in the early convalescent phase and compared them with influenza-specific memory responses from the same donors. All subjects showed SARS-CoV-2–specific T cell responses to at least one Ag. Both SARS-CoV-2–specific and influenza-specific CD4+ T cell responses were predominantly of the central memory phenotype; however SARS-CoV-2–specific CD4+ T cells exhibited a lower IFN-γ to TNF ratio compared with influenza-specific memory responses from the same donors, independent of disease severity. SARS-CoV-2–specific T cells were less multifunctional than influenza-specific T cells, particularly in severe cases, potentially suggesting exhaustion. Most SARS-CoV-2–convalescent subjects also produced IFN-γ in response to seasonal OC43 S protein. We observed granzyme B+/IFN-γ+, CD4+, and CD8+ proliferative responses to peptide pools in most individuals, with CD4+ T cell responses predominating over CD8+ T cell responses. Peripheral T follicular helper (pTfh) responses to S or N strongly correlated with serum neutralization assays as well as receptor binding domain–specific IgA; however, the frequency of pTfh responses to SARS-CoV-2 was lower than the frequency of pTfh responses to influenza virus. Overall, T cell responses to SARS-CoV-2 are robust; however, CD4+ Th1 responses predominate over CD8+ T cell responses, have a more inflammatory profile, and have a weaker pTfh response than the response to influenza virus within the same donors, potentially contributing to COVID-19 disease.