Numerous assays evaluating serological and cellular responses have been developed to characterize immune responses against SARS-CoV-2. Serological assays are both cost- and time-effective compared to cellular assays, but cellular immune responses may provide a diagnostic value to determine previous SARS-CoV-2 infection in seronegative individuals. However, potential cross-reactive T cell responses stemming from prior encounters with human coronaviruses (HCoVs) may affect assay specificity. In this study, we evaluated the specificity and sensitivity of a SARS-CoV-2 IFN-γ Release Assay (IGRA) based on the FluoroSpot method employing commercially available SARS-CoV-2-specific peptide pools, as well as an in-house designed SARS-CoV-2 peptide pool restricted to 5 amino acid stretches or less aligning with endemic HCoVs. Blood samples were obtained from healthcare workers (HCW) 5–6 months post SARS-CoV-2 spike (S) IgG and nucleocapsid (N) IgG dual seroconversion (n = 187) and HCW who had been S IgG and N IgG dual seronegative at repeated occasions, including the current sampling time point (n = 102). In addition, samples were obtained 4 to 5 months post infection from 55 polymerase chain reaction (PCR)-confirmed COVID-19 patients. Assay specificity and sensitivity were calculated with serology as a reference standard for HCW. The in-house generated peptide pool displayed a specificity of 96.1%, while the commercially available peptide pools displayed specificities of 80.4% and 85.3%, respectively. Sensitivity was higher in a cohort of previously hospitalized COVID-19 patients (96.4% and 84.0% for the commercially available peptide pools and 92.7% for the in-house generated peptide pool) compared to the HCW cohort (92.0% and 66.8% for the commercially available peptide pools and 76.0% for the in-house generated peptide pool). Based on these findings, the individual diagnostic value of T cell immune responses against SARS-CoV-2 currently appears to be limited but remain an important research tool ahead.