Cutaneous squamous cell carcinoma (cSCC) is an increasing health burden in white populations. We prospectively assessed risk factors for tumor-specific and overall survival in 1,434 patients who underwent surgery for cSCC between January 24, 2005, and May 29, 2015. A total of 2,149 invasive cSCCs were analyzed. Univariate and multivariate survival analyses included tumor thickness, horizontal size, body site, histological differentiation, desmoplastic growth, history of multiple cSCCs, and immunosuppression. The primary endpoint was time to tumor-specific death. During a median follow-up period of 36.5 months (range = 0–137 months), 515 patients died; 40 because of cSCC (2.8%). Of those, 12 died because of visceral metastases and 28 because of tumor growth by local infiltration. On multivariate analyses, prognostic factors for tumor-specific survival were increased vertical tumor thickness (hazard ratio = 6.73; 95% confidence interval = 3.47–13.08; P < 0.0001), desmoplastic growth (hazard ratio = 4.14; 95% confidence interval = 2.68–9.83; P < 0.0001), and immunosuppression (hazard ratio = 2.07; 95% confidence interval = 1.04–4.12; P = 0.039). Defining a point list out of those factors and grouping them into four cohorts resulted in comprehensively separating survival curves (P < 0.001). Using a cut-off for tumor thickness of 6 mm or greater, the presence of desmoplastic growth and immunosuppression identifies patients at high risk for tumor-specific death.