We explored the association between liver metastases, tumor CD8⁺ T-cell count, and response in patients with melanoma or lung cancer treated with the anti-PD-1 antibody, pembrolizumab. The melanoma discovery cohort was drawn from the phase I Keynote 001 trial, whereas the melanoma validation cohort was drawn from Keynote 002, 006, and EAP trials and the non–small cell lung cancer (NSCLC) cohort from Keynote 001. Liver metastasis was associated with reduced response and shortened progression-free survival [PFS; objective response rate (ORR), 30.6%; median PFS, 5.1 months] compared with patients without liver metastasis (ORR, 56.3%; median PFS, 20.1 months) P ≤ 0.0001, and confirmed in the validation cohort (P = 0.0006). The presence of liver metastasis significantly increased the likelihood of progression (OR, 1.852; P < 0.0001). In a subset of biopsied patients (n = 62), liver metastasis was associated with reduced CD8⁺ T-cell density at the invasive tumor margin (liver metastasis⁺ group, n = 547 ± 164.8; liver metastasis⁻ group, n = 1,441 ± 250.7; P < 0.016). A reduced response rate and shortened PFS was also observed in NSCLC patients with liver metastasis [median PFS, 1.8 months; 95% confidence interval (CI), 1.4–2.0], compared with those without liver metastasis (n = 119, median PFS, 4.0 months; 95% CI, 2.1–5.1), P = 0.0094. Thus, liver metastatic patients with melanoma or NSCLC that had been treated with pembrolizumab were associated with reduced responses and PFS, and liver metastases were associated with reduced marginal CD8⁺ T-cell infiltration, providing a potential mechanism for this outcome. Cancer Immunol Res; 5(5); 417–24. ©2017 AACR.