Multiple studies have described associations between infiltrating immune cells and prognosis in cancer; however, the clinical relevance has most often been attributed to the T‐cell linage. This study aimed to further investigate the clinicopathological correlates and prognostic impact of B cell and plasma cell infiltration in CRC. Immunohistochemical expression of CD20, CD138 and immunoglobulin kappa C (IGKC) was analysed in tissue microarrays with tumours from 557 incident cases of CRC from a prospective population‐based cohort. Kaplan–Meier analysis and Cox regression analysis were used to determine the impact of CD20, CD138 and IGKC expression on 5‐year overall survival. Immune cell‐specific CD20, CD138, and IGKC expression correlated significantly with lower T‐stage (p < 0.001, p < 0.001, and p = 0.006, respectively). A higher density of CD20+ cells correlated significantly with an improved OS (HR = 0.53, 95% CI 0.36–0.78), remaining significant in multivariable analysis adjusted for age, TNM stage, differentiation grade and vascular invasion (HR = 0.51; 95% CI 0.33–0.80). Immune cell‐specific CD138 and IGKC expression correlated significantly with an improved OS in univariable Cox regression analysis; however, these associations did not remain significant in multivariable analysis. Finally, tumour cell‐specific CD138 expression was found to be an independent factor of poor prognosis (HR 1.52; 95% CI 1.03–2.24). The results from the present study demonstrate that B cell infiltration in CRC has a significant impact on tumour progression and prognosis. These findings supplement and extend the current knowledge of the immune landscape in colorectal cancer, and merit further study.