The inherent complexity of multifactorial diseases such as cancer renders the process of patient prognosis and prediction of response to therapy extremely difficult. Many markers, signatures, and methods have been described to evaluate the prognosis of cancer patients, yet very few translate into the clinic. Systems biology approaches have facilitated analysis of the complex interaction between tumors and the host-immune response, and allowed the definition of the immune contexture. Here we review the potential of the immune contexture, quantified by the Immunoscore, to provide a statistically strong parameter for prognosis. Finally we introduce the concept that the host-immune reaction could be the critical element in determining response to therapy. The effect on the immune response could be the underlying factor behind many of the predictive markers.