Immunotherapy is the new trend in cancer treatment due to the selectivity, long lasting effects, and demonstrated improved overall survival and tolerance, when compared to patients treated with conventional chemotherapy. Despite these positive results, immunotherapy is still far from becoming the perfect magic bullet to fight cancer, largely due to the facts that immunotherapy is not effective in all patients nor in all cancer types. How and when will immunotherapy overcome these hurdles? In this review we take a step back to walk side by side with the pioneers of immunotherapy in order to understand what steps need to be taken today to make immunotherapy effective across all cancers. While early scientists, such as Coley, elicited an unselective but effective response against cancer, the search for selectivity pushed immunotherapy to the side in favor of drugs focused on targeting cancer cells. Fortunately, the modern era would revive the importance of the immune system in battling cancer by releasing the brakes or checkpoints (anti-CTLA-4 and anti-PD-1/PD-L1) that have been holding the immune system at bay. However, there are still many hurdles to overcome before immunotherapy becomes a universal cancer therapy. For example, we discuss how the redundant and complex nature of the immune system can impede tumor elimination by teeter tottering between different polarization states: one eliciting anti-cancer effects while the other promoting cancer growth and invasion. In addition, we highlight the incapacity of the immune system to choose between a fight or repair action with respect to tumor growth. Finally we combine these concepts to present a new way to think about the immune system and immune tolerance, by introducing two new metaphors, the “push the accelerator” and “repair the car” metaphors, to explain the current limitations associated with cancer immunotherapy.