Cancer microenvironment is created not only by malignant epithelial cells, but also by several kinds of stromal cells. Since Paget proposed the “seed and soil” hypothesis, the biological importance of the cancer microenvironment has come to be widely accepted. The main compartment of host stromal cells are fibroblasts (Cancer-Associated Fibroblasts; CAFs), which are the main source of the collagen-producing cells. CAFs directly communicate with the cancer cells and other types of stromal cells to acquire a specific biological phenotype. CAFs play important roles in several aspects of the tumor progression process and the chemotherapeutic process. However, CAFs have heterogeneous origins, phenotypes, and functions under these conditions. A crucial challenge is to understand how much of this heterogeneity serves different biological responses to cancer cells. In this review, we highlight the issue of how diverse and heterogeneous functions given by CAFs can exert potent influences on tumor progression and therapeutic response. Furthermore, we also discuss the current advances in the development of novel therapeutic strategies against CAFs.