Mounting evidence suggested that histone H4K20‑specific methyltransferase SET8 is required to maintain the malignant phenotype of various cancer types; however, the role of SET8 in mediating tumor metastasis in prostate cancer (PCa) has remained elusive. The present study demonstrated that small interfering RNA‑mediated knockdown of SET8 inhibited the invasive potential of the PCa cell line PC‑3 in vitro. Knockdown of SET8 reduced sphere formation, downregulated E‑cadherin and α‑catenin, and upregulated N‑cadherin and vimentin expression in CaP cells, while upregulation of SET8 expression with a recombinant plasmid had the opposite effect. Furthermore, SET8 was shown to be physically associated with the epithelial‑mesenchymal transition (EMT) inducer zinc finger E‑box‑binding homeobox 1 (ZEB1) in PCa cell lines. Chromatin immunoprecipitation suggested that SET8 binds to the promoter of cell adhesion molecule E‑cadherin and vimentin. Luciferase reporter assays suggested that E‑cadherin and vimentin are direct targets of SET8; furthermore, loss‑and gain‑of function studies of SET8 and ZEB1 indicated that suppression of downstream E‑cadherin and activation of vimentin are important mechanisms by which SET8 and ZEB1 cooperatively trigger metastasis. Furthermore, SET8‑induced methylated H4K20 was indicated to exert a dual function in ZEB1‑regulated gene expression. In conclusion, the present study revealed that SET8 and ZEB1 are functionally interdependent in promoting the EMT and enhancing the invasive potential of PCa cells in vitro.