Normal function of the placenta depends on the earliest developmental stages when trophoblast cells differentiate and invade into the endometrium to establish the definitive maternal-fetal interface. Previously, we identified the ubiquitously expressed tumour suppressor BRCA1-associated protein 1 (BAP1) as a central factor of a novel molecular node controlling early mouse placentation. However, functional insights into how BAP1 regulates trophoblast biology are still missing. Using CRISPR/Cas9 knockout and overexpression technology in mouse trophoblast stem cells, here we demonstrate that the downregulation of BAP1 protein is essential to trigger epithelial-mesenchymal transition (EMT) during trophoblast differentiation associated with a gain of invasiveness. Moreover, we show that the function of BAP1 in suppressing EMT progression is dependent on the binding of BAP1 to additional sex comb-like (ASXL1/2) proteins to form the polycomb repressive deubiquitinase (PR-DUB) complex. Finally, both endogenous expression patterns and BAP1 overexpression experiments in human trophoblast stem cells suggest that the molecular function of BAP1 in regulating trophoblast differentiation and EMT progression is conserved in mice and humans. Our results reveal that the physiological modulation of BAP1 determines the invasive properties of the trophoblast, delineating a new role of the BAP1 PR-DUB complex in regulating early placentation.