Epigenetic histone modifications play critical roles in the control of self-renewal and differentiation of hematopoietic stem cells (HSCs). Mysm1 is a recently identified histone H2A deubiquitinase with essential and intrinsic roles for maintaining functional HSCs. In this study, in addition to confirming this function of Mysm1, by using Mysm1-deficient (Mysm1^−/−) mice, we provide more evidence for how Mysm1 controls HSC homeostasis. Mysm1 deletion drives HSCs from quiescence into rapid cycling and increases their apoptotic rate, resulting in an exhaustion of the stem cell pool, which leads to an impaired self-renewal and lineage reconstituting abilities in the Mysm1-deficient mice. Our study identified Gfi1 as one of the candidate genes responsible for the HSC defect in Mysm1-deficient mice. Mechanistic studies revealed that Mysm1 modulates histone modifications and directs the recruitment of key transcriptional factors such as Gata2 and Runx1 to the Gfi1 locus in HSCs. We found that Mysm1 directly associates with the Gfi1 enhancer element and promotes its transcription through Gata2 and Runx1 transactivation. Thus, our study not only elaborates on the initial reports of Mysm1 association with HSC homeostasis but also delineates a possible epigenetic mechanism through which Mysm1 carries out this function in the HSCs.