Small leucine‐rich proteoglycans (SLRPs) regulate extracellular matrix organization, a process essential in development, tissue repair, and metastasis. In vivo interactions of biglycan and fibromodulin, two SLRPs highly expressed in tendons and bones, were investigated by generating biglycan/fibromodulin double‐deficient mice. Here we show that collagen fibrils in tendons from mice deficient in biglycan and/or fibromodulin are structurally and mechanically altered resulting in unstable joints. As a result, the mice develop successively and progressively 1) gait impairment, 2) ectopic tendon ossification, and 3) severe premature osteoarthritis. Forced use of the joints increases ectopic ossification and osteoarthritis in the double‐deficient mice, further indicating that structurally weak tendons cause the phenotype. The study shows that mutations in SLRPs may predispose to osteoarthritis and offers a valuable and unique animal model for spontaneous osteoarthritis characterized by early onset and a rapid progression of the disease