E-cadherin (CDH1) is a putative tumor suppressor gene implicated in breast carcinogenesis. Yet, whether risk factors or survival differ by E-cadherin tumor expression is unclear. We evaluated E-cadherin tumor immunohistochemistry expression using tissue microarrays of 5,933 female invasive breast cancers from 12 studies from the Breast Cancer Consortium. H-scores were calculated and case-case odds ratios (OR) and 95% confidence intervals (CIs) were estimated using logistic regression. Survival analyses were performed using Cox regression models. All analyses were stratified by estrogen receptor (ER) status and histologic subtype. E-cadherin low cases (N = 1191, 20%) were more frequently of lobular histology, low grade, >2 cm, and HER2-negative. Loss of E-cadherin expression (score < 100) was associated with menopausal hormone use among ER-positive tumors (ever compared to never users, OR = 1.24, 95% CI = 0.97–1.59), which was stronger when we evaluated complete loss of E-cadherin (i.e. H-score = 0), OR = 1.57, 95% CI = 1.06–2.33. Breast cancer specific mortality was unrelated to E-cadherin expression in multivariable models. E-cadherin low expression is associated with lobular histology, tumor characteristics and menopausal hormone use, with no evidence of an association with breast cancer specific survival. These data support loss of E-cadherin expression as an important marker of tumor subtypes.