Within the Ca_v1 family of voltage-gated calcium channels, Ca_v1.2 and Ca_v1.3 channels are the predominant subtypes in the brain. Whereas specific functions for each subtype were described in the adult brain, their role in brain development is poorly understood. Here we assess the role of Ca_v1.3 subunits in the activity-dependent development of the auditory brainstem. We used Ca_v1.3-deficient (Ca_v1.3^−/−) mice because these mice lack cochlea-driven activity that deprives the auditory centers from peripheral input. We found a drastically reduced volume in all auditory brainstem centers (range 25–59%, total 35%), which was manifest before hearing onset. A reduction was not obvious outside the auditory system. The lateral superior olive (LSO) was strikingly malformed in Ca_v1.3^−/− mice and had fewer neurons (1/3 less). The remaining LSO neurons displayed normal dendritic trees and received functional glutamatergic input, yet they fired action potentials predominantly with a multiple pattern upon depolarization, in contrast to the single firing pattern prevalent in controls. The latter finding appears to be due to a reduction of dendrototoxin-sensitive potassium conductances, presumably mediated through the K_v1.2 subtype. Fura2 imaging provided evidence for functional Ca_v1.3 channels in the LSO of wild-type mice. Our results imply that Ca_v1.3 channels are indispensable for the development of the central auditory system. We propose that the unique LSO phenotype in Ca_v1.3^−/− mice, which hitherto was not described in other hereditary deafness models, is caused by the synergistic contribution of two factors: on-site loss of Ca_v1.3 channels in the neurons plus lack of peripheral input.