Despite the prevalence of poverty worldwide, little is known about how early socioeconomic adversity affects auditory brain function. Socioeconomically disadvantaged children are underexposed to linguistically and cognitively stimulating environments and overexposed to environmental toxins, including noise pollution. This kind of sensory impoverishment, we theorize, has extensive repercussions on how the brain processes sound. To characterize how this impoverishment affects auditory brain function, we compared two groups of normal-hearing human adolescents who attended the same schools and who were matched in age, sex, and ethnicity, but differed in their maternal education level, a correlate of socioeconomic status (SES). In addition to lower literacy levels and cognitive abilities, adolescents from lower maternal education backgrounds were found to have noisier neural activity than their classmates, as reflected by greater activity in the absence of auditory stimulation. Additionally, in the lower maternal education group, the neural response to speech was more erratic over repeated stimulation, with lower fidelity to the input signal. These weaker, more variable, and noisier responses are suggestive of an inefficient auditory system. By studying SES within a neuroscientific framework, we have the potential to expand our understanding of how experience molds the brain, in addition to informing intervention research aimed at closing the achievement gap between high-SES and low-SES children.