Learning to read proceeds smoothly for most children, yet others struggle to translate verbal language into its written form. Poor readers often have a host of auditory, linguistic, and attention deficits, including abnormal neural representation of speech and inconsistent performance on psychoacoustic tasks. We hypothesize that this constellation of deficits associated with reading disorders arises from the human auditory system failing to respond to sound in a consistent manner, and that this inconsistency impinges upon the ability to relate phonology and orthography during reading. In support of this hypothesis, we show that poor readers have significantly more variable auditory brainstem responses to speech than do good readers, independent of resting neurophysiological noise levels. Thus, neural variability may be an underlying biological contributor to well established behavioral and neural deficits found in poor readers.