While reading, our mind can wander to unrelated autobiographical information, creating a perceptually decoupled state detrimental to narrative comprehension. To understand how this mind-wandering state emerges, we asked whether retrieving autobiographical content necessitates functional disengagement from visual input. In Experiment 1, brain activity was recorded using functional magnetic resonance imaging (fMRI) in an experimental situation mimicking naturally occurring mind-wandering, allowing us to precisely delineate neural regions involved in memory and reading. Individuals read expository texts and ignored personally relevant autobiographical memories, as well as the opposite situation. Medial regions of the default mode network (DMN) were recruited during memory retrieval. In contrast, left temporal and lateral prefrontal regions of the DMN, as well as ventral visual cortex, were recruited when reading for comprehension. Experiment two used functional connectivity both at rest and during tasks to establish that (i) DMN regions linked to memory are more functionally decoupled from regions of ventral visual cortex than regions in the same network engaged when reading; and (ii) individuals with more self-generated mental contents and poorer comprehension, while reading in the lab, showed more decoupling between visually connected DMN sites important for reading and primary visual cortex. A similar pattern of connectivity was found in Experiment 1, with greater coupling between this DMN site and visual cortex when participants reported greater focus on reading in the face of conflict from autobiographical memory cues; moreover, the retrieval of personally relevant memories increased the decoupling of these sites. These converging data suggest we lose track of the narrative when our minds wander because generating autobiographical mental content relies on cortical regions within the DMN which are functionally decoupled from ventral visual regions engaged during reading.