We investigated the extent to which decline in memory and working memory in beta-amyloid (Aβ) positive non-demented individuals was related to hippocampal atrophy and Aβ accumulation over 36 months. Cognitively normal older adults (CN) (n = 178) and adults with mild cognitive impairment (MCI) (n = 49) underwent positron emission tomography neuroimaging, magnetic resonance imaging, and cognitive assessments at baseline, 18- and 36-months. Relative to Aβ- CNs, Aβ+ CNs and Aβ+ MCIs showed greater rates of cognitive decline, Aβ accumulation, and hippocampal atrophy. Analysis of interrelationships between these Alzheimer’s disease markers in Aβ+ CNs and MCIs indicated that rate of Aβ accumulation was associated with rate of hippocampal atrophy (β = -0.05, p = .037), which was in turn associated independently with rate of decline in memory (β = -0.03, p = .032). This suggests that Aβ accumulation precedes any neurodegeneration or clinical symptoms, and that the relationship between Aβ and cognitive decline is mediated by hippocampal atrophy.