The apolipoprotein E (APOE) ɛ4 allele and high levels of beta-amyloid (Aβ) are associated with episodic memory decline and risk for Alzheimer’s disease. However, there is debate about independent or interactive effects of ɛ4 on Aβ-related memory decline in healthy older adults. Healthy older adults with high Aβ burden (n = 84) enrolled in Australian Imaging, Biomarkers, and Lifestyle Study were included in this study. Cognition was measured using the computerized Cogstate Brief Battery at baseline, 18-, 36-, and 54-month follow-ups. Mini Mental State Examination and Clinical Dementia Rating scales were also administered at baseline and each follow-up timepoint. Relative to Aβ+ ɛ4 noncarriers (n = 36), Aβ+ ɛ4 carriers (n = 48) showed significantly faster decline on memory tasks, which was by convention, moderate in magnitude (d = 0.40-0.47). Aβ positivity coupled with APOE ɛ4 was associated with moderately increased decline in memory over a 54-month assessment period, suggesting that, in the preclinical stages of Alzheimer’s disease, the manifestation of memory decline in older adults with high Aβ is exacerbated by the presence of APOE ɛ4.