There has been significant interest in the effects of television on cognition in children, but much less research has been carried out into the effects in older adults. This study aimed to explore whether television viewing behaviours in adults aged 50 or over are associated with a decline in cognition. Using data from the English Longitudinal Study of Aging involving 3,662 adults aged 50+, we used multivariate linear regression models to explore longitudinal associations between baseline television watching (2008/2009) and cognition 6 years later (2014/2015) while controlling for demographic factors, socio-economic status, depression, physical health, health behaviours and a range of other sedentary behaviours. Watching television for more than 3.5 hours per day is associated with a dose-response decline in verbal memory over the following six years, independent of confounding variables. These results are found in particular amongst those with better cognition at baseline and are robust to a range of sensitivity analyses exploring reverse causality, differential non-response and stability of television viewing. Watching television is not longitudinally associated with changes in semantic fluency. Overall our results provide preliminary data to suggest that television viewing for more than 3.5 hours per day is related to cognitive decline.