Assessing the Stability of Cognitive and Attentional Computerized Tests

  1. Lira Delcore, Aranza 3
  2. Tapia Iglesias, José Luis 12
  3. Duñabeitia, Jon Andoni 45
  1. 1 Universidad Antonio de Nebrija (2020)
  2. 2 ERI Lectura, Universitat de València
  3. 3 Universidad Nebrija
    info
    Universidad Nebrija

    Madrid, España

    ROR https://ror.org/03tzyrt94

    Geographic location of the organization Universidad Nebrija
  4. 4 Universidad Antonio de Nebrija (2018)
  5. 5 Basque Center on Cognition, Brain and Language
    info
    Basque Center on Cognition, Brain and Language

    San Sebastián, España

    ROR https://ror.org/01a28zg77

    Geographic location of the organization Basque Center on Cognition, Brain and Language
Show affiliations +
Journal:
Psychologica

ISSN: 1647-8606,  0871-4657

Year of publication: 2025

Volume: 68

Pages: e068007

Type: Article

DOI: 10.14195/1647-8606_68_7 GOOGLE SCHOLAR lock_openOpen access editor

More publications in: Psychologica

Abstract

This study evaluates the test-retest reliability of four computerized cognitive tests designed to measure selective attention, sustained attention, visual search, and visual scanning. The sample consisted of 39 young adults who completed computerized versions of the Trail Making Test, Visual Search Test, Sustained Attention to Response Test, and Selec-tive Attention Test developed by CogniFit Inc. across three sessions. The study included two comparisons: test-retest on different days and immediate test-retest on the same day. Results indicated consistent outcomes across sessions for all tests, except for certain variables where immediate repetition likely induced learning effects and for variables not crucial for the task. These findings support the reliability of the tests over short intervals, consistent with prior research on cognitive assessments. The study highlights their utility as reliable and accessible tools for broad application. Future research should examine longer intervals and diverse populations to further validate their reliability.

Bibliographic References

  • Asensio, D., & Duñabeitia, J. A. (2023). The necessary, albeit belated, transition to computerized cognitive assessment. Frontiers in Psychology, 14, 1160554. https://doi.org/10.3389/fpsyg.2023.1160554
  • Aul, C., Brau, J. M., Sugarman, A., DeGutis, J. M., Germine, L. T., Esterman, M., McGlinchey, R. E., & Fortenbaugh, F. C. (2023). The functional relevance of visuospatial processing speed across he lifespan. Cognitive Research: Principles and Implications, 8(1), 51. https://doi.org/10.1186/s41235-023-00504-y
  • Bird, C. M., Papadopoulou, K., Ricciardelli, P., Rossor, M. N., & Cipolotti, L. (2004). Monitoring cognitive changes: Psychometric properties of six cognitive tests. British Journal of Clinical Psychology, 43(2), 197–210. https://doi.org/10.1348/014466504323088051
  • Bolarinwa, O. (2015). Principles and methods of validity and reliability testing of questionnaires used in social and health science researches. Nigerian Postgraduate Medical Journal, 22(4), 195. https://doi.org/10.4103/1117-1936.173959
  • Bowie, C. R., & Harvey, P. D. (2006). Administration and interpretation of the Trail Making Test. Nature Protocols, 1(5), 2277–2281. https://doi.org/10.1038/nprot.2006.390
  • Brickenkamp, R. (1962). Test d2: Attention stress test.Chen, K.-W., Lin, G.-H., Chen, N.-C., Wang, J.-K., & Hsieh, C.-L. (2020). Practice Effects and Test–Retest Reliability of the Continuous Performance Test, Identical Pairs Version in Patients with Schizophrenia over Four Serial Assessments. Archives of Clinical Neuropsychology, 35(5), 545–552. https://doi.org/10.1093/arclin/acaa004
  • Conesa, P. J., & Duñabeitia, J. A. (2021). Effects of computer-based training on children’s executive functions and academic achievement. The Journal of Educational Research, 114(6), 562–571. https://doi.org/10.1080/00220671.2021.1998881
  • Correa-Rojas, J. (2021). Coeficiente de Correlación Intraclase: Aplicaciones para estimar la estabili-dad temporal de un instrumento de medida. Ciencias Psicológicas, 15(2), e-2318. https://doi.org/10.22235/cp.v15i2.2318
  • Cronbach, L. J. (1947). Test “Reliability”: Its Meaning and Determination. Psychometrika, 12(1), 1–16. https://doi.org/10.1007/BF02289289
  • Cullen, B., O’Neill, B., Evans, J. J., Coen, R. F., & Lawlor, B. A. (2007). A review of screening tests for cognitive impairment. Journal of Neurology, Neurosurgery & Psychiatry, 78(8), 790 –799. https://doi.org/10.1136/jnnp.2006.095414
  • da Cunha, D., Rocha, C., Leite, D. C., Schoenfeld, B. J., & Prestes, J. (2019). New insights for statistical analysis of blood pressure response to resistance training in elderly hypertensive women. Journal of Physical Education, 30, e3025. https://doi.org/10.4025/jphyseduc.v30i1.3025
  • Erez, A. B.-H., Katz, N., Ring, H., & Soroker, N. (2009). Assessment of spatial neglect using compu-terised feature and conjunction visual search tasks. Neuropsychological Rehabilitation, 19(5), 677– 695. https://doi.org/10.1080/09602010802711160
  • Feinkohl, I., Borchers, F., Burkhardt, S., Krampe, H., Kraft, A., Speidel, S., Kant, I. M. J., Van Mon-tfort, S. J. T., Aarts, E., Kruppa, J., Slooter, A., Winterer, G., Pischon, T., & Spies, C. (2020). Stability of neuropsychological test performance in older adults serving as normative controls for a study on postoperative cognitive dysfunction. BMC Research Notes, 13(1), 55. https://doi.org/10.1186/s13104-020-4919-3
  • Flanagan, D. P., & McDonough, E. M. (2022). Contemporary Intellectual Assessment: Theories, Tests, and Issues. Guilford Publications.
  • Flanagan, Dawn. P., Alfonso, V. C., Ortiz, S. O., & Dynda, A. M. (2013). Cognitive Assessment: Pro-gress in Psychometric Theories of Intelligence, the Structure of Cognitive Ability Tests, and Interpretive Approaches to Cognitive Test Performance. In D. H. Saklofske, C. R. Reynolds, V. Schwean (Eds.), The Oxford handbook of child psychological assessment (pp. 239–285). Oxford University Press.
  • Lachman, M. E., Agrigoroaei, S., Tun, P. A., & Weaver, S. L. (2014). Monitoring Cognitive Functioning: Psychometric Properties of the Brief Test of Adult Cognition by Telephone. Assessment, 21(4), 404–417. https://doi.org/10.1177/1073191113508807
  • Lee, P., Li, P.-C., Liu, C.-H., Lin, H.-Y., Huang, C.-Y., & Hsieh, C.-L. (2021). Practice Effects, Test–Retest Reliability, and Minimal Detectable Change of the Ruff 2 and 7 Selective Attention Test in Patients with Schizophrenia. International Journal of Environmental Research and Public Health, 18(18), 9440. https://doi.org/10.3390/ijerph18189440
  • Lee, P., Lu, W.-S., Liu, C.-H., Lin, H.-Y., & Hsieh, C.-L. (2018). Test–Retest Reliability and Minimal Detectable Change of the D2 Test of Attention in Patients with Schizophrenia. Archives of Clinical Neuropsychology, 33(8), 1060–1068. https://doi.org/10.1093/arclin/acx123
  • Lee, S.-C., Chien, T.-H., Chu, C.-P., Lee, Y., & Chiu, E.-C. (2023). Practice effect and test–retest relia-bility of the Wechsler Memory Scale-Fourth Edition in people with dementia. BMC Geriatrics, 23(1), 209. https://doi.org/10.1186/s12877-023-03913-2
  • MacKeben, M., & Fletcher, D. C. (2011). Target Search and Identification Performance in Low Vision Patients. Investigative Opthalmology & Visual Science, 52(10), 7603. https://doi.org/10.1167/iovs.10-6728
  • Marques-Costa, C., Almiro, P. A., & Simões, M. R. (2018). Computerized Cognitive Tests (CCT) in elderly: A psychometric review. European Review of Applied Psychology, 68(2), 61–68. https://doi.org/10.1016/j.erap.2018.04.002
  • Meeker, W. Q., Escobar, L. A., & Pascual, F. G. (2021). Statistical Methods for Reliability Data. John Wiley & Sons.
  • Morgan, C., Honan, I., Allsop, A., Novak, I., & Badawi, N. (2019). Psychometric Properties of Asses-sments of Cognition in Infants With Cerebral Palsy or Motor Impairment: A Systematic Review. Journal of Pediatric Psychology, 44(2), 238–252. https://doi.org/10.1093/jpepsy/jsy068
  • Murray, N. P., & Janelle, C. M. (2003). Anxiety and Performance: A Visual Search Examination of the Processing Efficiency Theory. Journal of Sport and Exercise Psychology, 25(2), 171–187. https://doi.org/10.1123/jsep.25.2.171
  • Noble, S., Scheinost, D., & Constable, R. T. (2021). A guide to the measurement and interpretation of f MRI test-retest reliability. Current Opinion in Behavioral Sciences, 40, 27–32. https://doi.org/10.1016/j.cobeha.2020.12.012
  • Paiva, C. E., Barroso, E. M., Carneseca, E. C., De Pádua Souza, C., Dos Santos, F. T., Mendoza López, R. V., & Ribeiro Paiva, S. B. (2014). A critical analysis of test-retest reliability in instrumentvalidation studies of cancer patients under palliative care: A systematic review. BMC Medical Research Methodology, 14(1), 8. https://doi.org/10.1186/1471-2288-14-8
  • Pautex, S., Berger, A., Chatelain, C., Herrmann, F., & Zulian, G. B. (2003). Symptom assessment in elderly cancer patients receiving palliative care. Critical Reviews in Oncology/Hematology, 47(3), 281–286. https://doi.org/10.1016/S1040-8428(03)00043-X
  • Portney, L. G. (2020). Foundations of Clinical Research: Applications to Evidence-Based Practice.
  • F.A. Davis.Proust-Lima, C., Amieva, H., Dartigues, J.-F., & Jacqmin-Gadda, H. (2006). Sensitivity of Four Psycho-metric Tests to Measure Cognitive Changes in Brain Aging-Population-based Studies. American Journal of Epidemiology, 165(3), 344–350. https://doi.org/10.1093/aje/kwk017
  • Reitan, R. M. (1958). Validity of the Trail Making Test as an Indicator of Organic Brain Damage. Perceptual and Motor Skills, 8(3), 271–276. https://doi.org/10.2466/pms.1958.8.3.271
  • Robertson, I. H., Manly, T., Andrade, J., Baddeley, B. T., & Yiend, J. (1997). Sustained Attention to Response Task [Dataset].APA PsycTests. https://doi.org/10.1037/t28308-000
  • Rollè, L., Gullotta, G., Trombetta, T., Curti, L., Gerino, E., Brustia, P., & Caldarera, A. M. (2019). Father Involvement and Cognitive Development in Early and Middle Childhood: A Systematic Rev iew. Frontiers in Psychology, 10, 2405. https://doi.org/10.3389/fpsyg.2019.02405
  • Sánchez-Vincitore, L. V., Cubilla-Bonnetier, D., Marte-Santana, H., & Duñabeitia, J. A. (2023). Cog-nitive decline monitoring through a web-based application. Frontiers in Aging Neuroscience, 15, 1212496. https://doi.org/10.3389/fnagi.2023.1212496
  • Steinborn, M. B., Langner, R., Flehmig, H. C., & Huestegge, L. (2018). Methodology of performance scoring in the d2 sustained-attention test: Cumulative-reliability functions and practical guidelines. Psychological Assessment, 30(3), 339–357. https://doi.org/10.1037/pas0000482
  • Sürücü, L., & Maslakçi, A. (2020). Validity and reliability in quantitative research. Business & Manage-ment Studies: An International Journal, 8(3), 2694–2726. https://doi.org/10.15295/bmij.v8i3.1540
  • Tikhomirova, T., Malykh, A., & Malykh, S. (2020). Predicting Academic Achievement with Cognitive Abilities: Cross-Sectional Study across School Education. Behavioral Sciences, 10(10), 158 . https://doi.org/10.3390/bs10100158
  • Treisman, A. M., & Gelade, G. (1980). A feature-integration theory of attention. Cognitive Psychology, 12(1), 97–136. https://doi.org/10.1016/0010-0285(80)90005-5
  • Van Patten, R., Iverson, G. L., Muzeau, M. A., & VanRavenhorst-Bell, H. A. (2021). Test–Retest Relia-bility and Reliable Change Estimates for Four Mobile Cognitive Tests Administered Virtually in Community-Dwelling Adults. Frontiers in Psychology, 12, 734947. https://doi.org/10.3389/fpsyg.2021.734947
  • VandenBos, G. R. (Ed.). (2015). APA Dictionary of Psychology, 2nd Edition. American Psychological Association. https://doi.org/10.1037/14646-000
  • Wagner, S., Helmreich, I., Dahmen, N., Lieb, K., & Tadic, A. (2011). Reliability of Three Alternate Forms of the Trail Making Tests A and B. Archives of Clinical Neuropsychology, 26(4), 314 –321. https://doi.org/10.1093/arclin/acr024
  • Weir, J. P. (2005). Quantifying Test-Retest Reliability Using the Intraclass Correlation Coefficient and the SEM. The Journal of Strength and Conditioning Research, 19(1), 231. https://doi.org/10.1519/15184.1
  • Woods, D. L., Wyma, J. M., Herron, T. J., & Yund, E. W. (2015). The Effects of Aging, Malingering, and Traumatic Brain Injury on Computerized Trail-Making Test Performance. PLOS ONE, 10(6), e0124345. https://doi.org/10.1371/journal.pone.0124345
  • Zygouris, S., & Tsolaki, M. (2015). Computerized Cognitive Testing for Older Adults: A Review. American Journal of Alzheimer’s Disease & Other Dementias®, 30(1), 13–28. https://doi.org /10.1177/1533317514522852