Publication: Reliability generalization meta‑analysis: comparing different statistical methods
Authors
López‑Ibáñez, Carmen ; López Nicolás, Rubén ; Blázquez‑Rincón, Desirée M. ; Sánchez Meca, Julio
item.page.secondaryauthor
Facultad de Psicología y Logopedia
item.page.director
Publisher
Springer
publication.page.editor
publication.page.department
DOI
https://doi.org/10.1007/s12144-023-05604-y
item.page.type
info:eu-repo/semantics/article
Description
Abstract
Reliability generalization (RG) is a kind of meta-analysis that aims to characterize how reliability varies from one test application to the next. A wide variety of statistical methods have typically been applied in RG meta-analyses, regarding statistical model (ordinary least squares, fixed-effect, random effects, varying-coefficient models), weighting scheme (inverse variance, sample size, not weighting), and transformation method (raw, Fisher’s Z, Hakstian and Whalen’s and Bonett’s transformation) of reliability coefficients. This variety of methods compromise the comparability of RG meta-analyses results and their reproducibility. With the purpose of examining the influence of the different statistical methods applied, a methodological review was conducted on 138 published RG meta-analyses of psychological tests, amounting to a total of 4,350 internal consistency coefficients. Among all combinations of procedures that made theoretical sense, we compared thirteen strategies for calculating the average coefficient, eighteen for calculating the confidence intervals of the average coefficient and calculated the heterogeneity indices for the different transformations of the coefficients. Our findings showed that transformation methods of the reliability coefficients improved the normality adjustment of the coefficient distribution. Regarding the average reliability coefficient and the width of confidence intervals, clear differences among methods were found. The largest discrepancies were found between the different strategies for calculating confidence intervals. Our findings point towards the need for the meta-analyst to justify the statistical model assumed, as well as the transformation method of the reliability coefficients and the weighting scheme.
publication.page.subject
Citation
López-Ibáñez, C., López-Nicolás, R., Blázquez-Rincón, D.M. et al. Reliability generalization meta-analysis: comparing different statistical methods. Curr Psychol 43, 18275–18293 (2024). https://doi.org/10.1007/s12144-023-05604-y
item.page.embargo
Collections
Ir a Estadísticas
Este ítem está sujeto a una licencia Creative Commons. http://creativecommons.org/licenses/by/4.0/