Browsing by Subject "Reliability coefficient"

Now showing 1 - 2 of 2
  • Thumbnail Image
    Publication
    Restricted
    Improving the reporting quality of reliability generalization meta-analyses: The REGEMA checklist
    (Wiley Online Library, 2021-03-20) Sánchez-Meca, Julio; Marín-Martínez, Fulgencio; López-López, José Antonio; Núñez-Núñez, Rosa Maria; Rubio-Aparicio, María; López-García, Juan José; López-Pina, José Antonio; Blázquez-Rincón, Desirée Mª; López-Ibáñez, Carmen; López-Nicolás, Rubén; Psicología Básica y Metodología
    Reliability generalization (RG) is a meta-analytic approach that aims to characterize how reliability estimates from the same test vary across different applications of the instrument. With this purpose RG meta-analyses typically focus on a particular test and intend to obtain an overall reliability of test scores and to investigate how the composition and variability of the samples affect reliability. Although several guidelines have been proposed in the meta-analytic literature to help authors improve the reporting quality of meta-analyses, none of them were devised for RG meta-analyses. The purpose of this investigation was to develop REGEMA (REliability GEneralization Meta-Analysis), a 30-item checklist (plus a flow chart) adapted to the specific issues that the reporting of an RG meta-analysis must take into account. Based on previous checklists and guidelines proposed in the meta-analytic arena, a first version was elaborated by applying the nominal group methodology. The resulting instrument was submitted to a list of independent meta-analysis experts and, after discussion, the final version of the REGEMA checklist was reached. In a pilot study, four pairs of coders applied REGEMA to a random sample of 40 RG meta-analyses in Psychology, and results showed satisfactory inter-coder reliability. REGEMA can be used by: (a) meta-analysts conducting or reporting an RG meta-analysis and aiming to improve its reporting quality; (b) consumers of RG meta-analyses who want to make informed critical appraisals of their reporting quality, and (c) reviewers and editors of journals who are considering submissions where an RG meta-analysis was reported for potential publication.
  • Thumbnail Image
    Publication
    Open Access
    Reliability generalization meta‑analysis: comparing different statistical methods
    (Springer, 2024-01-22) López‑Ibáñez, Carmen; López Nicolás, Rubén; Blázquez‑Rincón, Desirée M.; Sánchez Meca, Julio; Psicología Básica y Metodología; Facultad de Psicología y Logopedia
    Reliability generalization (RG) is a kind of meta-analysis that aims to characterize how reliability varies from one test application to the next. A wide variety of statistical methods have typically been applied in RG meta-analyses, regarding statistical model (ordinary least squares, fixed-effect, random effects, varying-coefficient models), weighting scheme (inverse variance, sample size, not weighting), and transformation method (raw, Fisher’s Z, Hakstian and Whalen’s and Bonett’s transformation) of reliability coefficients. This variety of methods compromise the comparability of RG meta-analyses results and their reproducibility. With the purpose of examining the influence of the different statistical methods applied, a methodological review was conducted on 138 published RG meta-analyses of psychological tests, amounting to a total of 4,350 internal consistency coefficients. Among all combinations of procedures that made theoretical sense, we compared thirteen strategies for calculating the average coefficient, eighteen for calculating the confidence intervals of the average coefficient and calculated the heterogeneity indices for the different transformations of the coefficients. Our findings showed that transformation methods of the reliability coefficients improved the normality adjustment of the coefficient distribution. Regarding the average reliability coefficient and the width of confidence intervals, clear differences among methods were found. The largest discrepancies were found between the different strategies for calculating confidence intervals. Our findings point towards the need for the meta-analyst to justify the statistical model assumed, as well as the transformation method of the reliability coefficients and the weighting scheme.