Inter-rater agreement and adjusted overall degree of distinguishability for ordered categories


Creative Commons License

YILMAZ A. E.

Chilean Journal of Statistics, cilt.13, sa.2, ss.165-186, 2022 (Scopus) identifier

  • Yayın Türü: Makale / Tam Makale
  • Cilt numarası: 13 Sayı: 2
  • Basım Tarihi: 2022
  • Doi Numarası: 10.32372/chjs.13-02-03
  • Dergi Adı: Chilean Journal of Statistics
  • Derginin Tarandığı İndeksler: Scopus
  • Sayfa Sayıları: ss.165-186
  • Anahtar Kelimeler: AC2 coefficient, Category distinguishability, Polynomial regression, Square contingency tables, Weighted kappa coefficient
  • Hacettepe Üniversitesi Adresli: Evet

Özet

In square contingency tables, weighted kappa and AC2 coefficients are used to summarize the degree of agreement between raters of an ordered square contingency table. In addition to investigate the agreement between raters, category distinguishability should be considered to check the reliability of the study. The overall degree of distinguishability is used for R×R tables. In some applications, the value of overall degree of distinguishability is calculated outside the defined range as negative values. Since overall degree of distinguishability is calculated by using all the category pairs, there occurs inflation on its value. In this study, adjusted overall degree of distinguishability is suggested to solve these two problems. Furthermore, interpretation of category distinguishability is outlined and benchmark scales for overall degree of distinguishability are developed. A simulation study is performed to compare the accuracy of the adjusted overall degree of distinguishability and the classical one. The tables to find the adjusted overall degrees of distinguishability for certain values of agreement coefficients are generated. The results are discussed over medical data sets.