Inter-Coder Agreement in Qualitative Coding: Considerations for its Use

Sean N.  Halpin

doi:10.29333/ajqr/14887

Sean N. Halpin ¹ ^*

More Detail

AM J QUALITATIVE RES, Volume 8, Issue 3, pp. 23-43

https://doi.org/10.29333/ajqr/14887

OPEN ACCESS 1088 Views 1189 Downloads

Download Full Text (PDF)

Abstract

The historically quantitative-dominated field of health sciences has increasingly embraced qualitative methods. However, calls for quantitative measures of rigor, such as Inter-coder Agreement (ICA), remain. The aim of this manuscript is to demystify ICA and provide practical guidance. I begin by describing considerations while planning for ICA, including differences between various ICA tests (i.e., percent agreement, Holsti Method, Cohen’s kappa, Krippendorf’s alpha, and Gwet’s AC1 and AC2), setting the threshold of acceptability for your chosen test, deciding whether to use qualitative data analysis software, choosing the number of coders, selecting what data will be coded by more than one coder, developing a deductive codebook, creating a process for resolving coding disagreements, and establishing an audit trail for codebook changes. Next, I provide step-by-step guidance on an iterative process used for enacting ICA. Finally, I discuss the importance of reporting, emphasizing clarity, conciseness, completeness, and accuracy.

Keywords: Trustworthiness, rigor, inter-rater reliability, qualitative coding

References

Allsop, D. B., Chelladurai, J. M., Kimball, E. R., Marks, L. D., & Hendricks, J. J. (2022). Qualitative methods with Nvivo software: A practical guide for analyzing qualitative data. Psych, 4(2), 142–159. https://doi.org/10.3390/psych4020013
Armstrong, D., Gosling, A., Weinman, J., & Marteau, T. (1997). The place of inter-rater reliability in qualitative research: An empirical study. Sociology, 31(3), 597–606. https://doi.org/10.1177/0038038597031003015
Belur, J., Tompson, L., Thornton, A., & Simon, M. (2021). Interrater reliability in systematic review methodology: Exploring variation in coder decision-making. Sociological Methods and Research, 50(2), 837–865. https://doi.org/10.1177/0049124118799372
Bransford, J. D., Haynes, A. F., Stein, B. S., & Lin, X. (1998). The IDEAL workplace: Strategies for improving learning, problem solving, and creativity. NashvilleREAD.Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101. https://doi.org/10.1191/1478088706qp063oa
Braun, V., & Clarke, V. (2021a). One size fits all? What counts as quality practice in (reflexive) thematic analysis? Qualitative Research in Psychology, 18(3), 328–252. https://doi.org/10.1080/14780887.2020.1769238
Braun, V., & Clarke, V. (2021b). To saturate or not to saturate? Questioning data saturation as a useful concept for thematic analysis and sample-size rationales. Qualitative Research in Sport, Exercise and Health, 13(2), 201–216. https://doi.org/10.1080/2159676X.2019.1704846
Campbell, J. L., Quincy, C., Osserman, J., & Pedersen, O. K. (2013). Coding in-depth semi-structured interviews: Problems of unitization and intercoder reliability and agreement. Sociological Methods and Research, 42(3), 294–320. https://doi.org/10.1177/0049124113500475
Cascio, M. A., Lee, E., Vaudrin, N., & Freedman, D. A. (2019). A team-based approach to open coding: Considerations for creating intercoder consensus. Field Methods, 31(2), 116–130. https://doi.org/10.1177/1525822X19838237
Charmaz, K. (2006). Constructing grounded theory: A practical guide through qualitative analysis (Introducing Qualitative Methods series). SAGE Publications.
Cheung, K. K. C., & Tai, K. W. H. (2023). The use of intercoder reliability in qualitative interview data analysis in science education. Research in Science and Technological Education, 41(3), 1155–1175. https://doi.org/10.1080/02635143.2021.1993179
Clarke, S. N., Sushil, S., Dennis, K., Lee, U. S., Gomoll, A., & Gates, Z. (2023). Developing shared ways of seeing data: The perils and possibilities of achieving intercoder agreement. International Journal of Qualitative Methods, 22(1), 1–10. https://doi.org/10.1177/16094069231160973
Cofie, N., Braund, H., & Dalgarno, N. (2022). Eight ways to get a grip on intercoder reliability using qualitative-based measures. Canadian Medical Education Journal, 13(2), 73–76. https://doi.org/10.36834/cmej.72504
Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37–46. https://doi.org/10.1177/001316446002000104
Cohen, J. (1968). Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychological Bulletin, 70(4), 213–220. https://doi.org/10.1037/h0026256
Cole, R. (2023). Inter-rater reliability methods in qualitative case study research. Sociological Methods and Research.. Advance online publication. https://doi.org/10.1177/00491241231156971
Compton, D., Love, T. P., & Sell, J. (2012). Developing and assessing intercoder reliability in studies of group interaction. Sociological Methodology, 42(1), 348–364. https://doi.org/10.1177/0081175012444860
Cook, K. E. (2012). Reliability assessments in qualitative health promotion research. In Health Promotion International, 27(1), 90–101. https://doi.org/10.1093/heapro/dar027
Creswell, J. W. (2003). Research design: Qualitative quantitative and mixed methods approaches (2nd ed.). SAGE Publications.
Crotty, M., Shakespeare, W., & Henry, V. (2020). The foundations of social researchRCH: Meaning and perspective in the research process. SAGE Publications. https://doi.org/10.4324/9781003115700
Cypress, B. S. (2017). Rigor or reliability and validity in qualitative research: Perspectives, strategies, reconceptualization, and recommendations. Dimensions of Critical Care Nursing, 36(4), 253–263. https://doi.org/10.1097/DCC.0000000000000253
De Munck, V. C. (2000). Handbook of methods in cultural anthropology. American Anthropologist, 102(1), 183–186. https://doi.org/10.1525/aa.2000.102.1.183
Denzin, N. K. (2017). The Research Act: A Theoretical Introduction to Sociological Methods. In The Research Act: A Theoretical Introduction to Sociological Methods. https://doi.org/10.4324/9781315134543
Devotta, K., & Pedersen, C. (2015). Coding qualitative data: Working with a team of coders. Cultural Anthropology Methods, 10(2), 31–36 http://sru.crich.ca
Díaz, J., Pérez, J., Gallardo, C., & González-Prieto, Á. (2023). Applying inter-rater reliability and agreement in collaborative grounded theory studies in software engineering. Journal of Systems and Software, 195, Article 111520 https://doi.org/10.1016/j.jss.2022.111520
Feeley, N., & Gottlieb, L. N. (1998). Classification systems for health concerns, nursing strategies, and Client Outcomes: Nursing practice with families who have a child with a chronic illness. Canadian Journal of Nursing Research, 30(1), 45–60.
Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5), 378–382. https://doi.org/10.1037/h0031619
Francis, J. J., Johnston, M., Robertson, C., Glidewell, L., Entwistle, V., Eccles, M. P., & Grimshaw, J. M. (2010). What is an adequate sample size? Operationalising data saturation for theory-based interview studies. Psychology and Health, 25(10), 1229–1245. https://doi.org/10.1080/08870440903194015
Gabay, M. (2017). 21st century cures act. Hospital Pharmacy, 52(4), 264–265. https:// doi.org/10.1310/hpj5204-264.
Geertz, C. (2021). Thick description: Toward an interpretive theory of culture [1973]. In Readings for a History of Anthropological Theory, Sixth Edition.
González-Prieto, Á., Perez, J., Diaz, J., & López-Fernández, D. (2023). Reliability in software engineering qualitative research through Inter-Coder Agreement. Journal of Systems and Software, 202(1) 1–35. https://doi.org/10.1016/j.jss.2023.111707
Graneheim, U. H., & Lundman, B. (2004). Qualitative content analysis in nursing research: Concepts, procedures and measures to achieve trustworthiness. Nurse Education Today, 24(2), 105–112. https://doi.org/10.1016/j.nedt.2003.10.001
Guest, G., Bunce, A., & Johnson, L. (2006). How many interviews are enough? An experiment with data saturation and variability. Field Methods, 18(1), 59–82. https://doi.org/10.1177/1525822X05279903
Gwet, K. L. (2010). Handbook of inter-rater reliability: The definitive guide to measuring the extent of agreement among raters (4th ed.). Advanced Analytics LLC.
Halpin, S. N. (2023). Inter-rater Reliability in Qualitative Coding: Considerations for its Use. https://qualpage.com/2023/08/31/inter-rater-reliability-in-qualitative-coding-considerations-for-its-use/
Halpin, S. N., & Konomos, M. (2022). An iterative formative evaluation of medical education for multiple myeloma patients receiving autologous stem cell transplant. Journal of Cancer Education 37 (3), 779-787. https://link.springer.com/article/10.1007/s13187-020-01882-3
Halpin, S. N., Konomos, M., & Jowers, I. (2021). Interrupted identities: Autologous stem cell transplant in patients with multiple myeloma. Journal of Patient Experience 8. https://doi.org/10.1177/237437352199886
Halpin, S. N., Dillard, R. L., & Puentes, W. J. (2017). Socio-emotional adaptation theory: charting the emotional process of Alzheimer’s disease. The Gerontologist 57 (4), 696-706. https://doi.org/10.1093/geront/gnw046
Hoddy, E. T. (2019). Critical realism in empirical research: Employing techniques from grounded theory methodology. International Journal of Social Research Methodology, 22(1), 111–124. https://doi.org/10.1080/13645579.2018.1503400
Joffe, H., & Yardley, L. (2004). Content and thematic analysis. In D. F. Marks & L. Yardley (Eds.), Research methods for clinical and health psychology (pp. 56–68). SAGE Publications.
Johnson, J. L., Adkins, D., & Chauvin, S. (2020). A review of the quality indicators of rigor in qualitative research. In American Journal of Pharmaceutical Education, 84(1), Article 7120. https://doi.org/10.5688/ajpe7120
Johnson, R. B., & Onwuegbuzie, A. J. (2007). Toward a definition of mixed methods research. Journal of Mixed Methods Research, 1(2) 112–133. https://doi.org/10.1177/1558689806298224
Krippendorff, K. (2004). Reliability in content analysis: Some common misconceptions and recommendations. Human Communication Research, 30(3) 411–433. https://doi.org/10.1093/hcr/30.3.411
Krippendorff, K. (2022). Content analysis: An introduction to its methodology (4th ed.). SAGE Publications. https://doi.org/10.4135/9781071878781
Kurasaki, K. S. (2000). Field methods intercoder reliability for validating conclusions drawn from open-ended interview data. Field Methods, 12(1) 179–194. http://fmx.sagepub.comhttp://fmx.sagepub.com/cgi/content/abstract/12/3/179http://www.sagepublications.comhttp://fmx.sagepub.com/cgi/alertsEmailAlerts:http://fmx.sagepub.com/
Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174. https://doi.org/10.2307/2529310
Leung, L. (2015). Validity, reliability, and generalizability in qualitative research. Journal of Family Medicine and Primary Care, 4(3), 324–327. https://doi.org/10.4103/2249-4863.161306
Lincoln, Y. S., Guba, E. G., & Pilotta, J. J. (1985). Naturalistic inquiry. International Journal of Intercultural Relations, 9(4), 438–439. https://doi.org/10.1016/0147-1767(85)90062-8
Lombard, M., Snyder-Duch, J., & Bracken, C. C. (2002). Content analysis in mass communication: Assessment and reporting of intercoder reliability. In Human Communication Research, 28(4), 587–604. https://doi.org/10.1093/hcr/28.4.587
Long, H. A., French, D. P., & Brooks, J. M. (2020). Optimising the value of the critical appraisal skills programme (CASP) tool for quality appraisal in qualitative evidence synthesis. Research Methods in Medicine & Health Sciences, 1(1), 31–42. https://doi.org/10.1177/2632084320947559
MacQueen, K. M., McLellan-Lemal, E., Bartholow, K., & Milstein, B. (2008). Team-based codebook development: Structure, process, and agreement. Handbook for team-based qualitative research 119, 119-135.
Marzi, G., Balzano, M., & Marchiori, D. (2024). K-Alpha calculator: Krippendorff’s Alpha Calculator: A user-friendly tool for computing Krippendorff’s Alpha inter-rater reliability coefficient. MethodsX, 12(1), 1–10. https://doi.org/10.1016/j.mex.2023.102545
Mays, N., & Pope, C. (1995). Qualitative research: Rigour and qualitative research. BMJ, 311, 109–112. https://doi.org/10.1136/bmj.311.6997.109
McAlister, A. M., Lee, D. M., Ehlert, K. M., Kajfez, R. L., Faber, C. J., & Kennedy, M. S. (2017). Qualitative coding: An approach to assess inter-rater reliability. ASEE Annual Conference and Exposition, Conference Proceedings. https://doi.org/10.18260/1-2--28777
McDonald, N., Schoenebeck, S., & Forte, A. (2019). Reliability and inter-rater reliability in qualitative research: Norms and guidelines for CSCW and HCI practice. In Proceedings of the ACM on Human-Computer Interaction, 72(3), 1–23. https://doi.org/10.1145/3359174
Miles, M. B., Huberman, M. A., & Saldaña, J. (1994). Qualitative data analysis: A method sourcebook. SAGE Publications.
Moret, M., Reuzel, R., Van Der Wilt, G. J., & Grin, J. (2007). Validity and reliability of qualitative data analysis: Interobserver agreement in reconstructing interpretative frames. In Field Methods, 19(1), 24–39. https://doi.org/10.1177/1525822X06295630
Morse, J. (2020). The changing face of qualitative inquiry. International Journal of Qualitative Methods, 19, 1–7. https://doi.org/10.1177/1609406920909938
Morse, J. M. (1997). “Perfectly healthy, but dead”: The myth of inter-rater reliability. Qualitative Health Research, 7(4), 445–447. https://doi.org/10.1177/104973239700700401
Morse, J. M. (2015). Critical analysis of strategies for determining rigor in qualitative inquiry. Qualitative Health Research, 25(9), 1212–1222. https://doi.org/10.1177/1049732315588501
Muskens, G. J. (1980). Frames of meaning, are they measurable? A methodological critique of the content analysis of illustrated periodical magazines [Doctoral dissertation, Radbound University]. https://repository.ubn.ru.nl/handle/2066/147940
Nili, A., Tate, M., Barros, A., & Johnstone, D. (2020). An approach for selecting and using a method of inter-coder reliability in information management research. International Journal of Information Management, 54(1), 1–13. https://doi.org/10.1016/j.ijinfomgt.2020.102154
O’Brien, B. C., Harris, I. B., Beckman, T. J., Reed, D. A., & Cook, D. A. (2014). Standards for reporting qualitative research: A synthesis of recommendations. Academic Medicine, 89(9), 1245–1251. https://doi.org/10.1097/ACM.0000000000000388
O’Connor, C., & Joffe, H. (2020). Intercoder reliability in qualitative research: Debates and practical guidelines. International Journal of Qualitative Methods, 19, 1–13. https://doi.org/10.1177/1609406919899220
O’Sullivan, T. A., & Jefferson, C. G. (2020). A review of strategies for enhancing clarity and reader accessibility of qualitative research results. American Journal of Pharmaceutical Education, 84(1), Article 7124. https://doi.org/10.5688/ajpe7124
Parker, E. B., & Holsti, O. R. (1970). Content analysis for the social sciences and humanities. American Sociological Review, 35(2), 356–357. https://doi.org/10.2307/2093233
Prasanth, M. (2021). Publication manual of the American Psychological Association: The official guide to APA style. Kelpro Bulletin, 25(2), 90–92.
Roberts, K., Dowell, A., & Nie, J. B. (2019). Attempting rigour and replicability in thematic analysis of qualitative research data: A case study of codebook development. BMC Medical Research Methodology, 19(1), 1–8. https://doi.org/10.1186/s12874-019-0707-y
Roulston, K., & Halpin, S. N. (2022). Designing qualitative research using interview data. The SAGE Handbook of Qualitative Research Design. SAGE publications.
Rousson, V., Gasser, T., & Seifert, B. (2002). Assessing intrarater, interrater and test-retest reliability of continuous measurements. Statistics in Medicine, 21(22), 3431–3446. https://doi.org/10.1002/sim.1253
Saldaña, J. (2016). The coding manual for qualitative researchers (3rd ed.). SAGE Publications.
Scott, W. A. (1955). Reliability of content analysis: The case of nominal scale coding. Public Opinion Quarterly, 19(3), 321–325. https://doi.org/10.1086/266577
Sim, J., Saunders, B., Waterfield, J., & Kingstone, T. (2018). Can sample size in qualitative research be determined a priori? In International Journal of Social Research Methodology, 21(5), 619–634. https://doi.org/10.1080/13645579.2018.1454643
Sword, H. (2015). The writer’s diet: A guide to fit prose. The University of Chicago Press. https://doi.org/10.7208/chicago/9780226352039.001.0001
Sword, H. (2018). Air and light and time and space: how successful academics write. Harvard University Press.
Tong, A., Sainsbury, P., & Craig, J. (2007). Consolidated criteria for reporting qualitative research (COREQ): A 32-item checklist for interviews and focus groups. International Journal for Quality in Health Care, 19(6). 349–357. https://doi.org/10.1093/intqhc/mzm042
Watt, D. (2015). On becoming a qualitative researcher: The value of reflexivity. The Qualitative Report, 12(2), 82–101. https://doi.org/10.46743/2160-3715/2007.1645
Wongpakaran, N., Wongpakaran, T., Wedding, D., & Gwet, K. L. (2013). A comparison of Cohen’s Kappa and Gwet’s AC1 when calculating inter-rater reliability coefficients: A study conducted with personality disorder samples. BMC Medical Research Methodology, 13(1), 1–7. https://doi.org/10.1186/1471-2288-13-61
Woods, M., Paulus, T., Atkins, D. P., & Macklin, R. (2016). Advancing qualitative research using qualitative data analysis software (QDAS)? Reviewing potential versus practice in published studies using ATLAS.ti and NVivo, 1994–2013. Social Science Computer Review, 34(5), 597–617. https://doi.org/10.1177/0894439315596311
Xie, Q. (2013, November). Agree or disagree? A demonstration of an alternative statistic to Cohen’s Kappa for measuring the extent and reliability of agreement between observers. In Proceedings of the Federal Committee on Statistical Methodology Research Conference (Vol. 4, pp. 294–306). https://nces.ed.gov/FCSM/pdf/J4_Xie_2013FCSM.pdf
Yardley, L. (2000). Dilemmas in qualitative health research. Psychology and Health, 15(2), 215–228. https://doi.org/10.1080/08870440008400302
Zade, H., Drouhard, M., Chinh, B., Gan, L., & Aragon, C. (2018). Conceptualizing disagreement in qualitative coding. Conference on Human Factors in Computing Systems - Proceedings, 2018-April. https://doi.org/10.1145/3173574.3173733
Zamawe, F. C. (2015). The implication of using NVivo software in qualitative data analysis: Evidence-based reflections. Malawi Medical Journal, 27(1), 13–15. https://doi.org/10.4314/mmj.v27i1.4

Citation

Halpin, S. N. (2024). Inter-Coder Agreement in Qualitative Coding: Considerations for its Use. American Journal of Qualitative Research, 8(3), 23-43. https://doi.org/10.29333/ajqr/14887

Halpin, S. N. (2024). Inter-Coder Agreement in Qualitative Coding: Considerations for its Use. American Journal of Qualitative Research, 8(3), pp. 23-43. https://doi.org/10.29333/ajqr/14887

Halpin SN. Inter-Coder Agreement in Qualitative Coding: Considerations for its Use. AM J QUALITATIVE RES. 2024;8(3), 23-43. https://doi.org/10.29333/ajqr/14887

Halpin, Sean N.. "Inter-Coder Agreement in Qualitative Coding: Considerations for its Use". American Journal of Qualitative Research 2024 8 no. 3 (2024): 23-43. https://doi.org/10.29333/ajqr/14887

Halpin, Sean N. "Inter-Coder Agreement in Qualitative Coding: Considerations for its Use". American Journal of Qualitative Research, vol. 8, no. 3, 2024, pp. 23-43. https://doi.org/10.29333/ajqr/14887

Halpin SN. Inter-Coder Agreement in Qualitative Coding: Considerations for its Use. AM J QUALITATIVE RES. 2024;8(3):23-43. https://doi.org/10.29333/ajqr/14887