Refresher

Methods tutorial #28831(a), module (political) communication research methods

Marko Bachl

Freie Universität Berlin

20. 10. 2025

Hello again

Agenda

  1. Orga: Short presentations, workstation, orga questions

  2. Traditional content analysis

  3. How to evaluate a standardized content analysis?

  4. Questions

Orga

Orga: Short presentations

  • Current status: Blackboard

  • Overview:

    • Current work about LLM-based zero-shot classification
    • One paper presented by two to three participants
    • Short presentations (10 to 15 minutes)
    • Not a detailed description, but a summary for the class
    • Consider seminar context: No need for all of you to start with what LLM based zero-shot analysis is, why it is important, … directly start with core content of the paper
    • Post presentation in Blackboard

Orga: Workstation access

Questions?

Traditional content analysis

Traditional content analysis workflow

(Neuendorf, 2017)

Bag-of-Words Machine Learning

Abbildungen von Philipp K. Masur; Bachl & Scharkow (2024) for an overview

Transfer Learning

Abbildungen von Philipp K. Masur; Bachl & Scharkow (2024) for an overview

Zero-shot classification

Abbildungen von Philipp K. Masur; Bachl & Scharkow (2024) for an overview

LLM-based content analysis workflow

(Törnberg, 2024)

Questions?

How to evaluate a standardized content analysis?

Terms

  • Validity: Measurement validity: measures what it is supposed to measure; corresponds to some external truth
  • Reliability: Repeated measurements taken from the same data yield similar results (by different coders: intercoder r.; same coders: intracoder r.)
  • For quantitative evaluation: Comparison to what?
    • “Gold” or “preferred” standard: Confusion matrix or misclassification matrix
    • Among equally valid measurements: Coincidence matrix

Confusion matrix or misclassification matrix

Gold Standard Negative Gold Standard Positive
Observed Negative True Negatives (TN) False Negatives (FN)
Observed Positive False Positives (FP) True Positives (TP)

Common metrics for confusion matrix

Gold Standard Negative Gold Standard Positive
Observed Negative 12 (TN) 3 (FN)
Observed Positive 5 (FP) 7 (TP)
  • Accuracy: (TN + TP) / (TN + TP + FP + FN) = (12 + 7) / (12 + 7 + 5 + 3) = 0.70
  • Recall: TP / (TP + FN) = 7 / (7 + 3) = 0.70
  • Precision: TP / (TP + FP) = 7 / (7 + 5) = 0.58
  • F1-Score: 2 / (1/Precision + 1/Recall) = 2 / (1/0.58 + 1/0.70) = 0.63

Coincidence matrix

Class A Class B
Class A AA AB
Class B BA BB

Coincidence matrix

Class A Class B
Class A 13 3
Class B 3 11
  • Agreement: (AA + BB) / (AA + BB + BA + AB) =
    = (13 + 11) / (13 + 11 + 3 + 3) = 0.8
  • Krippendorff’s \(\alpha\): \(\frac{(n-1)\sum o_{cc} - \sum n_c(n_c-1)}{n(n-1) - \sum n_c(n_c-1)}\), with \(n\) all pairs, \(o_{cc}\) diagonal pairs, and \(n_c\) marginal pair sums; \(\alpha = 0.61\)
c1 c2 c3
A A A
A A A
B B B
B B B
A A A
B B B
A A A
A A B
B B A
A B B

Questions?

Thank you — see you next week

Marko Bachl

References

Bachl, M., & Scharkow, M. (2024). Computational text analysis. OSF. https://doi.org/10.31219/osf.io/3yhu8
Neuendorf, K. A. (2017). The content analysis guidebook. SAGE Publications, Inc. https://doi.org/dz7p
Törnberg, P. (2024). Best practices for text annotation with large language models. Sociologica, 18(2), 67–85. https://doi.org/g9vgm7