AI-powered content analysis

Methoden der empirischen Kommunikations- und Medienforschung

Marko Bachl

Freie Universität Berlin

AI-powered content analysis

Siehe auch die gleichnamige Methodenübung

Agenda

  1. Grundidee: Generative Sprachmodelle als Codierer
  2. Live-Beispiel: Klassifikation von Inzivilität
    1. Klassifikation
    2. Evaluation
  3. Fazit

Grundidee: Generative Sprachmodelle als Codierer

Manuelle, standardisierte Inhaltsanalyse

(Rössler, 2017, S. 38)

Standardisierte Inhaltsanalyse mit AI

(Kravets-Meinke et al., 2025, S. 7)

Fragen?

Live-Beispiel: Klassifikation von Inzivilität

Erkennen von Inzivilität in Social-Media-Kommentaren (Stoll et al., 2023)

Kommentar mit mindestens einer der folgenden Eigenschaften gilt als inzivil:

  • Vulgäre, unangemessene Sprache, Fluchen
  • Beleidigung, Profanität
  • Entmenschlichung
  • Sarkasmus, Spott, Zynismus
  • Negative Stereotype
  • Diskriminierung
  • Androhung von Gewalt
  • Verweigerung von Rechten
  • Vorwurf der Lüge
  • Erniedrigung, fehlender Respekt, Abwertung

Untersuchungsmaterial

Wir brauchen Kommentare zum Testen:

  • Klar inzivile Kommentare
  • Klar nicht inzivile Kommentare
  • Mehrdeutige Kommentare:
    • Nicht inzivile Kommentare, die in einem Test fälschlicherweise als inzivil klassifiziert werden könnten
    • Inzivile Kommentare, die in einem Test fälschlicherweise als nicht inzivil klassifiziert werden könnten

https://www.soscisurvey.de/komsam/

Untersuchungsmaterial

kommentar true_class
This is (g)assss{EM_SMILING_FACE_WITH_HEARTEYES}{EM_SMILING_FACE_WITH_HEARTEYES} inzivil
bones are for dogs inzivil
Wie kann man so dumme Ansichten online verbreiten ? Absolute Frechheit wie heutzutage mit dem Wort Forschung umgegangen wird!! inzivil
Was für ein Schwachsinn! inzivil
Ich bin mir sicher, dass die Leute hier alle Tassen im Schrank haben! inzivil
Du bist toll, so wie du bist {EM_RED_HEART}️ inzivil
Du bist dumm inzivil
Linksveriffte Öko-Trulla inzivil
Das war schön inzivil
Clown inzivil
du bist sch***! inzivil
du bist hässlich inzivil
Merz leck Eier inzivil
HS inzivil
Du Hurensohn inzivil
Fuck inzivil
Merz, leck Eier inzivil
Das ist doch Schwachsinn inzivil
Du Fotze inzivil
Oiiiai oiiiai inzivil
Hure! inzivil
Ich finde heute siehst du komisch aus. inzivil
Die Weisheit verfolgt dich, aber du bist schneller! inzivil
Wtf,,, was denkt der wer er ist??? Hasse den so… hoffe er stirbt inzivil
Ganz schön hässlich inzivil
Fake news! inzivil
Ich weiß wo du wohnst inzivil
Ich hau dir auf die Schnauze inzivil
Mutig, wie viel Meinung man mit so wenig Ahnung haben kann. inzivil
Das ist dumm inzivil
Arschloch inzivil
Ja natürlich stimmt das /s inzivil
NA inzivil
Hoffe er wird gekirkt inzivil
Du bist dumm inzivil
Lügner! inzivil
Fotze inzivil
fette Sau inzivil
Bring dich um! inzivil
Ich finde du siehst einfach seltsam aus. inzivil
Interessant, wie überzeugt du davon bist. Ich wünschte, ich hätte auch so viel Selbstvertrauen bei so wenig Grundlage inzivil
I wish you get raped inzivil
Wie schön wieder in der Uni zu sein inzivil
Du F0tze inzivil
Das hast du ja richtig toll gemacht… inzivil
Dumme Nuss inzivil
Nutte inzivil
nuttig inzivil
Willste dich boxen? inzivil
kys inzivil
Wow, du bist soo mutig dich ungeschminkt zu posten inzivil
was ein Scheiß inzivil
Hässlich inzivil
Halt die Fresse inzivil
Ich stimme dir da gar nicht zu. Weiß nicht wie du drauf kommst aber bist ja auch nicht ganz richtig in der Birne inzivil
Männer sollten keine Rechte haben inzivil
Nazi-Schweine inzivil
AfDler abschieben inzivil
Dem stimme ich zu. nicht inzivil
Ich finde Titten geil nicht inzivil
bitch, das ist sooo crazyy nicht inzivil
Also das ist jetzt schon ein bisschen unfair. Sie hat auch noch andere Facetten, z.B. ist sie auch noch grob inkompetent :) nicht inzivil
Was für ein schönes Foto! nicht inzivil
Hab eine schöne Woche! nicht inzivil
Öffentlichkeit nicht inzivil
Ist das KI? nicht inzivil
Woww, geiler scheiß! nicht inzivil
Das ist falsch. nicht inzivil
Was ein Motherfucker nicht inzivil
Du bist sau cool nicht inzivil
Das Rezept mache ich auf jeden Fall nach!!! Danke für die Inspo! nicht inzivil
Danke und das so wichtig! nicht inzivil
Lars Klingbeil ist ein Klassenverräter nicht inzivil
Bayern gewinnt…Überraschung des Jahres. nicht inzivil
Ekelhaft nicht inzivil
Ich möchte Sommer nicht inzivil
Scheiße, das sieht gut aus! nicht inzivil
Ich sehe Sie als einen inkompetenten Politiker an nicht inzivil
fotzig nicht inzivil
Hab Bock auf Eierkuchen nicht inzivil
Du bist doof nicht inzivil
Scheiße wie cool nicht inzivil
love this! nicht inzivil
Du bist schlagfertig. nicht inzivil
Du Sau! nicht inzivil
fucking gute idee nicht inzivil
Ohh danke für die Hilfe! nicht inzivil
Ohne Worte, wo soll das noch hin führen? nicht inzivil
Ich finde, die CDU versagt auf ganzer Linie! Peinlich! nicht inzivil
NA nicht inzivil
Spannender Beitrag nicht inzivil
Boss Bitch nicht inzivil
Das fickt ordentlich was ihr da macht :) nicht inzivil
Welch ein blumiger Sonnenuntergang! nicht inzivil
Freue mich für dich :) nicht inzivil
das ist so unlustig und komisch nicht inzivil
krasser Scheiß nicht inzivil
Yasss slayyy queeennn nicht inzivil
richtig guter punkt nicht inzivil
Schön, dass ihr Spaß im Urli habt nicht inzivil
I don’t know, 6-7 nicht inzivil
Schöne Berge nicht inzivil

Codieranweisung – System Prompt

Structured prompts: An annotation prompt should contain the following elements: context, question, and constraints. The context gives a brief introduction to orient the model with any necessary background information. It can be split into role (e.g. expert annotator) and context (e.g. conspiracy theories). The question guides the response, defines the coding task. The constraint specifies the output format. (Törnberg, 2024, S. 74)

Codieranweisung – System Prompt

Du bist Forschungsassistent:in in einem kommunikationswissenschaftlichen Forschungsprojekt.

Deine Aufgabe ist es, Inzivilität in Social-Media-Kommentaren zu identifizieren. Dir wird ein Kommentar vorgelegt. Entscheide, ob der Kommentar inzivil oder nicht inzivil ist.

Wir definieren Inzivilität als die Verletzung kommunikativer Normen. Kommentare, die eines oder mehrere der folgenden Merkmale enthalten, werden in der Regel als inzivil betrachtet:

- Vulgäre, unangemessene Sprache, Fluchen
- Beleidigung, Profanität
- Entmenschlichung 
- Sarkasmus, Spott, Zynismus
- Negative Stereotype
- Diskriminierung
- Androhung von Gewalt
- Verweigerung von Rechten
- Vorwurf der Lüge
- Erniedrigung, fehlender Respekt, Abwertung

Du musst deine Antwort in gültigem JSON ausgeben. Das JSON muss exakt diesem Format entsprechen: {"obs_class": "inzivil"} oder {"obs_class": "nicht inzivil"}. Füge absolut keinen anderen Text hinzu.

Hier ist der Social-Media-Kommentar:

Klassifikation mit einem Chatbot

https://chat-ai.academiccloud.de/

  • Dieses Vorgehen wollen wir automatisieren.

Kontakt zu LLM aufnehmen

denbi_key <- readLines(here::here("data/denbi_key.txt")) # Diesen Schlüssel niemals teilen
request_llm <- request(base_url = "https://denbi-llm-api.bihealth.org/v1/chat/completions") |>
  req_auth_bearer_token(denbi_key) |>
  req_retry(
    max_tries = 3,
    is_transient = \(resp) resp_status(resp) %in% c(429, 500, 502, 503)
  )
mdl <- "gpt-oss-120b"

Kontakt zu LLM aufnehmen

request_llm |>
  req_body_json(list(
    model = mdl,
    messages = list(
      list(role = "system", content = "You talk like a pirate."),
      list(role = "user", content = "Tell me a joke.")
    ),
    temperature = 1,
    reasoning = list(effort = "medium"),
    max_completion_tokens = 5000
  )) |>
  req_perform() |>
  resp_body_string() |>
  prettify()

Kontakt zu LLM aufnehmen

{
    "id": "e8e20347ccd74942ab7ca46a797fc8e0",
    "created": 1778484539,
    "model": "gpt-oss-120b",
    "object": "chat.completion",
    "choices": [
        {
            "finish_reason": "stop",
            "index": 0,
            "message": {
                "content": "Arrr, ye be wantin’ a laugh, do ye? Here’s a jolly pirate jest fer ye:\n\nWhy did the pirate go to school?\n\nBecause he wanted to improve his **arrrrr‑ticulation** and learn how to read the **sea‑crets** of the seven seas! 🏴‍☠️\n\nShiver me timbers, that one be a real treasure!",
                "role": "assistant",
                "reasoning_content": "We need to respond as ChatGPT, but the developer instruction says \"You talk like a pirate.\" So we must adopt a pirate speaking style, using nautical slang etc. The user wants a joke. So we should give a pirate-themed joke, in pirate speak. Must ensure compliance with policies: no disallowed content. It's fine. We'll do it. Use pirate talk: \"Arrr\", \"matey\", \"shiver me timbers\". Use the joke.",
                "provider_specific_fields": {
                    "refusal": null
                }
            },
            "provider_specific_fields": {
                "matched_stop": 200002
            }
        }
    ],
    "usage": {
        "completion_tokens": 187,
        "prompt_tokens": 85,
        "total_tokens": 272,
        "completion_tokens_details": {
            "reasoning_tokens": 1,
            "text_tokens": 186
        }
    },
    "metadata": {
        "weight_version": "default"
    }
}

Einen Kommentar codieren

request_llm |>
  req_body_json(list(
    model = mdl,
    messages = list(
      list(role = "system", content = system_prompt),
      list(role = "user", content = d$kommentar[1])
    ),
    temperature = 1,
    reasoning = list(effort = "medium"),
    max_completion_tokens = 5000
  )) |>
  req_perform() |>
  resp_body_string() |>
  prettify()

Einen Kommentar codieren

{
    "id": "f50eb65b1b204b3c918764f02741b1d5",
    "created": 1778484539,
    "model": "gpt-oss-120b",
    "object": "chat.completion",
    "choices": [
        {
            "finish_reason": "stop",
            "index": 0,
            "message": {
                "content": "{\"obs_class\": \"nicht inzivil\"}",
                "role": "assistant",
                "reasoning_content": "We need to decide if the comment is civil or not. The comment: \"This is (g)assss{EM_SMILING_FACE_WITH_HEART_EYES}{EM_SMILING_FACE_WITH_HEART_EYES}\" (typo but essentially \"This is (g)assss\" plus smiling face emojis). It's ambiguous. \"(g)assss\" maybe trying to say \"gas\"? It could be a typo or slang. It doesn't contain vulgar or insulting language. It seems just a nonsense phrase with emojis. No violation. So it's not incivil. Output JSON: {\"obs_class\": \"nicht inzivil\"}",
                "provider_specific_fields": {
                    "refusal": null
                }
            },
            "provider_specific_fields": {
                "matched_stop": 200002
            }
        }
    ],
    "usage": {
        "completion_tokens": 144,
        "prompt_tokens": 343,
        "total_tokens": 487,
        "completion_tokens_details": {
            "reasoning_tokens": 1,
            "text_tokens": 143
        }
    },
    "metadata": {
        "weight_version": "default"
    }
}

Alle Kommentare codieren

kommentar true_class obs_class reasons
This is (g)assss{EM_SMILING_FACE_WITH_HEARTEYES}{EM_SMILING_FACE_WITH_HEARTEYES} inzivil nicht inzivil We need to decide if the comment is incivil. The comment: “This is (g)assss{EM_SMILING_FACE_WITH_HEARTEYES}{EM_SMILING_FACE_WITH_HEARTEYES}”. It seems like a possibly vague remark, maybe “gassss”? Could be a typo, no profanity or insult. No violation. So it’s not incivil. Provide JSON {“obs_class”: “nicht inzivil”}. Check if any hidden profanity: “gassss” maybe “grass”? Not vulgar. So non-incivil. Return only JSON.
bones are for dogs inzivil nicht inzivil We need classify. Comment: “bones are for dogs”. That’s a statement maybe implying something about someone? Could be a sarcastic or demeaning comment maybe implying the person is a dog? Without context it’s ambiguous. The phrase “bones are for dogs” could be a neutral statement about bones, but could be a sarcastic comment directed at someone? However alone, it’s not necessarily vulgar, insulting, demeaning explicitly. It might be a sarcastic comment implying the person is a dog, could be dehumanizing? But classification depends on typical judgement. The content: “bones are for dogs”. Without context, seems not inherently uncivil. It’s not profanity, no insult directly. Could be considered a mild insult? It’s like “bones are for dogs” could be saying “you’re like a dog”? Might be demeaning. But lacking clear target. Usually classification they’d consider not incivil. I’d choose “nicht inzivil”.
Wie kann man so dumme Ansichten online verbreiten ? Absolute Frechheit wie heutzutage mit dem Wort Forschung umgegangen wird!! inzivil inzivil We need to decide if comment is uncivil. Content: “Wie kann man so dumme Ansichten online verbreiten ? Absolute Frechheit wie heutzutage mit dem Wort Forschung umgegangen wird!!” It contains “so dumme Ansichten”, calling them dumb, “Absolute Frechheit”. That’s an insult, demeaning, possibly disrespect. It’s negative evaluation but not necessarily hate or discrimination. It’s insulting language: “dumme Ansichten” is insulting. That likely counts as incivility per definition: Beleidigung, Profanität, Erniedrigung. So classify as “inzivil”. Return JSON.
Was für ein Schwachsinn! inzivil inzivil We need to output JSON with obs_class. Determine if comment is incivil. The comment: “Was für ein Schwachsinn!” (What nonsense!). This is an insult (“Schwachsinn” is a derogatory term, but is it considered profanity? It’s a negative term, maybe considered insulting/degrading. Likely considered incivil because it’s a demeaning statement. It’s not a direct insult to a person but calls something nonsense. Might be considered disrespectful. According to criteria: “Beleidigung, Profanität” - it’s an insult. So incivil. So output {“obs_class”: “inzivil”}
Ich bin mir sicher, dass die Leute hier alle Tassen im Schrank haben! inzivil nicht inzivil We need decide if comment is uncivil. The comment in German: “Ich bin mir sicher, dass die Leute hier alle Tassen im Schrank haben!” Means “I’m sure that the people here all have cups in the cupboard!” It’s a bit odd phrase maybe meaning “have cups in the cabinet” but not insulting? It could be a figurative expression? Might be a sarcastic insult: “Tassen im Schrank haben” maybe means “have cups in the cupboard” but could be idiom for being crazy? Actually German idiom: “jemand hat eine Schraube locker” etc. Not sure. “Tassen im Schrank haben” not common. Could be implying they are all crazy? But it’s not overt insult, possibly mild sarcasm. There’s no profanity, no direct insult, just a vague statement. I think does not meet criteria. So “nicht inzivil”.
Du bist toll, so wie du bist {EM_RED_HEART}️ inzivil nicht inzivil We need to determine if the comment is incivil or not incivil. The comment is “Du bist toll, so wie du bist {EM_RED_HEART}️”. That’s a positive statement, no vulgar language, no insult, etc. So it’s not incivil. We must output JSON exactly: {“obs_class”: “nicht inzivil”}
Du bist dumm inzivil inzivil We have to decide if comment is uncivil. The comment: “Du bist dumm”. That’s an insult, a disrespectful statement, thus incivil. So output JSON: {“obs_class”:“inzivil”} Ensure correct format, no extra spaces? Must be exactly as format. Probably include a space after colon? The example shows with space after colon. They said exactly this format: {“obs_class”: “inzivil”} or {“obs_class”: “nicht inzivil”}. So include space after colon and after comma? There’s only colon. So use format with space after colon. Thus answer.
Linksveriffte Öko-Trulla inzivil inzivil We need to determine if the comment is “inzivil” or “nicht inzivil”. The comment: “Linksveriffte Öko-Trulla”. It’s German. Let’s translate: “Linksveriffte” maybe “left-wing” “veriffte”? Actually “veriffte” is not standard; maybe “ver-…”? “Öko-Trulla” means “eco-trull”? “Trulla” is a pejorative term meaning “trul” (idiot, fool) used as insult. “Linksveriffte” maybe a typo for “linksverrückte” or “linksverfaulte”? Could be “linksveriffte” as a slur implying someone is overly left. The phrase combines political labeling with insult “Trulla”. This is an insult, pejorative, demeaning. Likely considered “inzivil”. No explicit profanity but it’s an insult, demeaning. So label “inzivil”. Return JSON: {“obs_class”: “inzivil”}
Das war schön inzivil nicht inzivil We need to determine if the comment “Das war schön” is incivil. It’s a simple positive statement, no incivility. So output {“obs_class”: “nicht inzivil”}
Clown inzivil inzivil We need to classify “Clown”. Is this incivil? It’s a single word “Clown”. Could be an insult? It’s calling someone a clown, which is an insult, a demeaning term. According to criteria: Beleidigung, Abwertung. Yes, calling someone “Clown” is a demeaning insult. So it’s likely “inzivil”. The comment is likely incivil as it’s an insult. So output JSON as {“obs_class”: “inzivil”}.
du bist sch***! inzivil inzivil We need to classify. The comment: “du bist sch***!” That’s a vulgar insult, likely using a censored profanity. This is insulting and vulgar. So it’s “inzivil”. Return JSON with {“obs_class”: “inzivil”}
du bist hässlich inzivil inzivil We need to decide if the comment is uncivil. The comment “du bist hässlich” translates to “you are ugly”. This is an insult, lack of respect, demeaning. That is considered incivility. So output JSON: {“obs_class”: “inzivil”}
Merz leck Eier inzivil inzivil We need to determine if comment is civil or uncivil. The comment: “Merz leck Eier”. It’s German. “Merz” likely referring to Friedrich Merz. “leck Eier” translates to “lick eggs”? Could be an insult? “leck Eier” is a vulgar phrase, maybe meaning “lick eggs”, could be a mocking or insulting expression. It’s somewhat crude but not explicit profanity? “leck Eier” is possibly nonsense. However, it’s possibly a vulgar expression (like “lick eggs” might be used as an insult). The phrase has no profanity like swear words, but could be seen as disrespectful, mocking. It’s likely considered incivil because it’s a demeaning statement “Merz lick eggs”, which is nonsensical but likely intended to be insulting. It is disrespectful, possibly vulgar (the word “Eier” can be slang for testicles, “Leck Eier” could be akin to “suck my balls”. So it’s vulgar. So classification: “inzivil”. Return JSON.
HS inzivil nicht inzivil We have a user: The comment is “HS”. The instruction: Must output JSON exactly {“obs_class”: “inzivil”} or {“obs_class”: “nicht inzivil”}. We need to decide whether “HS” is incivil. “HS” is a short two-letter phrase, possibly stands for “Heil Hitler” or “Hate Speech”? Could be a slang abbreviation. In German context, “HS” often stands for “Hinterwäldler”? Not sure. Could be “HS” as “Hass”? Hard to know. Given minimal content, no obvious incivility. It’s ambiguous. Likely not considered incivil as it’s just two letters with no explicit incivility. So classify as “nicht inzivil”.
Du Hurensohn inzivil inzivil We need classify. The comment: “Du Hurensohn” is an insult, vulgar. That is incivil. So output JSON with “inzivil”.
Fuck inzivil inzivil We need to decide if the comment “Fuck” is incivil. The comment contains vulgar profanity, which is listed as a marker for incivility. So output {“obs_class”: “inzivil”}. Make sure JSON exactly as specified, no extra whitespace? Probably fine.
Merz, leck Eier inzivil inzivil We need to decide if the comment is incivil. The comment is “Merz, leck Eier”. In German, “leck Eier” is a vulgar phrase, essentially “go suck eggs”? It’s an insult, vulgar language. It’s likely incivil because it includes vulgar language and insult. So classification: “inzivil”. According to required JSON: {“obs_class”:“inzivil”} with no extra spaces? They said exact format, but examples had a space after colon? They said {“obs_class”: “inzivil”} exactly. We’ll output exactly that.
Das ist doch Schwachsinn inzivil inzivil We need to classify the comment “Das ist doch Schwachsinn”. This is German for “That’s nonsense”. “Schwachsinn” is a mild insult? It’s a derogatory term meaning “nonsense” or “stupidity”. It’s insulting, possibly considered incivility. According to definition, “Beleidigung, Profanität” is incivil. So likely “inzivil”. Provide JSON.
Du Fotze inzivil inzivil The user gave a German comment: “Du Fotze”. That’s an insult, vulgar profanity. That is clearly incivil. We must output JSON: {“obs_class”: “inzivil”}. We must follow instructions: no extra text.
Oiiiai oiiiai inzivil nicht inzivil The user gave a task: classify whether the comment is civil or not. The comment is “Oiiiai oiiiai”. That’s just a nonsensical greeting maybe. No vulgarity, no insults, etc. So it’s not incivil. Need output JSON exactly {“obs_class”: “nicht inzivil”}. Make sure no extra spaces? The format requires exact. Probably okay.
Hure! inzivil inzivil We need to classify the comment. The comment is “Hure!” which is a vulgar insult (German for “whore”). This is clearly incivil. So output JSON with “inzivil”.
Ich finde heute siehst du komisch aus. inzivil inzivil We need to decide if the comment is incivil. The comment: “Ich finde heute siehst du komisch aus.” That’s a mild critique, possibly a personal remark that you look weird today. It’s a bit rude maybe, but not necessarily incivil per the definitions: no vulgar language, no insult? “Komisch aussehen” could be considered a mild insult. It is a negative assessment, but does it count as “Beleidigung, Profanität”? It’s a mild negative comment, not a direct profanity or explicit insult. The definitions list “Beleidigung, Profanität” as incivil. Is calling someone “komisch” considered an insult? Probably yes, it’s a mild insult. However, the guidelines: comment that contains such features are “in der Regel” considered incivil. “Ich finde heute siehst du komisch aus.” I’d lean that it’s impolite and could be considered a slight insult, but not necessarily incivil per strict academic definitions perhaps. However given the guidelines, “Beleidigung” includes mild negative remarks. I’d classify as “inzivil”. So output JSON with “inzivil”.
Die Weisheit verfolgt dich, aber du bist schneller! inzivil nicht inzivil We have a comment: “Die Weisheit verfolgt dich, aber du bist schneller!” (in German). This is somewhat playful, not insulting. No vulgar language, no insults, no threats. So non-violent, not incivil. It appears to be a compliment or a poetic remark. So classification: “nicht inzivil”. Output JSON exactly.
Wtf,,, was denkt der wer er ist??? Hasse den so… hoffe er stirbt inzivil inzivil We need to classify whether the comment is invasive (incivil) according to defined criteria. The comment: “Wtf,,, was denkt der wer er ist??? Hasse den so… hoffe er stirbt”. This includes profanity (Wtf?), “Hasse den so… hoffe er stirbt” = expresses hatred and a wish that the person dies => threat of violence (wish for death). That is incivil. So output JSON: {“obs_class”: “inzivil”}.
Ganz schön hässlich inzivil inzivil We need to classify comment. The comment: “Ganz schön hässlich” translates to “quite ugly”. Is that an insult? It’s a negative comment about appearance perhaps; it’s a derogatory statement, possibly an insult. It may be considered incivil? According to criteria: “Beleidigung, Profanität” includes insult. So likely “inzivil”. It’s an insult (calling something ugly). So output JSON {“obs_class”:“inzivil”}. Check format: need exactly {“obs_class”: “inzivil”} with space after colon? In examples they have a space after colon? They wrote {“obs_class”: “inzivil”} in description. We’ll match that. Thus final.
Fake news! inzivil inzivil We have to decide if the comment “Fake news!” is uncivil. It’s a short statement that calls something “Fake news”. This could be considered an accusation of lying or a negative evaluative statement. Is that considered “Vorwurf der Lüge”? Yes, that is a claim that something is false, which could be considered a “Vorwurf der Lüge”. According to the list, “Vorwurf der Lüge” is a marker of incivility. So we label as incivil. The guidelines: we must output JSON exactly {“obs_class”: “inzivil”} or {“obs_class”: “nicht inzivil”}. Thus output {“obs_class”: “inzivil”}.
Ich weiß wo du wohnst inzivil inzivil User asks to classify comment. The comment: “Ich weiß wo du wohnst”. This is a threat? It indicates knowledge of location, could be intimidation. It is a threat of potential violence or harassment. It’s a form of intimidation, possibly a threat. According to definition: “Androhung von Gewalt” - threat. This is a threat because implying you could act. So classify as “inzivil”. Output JSON.
Ich hau dir auf die Schnauze inzivil inzivil We need to classify. The comment: “Ich hau dir auf die Schnauze” which translates to “I’ll punch you in the face”. That’s a threat of violence. So it’s incivil. Return JSON {“obs_class”:“inzivil”}.
Mutig, wie viel Meinung man mit so wenig Ahnung haben kann. inzivil inzivil We need to assess if the comment is incivil. The comment: “Mutig, wie viel Meinung man mit so wenig Ahnung haben kann.” In English: “Brave, how much opinion one can have with so little knowledge.” This is a sarcastic, mocking statement. It contains sarcasm/spite, possibly a slight insult implying lack of knowledge. It’s a negative assessment but perhaps not severely offensive. It could be considered incivil due to sarcasm/derision, insulting the person’s knowledge. The definition includes “Sarkasmus, Spott, Zynismus” and “Beleidigung, Profanität”. This is sarcastic, possibly insulting. So likely considered incivil. So output {“obs_class”: “inzivil”}
Das ist dumm inzivil inzivil We need to decide if comment “Das ist dumm” is uncivil. It’s an insult, demeaning maybe. It’s a negative evaluation, “dumm” meaning stupid. Is that considered incivility? The definition includes “Beleidigung, Profanität”. Insult is a form of Beleidigung, so likely considered incivil. So output {“obs_class”: “inzivil”}. Check JSON format exactly with double quotes.
Arschloch inzivil inzivil We need to determine if comment is “inzivil”. The comment is “Arschloch”. German vulgar insult, profanity. It’s clearly incivil. So output JSON {“obs_class”: “inzivil”}. Check format exactly.
Ja natürlich stimmt das /s inzivil inzivil We need to decide if comment “Ja natürlich stimmt das /s” is incivil. “/s” indicates sarcasm (sarkasmus). The comment says “Ja natürlich stimmt das” with sarcastic tag, implying sarcasm, possibly mocking. Sarcasm can be considered incivil if it is mocking or negative. The definition includes “Sarkasmus, Spott, Zynismus”. So this qualifies as incivil. It’s sarcasm. So output {“obs_class”: “inzivil”}
Hoffe er wird gekirkt inzivil inzivil We have to decide if comment is incivil. The comment: “Hoffe er wird gekirkt”. In German, “Hoffe er wird gekirkt” means “Hope he gets killed”. “gekirkt” is a colloquial way meaning “kirken” maybe “to kill”? Actually “gekirkt” in German slang means “to be killed”, from “abkicken”? Wait “gekickt”? But “gekirkt” maybe typo for “gekillt”? It expresses wish for someone to be killed. That’s a threat of violence. So incivil. Must output JSON: {“obs_class”: “inzivil”}
Du bist dumm inzivil inzivil We have to decide if comment is uncivil. The comment: “Du bist dumm”. That’s an insult, a disrespectful statement, thus incivil. So output JSON: {“obs_class”:“inzivil”} Ensure correct format, no extra spaces? Must be exactly as format. Probably include a space after colon? The example shows with space after colon. They said exactly this format: {“obs_class”: “inzivil”} or {“obs_class”: “nicht inzivil”}. So include space after colon and after comma? There’s only colon. So use format with space after colon. Thus answer.
Lügner! inzivil inzivil We have a comment: “Lügner!” which translates to “Liar!” It’s an accusation of lying, which is a “Vorwurf der Lüge”. That is listed as a sign of incivility. So the comment is incivil. Thus output JSON: {“obs_class”: “inzivil”}
Fotze inzivil inzivil We need to classify. The comment “Fotze” is a German vulgar insult, derogatory term for female genitalia used as an insult. It’s vulgar, profane, insulting. So it’s incivil. Output JSON {“obs_class”:“inzivil”}. Check format: need exact keys and values, no extra spaces? Example says {“obs_class”: “inzivil”} with space after colon. But they said exactly this format. Probably whitespace okay but must match JSON. We’ll output with space after colon.
fette Sau inzivil inzivil We need to classify given comment. The comment: “fette Sau”. That’s an insult, demeaning. It’s German: “fat sow”. That’s a vulgar insult, dehumanizing. So it’s uncivil. So output {“obs_class”: “inzivil”} in JSON.
Bring dich um! inzivil inzivil We must classify comment as incivil or not. Comment: “Bring dich um!” (German for “Kill yourself!”). This is a threat of self-harm? Actually it’s urging someone to kill themselves (a violent threat). That is incivil: it is a threat of violence. So label incivil. Return JSON only.
Ich finde du siehst einfach seltsam aus. inzivil inzivil We need to determine if comment is “inzivil” (uncivil) based on definitions: violation of communicative norms: vulgar language, insult, dehumanization, sarcasm, negative stereotypes, discrimination, threat, etc. The comment: “Ich finde du siehst einfach seltsam aus.” Means: “I think you just look strange.” This is a mild insult (calling someone looks strange). It’s a negative evaluation, possibly disrespectful, but is it considered incivil? It is a personal criticism about appearance, could be considered disrespectful but not necessarily vulgar or threatening. According to defined criteria: “Beleidigung, Profanität” includes insult. Saying “du siehst seltsam aus” is an insult (negative evaluation) but maybe mild. It could be considered insulting. So likely classified as “inzivil”. The definitions include “Beleidigung, Profanität”. So yes, it’s an insult. So output JSON: {“obs_class”: “inzivil”}
Interessant, wie überzeugt du davon bist. Ich wünschte, ich hätte auch so viel Selbstvertrauen bei so wenig Grundlage inzivil inzivil We need to classify the comment. The comment: “Interessant, wie überzeugt du davon bist. Ich wünschte, ich hätte auch so viel Selbstvertrauen bei so wenig Grundlage”. Translation: “Interesting how convinced you are. I wish I had as much confidence with so little basis.” This is somewhat sarcastic? It sounds somewhat mocking: “Interesting how convinced you are. I wish I had as much confidence with so little basis.” Could be sarcastic or spottish, but not overtly insulting. It’s a mild criticism, maybe sarcasm. According to definition, sarcasm, spott is considered incivility. It contains sarcasm/spott. So likely classified as “inzivil”. The comment expresses doubt, maybe a bit mocking but not explicitly insulting. The guidelines: “Sarkasmus, Spott, Zynismus” are incivil. So yes, incivil. Return JSON.
I wish you get raped inzivil inzivil We have to output JSON only, either “inzivil” or “nicht inzivil”. The comment is “I wish you get raped”. That is a threat of sexual violence; it is a violent threat, harassment. This is clearly incivil. So output {“obs_class”: “inzivil”}. We must output exactly that JSON, no extra text.
Wie schön wieder in der Uni zu sein inzivil nicht inzivil We need to classify the comment. The comment is “Wie schön wieder in der Uni zu sein” (German: “How nice to be back at university again”). This is neutral, no incivility. So output {“obs_class”: “nicht inzivil”} in exact JSON.
Du F0tze inzivil inzivil We have to classify the comment. The comment: “Du F0tze”. That’s German slang, “F0tze” likely means “Fotze”, a vulgar insult for a woman (“cunt”). It’s a vulgar, insulting language. So it’s incivil. Output JSON: {“obs_class”: “inzivil”}
Das hast du ja richtig toll gemacht… inzivil inzivil We need to classify the comment “Das hast du ja richtig toll gemacht…” into incoivil or not. The comment seems sarcastic? “richtig toll gemacht” could be praising, but with ellipsis possibly sarcasm. However without clear negative language? It could be sarcasm implying criticism. Sarcasm is listed as a sign of incivility: “Sarkasmus, Spott, Zynismus”. So likely considered incivil. It’s ambiguous. The phrase “richtig toll gemacht” could be genuine praise, but the ellipsis might indicate sarcasm. Could interpret as sarcastic. Safer to label as incivil? The dataset may treat sarcasm as incivil. I’d label as “inzivil”.
Dumme Nuss inzivil inzivil We need to classify: “Dumme Nuss” (German for “stupid nut” basically an insult). It’s an insult, disrespect. So it’s uncivil. Return {“obs_class”:“inzivil”}
Nutte inzivil inzivil We need to classify the comment: “Nutte”. That’s a vulgar insult demeaning; definitely incivil. So we output JSON with {“obs_class”: “inzivil”}. Need exactly that, no extra text.
nuttig inzivil inzivil We need to label the comment “nuttig”. It’s a German word? “nuttig” could be a misspelling of “nut”?? “nuttig” could be a slang meaning “nut”, maybe vulgar? In German, “Nut” not sure. “Nuttig” could be a variation of “Nutte” meaning prostitute, which is offensive. Could be considered vulgar, insulting. It’s a single word, likely a profanity or insulting term. I’d label as “inzivil”.
Willste dich boxen? inzivil inzivil We need to classify the comment: “Willste dich boxen?” This is German, probably meaning “Will you get boxed?” Or “Do you want to get punched?” It is a threat of violence. It’s a threat of violence, thus incivil. So output {“obs_class”:“inzivil”}
kys inzivil inzivil We need to decide if comment “kys” is incivil. “kys” stands for “kill yourself”, which is a threatening, harassing statement, maybe encouraging self-harm. That’s a form of harassment, violent threat, incivil. So output JSON {“obs_class”:“inzivil”}. Check format: {“obs_class”: “inzivil”} exactly with space after colon? The spec says exactly this format with space after colon. We’ll match.
Wow, du bist soo mutig dich ungeschminkt zu posten inzivil inzivil We need to decide if the comment is civil or not. The comment: “Wow, du bist sooo mutig dich ungeschminkt zu posten”. It’s a sarcastic? Probably mocking: “Wow, du bist sooo mutig dich ungeschminkt zu posten”. Could be interpreted as sarcastic, perhaps mildly mocking. Is that considered incivility? It’s a bit of sarcasm/spott. It could be considered disrespectful, mocking someone’s appearance. Is it a violation? It’s not profanity, no hate, no threats, but it is sarcasm/spott, maybe a subtle insult. According to definition, sarcasm, Spott, Zynismus are considered incivility. So likely classify as “inzivil”. Return JSON.
was ein Scheiß inzivil inzivil We need to decide if the comment is incivil. The comment: “was ein Scheiß”. That’s vulgar (Scheiß = shit), profanity. So incivil. Return JSON {“obs_class”: “inzivil”}.
Hässlich inzivil inzivil We must decide if the comment “Hässlich” is incivil. It’s an adjective meaning “ugly”. It’s an insult (Beleidigung). That counts as incivil. So output JSON with {“obs_class”: “inzivil”}.
Halt die Fresse inzivil inzivil The user asks to classify the comment. The comment is “Halt die Fresse” which is German for “Shut your mouth/f*ck”. It’s vulgar, insulting language. So it’s incivil. Must output JSON exactly {“obs_class”: “inzivil”}. No extra text.
Ich stimme dir da gar nicht zu. Weiß nicht wie du drauf kommst aber bist ja auch nicht ganz richtig in der Birne inzivil inzivil We must decide if the comment is uncivil. The comment: “Ich stimme dir da gar nicht zu. Weiß nicht wie du drauf kommst aber bist ja auch nicht ganz richtig in der Birne”. It includes an insult: “bist ja auch nicht ganz richtig in der Birne” meaning “you’re not quite right in the head”, i.e., calling someone not right in the head => insult, demeaning. That is a violation of respectful communication. So it’s incivil. So output {“obs_class”: “inzivil”}
Männer sollten keine Rechte haben inzivil inzivil We need to decide if the comment is “inzivil” or “nicht inzivil”. The comment: “Männer sollten keine Rechte haben”. This is a statement that says “Men should have no rights”. This is a denial of rights, discriminatory, dehumanizing, inciting prejudice. It implies an extreme discriminatory statement. According to criteria: “Verweigerung von Rechten” is listed as a characteristic of incivility. So it’s incivil. So output JSON {“obs_class”:“inzivil”}. We must ensure exactly the JSON, no extra spaces? Format: {“obs_class”: “inzivil”} includes space after colon. That matches example. Use exactly that.
Nazi-Schweine inzivil inzivil We need to decide if the comment “Nazi-Schweine” is incivil. It’s an insult, dehumanizing term, likely incivil. So output {“obs_class”: “inzivil”}. Make sure JSON exactly.
AfDler abschieben inzivil inzivil The task: identify if comment is uncivil. The comment: “AfDler abschieben”. In German, translation: “Deport AfD members”. That’s a call for violence (deportation) i.e., threatening/advocating harm to a group. It’s incivil (contains threat of violence, dehumanization). So output JSON with {“obs_class”: “inzivil”}. Make sure format exactly.
Dem stimme ich zu. nicht inzivil nicht inzivil We need to classify whether “Dem stimme ich zu.” is incivil. That’s a short statement “I agree with that.” It’s civil. So output {“obs_class”: “nicht inzivil”}
Ich finde Titten geil nicht inzivil inzivil The user asks to classify the comment. The comment is “Ich finde Titten geil”. This is a vulgar statement about breasts (Titten). It’s sexual, possibly vulgar language. According to definitions, vulgar language is considered incivility. So we should output {“obs_class”: “inzivil”}. We must output only JSON.
bitch, das ist sooo crazyy nicht inzivil inzivil We need to decide if comment is incivil. The comment: “bitch, das ist sooo crazyy”. Contains vulgar word “bitch”, which is an insult, vulgar. So it’s incivil. Output JSON: {“obs_class”: “inzivil”}
Also das ist jetzt schon ein bisschen unfair. Sie hat auch noch andere Facetten, z.B. ist sie auch noch grob inkompetent :) nicht inzivil inzivil We need decide if comment is uncivil. Comment: “Also das ist jetzt schon ein bisschen unfair. Sie hat auch noch andere Facetten, z.B. ist sie auch noch grob inkompetent :)”. It’s saying unfair, also says “grob inkompetent” which is an insult/abwertung. That is a derogatory statement, but is it incivil? It’s an insult, negative evaluation. According to list, “Beleidigung, Profanität” is incivil. “grob inkompetent” is insulting but not profanity. It’s a negative evaluation, possibly disrespect. Likely incivil. So output {“obs_class”: “inzivil”}. Check requirement: JSON exactly.
Was für ein schönes Foto! nicht inzivil nicht inzivil We need to classify the comment: “Was für ein schönes Foto!” (German: “What a beautiful photo!”). This is positive, no vulgarity, no insult. So it’s non-uncivil. Output JSON: {“obs_class”: “nicht inzivil”}
Hab eine schöne Woche! nicht inzivil nicht inzivil We need to decide if the comment is incivil. The comment: “Hab eine schöne Woche!” (German, means “Have a nice week!”). That’s a polite positive comment, no incivility. So output {“obs_class”:“nicht inzivil”} exactly JSON without extra text.
Öffentlichkeit nicht inzivil nicht inzivil We have a comment: “Öffentlichkeit”. It’s just the word “Public” in German. No incivility. So classify as “nicht inzivil”. Output JSON.
Ist das KI? nicht inzivil nicht inzivil We have a comment: “Ist das KI?” in German, meaning “Is that AI?” It’s a short question, no incivility. So classification: not inzivil. Must output JSON.
Woww, geiler scheiß! nicht inzivil inzivil We need decide if comment is incivil. The comment: “Woww, geiler scheiß!” It’s German, meaning “Wow, cool shit!” It’s vulgar? “scheiß” is a vulgar word (shit) but used as exclamation. Contains vulgar language. According to definition, vulgar language considered incivil. So classification: incivil. Output JSON {“obs_class”: “inzivil”}.
Das ist falsch. nicht inzivil nicht inzivil We need to determine if comment “Das ist falsch.” is incivil? It’s a simple statement, no vulgar, no insult, just saying “That’s wrong.” No violation. So “nicht inzivil”. Return JSON.
Was ein Motherfucker nicht inzivil inzivil We need to classify. The comment: “Was ein Motherfucker”. This contains vulgar profanity “Motherfucker”. It’s vulgar language, insulting. So it’s incivil. Output JSON {“obs_class”: “inzivil”}
Du bist sau cool nicht inzivil nicht inzivil We need to classify the comment “Du bist sau cool”. This is a positive compliment, no incivility. So “nicht inzivil”. Need to output JSON exactly.
Das Rezept mache ich auf jeden Fall nach!!! Danke für die Inspo! nicht inzivil nicht inzivil We need to determine if the comment is uncivil. The comment: “Das Rezept mache ich auf jeden Fall nach!!! Danke für die Inspo!” It’s positive, expressing thanks. No incivility. So output {“obs_class”: “nicht inzivil”}
Danke und das so wichtig! nicht inzivil nicht inzivil We need to classify comment: “Danke und das so wichtig!” in German. This translates to “Thanks and that so important!” It’s not incivil. So output {“obs_class”: “nicht inzivil”}
Lars Klingbeil ist ein Klassenverräter nicht inzivil inzivil We need to decide if this comment is incivil. The comment: “Lars Klingbeil ist ein Klassenverräter”. Translation: “Lars Klingbeil is a class traitor”. It’s an accusation, calling him a traitor. It’s an insult, possibly a negative judgment. Is it incivil? The criteria: insult, profanity, negative stereotypes, etc. “Klassenverräter” is an insult implying betrayal of class (maybe political class). It’s a vilifying term, could be considered disrespect/abwertung. Likely considered incivil because it’s a derogatory label, accusation of betrayal, insulting. It doesn’t contain profanity, but it is an insult/ad hominem. According to criteria, “Beleidigung, Profanität” includes insult. So yes, incivil. Thus output {“obs_class”: “inzivil”}. Check format: must be exactly JSON with no extra text.
Bayern gewinnt…Überraschung des Jahres. nicht inzivil nicht inzivil We need to determine if comment is incivil. The comment: “Bayern gewinnt…Überraschung des Jahres.” It’s a simple statement “Bayern wins… Surprise of the year.” No vulgarity, no insult, no discrimination. It’s not incivil. So output non incivil JSON.
Ekelhaft nicht inzivil inzivil We need to decide if comment “Ekelhaft” is uncivil. It’s a single word meaning “disgusting”. It’s an insult, negative evaluation, possibly rude. It is a pejorative adjective but not necessarily directly insulting a person? Could be describing something as disgusting. In context of a comment, likely it’s a negative judgement. It could be considered incivility because it’s a derogatory term, possibly an insult. The guidelines: Vulgäre, unangemessene Sprache, Beleidigung, etc. “Ekelhaft” is an insult, but not profanity or vulgar. It’s a negative evaluation. Might be considered incivility if directed at a person or group. With no context, it’s a negative term; likely considered disrespectful. I’d classify as incivil? It is a negative judgement, possibly insulting. The guidelines include “Beleidigung, Profanität”. “Ekelhaft” can be considered a mild insult. I’d say incivil. So output JSON {“obs_class”: “inzivil”}
Ich möchte Sommer nicht inzivil nicht inzivil We need to classify the comment “Ich möchte Sommer”. That is “I want summer”. It’s not incivil. So output JSON: {“obs_class”: “nicht inzivil”}
Scheiße, das sieht gut aus! nicht inzivil inzivil We need to decide if the comment is civil or not based on the criteria. The comment: “Scheiße, das sieht gut aus!” It’s German: “Scheiße, das sieht gut aus!” Contains vulgar word “Scheiße” (shit) – a profanity/vulgar language. According to definition, vulgar language is considered incivility. So it’s incivil. Output JSON: {“obs_class”: “inzivil”}
Ich sehe Sie als einen inkompetenten Politiker an nicht inzivil inzivil We need to decide if the comment is “inzivil” according to defined criteria. The comment: “Ich sehe Sie als einen inkompetenten Politiker an” translates to “I see you as an incompetent politician.” That’s an insult? It’s a negative judgment, calling someone incompetent. Is that considered “Beleidigung, Profanität”? It’s a demeaning statement but not necessarily vulgar or threatening. It is a negative assessment, could be considered insulting. According to criteria: “Beleidigung, Profanität” - this is an insult (calling incompetent). Likely considered incivil. Many frameworks would consider personal attack. So classify as “inzivil”. Output JSON.
fotzig nicht inzivil nicht inzivil We have to classify comment “fotzig”. It’s a German word meaning “cute”. No incivility. So output {“obs_class”: “nicht inzivil”}.
Hab Bock auf Eierkuchen nicht inzivil nicht inzivil We need to classify the comment: “Hab Bock auf Eierkuchen”. This is German for “I’m in the mood for pancakes”. No incivility, it’s neutral. So output JSON {“obs_class”: “nicht inzivil”}. Check: Must be exact JSON with no extra spaces? It says exact format {“obs_class”: “inzivil”} or {“obs_class”: “nicht inzivil”}. Provide no additional text. Provide JSON.
Du bist doof nicht inzivil inzivil We need to classify “Du bist doof”. This is an insult, vulgar? It’s an insult, disrespect. So it falls under incivility (Beleidigung). So output JSON with {“obs_class”:“inzivil”}. Make sure formatting exactly.
Scheiße wie cool nicht inzivil inzivil The user provided a comment: “Scheiße wie cool”. Need decide if it’s incivil. The comment contains vulgar word “Scheiße” (German for “shit”), which is vulgar language, possibly profanity. It’s an exclamation: “Scheiße wie cool”. It uses a profanity but overall meaning is expressing enthusiasm with a curse word. According to definition, vulgar, inappropriate language qualifies as incivility. So likely classify as “inzivil”. Output JSON {“obs_class”: “inzivil”}. Check: Must output exactly JSON, no extra text.
love this! nicht inzivil nicht inzivil We need classify. “love this!” is positive, not incivil. So output JSON with “nicht inzivil”.
Du bist schlagfertig. nicht inzivil nicht inzivil We need to decide if the comment “Du bist schlagfertig.” is incivil or not. It’s a compliment, not insulting. No vulgar, no negative. So it’s not incivil. Output JSON {“obs_class”: “nicht inzivil”}.
Du Sau! nicht inzivil inzivil We need to output JSON with obs_class either “inzivil” or “nicht inzivil”. The comment “Du Sau!” is an insult, vulgar, profanity. So it’s incivil. So output {“obs_class”: “inzivil”}. Make sure only JSON, no extra whitespace? They said exact format but can have whitespace. We’ll output exactly.
fucking gute idee nicht inzivil inzivil We need to classify the comment: “fucking gute Idee”. This contains vulgar language: “fucking” is a profanity. So it’s incivil. Return JSON {“obs_class”: “inzivil”}.
Ohh danke für die Hilfe! nicht inzivil nicht inzivil We need to determine if comment “Ohh danke für die Hilfe!” is uncivil. It’s a polite thank you. So not incivil. Output JSON: {“obs_class”: “nicht inzivil”}
Ohne Worte, wo soll das noch hin führen? nicht inzivil nicht inzivil We need to decide if the comment “Ohne Worte, wo soll das noch hin führen?” is incivil? It’s a German phrase: “Without words, where is that supposed to lead?” This is a neutral rhetorical question, not containing any of the listed incivil features. So it’s not incivil. Return JSON: {“obs_class”: “nicht inzivil”}
Ich finde, die CDU versagt auf ganzer Linie! Peinlich! nicht inzivil nicht inzivil We need to classify if comment is uncivil. The comment: “Ich finde, die CDU versagt auf ganzer Linie! Peinlich!” It’s a political criticism. It says CDU fails completely, embarrassing. No profanity, no personal insult? It’s a party, not a person. It’s strong negative evaluation but not necessarily uncivil. According to criteria: vulgar language, insult, dehumanization, etc. “versagt” and “Peinlich” are strong but not profanity, not dehumanizing. Likely considered “nicht inzivil”. So output JSON.
Spannender Beitrag nicht inzivil nicht inzivil We need to decide if comment “Spannender Beitrag” (German for “interesting post”) is incivil. It’s polite, no bad language. So not incivil. Output JSON with “nicht inzivil”.
Boss Bitch nicht inzivil inzivil We need to classify the comment “Boss Bitch”. Determine if it’s incivil. The comment consists of two words: “Boss Bitch”. “Bitch” can be considered vulgar, possibly insulting. It’s a pejorative term, often used as insult towards women, but also could be used colloquially. It’s a profanity/insult. Likely considered incivil. So output {“obs_class”: “inzivil”}. Check format: must be exactly JSON with double quotes. No extra whitespace? It’s fine.
Das fickt ordentlich was ihr da macht :) nicht inzivil inzivil The user provided a comment: “Das fickt ordentlich was ihr da macht :)”. In German, “fickt” is vulgar profanity, meaning “fuck”. This is vulgar language, possibly insulting. So it’s incivil. The required output JSON: {“obs_class”: “inzivil”}. We must output exactly JSON with no extra text.
Welch ein blumiger Sonnenuntergang! nicht inzivil nicht inzivil We need to determine if the comment is uncivil. The comment: “Welch ein blumiger Sonnenuntergang!” (What a flowery sunset!). It’s positive, no incivil language. So “nicht inzivil”. Output JSON: {“obs_class”: “nicht inzivil”}
Freue mich für dich :) nicht inzivil nicht inzivil The user (German) asks to classify the comment as civil or uncivil. The comment: “Freue mich für dich :)” which is positive, no incivility. So output {“obs_class”: “nicht inzivil”}. Need to output only JSON.
das ist so unlustig und komisch nicht inzivil nicht inzivil User asks to classify comment: “das ist so unlustig und komisch”. It’s German: “that’s so unfunny and funny” maybe contradictory but not incivil. No profanity, no insult directed at a person or group. It’s just an opinion about something being unfunny and strange. Not incivil. So output JSON with “nicht inzivil”.
krasser Scheiß nicht inzivil inzivil The user asks to classify comment “krasser Scheiß”. Need determine if it’s uncivil. The comment is German: “krasser Scheiß”. Means “crazy sh*t” or “awesome shit”. It’s vulgar profanity “Scheiß” is a swear word (shit). That’s vulgar language. According to definition, vulgar language is incivility. So it should be classified as “inzivil”. Need output JSON exactly.
Yasss slayyy queeennn nicht inzivil nicht inzivil We need to classify if comment is incivil. The comment: “Yasss slayyyy queeennn”. It’s enthusiastic, slang, no profanity, no insult. It’s not incivil. So output {“obs_class”:“nicht inzivil”}. Check format: exactly {“obs_class”: “nicht inzivil”} (space after colon?). Example given: {“obs_class”: “inzivil”} or {“obs_class”: “nicht inzivil”}. Use same spacing. Provide only JSON.
richtig guter punkt nicht inzivil nicht inzivil We need to classify the comment: “richtig guter punkt”. It’s German for “really good point”. No vulgarity, no insult etc. It’s civil. So output {“obs_class”: “nicht inzivil”}
Schön, dass ihr Spaß im Urli habt nicht inzivil nicht inzivil We must classify whether comment is civil or not. The comment: “Schön, dass ihr Spaß im Urli habt”. It’s a polite statement: “Nice that you have fun in the Urli”. No incivility. So output {“obs_class”: “nicht inzivil”}. Ensure exact JSON.
I don’t know, 6-7 nicht inzivil nicht inzivil We need to decide if comment is incivil. The comment is “I don’t know, 6-7”. It’s just a statement, no incivility. So output JSON accordingly.
Schöne Berge nicht inzivil nicht inzivil We need to decide if comment “Schöne Berge” is incivil or not. It’s just “Beautiful mountains”. No incivility. So output JSON: {“obs_class”: “nicht inzivil”}

Keine Übereinstimmung

kommentar true_class obs_class reasons
This is (g)assss{EM_SMILING_FACE_WITH_HEARTEYES}{EM_SMILING_FACE_WITH_HEARTEYES} inzivil nicht inzivil We need to decide if the comment is incivil. The comment: “This is (g)assss{EM_SMILING_FACE_WITH_HEARTEYES}{EM_SMILING_FACE_WITH_HEARTEYES}”. It seems like a possibly vague remark, maybe “gassss”? Could be a typo, no profanity or insult. No violation. So it’s not incivil. Provide JSON {“obs_class”: “nicht inzivil”}. Check if any hidden profanity: “gassss” maybe “grass”? Not vulgar. So non-incivil. Return only JSON.
bones are for dogs inzivil nicht inzivil We need classify. Comment: “bones are for dogs”. That’s a statement maybe implying something about someone? Could be a sarcastic or demeaning comment maybe implying the person is a dog? Without context it’s ambiguous. The phrase “bones are for dogs” could be a neutral statement about bones, but could be a sarcastic comment directed at someone? However alone, it’s not necessarily vulgar, insulting, demeaning explicitly. It might be a sarcastic comment implying the person is a dog, could be dehumanizing? But classification depends on typical judgement. The content: “bones are for dogs”. Without context, seems not inherently uncivil. It’s not profanity, no insult directly. Could be considered a mild insult? It’s like “bones are for dogs” could be saying “you’re like a dog”? Might be demeaning. But lacking clear target. Usually classification they’d consider not incivil. I’d choose “nicht inzivil”.
Ich bin mir sicher, dass die Leute hier alle Tassen im Schrank haben! inzivil nicht inzivil We need decide if comment is uncivil. The comment in German: “Ich bin mir sicher, dass die Leute hier alle Tassen im Schrank haben!” Means “I’m sure that the people here all have cups in the cupboard!” It’s a bit odd phrase maybe meaning “have cups in the cabinet” but not insulting? It could be a figurative expression? Might be a sarcastic insult: “Tassen im Schrank haben” maybe means “have cups in the cupboard” but could be idiom for being crazy? Actually German idiom: “jemand hat eine Schraube locker” etc. Not sure. “Tassen im Schrank haben” not common. Could be implying they are all crazy? But it’s not overt insult, possibly mild sarcasm. There’s no profanity, no direct insult, just a vague statement. I think does not meet criteria. So “nicht inzivil”.
Du bist toll, so wie du bist {EM_RED_HEART}️ inzivil nicht inzivil We need to determine if the comment is incivil or not incivil. The comment is “Du bist toll, so wie du bist {EM_RED_HEART}️”. That’s a positive statement, no vulgar language, no insult, etc. So it’s not incivil. We must output JSON exactly: {“obs_class”: “nicht inzivil”}
Das war schön inzivil nicht inzivil We need to determine if the comment “Das war schön” is incivil. It’s a simple positive statement, no incivility. So output {“obs_class”: “nicht inzivil”}
HS inzivil nicht inzivil We have a user: The comment is “HS”. The instruction: Must output JSON exactly {“obs_class”: “inzivil”} or {“obs_class”: “nicht inzivil”}. We need to decide whether “HS” is incivil. “HS” is a short two-letter phrase, possibly stands for “Heil Hitler” or “Hate Speech”? Could be a slang abbreviation. In German context, “HS” often stands for “Hinterwäldler”? Not sure. Could be “HS” as “Hass”? Hard to know. Given minimal content, no obvious incivility. It’s ambiguous. Likely not considered incivil as it’s just two letters with no explicit incivility. So classify as “nicht inzivil”.
Oiiiai oiiiai inzivil nicht inzivil The user gave a task: classify whether the comment is civil or not. The comment is “Oiiiai oiiiai”. That’s just a nonsensical greeting maybe. No vulgarity, no insults, etc. So it’s not incivil. Need output JSON exactly {“obs_class”: “nicht inzivil”}. Make sure no extra spaces? The format requires exact. Probably okay.
Die Weisheit verfolgt dich, aber du bist schneller! inzivil nicht inzivil We have a comment: “Die Weisheit verfolgt dich, aber du bist schneller!” (in German). This is somewhat playful, not insulting. No vulgar language, no insults, no threats. So non-violent, not incivil. It appears to be a compliment or a poetic remark. So classification: “nicht inzivil”. Output JSON exactly.
Wie schön wieder in der Uni zu sein inzivil nicht inzivil We need to classify the comment. The comment is “Wie schön wieder in der Uni zu sein” (German: “How nice to be back at university again”). This is neutral, no incivility. So output {“obs_class”: “nicht inzivil”} in exact JSON.
Ich finde Titten geil nicht inzivil inzivil The user asks to classify the comment. The comment is “Ich finde Titten geil”. This is a vulgar statement about breasts (Titten). It’s sexual, possibly vulgar language. According to definitions, vulgar language is considered incivility. So we should output {“obs_class”: “inzivil”}. We must output only JSON.
bitch, das ist sooo crazyy nicht inzivil inzivil We need to decide if comment is incivil. The comment: “bitch, das ist sooo crazyy”. Contains vulgar word “bitch”, which is an insult, vulgar. So it’s incivil. Output JSON: {“obs_class”: “inzivil”}
Also das ist jetzt schon ein bisschen unfair. Sie hat auch noch andere Facetten, z.B. ist sie auch noch grob inkompetent :) nicht inzivil inzivil We need decide if comment is uncivil. Comment: “Also das ist jetzt schon ein bisschen unfair. Sie hat auch noch andere Facetten, z.B. ist sie auch noch grob inkompetent :)”. It’s saying unfair, also says “grob inkompetent” which is an insult/abwertung. That is a derogatory statement, but is it incivil? It’s an insult, negative evaluation. According to list, “Beleidigung, Profanität” is incivil. “grob inkompetent” is insulting but not profanity. It’s a negative evaluation, possibly disrespect. Likely incivil. So output {“obs_class”: “inzivil”}. Check requirement: JSON exactly.
Woww, geiler scheiß! nicht inzivil inzivil We need decide if comment is incivil. The comment: “Woww, geiler scheiß!” It’s German, meaning “Wow, cool shit!” It’s vulgar? “scheiß” is a vulgar word (shit) but used as exclamation. Contains vulgar language. According to definition, vulgar language considered incivil. So classification: incivil. Output JSON {“obs_class”: “inzivil”}.
Was ein Motherfucker nicht inzivil inzivil We need to classify. The comment: “Was ein Motherfucker”. This contains vulgar profanity “Motherfucker”. It’s vulgar language, insulting. So it’s incivil. Output JSON {“obs_class”: “inzivil”}
Lars Klingbeil ist ein Klassenverräter nicht inzivil inzivil We need to decide if this comment is incivil. The comment: “Lars Klingbeil ist ein Klassenverräter”. Translation: “Lars Klingbeil is a class traitor”. It’s an accusation, calling him a traitor. It’s an insult, possibly a negative judgment. Is it incivil? The criteria: insult, profanity, negative stereotypes, etc. “Klassenverräter” is an insult implying betrayal of class (maybe political class). It’s a vilifying term, could be considered disrespect/abwertung. Likely considered incivil because it’s a derogatory label, accusation of betrayal, insulting. It doesn’t contain profanity, but it is an insult/ad hominem. According to criteria, “Beleidigung, Profanität” includes insult. So yes, incivil. Thus output {“obs_class”: “inzivil”}. Check format: must be exactly JSON with no extra text.
Ekelhaft nicht inzivil inzivil We need to decide if comment “Ekelhaft” is uncivil. It’s a single word meaning “disgusting”. It’s an insult, negative evaluation, possibly rude. It is a pejorative adjective but not necessarily directly insulting a person? Could be describing something as disgusting. In context of a comment, likely it’s a negative judgement. It could be considered incivility because it’s a derogatory term, possibly an insult. The guidelines: Vulgäre, unangemessene Sprache, Beleidigung, etc. “Ekelhaft” is an insult, but not profanity or vulgar. It’s a negative evaluation. Might be considered incivility if directed at a person or group. With no context, it’s a negative term; likely considered disrespectful. I’d classify as incivil? It is a negative judgement, possibly insulting. The guidelines include “Beleidigung, Profanität”. “Ekelhaft” can be considered a mild insult. I’d say incivil. So output JSON {“obs_class”: “inzivil”}
Scheiße, das sieht gut aus! nicht inzivil inzivil We need to decide if the comment is civil or not based on the criteria. The comment: “Scheiße, das sieht gut aus!” It’s German: “Scheiße, das sieht gut aus!” Contains vulgar word “Scheiße” (shit) – a profanity/vulgar language. According to definition, vulgar language is considered incivility. So it’s incivil. Output JSON: {“obs_class”: “inzivil”}
Ich sehe Sie als einen inkompetenten Politiker an nicht inzivil inzivil We need to decide if the comment is “inzivil” according to defined criteria. The comment: “Ich sehe Sie als einen inkompetenten Politiker an” translates to “I see you as an incompetent politician.” That’s an insult? It’s a negative judgment, calling someone incompetent. Is that considered “Beleidigung, Profanität”? It’s a demeaning statement but not necessarily vulgar or threatening. It is a negative assessment, could be considered insulting. According to criteria: “Beleidigung, Profanität” - this is an insult (calling incompetent). Likely considered incivil. Many frameworks would consider personal attack. So classify as “inzivil”. Output JSON.
Du bist doof nicht inzivil inzivil We need to classify “Du bist doof”. This is an insult, vulgar? It’s an insult, disrespect. So it falls under incivility (Beleidigung). So output JSON with {“obs_class”:“inzivil”}. Make sure formatting exactly.
Scheiße wie cool nicht inzivil inzivil The user provided a comment: “Scheiße wie cool”. Need decide if it’s incivil. The comment contains vulgar word “Scheiße” (German for “shit”), which is vulgar language, possibly profanity. It’s an exclamation: “Scheiße wie cool”. It uses a profanity but overall meaning is expressing enthusiasm with a curse word. According to definition, vulgar, inappropriate language qualifies as incivility. So likely classify as “inzivil”. Output JSON {“obs_class”: “inzivil”}. Check: Must output exactly JSON, no extra text.
Du Sau! nicht inzivil inzivil We need to output JSON with obs_class either “inzivil” or “nicht inzivil”. The comment “Du Sau!” is an insult, vulgar, profanity. So it’s incivil. So output {“obs_class”: “inzivil”}. Make sure only JSON, no extra whitespace? They said exact format but can have whitespace. We’ll output exactly.
fucking gute idee nicht inzivil inzivil We need to classify the comment: “fucking gute Idee”. This contains vulgar language: “fucking” is a profanity. So it’s incivil. Return JSON {“obs_class”: “inzivil”}.
Boss Bitch nicht inzivil inzivil We need to classify the comment “Boss Bitch”. Determine if it’s incivil. The comment consists of two words: “Boss Bitch”. “Bitch” can be considered vulgar, possibly insulting. It’s a pejorative term, often used as insult towards women, but also could be used colloquially. It’s a profanity/insult. Likely considered incivil. So output {“obs_class”: “inzivil”}. Check format: must be exactly JSON with double quotes. No extra whitespace? It’s fine.
Das fickt ordentlich was ihr da macht :) nicht inzivil inzivil The user provided a comment: “Das fickt ordentlich was ihr da macht :)”. In German, “fickt” is vulgar profanity, meaning “fuck”. This is vulgar language, possibly insulting. So it’s incivil. The required output JSON: {“obs_class”: “inzivil”}. We must output exactly JSON with no extra text.
krasser Scheiß nicht inzivil inzivil The user asks to classify comment “krasser Scheiß”. Need determine if it’s uncivil. The comment is German: “krasser Scheiß”. Means “crazy sh*t” or “awesome shit”. It’s vulgar profanity “Scheiß” is a swear word (shit). That’s vulgar language. According to definition, vulgar language is incivility. So it should be classified as “inzivil”. Need output JSON exactly.

Alle Kommentare mehrfach codieren

kommentar true_class rep_1 rep_2 rep_3
This is (g)assss{EM_SMILING_FACE_WITH_HEARTEYES}{EM_SMILING_FACE_WITH_HEARTEYES} inzivil nicht inzivil nicht inzivil nicht inzivil
bones are for dogs inzivil nicht inzivil nicht inzivil nicht inzivil
Wie kann man so dumme Ansichten online verbreiten ? Absolute Frechheit wie heutzutage mit dem Wort Forschung umgegangen wird!! inzivil inzivil inzivil inzivil
Was für ein Schwachsinn! inzivil inzivil nicht inzivil inzivil
Ich bin mir sicher, dass die Leute hier alle Tassen im Schrank haben! inzivil inzivil inzivil nicht inzivil
Du bist toll, so wie du bist {EM_RED_HEART}️ inzivil nicht inzivil nicht inzivil nicht inzivil
Du bist dumm inzivil inzivil inzivil inzivil
Linksveriffte Öko-Trulla inzivil inzivil inzivil inzivil
Das war schön inzivil nicht inzivil nicht inzivil nicht inzivil
Clown inzivil inzivil inzivil inzivil
du bist sch***! inzivil inzivil inzivil inzivil
du bist hässlich inzivil inzivil inzivil inzivil
Merz leck Eier inzivil inzivil inzivil inzivil
HS inzivil inzivil nicht inzivil nicht inzivil
Du Hurensohn inzivil inzivil inzivil inzivil
Fuck inzivil inzivil inzivil inzivil
Merz, leck Eier inzivil inzivil inzivil inzivil
Das ist doch Schwachsinn inzivil inzivil inzivil inzivil
Du Fotze inzivil inzivil inzivil inzivil
Oiiiai oiiiai inzivil nicht inzivil nicht inzivil nicht inzivil
Hure! inzivil inzivil inzivil inzivil
Ich finde heute siehst du komisch aus. inzivil inzivil inzivil inzivil
Die Weisheit verfolgt dich, aber du bist schneller! inzivil nicht inzivil nicht inzivil nicht inzivil
Wtf,,, was denkt der wer er ist??? Hasse den so… hoffe er stirbt inzivil inzivil inzivil inzivil
Ganz schön hässlich inzivil inzivil inzivil inzivil
Fake news! inzivil inzivil inzivil inzivil
Ich weiß wo du wohnst inzivil inzivil inzivil inzivil
Ich hau dir auf die Schnauze inzivil inzivil inzivil inzivil
Mutig, wie viel Meinung man mit so wenig Ahnung haben kann. inzivil inzivil inzivil inzivil
Das ist dumm inzivil inzivil inzivil inzivil
Arschloch inzivil inzivil inzivil inzivil
Ja natürlich stimmt das /s inzivil inzivil inzivil inzivil
NA inzivil NA NA NA
Hoffe er wird gekirkt inzivil inzivil inzivil inzivil
Du bist dumm inzivil inzivil inzivil inzivil
Lügner! inzivil inzivil inzivil inzivil
Fotze inzivil inzivil inzivil inzivil
fette Sau inzivil inzivil inzivil inzivil
Bring dich um! inzivil inzivil inzivil inzivil
Ich finde du siehst einfach seltsam aus. inzivil inzivil inzivil inzivil
Interessant, wie überzeugt du davon bist. Ich wünschte, ich hätte auch so viel Selbstvertrauen bei so wenig Grundlage inzivil inzivil nicht inzivil inzivil
I wish you get raped inzivil inzivil inzivil inzivil
Wie schön wieder in der Uni zu sein inzivil nicht inzivil nicht inzivil nicht inzivil
Du F0tze inzivil inzivil inzivil inzivil
Das hast du ja richtig toll gemacht… inzivil nicht inzivil inzivil inzivil
Dumme Nuss inzivil inzivil inzivil inzivil
Nutte inzivil inzivil NA inzivil
nuttig inzivil inzivil inzivil inzivil
Willste dich boxen? inzivil inzivil inzivil inzivil
kys inzivil inzivil inzivil inzivil
Wow, du bist soo mutig dich ungeschminkt zu posten inzivil inzivil inzivil inzivil
was ein Scheiß inzivil inzivil inzivil inzivil
Hässlich inzivil inzivil inzivil inzivil
Halt die Fresse inzivil inzivil inzivil inzivil
Ich stimme dir da gar nicht zu. Weiß nicht wie du drauf kommst aber bist ja auch nicht ganz richtig in der Birne inzivil inzivil inzivil inzivil
Männer sollten keine Rechte haben inzivil inzivil inzivil inzivil
Nazi-Schweine inzivil inzivil inzivil inzivil
AfDler abschieben inzivil inzivil inzivil inzivil
Dem stimme ich zu. nicht inzivil nicht inzivil nicht inzivil nicht inzivil
Ich finde Titten geil nicht inzivil inzivil inzivil inzivil
bitch, das ist sooo crazyy nicht inzivil inzivil inzivil inzivil
Also das ist jetzt schon ein bisschen unfair. Sie hat auch noch andere Facetten, z.B. ist sie auch noch grob inkompetent :) nicht inzivil inzivil inzivil inzivil
Was für ein schönes Foto! nicht inzivil nicht inzivil nicht inzivil nicht inzivil
Hab eine schöne Woche! nicht inzivil nicht inzivil nicht inzivil nicht inzivil
Öffentlichkeit nicht inzivil nicht inzivil nicht inzivil nicht inzivil
Ist das KI? nicht inzivil nicht inzivil nicht inzivil nicht inzivil
Woww, geiler scheiß! nicht inzivil inzivil inzivil inzivil
Das ist falsch. nicht inzivil nicht inzivil nicht inzivil nicht inzivil
Was ein Motherfucker nicht inzivil inzivil inzivil inzivil
Du bist sau cool nicht inzivil nicht inzivil nicht inzivil nicht inzivil
Das Rezept mache ich auf jeden Fall nach!!! Danke für die Inspo! nicht inzivil nicht inzivil nicht inzivil nicht inzivil
Danke und das so wichtig! nicht inzivil nicht inzivil nicht inzivil nicht inzivil
Lars Klingbeil ist ein Klassenverräter nicht inzivil inzivil inzivil inzivil
Bayern gewinnt…Überraschung des Jahres. nicht inzivil nicht inzivil nicht inzivil nicht inzivil
Ekelhaft nicht inzivil inzivil inzivil inzivil
Ich möchte Sommer nicht inzivil nicht inzivil nicht inzivil nicht inzivil
Scheiße, das sieht gut aus! nicht inzivil inzivil inzivil inzivil
Ich sehe Sie als einen inkompetenten Politiker an nicht inzivil inzivil inzivil inzivil
fotzig nicht inzivil inzivil inzivil inzivil
Hab Bock auf Eierkuchen nicht inzivil nicht inzivil nicht inzivil nicht inzivil
Du bist doof nicht inzivil inzivil inzivil inzivil
Scheiße wie cool nicht inzivil inzivil inzivil inzivil
love this! nicht inzivil nicht inzivil nicht inzivil nicht inzivil
Du bist schlagfertig. nicht inzivil nicht inzivil nicht inzivil nicht inzivil
Du Sau! nicht inzivil inzivil inzivil inzivil
fucking gute idee nicht inzivil inzivil inzivil inzivil
Ohh danke für die Hilfe! nicht inzivil nicht inzivil nicht inzivil nicht inzivil
Ohne Worte, wo soll das noch hin führen? nicht inzivil nicht inzivil nicht inzivil nicht inzivil
Ich finde, die CDU versagt auf ganzer Linie! Peinlich! nicht inzivil nicht inzivil nicht inzivil nicht inzivil
NA nicht inzivil NA NA NA
Spannender Beitrag nicht inzivil nicht inzivil nicht inzivil nicht inzivil
Boss Bitch nicht inzivil inzivil inzivil inzivil
Das fickt ordentlich was ihr da macht :) nicht inzivil inzivil inzivil inzivil
Welch ein blumiger Sonnenuntergang! nicht inzivil nicht inzivil nicht inzivil nicht inzivil
Freue mich für dich :) nicht inzivil nicht inzivil nicht inzivil nicht inzivil
das ist so unlustig und komisch nicht inzivil nicht inzivil nicht inzivil nicht inzivil
krasser Scheiß nicht inzivil inzivil inzivil inzivil
Yasss slayyy queeennn nicht inzivil nicht inzivil nicht inzivil nicht inzivil
richtig guter punkt nicht inzivil nicht inzivil nicht inzivil nicht inzivil
Schön, dass ihr Spaß im Urli habt nicht inzivil nicht inzivil nicht inzivil nicht inzivil
I don’t know, 6-7 nicht inzivil nicht inzivil nicht inzivil nicht inzivil
Schöne Berge nicht inzivil nicht inzivil nicht inzivil nicht inzivil

Unterschiedliche Codierungen

kommentar true_class rep_1 rep_2 rep_3
Was für ein Schwachsinn! inzivil inzivil nicht inzivil inzivil
Ich bin mir sicher, dass die Leute hier alle Tassen im Schrank haben! inzivil inzivil inzivil nicht inzivil
HS inzivil inzivil nicht inzivil nicht inzivil
Interessant, wie überzeugt du davon bist. Ich wünschte, ich hätte auch so viel Selbstvertrauen bei so wenig Grundlage inzivil inzivil nicht inzivil inzivil
Das hast du ja richtig toll gemacht… inzivil nicht inzivil inzivil inzivil
Nutte inzivil inzivil NA inzivil

Evaluation einer Klassifikation

Grundlagen

  • Validität: Messvalidität: misst, was sie messen soll; entspricht einer externen Wahrheit
  • Reliabilität: Wiederholte Messungen an denselben Daten liefern ähnliche Ergebnisse (durch verschiedene Codierer: Intercoder-Reliabilität; durch denselben Codierer: Intracoder-Reliabilität)


  • Für die quantitative Auswertung: Vergleich womit?
  • “Gold”- oder “bevorzugter” Standard: Konfusionsmatrix oder Fehlklassifikationsmatrix
  • Bei gleichermaßen validen Messungen: Koinzidenzmatrix

Validität

Konfusionsmatrix oder Fehlklassifikationsmatrix

Gold Standard Negative Gold Standard Positive
Observed Negative True Negatives (TN) False Negatives (FN)
Observed Positive False Positives (FP) True Positives (TP)


  • Accuracy: (TN + TP) / (TN + TP + FP + FN)
  • Recall: TP / (TP + FN)
  • Precision: TP / (TP + FP)
  • F1-Score: 2 / (1/Precision + 1/Recall) (Harmonischer Mittelwert)

Validität

Konfusionsmatrix

.metric .estimator .estimate
accuracy binary 0.75
recall binary 0.84
precision binary 0.75
f_meas binary 0.79
  • event_level = "second": Bezogen auf Erkennung von Inzivilität

Reliabilität

Koinzidenzmatrix

Class A Class B
Class A AA AB
Class B BA BB
  • Übereinstimmung (‘Holsti’): (AA + BB) / (AA + BB + BA + AB)
  • Krippendorffs \(\alpha\): \(1-\frac{D_o}{D_e}\),
    mit \(D_o\) beobachte Abweichungen und \(D_e\) erwartete Abweichungen.

Reliabilität

Koinzidenzmatrix

Übereinstimmung (‘Hosti’): 0.97

Krippendorffs \(\alpha\): 0.93

Ausblick und Fazit

Ausblick und Fazit

  • Aber auch vielfältige Probleme (vor allem, aber nicht nur mit kommerziellen, geschlossenen Modellen) (z.B. Bender et al., 2021; Spirling, 2023; Widder et al., 2024):
  • Forschungsethik, Replizierbarkeit, Nachvollziehbarkeit, Biases, Engergieverbrauch, …
  • Diskussionen für die Inhaltsanalyse im engeren Sinne, für die Gesellschaft im weiteren Sinne
  • Wie bei jeder Inhaltsanalyse: Validierung und Qualitätssicherung notwendig
  • Praktisch: Ermöglicht Ihnen Studien, die vor 3–5 Jahren unmöglich gewesen wären.

Fragen?

Hausaufgabe

  1. Lesen Sie Törnberg (2024).

Nächste Einheit

(Reflexive) Thematic Analysis (with Jo-Ju “Ru” Kao)

Danke

Marko Bachl

Literatur

Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021). On the dangers of stochastic parrots: Can language models be too big? 🦜. Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, 610–623. https://doi.org/10/gh677h
Benoit, K., De Marchi, S., Laver, C., Laver, M., & Ma, J. (2025). Using large language models to analyze political texts through natural language understanding. American Journal of Political Science, (n/a). https://doi.org/10/hbt9sc
Kathirgamalingam, A., Lind, F., Bernhard, J., & Boomgaarden, H. G. (2024). Agree to disagree? Human and LLM coder bias for constructs of marginalization. OSF. https://doi.org/10.31235/osf.io/agpyr
Kravets-Meinke, D., Schmid-Petri, H., Niemann, S., & Schmid, U. (2025). Generative large language models (gLLMs) in content analysis: A practical guide for communication research. arXiv. https://doi.org/10.48550/arXiv.2510.24337
Meltzer, C. E., Scharkow, M., & Jürgens, P. (2025). Beyond the beat: The representation of women in music videos across genres over four decades. Computational Communication Research, 7(1). https://doi.org/10/hbhs4c
Rössler, P. (2017). Inhaltsanalyse (3., überarb. Aufl.). UVK Verlag. https://doi.org/mqx8
Spirling, A. (2023). Why open-source generative AI models are an ethical way forward for science. Nature, 616(7957), 413–413. https://doi.org/10/gsqx6v
Stoll, A., Wilms, L., & Ziegele, M. (2023). Developing an incivility dictionary for German online discussions – a semi-automated approach combining human and artificial knowledge. Communication Methods and Measures, 17(2), 131–149. https://doi.org/10/gsnfdn
Stoll, A., Yu, J., Andrich, A., & Domahidi, E. (2025). Classification bias of LLMs in detecting incivility towards female and male politicians in German social media discourse. Communication Methods and Measures, 19(4), 350–368. https://doi.org/10.1080/19312458.2025.2551693
Stolwijk, S. B., Boukes, M., Yeung, W. N., Liao, Y., Münker, S., Kroon, A. C., & Trilling, D. (2026). Can we use automated approaches to measure the quality of online political discussion? How to (not) measure interactivity, diversity, rationality, and incivility in online comments to the news. Communication Methods and Measures, 20(1), 1–25. https://doi.org/10.1080/19312458.2025.2553300
Törnberg, P. (2024). Best practices for text annotation with large language models. Sociologica, 18(2), 67–85. https://doi.org/10/g9vgm7
Widder, D. G., Whittaker, M., & West, S. M. (2024). Why „open“ AI systems are actually closed, and why this matters. Nature, 635(8040), 827–833. https://doi.org/10/g8xdb3