11 Tutorial V: Performance evaluation

11.1 Virtual RStudio

https://simplevm-proxy.bihealth.org/causalmink_100/

11.2 Materials for this session

Session 2: Presentation on validation methods
Chapter Validation and Best Practices in Van Atteveldt et al. (2022)
Get started with {yardstick}: Package website
Explainable Detection of Online Sexism (EDOS)
- Github
- Paper
- Challenge (Part of SemEval 2023)
Model: ministral-3

11.3 Code snippets to select cases and write to csv file

d |> 
  subset(label_sexist != classification_sexist, # Select cases which disagree in their classification
         select = c(text, label_sexist, classification_sexist, reason_sexist)) |>  # Select relevant columns of the data
  write.csv2("errors_sexist.csv") # Write data to file; .csv2 is the format that "German Excel" can open directly

d |> 
  subset(label_sexist == classification_sexist, # Select cases which agree in their classification
         select = c(text, label_sexist, classification_sexist, reason_sexist)) |>  # Select relevant columns of the data
  write.csv2("agreements_sexist.csv") # Write data to file; .csv2 is the format that "German Excel" can open directly

d |> 
  subset(
    (label_category == "2. derogation" &  # Select cases which confuse categories 2 & 3
      classification_category == "3. animosity") |
      (label_category == "3. animosity" & 
         classification_category == "2. derogation"),
    select = c(text, label_category, classification_category, reason_category)  # Select relevant columns of the data
  ) |> 
  write.csv2("category_confusion_23.csv")  # Write data to file; .csv2 is the format that "German Excel" can open directly