Current NLP systems largely rely on single “gold” labels, overlooking the diversity of human perspectives inherent in language interpretation. This simplification obscures demographic and sociocultural nuances and limits the ability of models to represent minority viewpoints. In this project, we investigate how multiple perspectives can be systematically integrated across the NLP pipeline, from data collection and modeling to evaluation and explanation.
We examine the requirements for perspectivist corpus design, including annotator diversity, disaggregated labeling, and the role of annotation paradigms in shaping perspective representation. Furthermore, we explore modeling approaches that incorporate user and demographic information, and assess how disagreement can be distinguished from meaningful variation in viewpoints. With the rise of Large Language Models, we also study whether persona-based prompting can simulate human-like annotations and to what extent such generated perspectives align with real annotator behavior.
Finally, we address the challenge of evaluating perspectivist systems, proposing alternatives to majority-based metrics that better capture individual viewpoints, and investigate how explanations can complement labels to more fully represent perspectives. Our findings aim to advance the development of NLP systems that more accurately reflect the diversity of human interpretation.
Beyond exploring disaggregated labels for modeling perspectives, annotator rationales provide fine-grained signals of individual perspectives. In this work, we propose a framework for jointly modeling annotator-specific label prediction and corresponding explanations, fine-tuned on the annotators’ provided rationales. Using a dataset with disaggregated natural language inference (NLI) annotations and annotator-provided explanations, we condition predictions on both annotator identity and demographic metadata through a representation-level User Passport mechanism. We further introduce two explainer architectures: a post-hoc prompt-based explainer and a prefixed bridge explainer that transfers annotator-conditioned classifier representations directly into a generative model. This design enables explanation generation aligned with individual annotator perspectives. Our results show that incorporating explanation modeling substantially improves predictive performance over a baseline annotator-aware classifier, with the prefixed bridge approach achieving more stable label alignment and higher semantic consistency, while the post-hoc approach yields stronger lexical similarity. These findings indicate that modeling explanations as expressions of fine-grained perspective provides a richer and more faithful representation of disagreement. The proposed approaches advance perspectivist modeling by integrating annotator-specific rationales into both predictive and generative components.
@inproceedings{sarumi-etal-2026-fine,title={Fine-Grained Perspectives: Modeling Explanations with Annotator-Specific Rationales },author={Sarumi, Olufunke O. and Welch, Charles and Braun, Daniel},booktitle={Proceedings of the the fifth edition of NLPerspectives},month=may,year={2026},pages={66--75},address={Palma, Mallorca, Spain},publisher={European Language Resources Association (ELRA)},editor={Dudy, Shiran and Abercrombie, Gavin and Basile, Valerio and Leonardelli, Elisa and Frenda, Simona},doi={10.63317/4s7kqwvy5i6x},}
Recent works in Natural Language Processing have focused on developing methods to model annotator perspectives within subjective datasets, aiming to capture opinion diversity. This has led to the development of various approaches that learn from disaggregated labels, leading to the question of what factors most influence the performance of these models. While dataset characteristics are a critical factor, the choice of evaluation metric is equally crucial, especially given the fluid and evolving concept of perspectivism. A model considered state-of-the-art under one evaluation scheme may not maintain its top-tier status when assessed with a different set of metrics, highlighting a potential challenge between model performance and the evaluation framework. This paper presents a performance analysis of annotator modeling approaches using the evaluation metrics of the 2025 Learning With Disagreement (LeWiDi) shared task and additional metrics. We evaluate five annotator-aware models under the same configurations. Our findings demonstrate a significant metric-induced shift in model rankings. Across four datasets, no single annotator modeling approach consistently outperformed others using a single metric, revealing that the “best” model is highly dependent on the chosen evaluation metric. This study systematically shows that evaluation metrics are not agnostic in the context of perspectivist model assessment.
@inproceedings{sarumi-etal-2025-nlp,title={{NLP}-{R}es{T}eam at {L}e{W}i{D}i-2025:Performance Shifts in Perspective Aware Models based on Evaluation Metrics},author={Sarumi, Olufunke O. and Welch, Charles and Braun, Daniel},editor={Abercrombie, Gavin and Basile, Valerio and Frenda, Simona and Tonelli, Sara and Dudy, Shiran},booktitle={Proceedings of the The 4th Workshop on Perspectivist Approaches to NLP},month=nov,year={2025},address={Suzhou, China},publisher={Association for Computational Linguistics},url={https://aclanthology.org/2025.nlperspectives-1.19/},doi={10.18653/v1/2025.nlperspectives-1.19},pages={219--227},isbn={979-8-89176-350-0},}
Olufunke O.
Sarumi, Charles
Welch, Daniel
Braun, and
1 more author
In Proceedings of the 8th International Conference on Natural Language and Speech Processing (ICNLSP-2025). Association for Computational Linguistics, 2025
In this work, we explore the capability of Large Language Models (LLMs) to annotate hate speech and abusiveness while considering predefined annotator personas within the strong-to-weak data perspectivism spectra. We evaluated LLM-generated annotations against existing annotator modeling techniques for perspective modeling. Our findings show that LLMs selectively use demographic attributes from the personas. We identified prototypical annotators, with persona features that show varying degrees of alignment with the original human annotators. Within the data perspectivism paradigm, annotator modeling techniques that do not explicitly rely on annotator information performed better under weak data perspectivism compared to both strong data perspectivism and human annotations, suggesting LLM-generated views tend towards aggregation despite subjective prompting. However, for more personalized datasets tailored to strong perspectivism, the performance of LLM annotator modeling approached, but did not exceed, human annotators.
@inproceedings{sarumi-etal-2025-impact,title={The Impact of Annotator Personas on {LLM} Behavior Across the Perspectivism Spectrum},author={Sarumi, Olufunke O. and Welch, Charles and Braun, Daniel and Schl{\"o}tterer, J{\"o}rg},editor={Abbas, Mourad and Yousef, Tariq and Galke, Lukas},booktitle={Proceedings of the 8th International Conference on Natural Language and Speech Processing (ICNLSP-2025)},month=aug,year={2025},address={Southern Denmark University, Odense, Denmark},publisher={Association for Computational Linguistics},url={https://aclanthology.org/2025.icnlsp-1.14/},pages={121--136},doi={10.48550/arXiv.2508.17164},}
