Quantitative Assessment of Intersectional Empathetic Bias and Understanding
| Autoři | |
|---|---|
| Rok publikování | 2024 |
| Druh | Článek ve sborníku |
| Konference | Recent Advances in Slavonic Natural Language Processing, RASLAN 2024 |
| Fakulta / Pracoviště MU | |
| Citace | |
| www | Konferenční sborník |
| Klíčová slova | large language models, artificial empathy, evaluation, bias |
| Popis | A growing amount of critique concerns the current operationalizations of empathy based on loose definitions of the construct. Such definitions negatively affect dataset quality, model robustness, and evaluation reliability. We propose an empathy evaluation framework that operationalizes empathy close to its psychological origins. The framework measures the variance in responses of LLMs to prompts using existing metrics for empathy and emotional valence. We introduce the variance by varying social biases in the prompts, which affect context understanding and thus impact empathetic understanding. Our method maintains high control over the prompt generation, ensuring the theoretical validity of the constructs in the prompt dataset. Also, it makes high-quality translation, especially into languages with little to no way of evaluating empathy or bias, such as the Slavonic family, more manageable. Using chosen LLMs and various prompt types, we demonstrate the empathy evaluation with the framework, including multiple-choice answers and free generation. The measured variance in our initial evaluation sample is small, and we were unable to find the expected differences between the empathetic understanding given the differences in context for distinct social groups. However, the models showed significant alterations in their reasoning chains that were needed to capture the relatively subtle changes in the prompts. This provides the basis for future research into the construction of the evaluation sample and statistical methods for measuring the results. |
| Související projekty: |