Bases para evaluar la anotación de corpus de emociones espontáneas
ISSN: 1135-5948
Argitalpen urtea: 2009
Zenbakia: 43
Orrialdeak: 66-73
Mota: Artikulua
Beste argitalpen batzuk: Procesamiento del lenguaje natural
Laburpena
In this paper we propose the use of several statistical coefficients and information sources to create an appropriate basis for the evaluation of the annotations of non-acted emotional corpora. Experimental results over a corpus of spontaneous emotions show that traditional interpretations that can be found in the literature are not valid for this type of corpora in which the neutral category is inherently predominant. Our proposals provide sufficient information to obtain reliable interpretations of acceptability.
Erreferentzia bibliografikoak
- Artstein, Ron y Massimo Poesio. 2005. kappa3 = alpha (or beta). Informe tecnico, University of Essex.
- Callejas, Zoraida y Ramon Lopez-Cozar. 2008. Relations between de-facto criteria in the evaluation of a spoken dialogue system. Speech Communication, 50(89):646–665.
- Cicchetti, Domenic V. y Alvan R. Feinstein. 1990. High agreement but low Kappa: II. Resolving the paradoxes. Journal of Clinical Epidemiology, 43(6):551–558.
- Cohen, Jacob. 1968. Weighted kappa: nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin, 70(4):213–220. Davies, Mark y Joseph L. Fleiss. 1982. Measuring agreement for multinomial data. Biometrics, 38(4):1047–1051.
- Douglas-Cowie, Ellen, Nick Campbell, Roddy Cowie, y Peter Roach. 2003. Emotional speech: towards a new generation of databases. Speech Communication, 40:33–60.
- Feinstein, Alvan R. y Domenic V. Cicchetti. 1990. High agreement but low Kappa: I. The problems of two paradoxes. Journal of Clinical Epidemiology, 43(6):543–549.
- Fleiss, Joseph L. 1971. Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5):378–382.
- Fleiss, Joseph L. y Jacob Cohen. 1973. The equivalence of weighted kappa and the interclass correlation coefficient as measures of reliability. Educational and Psychological Measurement, 33:613–619.
- Johnstone, T. 1996. Emotional speech elicited using computer games. En Proceedings of the 4th International Conference on Spoken Language Processing (ICSLP 1996), volumen 3, paginas 1985– 1988, Philadelphia, PA.
- Krippendorff, Klaus. 2003. Content Analysis: An Introduction to its Methodology. Sage Publications, Inc.
- Landis, J. R. y G. G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics, 33:159–174.
- Lantz, Charles A. y Elliott Nebenzahl. 1996. Behavior and interpretation of the κ statistic: Resolution of the two paradoxes. Journal of Clinical Epidemiology, 49(4):431–434.
- Manning, Christopher D. y Hinrich Schutze. 2000. Foundations of statistical natural language processing. The MIT Press.
- Nesterenko, Irina y Stephane Rauzy. 2007. On the use of probabilistic grammars in speech annotation and segmentation tasks. En Proceedings of SPECOM 2007, Moscu, Rusia.
- Plutchik, Robert. 1980. EMOTION: A psychoevolutionary synthesis. Harper and Row publishers.
- Russell, J A. 1980. A circumplex model of affect. Journal of Personality and Social Psychology, 39:1161–1178.
- Shannon, C. E. 1948. A mathematical theory of communication. Bell System Technical Journal, 27:379–423.
- Steidl, Stefan, Michael Levit, Anton Batliner, Elmar Noth, y Heinrich Niemann. 2005. Of all things the measure is man. automatic classification of emotions and inter-labeler consistency. En Proceedings of ICASSP 2005, paginas 317–320, Philadelphia, USA.
- Vogt, Thurid y Elisabeth Andre. 2005. Comparing feature sets for acted and spontaneous speech in view of automatic emotion recognition. En Proceedings of IEEE International Conference on Multimedia and Expo, paginas 474–477.
- Wilting, Janneke, Emiel Krahmer, y Marc Swerts. 2006. Real vs. acted emotional speech. En Proceedings of Interspeech 2006, paginas 805–808, Pittsburgh PA, USA