Extrinsic Corpus Evaluation with a Collocation Dictionary Task

This publication doesn't include Faculty of Arts. It includes Faculty of Informatics. Official publication website can be found on muni.cz.

Authors

KILGARRIFF Adam RYCHLÝ Pavel JAKUBÍČEK Miloš KOVÁŘ Vojtěch BAISA Vít KOCINCOVÁ Lucia

Type Article in Proceedings
Conference Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14)
MU Faculty or unit

Faculty of Informatics

Citation
WWW http://www.lrec-conf.org/proceedings/lrec2014/summaries/52.html
Field Informatics
Keywords corpus; evaluation; collocation
Description The NLP researcher or application-builder often wonders ``what corpus should I use, or should I build one of my own? If I build one of my own, how will I know if I have done a good job?'' Currently there is very little help available for them. They are in need of a framework for evaluating corpora. We develop such a framework, in relation to corpora which aim for good coverage of `general language'. The task we set is automatic creation of a publication-quality collocations dictionary. For a sample of 100 headwords of Czech and 100 of English, we identify a gold standard dataset of (ideally) all the collocations that should appear for these headwords in such a dictionary. The datasets are being made available alongside this paper. We then use them to determine precision and recall for a range of corpora, with a range of parameters.
Related projects: