Difference between revisions of "User:Eortici1/Final project"

Revision as of 04:35, 10 May 2017

This page documents my final project, a larger Universal Dependencies corpus and annotation guidelines for Bikol.

The repository for this project is modeled closely after the repositories used by Universal Dependencies (example here).

Introduction to UD

The Universal Dependencies project was created to "provide a universal inventory of categories and guidelines to facilitate consistent annotation of similar constructions across languages". UD uses a set of typed dependency relations between words, where one word is the head of the sentence and all other words are dependent on another word in the sentence. These dependencies are generally universal and are meant to bring out cross-lingual typological parallels among the world's languages.

For examples of the relations used in the Bikol treebank, see the Annotation guidelines for Bikol.

For examples of annotated Bikol sentences, see the Bikol corpus.

Corpus Information

The annotated corpus can be found in .txt and .conllu (with and without morphological information) in the project repository.

The corpus contains extracts from the following sources:

• Various Bikol Wikipedia articles, accessed from 01/05/2017 to 08/05/2017
• Universal Declaration of Human Rights (used English translation for glosses)

Process

In the following section, I address solutions to specific problems I encountered while annotating my corpus.

Pluralization of nouns with 'mga'

Linker in Modification Structures: 'na'

• aux:amod, aux:nmod, aux:advmod, aux:nummod
• check @dep and anything labeled !*

UD Annotation Guidelines

The final component of the project was to create annotation guidelines for the Bikol UD Treebank. This documentation would be useful to anyone who wished to contribute to the Treebank.

Again, I used the same format that is used by the Universal Dependencies Project, using the repository of Swedish guidelines as a guide.

The annotation guidelines can be found in the bcl-dep directory of the project's repository, and feature documentation on 25 of the most commonly used relations I used in my corpus, including the four language-specific relations I mentioned earlier in the Process section.

Evaluation

Parsers are typically evaluated with the LAS (Label Attachment Score) and UAS (Unlabeled Attachment Score) obtained by testing on a separate corpus. However, making an additional, separate corpus would be time consuming. To get around this, I divided the corpus into 10 roughly equal parts (10-11 sentences each). Then, I used each part as a test corpus for two parsers (with and without morphological information) which were trained on the remaining corpus.

The partial corpora, conllu and udpipe files can all be found in the bcl-eval directory of the repository. The data from all the individual tests can be found in the Appendix.

To evaluate the parsers trained on the entire corpus, I took the average of all the UAS/LAS scores obtained on the 10 test sets, ± the standard deviation. The results are displayed below.

Further Work

Further work on this project would mainly involve increasing the size of the annotated corpus. My corpus of ~100 sentences and ~1200 tokens is very small in comparison to completed UD Treebanks, due to the time constraints of the project and the nature of the material I was working with (Wikipedia articles, no glosses). The most fully developed UD treebanks have 100-200K+ tokens annotated, which would yield much, much more accurate parsers.

Appendix