Difference between revisions of "User:Eortici1/Final project"

Revision as of 04:18, 10 May 2017

This page documents my final project, a larger UD corpus and annotation guidelines for Bikol.

The repository for this project is modeled closely after the repositories used by Universal Dependencies.

Issues

• links
• mga

Re-evaluate NA !!!

• aux:amod, aux:nmod, aux:advmod, aux:nummod
• check @dep and anything labeled !*

Corpus Information

This corpus contains extracts from the following sources:

• Universal Declaration of Human Rights (used English translation for glosses)
• Various Bikol Wikipedia articles, accessed from 01/05/2017 to 08/05/2017

Process

UD Annotation Guidelines

The final component of the project was to create annotation guidelines for the Bikol UD Treebank. This documentation would be useful to anyone who wished to contribute to the Treebank.

Again, I used the same format that is used by the Universal Dependencies Project, using the repository of Swedish guidelines as a guide.

The annotation guidelines can be found in the bcl-dep directory of the project's repository, and feature documentation on 25 of the most commonly used relations I used in my corpus, including the four language-specific relations I mentioned earlier in the Process section.

Evaluation

Parsers are typically evaluated with the LAS (Label Attachment Score) and UAS (Unlabeled Attachment Score) obtained by testing on a separate corpus. However, making an additional, separate corpus would be time consuming. To get around this, I divided the corpus into 10 roughly equal parts (10-11 sentences each). Then, I used each part as a test corpus for two parsers (with and without morphological information) which were trained on the remaining corpus.

The partial corpora, conllu and udpipe files can all be found in the bcl-eval directory of the repository. The data from all the individual tests can be found in the Appendix of this Wiki page.

To evaluate the parsers trained on the entire corpus, I took the average of all the UAS/LAS scores obtained on the 10 test sets, ± the standard deviation. The results are displayed below.

Further Work

Further work on this project would mainly involve increasing the size of the annotated corpus. My corpus of ~100 sentences and ~1200 tokens is very small in comparison to completed UD Treebanks, due to the time constraints of the project and the nature of the material I was working with (Wikipedia articles, no glosses). The most fully developed UD treebanks have 100-200K+ tokens annotated, which would yield much, much more accurate parsers.

Appendix