User:Eortici1/Final project
This page documents my final project, a larger Universal Dependencies corpus and annotation guidelines for Bikol.
The repository for this project is modeled closely after the repositories used by Universal Dependencies (example here).
Contents
Introduction to UD
The Universal Dependencies project was created to "provide a universal inventory of categories and guidelines to facilitate consistent annotation of similar constructions across languages". UD uses a set of typed dependency relations between words, where one word is the head of the sentence and all other words are dependent on another word in the sentence. These dependencies are generally universal and are meant to bring out cross-lingual typological parallels among the world's languages.
For examples of the relations used in the Bikol treebank, see the Annotation guidelines for Bikol.
For examples of annotated Bikol sentences, see the Bikol corpus.
Corpus Information
The annotated corpus can be found in .txt and .conllu (with and without morphological information) in the project repository.
| Sentences | Tokens |
| 103 | 1163 |
The corpus contains extracts from the following sources:
- Various Bikol Wikipedia articles, accessed from 01/05/2017 to 08/05/2017
- Universal Declaration of Human Rights (used English translation for glosses)
Process
In the following section, I address solutions to specific problems I encountered while annotating my corpus.
Pluralization of nouns with 'mga'
To make nouns plural in Bikol, you add the word 'mga' before it.
Halion mo an mga papers sa lamesa ta pigkorehiran na iyan.
When deciding which relation to use for this scenario, my initial thought was to use the @nummod (numeric modifier) relation, because this is a separate word that modifies the quantity of a noun. However, 'mga' has no isolated meaning in Bikol and is therefore not a numeral. So, I decided to use the @det (determiner) relation. In UD's documentation of what constitutes a determiner, it says that quantity determiners (quantifiers) qualify for this relation.
Linker in Modification Structures: 'na'
The biggest challenge of this project (besides the time constraint) was deciding on the relation to use for the linking word, 'na'. In the grammar books I have, this word is simply described as a linking word in modification structures. The word is used very frequently in Bikol in a variety of different modification structures. For this word, I decided to create four language-specific relations. These are outlined below.
amod
May turista na puti nagduman sa San Jose. There is a white tourist that visited San Jose.
This sentence demonstrates an @amod relation, in which an adjectival modifier (puti, or white) modifies the meaning of a nominal (turista, or tourist). The linking word 'na' is assigned an aux:amod relation dependent on the modifier. Language-specific relations are, by convention, composed in this manner (universal:extension). I chose to make this relation an extension of the @aux (auxiliary) relation because the linking word serves as an auxiliary word to the modifier.
For more documentation on this relation, see the aux:amod documentation.
nmod
advmod
nummod
UD Annotation Guidelines
The final component of the project was to create annotation guidelines for the Bikol UD Treebank. This documentation would be useful to anyone who wished to contribute to the Treebank.
Again, I used the same format that is used by the Universal Dependencies Project, using the repository of Swedish guidelines as a guide.
The annotation guidelines can be found in the bcl-dep directory of the project's repository, and feature documentation on 25 of the most commonly used relations I used in my corpus, including the four language-specific relations I mentioned earlier in the Process section.
Evaluation
Parsers are typically evaluated with the LAS (Label Attachment Score) and UAS (Unlabeled Attachment Score) obtained by testing on a separate corpus. However, making an additional, separate corpus would be time consuming. To get around this, I divided the corpus into 10 roughly equal parts (10-11 sentences each). Then, I used each part as a test corpus for two parsers (with and without morphological information) which were trained on the remaining corpus.
The partial corpora, conllu and udpipe files can all be found in the bcl-eval directory of the repository. The data from all the individual tests can be found in the Appendix.
To evaluate the parsers trained on the entire corpus, I took the average of all the UAS/LAS scores obtained on the 10 test sets, ± the standard deviation. The results are displayed below.
| UAS | LAS |
| 60.87 ± 5.67 | 50.00 ± 4.22 |
| UAS | LAS |
| 50.51 ± 5.53 | 36.01 ± 6.59 |
Further Work
Further work on this project would mainly involve increasing the size of the annotated corpus. My corpus of ~100 sentences and ~1200 tokens is very small in comparison to completed UD Treebanks, due to the time constraints of the project and the nature of the material I was working with (Wikipedia articles, no glosses). The most fully developed UD treebanks have 100-200K+ tokens annotated, which would yield much, much more accurate parsers.
Appendix
| UAS with morphology | LAS with morphology | UAS without morphology | LAS without morphology | |
|---|---|---|---|---|
| Test 1 | 55.37 | 45.45 | 48.76 | 31.40 |
| Test 2 | 56.34 | 46.48 | 50.70 | 38.03 |
| Test 3 | 64.29 | 54.46 | 57.14 | 43.75 |
| Test 4 | 65.00 | 49.17 | 53.33 | 36.67 |
| Test 5 | 56.44 | 47.52 | 40.59 | 23.76 |
| Test 6 | 65.52 | 52.41 | 56.55 | 38.62 |
| Test 7 | 52.89 | 44.63 | 46.28 | 31.40 |
| Test 8 | 57.45 | 48.94 | 45.74 | 34.04 |
| Test 9 | 67.65 | 53.92 | 49.02 | 35.29 |
| Test 10 | 67.77 | 57.02 | 57.02 | 47.11 |
