Difference between revisions of "User:Eortici1/Final project"
(→Evaluation) |
|||
| Line 1: | Line 1: | ||
This page documents my final project, a larger UD corpus and annotation guidelines for Bikol. | This page documents my final project, a larger UD corpus and annotation guidelines for Bikol. | ||
| − | |||
| − | |||
Issues | Issues | ||
* links | * links | ||
* mga | * mga | ||
| − | |||
| − | |||
| − | |||
| − | |||
Re-evaluate NA !!! | Re-evaluate NA !!! | ||
| Line 15: | Line 9: | ||
* check @dep and anything labeled !* | * check @dep and anything labeled !* | ||
| − | |||
| − | |||
== Corpus Information == | == Corpus Information == | ||
| Line 29: | Line 21: | ||
|1163 | |1163 | ||
|} | |} | ||
| + | |||
This corpus contains extracts from the following sources: | This corpus contains extracts from the following sources: | ||
* [http://www.ohchr.org/EN/UDHR/Pages/Language.aspx?LangID=bkl Universal Declaration of Human Rights] (used [http://www.ohchr.org/EN/UDHR/Pages/Language.aspx?LangID=eng English translation] for glosses) | * [http://www.ohchr.org/EN/UDHR/Pages/Language.aspx?LangID=bkl Universal Declaration of Human Rights] (used [http://www.ohchr.org/EN/UDHR/Pages/Language.aspx?LangID=eng English translation] for glosses) | ||
| − | * [http://bcl.wikipedia.org Various Bikol Wikipedia articles], accessed from 01/05/2017 to | + | * [http://bcl.wikipedia.org Various Bikol Wikipedia articles], accessed from 01/05/2017 to 08/05/2017 |
== Evaluation == | == Evaluation == | ||
| + | |||
| + | Parsers are typically evaluated with the LAS (Label Attachment Score) and UAS (Unlabeled Attachment Score) obtained by testing on a separate corpus. However, making an additional, separate corpus would be time consuming. To get around this, I divided the corpus into 10 roughly equal parts (10-11 sentences each). Then, I used each part as a test corpus for two parsers (with and without morphological information) which were trained on the remaining corpus. | ||
| + | |||
| + | The partial corpora, conllu and udpipe files can all be found in the [https://github.swarthmore.edu/eortici1/UD_Bikol/tree/master/bcl-eval bcl-eval directory] of the repository. The data from all the individual tests can be found in the Appendix of this Wiki page. | ||
| + | |||
| + | To evaluate the [https://github.swarthmore.edu/eortici1/UD_Bikol/tree/master/dev parsers trained on the entire corpus], I took the average of all the UAS/LAS scores obtained on the 10 test sets, ± the standard deviation. The results are displayed below. | ||
{| class="wikitable" | {| class="wikitable" | ||
Revision as of 04:00, 10 May 2017
This page documents my final project, a larger UD corpus and annotation guidelines for Bikol.
Issues
- links
- mga
Re-evaluate NA !!!
- aux:amod, aux:nmod, aux:advmod, aux:nummod
- check @dep and anything labeled !*
Corpus Information
| Sentences | Forms |
| 103 | 1163 |
This corpus contains extracts from the following sources:
- Universal Declaration of Human Rights (used English translation for glosses)
- Various Bikol Wikipedia articles, accessed from 01/05/2017 to 08/05/2017
Evaluation
Parsers are typically evaluated with the LAS (Label Attachment Score) and UAS (Unlabeled Attachment Score) obtained by testing on a separate corpus. However, making an additional, separate corpus would be time consuming. To get around this, I divided the corpus into 10 roughly equal parts (10-11 sentences each). Then, I used each part as a test corpus for two parsers (with and without morphological information) which were trained on the remaining corpus.
The partial corpora, conllu and udpipe files can all be found in the bcl-eval directory of the repository. The data from all the individual tests can be found in the Appendix of this Wiki page.
To evaluate the parsers trained on the entire corpus, I took the average of all the UAS/LAS scores obtained on the 10 test sets, ± the standard deviation. The results are displayed below.
| UAS | LAS |
| 60.87 ± 5.67 | 50.00 ± 4.22 |
| UAS | LAS |
| 50.51 ± 5.53 | 36.01 ± 6.59 |
UD Annotation Guidelines
Appendix
Evaluation
| UAS with morphology | LAS with morphology | UAS without morphology | LAS without morphology | |
|---|---|---|---|---|
| Test 1 | 55.37 | 45.45 | 48.76 | 31.40 |
| Test 2 | 56.34 | 46.48 | 50.70 | 38.03 |
| Test 3 | 64.29 | 54.46 | 57.14 | 43.75 |
| Test 4 | 65.00 | 49.17 | 53.33 | 36.67 |
| Test 5 | 56.44 | 47.52 | 40.59 | 23.76 |
| Test 6 | 65.52 | 52.41 | 56.55 | 38.62 |
| Test 7 | 52.89 | 44.63 | 46.28 | 31.40 |
| Test 8 | 57.45 | 48.94 | 45.74 | 34.04 |
| Test 9 | 67.65 | 53.92 | 49.02 | 35.29 |
| Test 10 | 67.77 | 57.02 | 57.02 | 47.11 |
