Difference between revisions of "Evenki and English/Structural transfer"
From LING073
(Created page with "=== Pre=Evaluation === {| class="wikitable" |- ! WER Test !! PER Test !! Coverage Test |- | Output || Output || Output |- |} Before you start development of structural...") |
|||
| (10 intermediate revisions by one other user not shown) | |||
| Line 1: | Line 1: | ||
| − | === Pre | + | === Pre-Evaluation === |
| − | + | Note: Due to the previously mentioned orthography issues with the Evenki transducer, our test results are artificially low. Since the transducer is imported from apertium and we did not created, this currently not a problem we can fix. However, our work could be applied in future work on Evenki if this issue were to be fixed. | |
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | + | ------------------------------------------------------- | |
| + | Number of words in reference: 20 | ||
| + | Number of words in test: 22 | ||
| + | Number of unknown words (marked with a star) in test: 20 | ||
| + | Percentage of unknown words: 90.91 % | ||
| − | + | Results when removing unknown-word marks (stars) | |
| + | ------------------------------------------------------- | ||
| + | Edit distance: 22 | ||
| + | Word error rate (WER): 110.00 % | ||
| + | Number of position-independent correct words: 0 | ||
| + | Position-independent word error rate (PER): 110.00 % | ||
| − | |||
=== Structural Transfer Rules === | === Structural Transfer Rules === | ||
| − | + | We added a structural transfer rule to account for the different subject complement verb orders in Evenki and English. | |
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
=== Mode Outputs For Each Sentence=== | === Mode Outputs For Each Sentence=== | ||
| − | |||
| − | |||
{| class="wikitable" | {| class="wikitable" | ||
|- | |- | ||
! sentence !! tagger !! lex !! transfer !! evn-eng | ! sentence !! tagger !! lex !! transfer !! evn-eng | ||
|- | |- | ||
| − | | | + | | Тадук нунгартын аичимни чатын. || ^*Тадук$ ^*нунгартын$ ^*аичимни$ ^*чатын$. || ^*Тадук/*Тадук$ ^*нунгартын/*нунгартын$ ^*аичимни/*аичимни$ ^*чатын/*чатын$. || ^*Тадук$ ^*нунгартын$ ^*аичимни$ ^*чатын$. || *Тадук *нунгартын *аичимни *чатын. |
| − | | | ||
| − | | | ||
| − | |||
| − | |||
|- | |- | ||
|} | |} | ||
| Line 44: | Line 33: | ||
| − | + | ------------------------------------------------------- | |
| − | + | Number of words in reference: 20 | |
| − | + | Number of words in test: 22 | |
| − | + | Number of unknown words (marked with a star) in test: 20 | |
| − | + | Percentage of unknown words: 90.91 % | |
| − | + | ||
| − | |} | + | Results when removing unknown-word marks (stars) |
| + | ------------------------------------------------------- | ||
| + | Edit distance: 22 | ||
| + | Word error rate (WER): 110.00 % | ||
| + | Number of position-independent correct words: 0 | ||
| + | Position-independent word error rate (PER): 110.00 % | ||
| + | |||
| + | Note: Nothing improved when implementing structural transfer but this is likely due to the same issue from the transducer preventing words in both monolingual and bilingual dictionaries from mapping to any translation. Should that error be resolved significant improvement would likely be seen however fixing that issue is currently outside of the scope of our capabilities. | ||
| + | |||
| + | {{comment|No, the rules were just put in the wrong file}} | ||
| − | [[Category:Sp23_StructuralTransfer ]] | + | [[Category:Sp23_StructuralTransfer ]][[Category:evenki]][[Category:english]] |
Latest revision as of 18:34, 11 June 2023
Contents
Pre-Evaluation
Note: Due to the previously mentioned orthography issues with the Evenki transducer, our test results are artificially low. Since the transducer is imported from apertium and we did not created, this currently not a problem we can fix. However, our work could be applied in future work on Evenki if this issue were to be fixed.
------------------------------------------------------- Number of words in reference: 20 Number of words in test: 22 Number of unknown words (marked with a star) in test: 20 Percentage of unknown words: 90.91 %
Results when removing unknown-word marks (stars) ------------------------------------------------------- Edit distance: 22 Word error rate (WER): 110.00 % Number of position-independent correct words: 0 Position-independent word error rate (PER): 110.00 %
Structural Transfer Rules
We added a structural transfer rule to account for the different subject complement verb orders in Evenki and English.
Mode Outputs For Each Sentence
| sentence | tagger | lex | transfer | evn-eng |
|---|---|---|---|---|
| Тадук нунгартын аичимни чатын. | ^*Тадук$ ^*нунгартын$ ^*аичимни$ ^*чатын$. | ^*Тадук/*Тадук$ ^*нунгартын/*нунгартын$ ^*аичимни/*аичимни$ ^*чатын/*чатын$. | ^*Тадук$ ^*нунгартын$ ^*аичимни$ ^*чатын$. | *Тадук *нунгартын *аичимни *чатын. |
Post-Evaluation
------------------------------------------------------- Number of words in reference: 20 Number of words in test: 22 Number of unknown words (marked with a star) in test: 20 Percentage of unknown words: 90.91 %
Results when removing unknown-word marks (stars) ------------------------------------------------------- Edit distance: 22 Word error rate (WER): 110.00 % Number of position-independent correct words: 0 Position-independent word error rate (PER): 110.00 %
Note: Nothing improved when implementing structural transfer but this is likely due to the same issue from the transducer preventing words in both monolingual and bilingual dictionaries from mapping to any translation. Should that error be resolved significant improvement would likely be seen however fixing that issue is currently outside of the scope of our capabilities.
No, the rules were just put in the wrong file
