Difference between revisions of "Kickapoo and English/Structural transfer"
From LING073
(→Pre-evaluation) |
|||
| Line 3: | Line 3: | ||
apertium-eval-translator-line -r abc.tests.txt -t xyz-abc.tests.txt | apertium-eval-translator-line -r abc.tests.txt -t xyz-abc.tests.txt | ||
| − | + | Baseline measurements before implementing any structural transfer rules using the transfer test sentences | |
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | + | = Pre-evaluation == | |
| − | ------------------------- | + | |
| − | + | Before implementing structural transfer rules, we evaluated the system on the short sentence test corpus using ''apertium-eval-translator-line''. | |
| − | Word | + | |
| − | + | === Test setup === | |
| − | Position-independent word | + | |
| + | Reference file: | ||
| + | <pre> | ||
| + | /home/kpeng1/ling073/ling073-kic-eng-corpus/eng.tests.txt | ||
| + | </pre> | ||
| + | |||
| + | Test translation file: | ||
| + | <pre> | ||
| + | /home/kpeng1/ling073/ling073-kic-eng/kic-eng.tests.txt | ||
| + | </pre> | ||
| + | |||
| + | Command used: | ||
| + | <pre> | ||
| + | apertium-eval-translator-line -r /home/kpeng1/ling073/ling073-kic-eng-corpus/eng.tests.txt -t /home/kpeng1/ling073/ling073-kic-eng/kic-eng.tests.txt | ||
| + | </pre> | ||
| + | |||
| + | === Results === | ||
| + | |||
| + | {| class="wikitable" | ||
| + | ! Metric | ||
| + | ! Value | ||
| + | |- | ||
| + | | Number of words in reference | ||
| + | | 41 | ||
| + | |- | ||
| + | | Number of words in test | ||
| + | | 25 | ||
| + | |- | ||
| + | | Number of unknown words in test | ||
| + | | 11 | ||
| + | |- | ||
| + | | Coverage | ||
| + | | 56.00% | ||
| + | |- | ||
| + | | Word Error Rate (WER) | ||
| + | | 100.00% | ||
| + | |- | ||
| + | | Position-independent Error Rate (PER) | ||
| + | | 100.00% | ||
| + | |} | ||
| + | |||
| + | Coverage was calculated from the unknown-word rate reported by the evaluator: | ||
| + | |||
| + | <pre> | ||
| + | Coverage = 100% - 44.00% = 56.00% | ||
| + | </pre> | ||
| + | |||
| + | === Interpretation === | ||
| + | |||
| + | The baseline system has very low performance on this test set. Almost half of the words in the test translation are unknown (44.00%), which gives a coverage of only 56.00%. In addition, both WER and PER are 100.00%, meaning that the current output does not match the reference translation either in word choice or in word order. | ||
| + | |||
| + | This suggests that before and during structural transfer development, we need to: | ||
| + | * add missing stems to the bilingual dictionary | ||
| + | * add missing lexical items to the relevant monolingual dictionaries | ||
| + | * improve analysis coverage on the transfer test sentences | ||
| + | * implement structural transfer rules for the contrastive grammar items being tested | ||
[[Category:Sp26_StructuralTransfer]] | [[Category:Sp26_StructuralTransfer]] | ||
[[Category:Kickapoo]] | [[Category:Kickapoo]] | ||
[[Category:English]] | [[Category:English]] | ||
Revision as of 12:15, 4 May 2026
Pre-evaluation
apertium-eval-translator-line -r abc.tests.txt -t xyz-abc.tests.txt
Baseline measurements before implementing any structural transfer rules using the transfer test sentences
Pre-evaluation =
Before implementing structural transfer rules, we evaluated the system on the short sentence test corpus using apertium-eval-translator-line.
Test setup
Reference file:
/home/kpeng1/ling073/ling073-kic-eng-corpus/eng.tests.txt
Test translation file:
/home/kpeng1/ling073/ling073-kic-eng/kic-eng.tests.txt
Command used:
apertium-eval-translator-line -r /home/kpeng1/ling073/ling073-kic-eng-corpus/eng.tests.txt -t /home/kpeng1/ling073/ling073-kic-eng/kic-eng.tests.txt
Results
| Metric | Value |
|---|---|
| Number of words in reference | 41 |
| Number of words in test | 25 |
| Number of unknown words in test | 11 |
| Coverage | 56.00% |
| Word Error Rate (WER) | 100.00% |
| Position-independent Error Rate (PER) | 100.00% |
Coverage was calculated from the unknown-word rate reported by the evaluator:
Coverage = 100% - 44.00% = 56.00%
Interpretation
The baseline system has very low performance on this test set. Almost half of the words in the test translation are unknown (44.00%), which gives a coverage of only 56.00%. In addition, both WER and PER are 100.00%, meaning that the current output does not match the reference translation either in word choice or in word order.
This suggests that before and during structural transfer development, we need to:
- add missing stems to the bilingual dictionary
- add missing lexical items to the relevant monolingual dictionaries
- improve analysis coverage on the transfer test sentences
- implement structural transfer rules for the contrastive grammar items being tested
