Difference between revisions of "Kickapoo and English/Structural transfer"

From LING073
Jump to: navigation, search
(→‎Pre-evaluation)
Line 1: Line 1:
 
== Pre-evaluation ==
 
== Pre-evaluation ==
−
 
−
apertium-eval-translator-line -r abc.tests.txt -t xyz-abc.tests.txt
 
−
 
−
Baseline measurements before implementing any structural transfer rules using the transfer test sentences
 
−
 
−
= Pre-evaluation ==
 
−
 
−
Before implementing structural transfer rules, we evaluated the system on the short sentence test corpus using ''apertium-eval-translator-line''.
 
−
 
−
=== Test setup ===
 
−
 
−
Reference file:
 
−
<pre>
 
−
/home/kpeng1/ling073/ling073-kic-eng-corpus/eng.tests.txt
 
−
</pre>
 
−
 
−
Test translation file:
 
−
<pre>
 
−
/home/kpeng1/ling073/ling073-kic-eng/kic-eng.tests.txt
 
−
</pre>
 
  
 
Command used:
 
Command used:
Line 57: Line 37:
 
</pre>
 
</pre>
  
−
=== Interpretation ===
+
== After lexical additions ==
  
−
The baseline system has very low performance on this test set. Almost half of the words in the test translation are unknown (44.00%), which gives a coverage of only 56.00%. In addition, both WER and PER are 100.00%, meaning that the current output does not match the reference translation either in word choice or in word order.
+
{| class="wikitable"
−
 
+
! Metric
−
This suggests that before and during structural transfer development, we need to:
+
! Value
−
* add missing stems to the bilingual dictionary
+
|-
−
* add missing lexical items to the relevant monolingual dictionaries
+
| Number of words in reference
−
* improve analysis coverage on the transfer test sentences
+
| 41
−
* implement structural transfer rules for the contrastive grammar items being tested
+
|-
 +
| Number of words in test
 +
| 27
 +
|-
 +
| Number of unknown words in test
 +
| 0
 +
|-
 +
| Coverage
 +
| 100.00%
 +
|-
 +
| Word Error Rate (WER)
 +
| 97.56%
 +
|-
 +
| Position-independent Error Rate (PER)
 +
| 97.56%
 +
|}
  
 
[[Category:Sp26_StructuralTransfer]]
 
[[Category:Sp26_StructuralTransfer]]
 
[[Category:Kickapoo]]
 
[[Category:Kickapoo]]
 
[[Category:English]]
 
[[Category:English]]

Revision as of 08:35, 5 May 2026

Pre-evaluation

Command used: 

apertium-eval-translator-line -r /home/kpeng1/ling073/ling073-kic-eng-corpus/eng.tests.txt -t /home/kpeng1/ling073/ling073-kic-eng/kic-eng.tests.txt

Results

Metric Value
Number of words in reference 41
Number of words in test 25
Number of unknown words in test 11
Coverage 56.00%
Word Error Rate (WER) 100.00%
Position-independent Error Rate (PER) 100.00%

Coverage was calculated from the unknown-word rate reported by the evaluator: 

Coverage = 100% - 44.00% = 56.00%

After lexical additions

Metric Value
Number of words in reference 41
Number of words in test 27
Number of unknown words in test 0
Coverage 100.00%
Word Error Rate (WER) 97.56%
Position-independent Error Rate (PER) 97.56%
Retrieved from "http://wikis.swarthmore.edu/ling073/index.php?title=Kickapoo_and_English/Structural_transfer&oldid=28430"