Difference between revisions of "Kickapoo and English/Structural transfer"

From LING073
Jump to: navigation, search
(→‎Pre-evaluation)
 
(3 intermediate revisions by the same user not shown)
Line 1: Line 1:
 
== Pre-evaluation ==
 
== Pre-evaluation ==
−
 
−
apertium-eval-translator-line -r abc.tests.txt -t xyz-abc.tests.txt
 
−
 
−
Baseline measurements before implementing any structural transfer rules using the transfer test sentences
 
−
 
−
= Pre-evaluation ==
 
−
 
−
Before implementing structural transfer rules, we evaluated the system on the short sentence test corpus using ''apertium-eval-translator-line''.
 
−
 
−
=== Test setup ===
 
−
 
−
Reference file:
 
−
<pre>
 
−
/home/kpeng1/ling073/ling073-kic-eng-corpus/eng.tests.txt
 
−
</pre>
 
−
 
−
Test translation file:
 
−
<pre>
 
−
/home/kpeng1/ling073/ling073-kic-eng/kic-eng.tests.txt
 
−
</pre>
 
  
 
Command used:
 
Command used:
Line 51: Line 31:
 
|}
 
|}
  
−
Coverage was calculated from the unknown-word rate reported by the evaluator:
+
 
 +
== After lexical additions ==
 +
 
 +
{| class="wikitable"
 +
! Metric
 +
! Value
 +
|-
 +
| Number of words in reference
 +
| 41
 +
|-
 +
| Number of words in test
 +
| 27
 +
|-
 +
| Number of unknown words in test
 +
| 0
 +
|-
 +
| Coverage
 +
| 100.00%
 +
|-
 +
| Word Error Rate (WER)
 +
| 97.56%
 +
|-
 +
| Position-independent Error Rate (PER)
 +
| 97.56%
 +
|}
 +
 
 +
 
 +
== After adding structural transfer rule ==
 +
 
 +
{| class="wikitable"
 +
! Metric
 +
! Value
 +
|-
 +
| Number of words in reference
 +
| 41
 +
|-
 +
| Number of words in test
 +
| 33
 +
|-
 +
| Number of unknown words in test
 +
| 0
 +
|-
 +
| Coverage
 +
| 100.00%
 +
|-
 +
| Word Error Rate (WER)
 +
| 87.80%
 +
|-
 +
| Position-independent Error Rate (PER)
 +
| 87.80%
 +
|}
 +
 
 +
=== Implemented Structural Transfer item ===
 +
 
 +
We implemented a structural transfer rule for the Kickapoo conjunct pattern illustrated by the sentence:
  
 
<pre>
 
<pre>
−
Coverage = 100% - 44.00% = 56.00%
+
manahi enoweechi
 
</pre>
 
</pre>
  
−
=== Interpretation ===
+
The corresponding English translation is:
 +
 
 +
<pre>
 +
This is what he said.
 +
</pre>
 +
 
 +
In this pattern, the Kickapoo source structure is more compact than the English target structure. The transfer rule expands the source into a fuller English clause by introducing additional overt material, including ''is'', ''what'', and ''he'', and by setting the verb to past tense.
 +
 
 +
==== Example outputs ====
 +
 
 +
Source sentence:
 +
<pre>
 +
manahi enoweechi
 +
</pre>
 +
 
 +
Tagger output:
 +
<pre>
 +
^manahi<prn><dem><sg>$ ^nowee<v><iv><aor_1><aa><cnj><s_3sg>$^.<sent>$
 +
</pre>
 +
 
 +
Lex output:
 +
<pre>
 +
^manahi<prn><dem><sg>/this<det><sg>$ ^nowee<v><iv><aor_1><aa><cnj><s_3sg>/say<vblex><aor_1><aa><cnj><s_3sg>$^.<sent>/.<sent>$
 +
</pre>
 +
 
 +
Transfer output:
 +
<pre>
 +
^this<det><sg>$ ^be<vblex><pres><p3><sg>$ ^what<prn>$ ^he<prn><p3><sg>$ ^say<vblex><past>$^.<sent>$
 +
</pre>
 +
 
 +
Final translation:
 +
<pre>
 +
this is what he said
 +
</pre>
  
−
The baseline system has very low performance on this test set. Almost half of the words in the test translation are unknown (44.00%), which gives a coverage of only 56.00%. In addition, both WER and PER are 100.00%, meaning that the current output does not match the reference translation either in word choice or in word order.
 
  
−
This suggests that before and during structural transfer development, we need to:
 
−
* add missing stems to the bilingual dictionary
 
−
* add missing lexical items to the relevant monolingual dictionaries
 
−
* improve analysis coverage on the transfer test sentences
 
−
* implement structural transfer rules for the contrastive grammar items being tested
 
  
 
[[Category:Sp26_StructuralTransfer]]
 
[[Category:Sp26_StructuralTransfer]]
 
[[Category:Kickapoo]]
 
[[Category:Kickapoo]]
 
[[Category:English]]
 
[[Category:English]]

Latest revision as of 10:37, 5 May 2026

Pre-evaluation

Command used: 

apertium-eval-translator-line -r /home/kpeng1/ling073/ling073-kic-eng-corpus/eng.tests.txt -t /home/kpeng1/ling073/ling073-kic-eng/kic-eng.tests.txt

Results

Metric Value
Number of words in reference 41
Number of words in test 25
Number of unknown words in test 11
Coverage 56.00%
Word Error Rate (WER) 100.00%
Position-independent Error Rate (PER) 100.00%


After lexical additions

Metric Value
Number of words in reference 41
Number of words in test 27
Number of unknown words in test 0
Coverage 100.00%
Word Error Rate (WER) 97.56%
Position-independent Error Rate (PER) 97.56%


After adding structural transfer rule

Metric Value
Number of words in reference 41
Number of words in test 33
Number of unknown words in test 0
Coverage 100.00%
Word Error Rate (WER) 87.80%
Position-independent Error Rate (PER) 87.80%

Implemented Structural Transfer item

We implemented a structural transfer rule for the Kickapoo conjunct pattern illustrated by the sentence: 

manahi enoweechi

The corresponding English translation is: 

This is what he said.

In this pattern, the Kickapoo source structure is more compact than the English target structure. The transfer rule expands the source into a fuller English clause by introducing additional overt material, including is, what, and he, and by setting the verb to past tense.

Example outputs

Source sentence: 

manahi enoweechi

Tagger output: 

^manahi<prn><dem><sg>$ ^nowee<v><iv><aor_1><aa><cnj><s_3sg>$^.<sent>$

Lex output: 

^manahi<prn><dem><sg>/this<det><sg>$ ^nowee<v><iv><aor_1><aa><cnj><s_3sg>/say<vblex><aor_1><aa><cnj><s_3sg>$^.<sent>/.<sent>$

Transfer output: 

^this<det><sg>$ ^be<vblex><pres><p3><sg>$ ^what<prn>$ ^he<prn><p3><sg>$ ^say<vblex><past>$^.<sent>$

Final translation: 

this is what he said
Retrieved from "http://wikis.swarthmore.edu/ling073/index.php?title=Kickapoo_and_English/Structural_transfer&oldid=28433"