Difference between revisions of "Evenki and English"

Latest revision as of 16:51, 11 May 2023

Resources for translation between Evenki and English.

External Resources

Language Pair Repo

English Transducer

Evenki to English Repo

Developed Resources

Lexical Selection

Contrastive Grammar

Structural Transfer

EVN → ENG Initial Evaluation

Monolingual Transducer Coverage

• English Coverage

TOP UNKNOWN WORDS:

coverage: 129 / 129 (1) remaining unknown forms: 0

• Evenki Coverage

TOP UNKNOWN WORDS:

     4 ^Ңи/*Ңи$
     4 ^вэл/*вэл$
     1 ^эңэсит/*эңэсит$
     1 ^умунупкиттула/*умунупкиттула$
     1 ^турӣнмумачин/*турӣнмумачин$
     1 ^са̄рича̄нинду/*са̄рича̄нинду$
     1 ^нуңан/*нуңан$
     1 ^мэ̄нңи/*мэ̄нңи$
     1 ^кэсэгӣчивунду/*кэсэгӣчивунду$
     1 ^куңака̄н/*куңака̄н$
     1 ^илэчэврэви/*илэчэврэви$
     1 ^ӣвувра/*ӣвувра$
     1 ^гражданствоват/*гражданствоват$
     1 ^государстваду/*государстваду$
     1 ^гиркумчэ̄/*гиркумчэ̄$
     1 ^а̄чина/*а̄чина$

coverage: 64 / 86 (~0.74418604651162790698) remaining unknown forms: 22

Bilingual Transducer Coverage

TOP UNKNOWN WORDS:

     5 ^вэл/*вэл$
     5 ^Илэтыкин/*Илэтыкин$
     4 ^Ңи/*Ңи$
     4 ^тэдет/*тэдет$
     3 ^эвки/*эвки$
     2 ^бинэ/*бинэ$
     2 ^бимчэ/*бимчэ$
     2 ^бидемэчин/*бидемэчин$
     2 ^авадытыкин/*авадытыкин$
     1 ^эңэсит/*эңэсит$
     1 ^эсин/*эсин$
     1 ^эрэ/*эрэ$
     1 ^эмэмчэ/*эмэмчэ$
     1 ^чина/*чина$
     1 ^чивунду/*чивунду$
     1 ^умунупкиттула/*умунупкиттула$
     1 ^тэдэт/*тэдэт$
     1 ^ты/*ты$
     1 ^тури/*тури$
     1 ^тартыки/*тартыки$

coverage: 20 / 99 (~0.20202020202020202020) remaining unknown forms: 79

Sentence Translation

Note: Many of the words with stems entered into the bilingual dictionary are not translating despite being in both the monolingual and bilingual dictionaries (ie, based on our entries to apertium-evn.evn.lexc "Сӣ ӈӣ?" should be translating to "you where from?" or "you no-one" accounting for lack of disambiguation). We are unsure why although it seems to be an orthography issue with the transducer we pulled from apertium. We have been unable to resolve this problem at this time despite working with a TA for several hours. This has lead to artificially poor performance from our whole RBMT system pipeline.

Additions

• Added 5 New Lexical Selection rules

EVN → ENG Final Evaluation

Monolingual Transducer

Note: We used a preexisting transducer for evenki on apertium and did not produce the transducer we used ourselves.

Updated precision and recall against the (updated) eval.test and eval.gold files:

We tried to run precision and recall against the bible corpus and received the following error:

ERROR: different number of annotated (158) and test (210) cohorts; check these: ,, :, «, ., », Иона‐ка, ;, Ина‐ка, н‐ка, ?, !, ка, Ина, н
 
Since we are already having issues with the transducer parsing words regardless of whether they are in the monolingual dictionary or if their stems are in the transducer, we decided to move onto the other evaluation metrics.

Coverage over the large corpus:

terminate called after throwing an instance of 'TransducerHasWrongTypeException'
TOP UNKNOWN WORDS:
coverage: 0 / 0 (Error 10002)
remaining unknown forms: 0
ambiguity: 0 / 0 (Error 10002)
evn.bible.txt Thu May 11 12:15:42 PM EDT 2023   0/0     Error 10002     \|      0/0     Error 10002

Note: This error and the previous error are likely related to the same transducer problem that has affected all of our work with Evenki.

The number of words in the large corpus: 1,180 words (from Universal Declaration of Human Rights)

The number of stems in the transducer: ~2000 (Original source unclear)

Machine Translation

WER and PER over longer corpus:

-------------------------------------------------------
Number of words in reference: 141
Number of words in test: 81
Number of unknown words (marked with a star) in test: 99
Percentage of unknown words: 122.22 %

Results when removing unknown-word marks (stars)
-------------------------------------------------------
Edit distance: 143
Word error rate (WER): 101.42 %
Number of position-independent correct words: 0
Position-independent word error rate (PER): 100.00 %

The proportion of stems translated correctly in the longer corpus: 0%

Trimmed coverage over longer corpora:

Number of tokenised words in the corpus: 100
Coverage: 1.00%
Top unknown words in the corpus:
5        Илэтыкин
5        вэл
4        тэдет
4        Ңи
3        эвки
2        бидемэчин
2        авадытыкин
2        бимчэ
2        бинэ
2        Мурин
2        орондук
2        хегдытмер
2        Тар
2        орон
2        ньами
1        индэ
1        ви
1        мэ
1        нӈи
1        илэчэврэви
Translation time: 0.05595231056213379 seconds

Trimmed coverage over large corpora:

Number of tokenised words in the corpus: 1274
Coverage: 0.08%
Top unknown words in the corpus:
35       о
30       статьян
25       н
23       Илэтыкин
21       тэдет
19       гу
19       вэл
16       мэ
14       илэ
14       илэл
13       1
13       2
12       да
11       бимчэ
10       нңи
10       Дян
10       Дюр
9        ты
9        тын
9        тар
Translation time: 0.04945206642150879 seconds

Final Note

We did our best to work around issues that arose with working with Evenki but ultimately we were not able to get good results from our RBMT system due to whatever flaw with the original transducer was preventing the vast majority of words from registering. Other issues we would like to note for anyone wanting to do future work on Evenki are the following:

• Very few resources exist for parallel corpuses and most are in Russian
• Grammar resources that relate to English that do exist almost exclusively transliterate the Cyrillic orthography into Latin characters
• The apertium supported transducer works well internally within itself but it seems to be incompatible with any Cyrillic from external sources for unclear reasons. This is the issue that most interfered with our progress.
• Documentation for all of our work on Evenki can be found throughout this wiki and on github