Difference between revisions of "Japhug/Japhug and English"
From LING073
(→Final evaluation) |
(→Monolingual transducer stats) |
||
| (4 intermediate revisions by 3 users not shown) | |||
| Line 215: | Line 215: | ||
<pre> | <pre> | ||
$ precisionRecall jya.eval.test.new.txt jya.eval.gold.txt | $ precisionRecall jya.eval.test.new.txt jya.eval.gold.txt | ||
| − | Totals: 79 forms, | + | Totals: 79 forms, 221 tp, 155 fp, 0 tn, 87 fn |
| − | Precision: | + | Precision: 58.77660% |
| − | Recall: | + | Recall: 71.75325% |
| + | </pre> | ||
*Coverage over the large corpus: | *Coverage over the large corpus: | ||
<pre> | <pre> | ||
| Line 247: | Line 248: | ||
</pre> | </pre> | ||
*The number of words in the large corpus: 3999 | *The number of words in the large corpus: 3999 | ||
| − | *The number of stems in the transducer | + | *The number of stems in the transducer: 6646 |
| − | |||
====MT in jya-eng stats==== | ====MT in jya-eng stats==== | ||
| Line 257: | Line 257: | ||
------------------------------------------------------- | ------------------------------------------------------- | ||
Number of words in reference: 3013 | Number of words in reference: 3013 | ||
| − | Number of words in test: | + | Number of words in test: 2450 |
| − | Number of unknown words (marked with a star) in test: | + | Number of unknown words (marked with a star) in test: 996 |
| − | Percentage of unknown words: | + | Percentage of unknown words: 40.65 % |
Results when removing unknown-word marks (stars) | Results when removing unknown-word marks (stars) | ||
------------------------------------------------------- | ------------------------------------------------------- | ||
| − | Edit distance: | + | Edit distance: 3116 |
| − | Word error rate (WER): | + | Word error rate (WER): 103.42 % |
| − | Number of position-independent correct words: | + | Number of position-independent correct words: 85 |
| − | Position-independent word error rate (PER): | + | Position-independent word error rate (PER): 97.18 % |
</pre> | </pre> | ||
| − | *The proportion of stems translated correctly in the longer corpus: | + | *The proportion of stems translated correctly in the longer corpus: 100% - 103.42% = -3.42% |
| − | *Trimmed coverage over longer corpora: ~0. | + | *Trimmed coverage over longer corpora: ~0.68006279434850863422 |
| − | *Trimmed coverage over large corpora: ~0. | + | *Trimmed coverage over large corpora: ~0.63545647558386411890 |
| − | *The number of tokens in longer corpora: | + | *The number of tokens in longer corpora: 3185 |
| − | *The number of tokens in large corpora: | + | *The number of tokens in large corpora: 4710 |
[[Category:Sp23_TranslationPairs ]] [[Category: Japhug]] | [[Category:Sp23_TranslationPairs ]] [[Category: Japhug]] | ||
Latest revision as of 02:48, 10 May 2023
Contents
Resources for machine translation between Japhug and English
Developed resources
External resources
Apertium language pair for translating between Jiarong and English
Jya → Eng evaluation
coverage (number of forms analysed) of Japhug transducer on jya.sentences.txt: 76.98 %
coverage of bilingual transducer (jya-eng.automorf.bin) on the same file: 28.61 %
10 sentences:
#1
The original sentence:
kɯɕɯŋgɯ tɕe, nɤki, lɯlu ci pjɤtu, tɕendɤre βʑɯ ci pjɤtu
The intended English translation: A long time ago, there was a cat and a mouse
Output of jya-eng-biltrans:
^kɯɕɯŋgɯ<n><ind>/long time ago<cnjadv><ind>$ ^tɕe<cnj>//as for<pr>$^,<cm>/,<cm>$ ^nɤki<dem>/that<det><dem><sg>$^,<cm>/,<cm>$ ^lɯlu<n><ind>/cat<n><sg><ind>$ ^ci<det>/a<det><ind><sg>/one<num><sg>$ ^pjɤtu<cop><other>/be<vbser><other>$^,<cm>/,<cm>$ ^tɕendɤre<cnj>/then<cnjadv>$ ^βʑɯ<n><ind>/mouse<n><sg><ind>$ ^ci<det>/a<det><ind><sg>/one<num><sg>$ ^pjɤtu<cop><other>/be<vbser><other>$^.<sent>/.<sent>$
Output of jya-eng:
#long time ago , that, #cat a #be, then #mouse a #be
#2
The original sentence:
lɯlu nɯ kɯ nɤkinɯ, βʑɯ ri tundze ɕti
The intended English translation: The cat ate the mouse,
Output of jya-eng-biltrans:
^lɯlu<n><ind>/cat<n><sg><ind>$ ^nɯ<dem>/that<det><dem><sg>$ ^kɯ<post>/with<pr>$ ^nɤkinɯ<fill>/um<ij>$^,<cm>/,<cm>$ ^βʑɯ<n><ind>/mouse<n><sg><ind>$ ^ri<post>/at<pr>$ ^ndze<v><tv><ipfv>/eat<vblex><ipfv>$ ^ɕti<v><iv>/be<vbser>$^.<sent>/.<sent>$
Output of jya-eng:
#cat that with um, #mouse at #eat #be
#3
The original sentence:
βʑɯ chondɤre nɤki, kumpɣɤtɕɯ nɯ kɯ βzaŋsa tonɯpandʑi,
The intended English translation: The mouse made friends with the sparrow,
Output of jya-eng-biltrans:
^βʑɯ<n><ind>/mouse<n><sg><ind>$ ^chondɤre<com>/with<pr>$ ^nɤki<dem>/that<det><dem><sg>$^,<cm>/,<cm>$ ^kumpɣɤtɕɯ<n><ind>/sparrow<n><sg><ind>$ ^nɯ<dem>/that<det><dem><sg>$ ^kɯ<post>/with<pr>$ ^βzaŋsa<n><ind>/friend<n><sg><ind>$ ^tonɯpa<v><tv><du>/made<vblex><pp><du>$^,<cm>/,<cm>$^.<sent>/.<sent>$
Output of jya-eng:
#mouse with that, #sparrow that with #friend #made,
#4
The original sentence:
tɕe βzaŋsa tonɯpandʑi tɕe, wuma pjɤkɤmɯmindʑici
The intended English translation: The two of them become friends and got along well
Output of jya-eng-biltrans:
^tɕe<cnj>//as for<pr>$ ^βzaŋsa<n><ind>/friend<n><sg><ind>$ ^tonɯpa<v><tv><du>/made<vblex><pp><du>$ ^tɕe<cnj>//as for<pr>$^,<cm>/,<cm>$ ^wuma<adv>/very<adv>$ ^ɤmɯmi<v><iv><pii><du>/get along well<vblex><pii><du>$^.<sent>/.<sent>$
Output of jya-eng:
#friend #made , very #get along well
#5
The original sentence:
maka cici ŋotɕu jɤarɯrindʑi ʑo tɯtɯrca ntsɯ juɕendʑi pjɤŋu
The intended English translation: they always got together
Output of jya-eng-biltrans:
^maka<adv>/at all<adv>$ ^cici<adv>/sometimes<adv>$ ^ŋotɕu<prn>/wherever<adv>$ ^*jɤarɯrindʑi/*jɤarɯrindʑi$ ^*ʑo/*ʑo$ ^*tɯtɯrca/*tɯtɯrca$ ^ntsɯ<adv>/always<adv>$ ^*juɕendʑi/*juɕendʑi$ ^ŋu<v><iv><pii>/be<vbser><pii>$^.<sent>/.<sent>$
Output of jya-eng:
at all sometimes wherever #each together always #go #be
#6
The original sentence:
kɤndza ci tu nɤ pjɯnɯkrondʑi
The intended English translation: There was food to eat, they shared,
Output of jya-eng-biltrans:
^kɤndza<n><ind>/food<n><sg><ind>$ ^ci<det>/a<det><ind><sg>/one<num><sg>$ ^tu<v><iv>/be<vbser>/exist<vblex>$ ^*nɤ/*nɤ$ ^*pjɯnɯkrondʑi/*pjɯnɯkrondʑi$^.<sent>/.<sent>$
Output of jya-eng:
#food a #be at #share
#7
The original sentence:
nɯ mɤʑɯ kɯmtɕhɯ kɯfse tɤtu nɤ, tɯɣrɤz ɲɯnɤɣrondʑi nɯ ra pjɤŋgrɤl
The intended English translation: Also, when there was something to play with, they played together
Output of jya-eng-biltrans:
^nɯ<dem>/that<det><dem><sg>$ ^mɤʑɯ<adv>/again<adv>$ ^kɯmtɕhɯ<n><ind>/toy<n><sg><ind>$ ^fse<v><tv><prc><sbj><ind>/resemble<vblex><prc><sbj><ind>$ ^tu<v><iv><aor>/be<vbser><aor>/exist<vblex><aor>$ ^nɤ<post>/at<pr>$^,<cm>/,<cm>$ ^tɯɣrɤz<adv>/together<adv>$ ^nɤɣro<v><tv><ipfv><du>/play<vblex><inf><ipfv><du>$ ^nɯ<dem>/that<det><dem><sg>$ ^ra<v><iv>/should<vbmod>$ ^ŋgrɤl<v><iv><pii>/usually like this<iv><pii>$^.<sent>/.<sent>$
Output of jya-eng:
that again #toy #resemble #be at, together #play that #should #usually like this
#8
The original sentence:
tɕhi kɤcha tukɤpjɤl kowa ntsɯ tuβzundʑi pjɤŋu tɕe
The intended English translation: They were always trying to think of ways to evade that cat
Output of jya-eng-biltrans:
^tɕhi<prn>/what<prn>$ ^cha<v><iv><prc><obj><ind>/able<adj><prc><obj><ind>/can<vbmod><prc><obj><ind>$ ^pjɤl<v><tv><inf><ipfv>/evade<vblex><inf><ipfv>$ ^kowa<n><ind>/method<n><sg><ind>$ ^ntsɯ<adv>/always<adv>$ ^βzu<v><tv><ipfv><du>/make<vblex><ipfv><du>$ ^ŋu<v><iv><pii>/be<vbser><pii>$ ^tɕe<cnj>//as for<pr>$^.<sent>/.<sent>$
Output of jya-eng:
#what #able #evade #method always #make #be
#9
The original sentence:
lɯlu nɯ kɯ βʑɯ ɯɕa tɕi pjɤrga, pɣɤtɕɯ ɯɕa tɕi pjɤrga ɕti tɕe.
The intended English translation: The cat likes eating rat meat, and also likes sparrow meat.
Output of jya-eng-biltrans:
^lɯlu<n><ind>/cat<n><sg><ind>$ ^nɯ<dem>/that<det><dem><sg>$ ^kɯ<post>/with<pr>$ ^βʑɯ<n><ind>/mouse<n><sg><ind>$ ^ɕa<n><px3sg>/meat<n><sg><px3sg>$ ^tɕi<adv>/also<adv>$ ^rga<v><iv><pii>/like<vblex><pii>$^,<cm>/,<cm>$ ^pɣɤtɕɯ<n><ind>/bird<n><sg><ind>$ ^ɕa<n><px3sg>/meat<n><sg><px3sg>$ ^tɕi<adv>/also<adv>$ ^rga<v><iv><pii>/like<vblex><pii>$ ^ɕti<v><iv>/be<vbser>$ ^tɕe<cnj>//as for<pr>$^.<sent>/.<sent>$^.<sent>/.<sent>$
Output of jya-eng:
#cat that with #mouse #meat also #like, #bird #meat also #like #be .
#10
The original sentence:
tɕe nɯ maʁ nɤ tɕe, βʑɯ nɯ kɯ wuma kɤrɤfsoʁ pjɤcha tɕe
The intended English translation: since then, that mouse was very good at accumulating food
Output of jya-eng-biltrans:
^tɕe<cnj>//as for<pr>$ ^nɯ maʁ nɤ<cnj>/or<cnjcoo>$ ^tɕe<cnj>//as for<pr>$^,<cm>/,<cm>$ ^βʑɯ<n><ind>/mouse<n><sg><ind>$ ^nɯ<dem>/that<det><dem><sg>$ ^kɯ<post>/with<pr>$ ^wuma<adv>/very<adv>$ ^rɤfsoʁ<v><iv><prc><obj><ind>/forage<vblex><prc><obj><ind>$ ^cha<v><iv><pii>/able<adj><pii>/can<vbmod><pii>$ ^tɕe<cnj>//as for<pr>$^.<sent>/.<sent>$
Output of jya-eng:
or , #mouse that with very #forage #able
Additions
These are some additions we made to our bilingual transducer beyond the labs.
- 5000+ lemmas to our Japhug transducer
- 5 new features covered in our Japhug morphological analyzer, including: imperfective prefixes, the peg circumfix, proximal/distal demonstratives, redesigned verb patterns to match the slot-based prefix system in Japhug, number suffixes on verbs.
- 2 new structural transfer rules: NP rule to combine nouns and possessed nouns; DP rule to place determiner before noun, and to take ergative case from postposition; vP rule to combine adverbs and verbs. These were made in the process of meeting our original goal, to match the SVO word order in English (Japhug is usually SOV or OSV)
(details listed for our own reference)
Final evaluation
Monolingual transducer stats
- Updated precision and recall against the (updated) eval.test and eval.gold files:
$ precisionRecall jya.eval.test.new.txt jya.eval.gold.txt Totals: 79 forms, 221 tp, 155 fp, 0 tn, 87 fn Precision: 58.77660% Recall: 71.75325%
- Coverage over the large corpus:
TOP UNKNOWN WORDS:
16 ^ɣɯ/*ɣɯ$
14 ^lɤβzaŋ/*lɤβzaŋ$
11 ^cho/*cho$
10 ^qartshɤs/*qartshɤs$
8 ^rɟɤlpu/*rɟɤlpu$
8 ^rcanɯ/*rcanɯ$
8 ^qhe/*qhe$
8 ^je/*je$
8 ^jamɤn/*jamɤn$
7 ^tɕɤn/*tɕɤn$
6 ^nɯmbala/*nɯmbala$
6 ^ndɤ/*ndɤ$
6 ^lɤari/*lɤari$
6 ^kɯki/*kɯki$
5 ^wo/*wo$
5 ^tɯɣjɤn/*tɯɣjɤn$
5 ^lɤɣe/*lɤɣe$
5 ^kɯsthɯci/*kɯsthɯci$
5 ^ciŋglɯɣ/*ciŋglɯɣ$
4 ^ɲɯsɯsɤm/*ɲɯsɯsɤm$
coverage: 4463 / 5019 (~0.88922096035066746364)
remaining unknown forms: 556
ambiguity: 9939 / 5019 (~1.98027495517035265989)
jya.corpus Mon May 8 02:13:49 PM EDT 2023 4463/5019 ~0.88922096035066746364 \| 9939/5019 ~1.98027495517035265989
- The number of words in the large corpus: 3999
- The number of stems in the transducer: 6646
MT in jya-eng stats
- WER and PER over longer corpus:
Statistics about input files ------------------------------------------------------- Number of words in reference: 3013 Number of words in test: 2450 Number of unknown words (marked with a star) in test: 996 Percentage of unknown words: 40.65 % Results when removing unknown-word marks (stars) ------------------------------------------------------- Edit distance: 3116 Word error rate (WER): 103.42 % Number of position-independent correct words: 85 Position-independent word error rate (PER): 97.18 %
- The proportion of stems translated correctly in the longer corpus: 100% - 103.42% = -3.42%
- Trimmed coverage over longer corpora: ~0.68006279434850863422
- Trimmed coverage over large corpora: ~0.63545647558386411890
- The number of tokens in longer corpora: 3185
- The number of tokens in large corpora: 4710
