Difference between revisions of "User:Scasada1/Final Project"
From LING073
(→Final Project Description) |
(→Selected Example Sentences from the Large Corpus) |
||
| (21 intermediate revisions by the same user not shown) | |||
| Line 6: | Line 6: | ||
We have released information for our project publicly on [https://github.com/apertium-uum GitHub (apertium-uum)] and [https://github.com/apertium-uum/sp2023-uum-poster/tree/main here] you may separately view a digital copy of our final presentation poster as it exists on GitHub. | We have released information for our project publicly on [https://github.com/apertium-uum GitHub (apertium-uum)] and [https://github.com/apertium-uum/sp2023-uum-poster/tree/main here] you may separately view a digital copy of our final presentation poster as it exists on GitHub. | ||
| − | [[File:Output-onlinepngtools (1).png| | + | [[File:Output-onlinepngtools (1).png|800px|thumb|right|SP2023 Urum "Polished RBMT System for Crimean Urum" Poster]] |
== Previous Work == | == Previous Work == | ||
| − | [https://wikis.swarthmore.edu/ling073/ | + | [https://wikis.swarthmore.edu/ling073/Urum Urum Transducer] |
| − | [https://wikis.swarthmore.edu/ling073/ | + | [https://wikis.swarthmore.edu/ling073/Urum_and_English Urum-English Machine Translation] |
== Additions == | == Additions == | ||
| − | === | + | === Urum Transducer === |
| − | * We added 100+ stems to the lexd | + | * We added 100+ stems to the .lexd file, covering common nouns, verbs, adjectives, proper nouns, etc. as they occurred in terms of frequency in appearance within (Олександр ГАРКАВЕЦЬ УРУМИ НАДАЗОВ’Я) |
| − | * | + | * Expanded our number of rules in .twol to cover the rest of Urum's vowel and consonant harmony: (Total Rules: 16 → 24) |
| + | * We added a large number of additional .lexd PATTERNS including: | ||
| + | **''VFutureDefinite, VFutureIndefinite, VTransgressive, VPastParticiple, NumOrdinal, NumDiscrete, CaseIns, CaseAbe, & CaseTerm'' | ||
| − | === | + | === Urum-English Machine Translation === |
| − | * Added one structural transfer rule to | + | * Added one structural transfer rule to with the formation of NPs in the case of proper nouns (prior not realizing correctly, omitting 'the' and instead having no output, ex: "king #Herod" |
| − | + | * Added 100+ new lexicon items to our bilingual dictionary, making the total ~250. | |
| − | * Added | ||
=== Corpus Construction === | === Corpus Construction === | ||
| Line 30: | Line 31: | ||
* Constructed large corpus with 100 sentences; the sentences in this corpus are randomly selected from the Bible and the Grammar book, and evaluating over this corpus shows the general accuracy of our machine translation system. | * Constructed large corpus with 100 sentences; the sentences in this corpus are randomly selected from the Bible and the Grammar book, and evaluating over this corpus shows the general accuracy of our machine translation system. | ||
| − | ==== | + | == Final Evaluation == |
| − | |||
| − | |||
| − | * | + | === Transducer Overview (uum) === |
| − | *: | + | *Coverage over the "Urums of North Azov: Collection of Narratives": 60.071% |
| + | *Precision: 55.678% | ||
| + | *Recall: 73.077% | ||
| + | *Total Stems: 471 | ||
| + | **N-Roots: 121 | ||
| + | **V-Roots: 61 | ||
| + | **Adv/Adj-Roots: 22 | ||
| − | * | + | === Transfer Overview (uum ‣ eng) === |
| − | *: | + | *Urum Bible: "Nativity": |
| − | + | ** WER: 81.29% | |
| − | * | + | ** PER: 57.19% |
| − | *: | + | (The amount of error present here is likely due to the type of language used in the King James translation of the bible, as we used this as gold for the English translation. We believe that the word-error rate, and more specifically PER, would be far lower with a version of the bible in English which is more accurate to the original text). |
| − | + | * Bilingual Dictionary Entries: 260 | |
| − | * | + | * Total LRX Rules: 8 |
| − | *: | ||
| − | |||
| − | |||
| − | === | + | === RBMT Evalutation === |
| − | |||
| − | |||
| − | |||
| − | = | + | {| class="wikitable" |
| − | + | !'''Corpus''' | |
| − | + | !'''Tokens''' | |
| − | + | !'''Coverage''' | |
| − | + | !'''Accuracy''' | |
| − | + | |- | |
| − | + | |Urum Bible: "Nativity" | |
| − | ( | + | |170 |
| + | |95.882% (163/170) | ||
| + | |91.765% | ||
| + | |- | ||
| + | |"Urums of North Azov: Collection of Narratives" | ||
| + | |18K | ||
| + | |28.487% (53K/186K) | ||
| + | |N/A | ||
| + | |} | ||
== Future Work == | == Future Work == | ||
| − | * | + | * We currently have a very rich analysis system, but we need to expand our current dictionary significantly (~1000+ words, ideally) |
| − | * | + | ** In order to solve this, we have a publicly available spreadsheet dictionary for Urum, based off a Ukranian-Urum dictionary translated to English ([https://docs.google.com/spreadsheets/d/1AowA3a4iWiNgHLxARw0QwWXgNq1_aegIrjV7eHxO9Xc/edit?usp=sharing which is available on Google Sheets]). |
| − | * | + | * Refine our Apertium (uum ‣ eng) transfer pipeline; i.e, robust expansion of lexical selection and lexical transfer as issues begin to arise. |
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
[[Category:Urum]] [[Category:Sp23_FinalProjects]] | [[Category:Urum]] [[Category:Sp23_FinalProjects]] | ||
Latest revision as of 00:25, 11 May 2023
Contents
Final Project Description
For our final project, we decided to work on a two-part expansion of our RBMT-system for our language (Crimean Urum). This work will consist of many parts, but we will outline our base goals here:
- Improve our morphological transducer for Crimean Urum, aiming for a corpus coverage above 80%.
- *Improve our Crimean Urum machine translation accuracy.
We believe that our morphological transducer and machine translation system will be potentially helpful for living Urum language communities with tasks like generating novel sentences, and/or helping with an auto-correction system, MT, text processing, etc.
We have released information for our project publicly on GitHub (apertium-uum) and here you may separately view a digital copy of our final presentation poster as it exists on GitHub.
.png)
Previous Work
Urum-English Machine Translation
Additions
Urum Transducer
- We added 100+ stems to the .lexd file, covering common nouns, verbs, adjectives, proper nouns, etc. as they occurred in terms of frequency in appearance within (Олександр ГАРКАВЕЦЬ УРУМИ НАДАЗОВ’Я)
- Expanded our number of rules in .twol to cover the rest of Urum's vowel and consonant harmony: (Total Rules: 16 → 24)
- We added a large number of additional .lexd PATTERNS including:
- VFutureDefinite, VFutureIndefinite, VTransgressive, VPastParticiple, NumOrdinal, NumDiscrete, CaseIns, CaseAbe, & CaseTerm
Urum-English Machine Translation
- Added one structural transfer rule to with the formation of NPs in the case of proper nouns (prior not realizing correctly, omitting 'the' and instead having no output, ex: "king #Herod"
- Added 100+ new lexicon items to our bilingual dictionary, making the total ~250.
Corpus Construction
- Constructed small corpus with 30 sentences; the lexicon in this corpus are mostly added to our transducer and the bilingual dictionary, which is useful for evaluateing how the effectiveness of the structural transfer rules when the lexicon are mostly known.
- Constructed large corpus with 100 sentences; the sentences in this corpus are randomly selected from the Bible and the Grammar book, and evaluating over this corpus shows the general accuracy of our machine translation system.
Final Evaluation
Transducer Overview (uum)
- Coverage over the "Urums of North Azov: Collection of Narratives": 60.071%
- Precision: 55.678%
- Recall: 73.077%
- Total Stems: 471
- N-Roots: 121
- V-Roots: 61
- Adv/Adj-Roots: 22
Transfer Overview (uum ‣ eng)
- Urum Bible: "Nativity":
- WER: 81.29%
- PER: 57.19%
(The amount of error present here is likely due to the type of language used in the King James translation of the bible, as we used this as gold for the English translation. We believe that the word-error rate, and more specifically PER, would be far lower with a version of the bible in English which is more accurate to the original text).
- Bilingual Dictionary Entries: 260
- Total LRX Rules: 8
RBMT Evalutation
| Corpus | Tokens | Coverage | Accuracy |
|---|---|---|---|
| Urum Bible: "Nativity" | 170 | 95.882% (163/170) | 91.765% |
| "Urums of North Azov: Collection of Narratives" | 18K | 28.487% (53K/186K) | N/A |
Future Work
- We currently have a very rich analysis system, but we need to expand our current dictionary significantly (~1000+ words, ideally)
- In order to solve this, we have a publicly available spreadsheet dictionary for Urum, based off a Ukranian-Urum dictionary translated to English (which is available on Google Sheets).
- Refine our Apertium (uum ‣ eng) transfer pipeline; i.e, robust expansion of lexical selection and lexical transfer as issues begin to arise.
