This page details the expansion of a Lingala transducer for use as a spell checker. The source code for the transducer is available <code>TODO</code>

==Building a Corpus==
Robust evaluation of any transducer requires a large corpus of authentic text. I created a corpus of 636,261 words drawn from four sources:
# Lingala [https://www.lds.org/bc/content/shared/content/lingala/pdf/language-materials/34406_lin.pdf?lang=li Book of Mormon] available online as a searchable PDF
# Lingala [http://gospelgo.com/f/lingala_nt.htm New Testament], available online.
# Lingala [http://gospelgo.com/q/Lingala%20Bible%20-%20Psalms.pdf Psalms]
# Lingala [https://dumps.wikimedia.org/lnwiki/20170501/ Wikipedia]

All of these required some preprocessing. Due to varying standards of tone markings, all accents were stripped from the corpus by converting from Unicode to ASCII using Unidecode ([[Lingala_and_Kikuyu#Installing_Unidecode_For_MT_Evaluation |installation instructions]]). Adtionally, the Psalms PDF was not searchable and had to first be converted to a text file using the free and open source Tesseract OCR software ([[User:Apowell1#Tesseract_OCR |instructions]]). To respect the licensing of these works, the processed corpora are stored only in a [https://github.swarthmore.edu/apowell1/ling073-lin-corpus/tree/master/final_project_corpus private repository] on Swarthmore's institutional Github server.

==Improving the Transducer (Analysis)==
The transducer was primarily improved by adding a substantial number of stems to the <code>lexc</code> file. Some modifications and expansions of the existing morphotactics were required for this.
===Adding Stems===
Through a system of systematically adding the most frequent unanalyzed words from the test corpus, a total of (approximately) 471 stems were adding to the transducer, bringing the total to 744.

===Expanding and Improving Morphology===
The transducer morphotactics were expanded in a few critical ways. The biggest change dealt with the way class IX-X nouns are handled. Previously, there had been separate sections for nouns prefaced by "m" and nouns prefaced by "n," but it turns out that what letter prefaces the noun can be determined by the following letter ("m" generally precedes "b" and "p"). Hence, the lexicons were combined and an archiphoneme with accompanying twol rule was used. I also realized that many class IX-X nouns sometimes lacked the preceding n/m, and a few even never had a prefix at all. Invariable nouns (always take a prefix or never take a prefix) were added to their own lexicon while nouns with spelling variations were placed in another lexicon. The "! Dir/LR" tag was then added to the null prefix in the morphotactic section so that both the prefixed and unprefixes versions of the noun could analyze but only the prefixed version could generate.

A small but important change was the introduction of a class Ia lexicon, which behaves similarly to class IX but is actually considered class I. I had read in my grammar books that such a class existed, but did not fully appreciate how widespread its use was until trying to make substantial increases in coverage. This additional section proved essential to increasing coverage.

A section was also added to attempt to deal with irregular nouns. It allows for verb that take one root in certain forms (infinitive, imperative, future), and a second root in everything else, and simply requires two entries per verb, one for each root.

Finally, many spelling variations were added with a "! Dir/LR" comment so that alternative spellings of the same word could be analyzed together but only a particular one would generate.

==Setting up the Spell Checker==

==Future Work==
Further transducer improvements should be probably fall into two priorities:
# Reviewing morphotactical and lexical structure to make everything as linguistically sound as possible and bring into concordance with other Apertium monolingual pack structures and tag use
# Add additional stems to the transducer
**Open source word lists: See [https://sourceforge.net/p/lingala/code/HEAD/tree/hunspell/ Kevin Scannell's past Lingala work] and [https://fr.wiktionary.org/wiki/Cat%C3%A9gorie:lingala French Wikitionary Lingala category].

==Evaluating the Results==
{| class="wikitable"
|+ Coverage over Corpus
|-
!
! Pre-evaluation
! Post-evaluation
|-
! Book of Mormon
| 61.87%
| 88.19%

|-
! New Testament
| 71.38%
| 96.12%

|-
! Psalms
| 72.31%
| 93.07%

|-
! Wikipedia
| 52.65%
| 70.34%

|-
! Combined Coverage
| 63.43%
| 86.96%
|}

Coverage over biblical texts is quite high since they use only a handful of words repeatedly. Wikipedia coverage is comparatively lower, in part because of the widely different domains covered by each article, and in part because of the substantial number of short articles that contain little more than a proper noun, the word is, and some prepositions.

{| class="wikitable"
|+ Precision and Recall (500 random froms)
|-
!
! Pre-evaluation
! Post-evaluation
|-
! Precision
| 96.22642%
| 100.00000%

|-
! Recall
| 19.84436%
| 39.29961%
|}

Note that the annotated corpus (available in the same corpus repository linked to earlier) is a somewhat rough approximation since it was not annotated by a native speaker. Self-annotating a corpus is a fairly difficult task since, in general, the transducer already encodes the entirety of my knowledge of the language. Hence, I assume every analysis given my transducer isgit s correct, which may inflate the precision. The recall is rather low, mainly due to the fact that a substantial number of words whose stems are not in the .lexc file were chosen at random. The fact that only forms that were not punctuation were considered will of course default the recall relative to the coverage. With the exception of words marked <code>unk</code> in the annotated corpus, for which I have no idea of the usage, any missing stems should be fairly trivial to add to the transducer.

[[Category:sp17_FinalProjects]]
[[Category:Lingala]]

User:Apowell1/Final project

2017-05-05T21:51:51Z

Apowell1: /* Building a Corpus */

User:Apowell1/Final project

2017-05-05T21:51:18Z

Apowell1:

2017-04-23T04:34:16Z

Lingala and Kikuyu

2017-04-13T15:35:14Z

Apowell1: /* Final Evaluation */

Resources for machine translation between [[Lingala]] and [[Kikuyu]]. Code is stored in a [https://github.swarthmore.edu/apowell1/ling073-kik-lin GitHub Repository].
==Developed Resources==
===Parallel Corpus===

A [https://github.swarthmore.edu/apowell1/ling073-lin-kik-corpus parallel corpus] created using two translations of Genesis is available on Github.

===Contrastive Grammar===
Contrastive Grammar is detailed on the wiki page [[Lingala and Kikuyu/Contrastive Grammar]].

===Lexical Selection===
Lexical selection is detailed on the wiki page [[Lingala and Kikuyu/Lexical Selection]].

===Structural Transfer===
Structural transfer is detailed on the wiki page [[Lingala and Kikuyu/Structural Transfer]]

==Installing Unidecode For MT Evaluation==
Most available Lingala text does not mark tones; however, the Lingala transducer has been configured using spellrelax to provide analyses and surface generation with tone markings included as accents on vowels. As a result, generated translations will include tone markings not present in the reference translation and artificially inflate evaluation metrics such as WER and PER. Fortunately, we can take advantage of the fact that accented characters are not present in ASCII to replace accented characters with their unaccented version on the command line.

[https://pypi.python.org/pypi/Unidecode Unidecode] is a Python package that implements a function to convert Unicode strings into their closest ASCII approximation. For simple accented characters, this is precisely the mapping we need to remove the accents for evaluation against a reference translation. Unidecode can also be used on the command line with our machine translation system as follows: <code>echo /path/to/kik-corpus | apertium -d . kik-lin | unidecode</code>. To install unidecode, it is recommended to use <code>pip</code>, as experience suggests that <code>apt-get</code> will not run the necessary installation scripts to make unidecode accessible in Bash.

On the Ling 073 virtual machines, Pip can be installed using <code>sudo apt-get install python-pip</code>. After installing pip, unidecode can be installed using <code> pip install unidecode</code>.

==Lexical Selection==
This section is part of the [[Lexical transfer]] assignment. In addition to adding many one-to-one mappings to our .dix file, we have included a couple of one-to-many mappings for each direction of translation. See [[Lingala and Kikuyu/Lexical Selection]] for further details.

From Lingala to Kikuyu, we consider four one-to-many mappings:
*The Lingala verb kokíma can be translated in English as both "to run" and "to run away." Kikuyu has two distinct verbs corresponding to these English translations, respectively, hanyũk and ũr.
*The Lingala verb kofánda can be translated in English as both "to sit" and "to live" (as in reside). Kikuyu has two distinct verbs corresponding to these English translations, respectively, ikar and tũũr.
*The Lingala verb kobánza can be translated in English as both "to think" and "to remember." Kikuyu has two distinct verbs corresponding to these English translations, respectively, ĩcir and ririkan.
*The Lingala noun mabele can be translated in English as both "sand" and "milk." Kikuyu has two distinct nouns corresponding to these English translations, respectively, thanga and ria.

From Kikuyu to Lingala, we consider three one-to-many-mappings:
*The Kikuyu verb igu can be translated in English as both "to obey" and "to hear." Lingala has two distinct verbs corresponding to these English translations, respectively, kotosa and koyoka.
*The Kikuyu verb rut can be translated in English as both "to learn" and "to remove." Lingala has two distinct verbs corresponding to these English translations, respectively, koyekola and kokukula.
*the Kikuyu noun ria can be translated in English as both "milk" and "pond." Lingala has two distinct nouns corresponding to these English translations, respectively, mabele and etimá.

==Initial Evaluation==
===lin → kik evaluation===
*Evaluation on mini corpus:
**WER: 87.50%
**PER: 87.50%
**Number of words: 9
**Number of unknown words: 0
**Proportion of stems translated correctly: 100%

*Evaluation of corpus:
**WER: 99.06%
**PER: 94.34%
**Number of words: 107
**Number of unknown words: 71
**Proportion of stems translated correctly:33.64%

===kik → lin evaluation===
*Evaluation on mini corpus:
**WER: 80%
**PER: 80%
**Number of words: 7 (This should really be 8, but the command we are supposed to use to count this outputs 7. The proportion remains unchanged in our case, however.)
**Number of unknown words: 0
**Proportion of stems translated correctly: 100%

*Evaluation of corpus:
**WER: 98.13%
**PER: 90.65%
**Number of words: 106
**Number of unknown words: 79 (Note: Using the commands on the wiki gives a slightly different count that using the apertium-eval-translator tool. We use the count given by the command, not the tool)
**Proportion of stems translated correctly: 25.47%

==Final Evaluation==
===lin → kik Implementation Details===

====Bilingual Dictionary Expansion====
Difference can be seen by looking at the Github commit history.

===kik → lin Implementation Details===
This direction of the RBMT system was improved by adding items to the bilingual dictionary, adding additional morphology to the transducer, writing more constraint grammar rules, and writing additional transfer rules. Specific implementation details with examples of improved output are detailed in the sections that follow.

====Bilingual Dictionary Expansion====
Difference can be seen by looking at the Github commit history.

====Morphology Expansion====
*Support for proper nouns
**Example sentence: "Nzambe" (God)
**Old output: <code>^Nzambe/*Nzambe$^./.<sent>$</code>
**New output: <code>^Nzambe/Nzambe<n><prop>$^./.<sent>$</code>

*Support for imperative mood
**Example sentence: "Kíma!" (Run!)
**Old output: <code>^Kíma/*Kíma$^!/!<sent>$^./.<sent>$</code>
**New output: <code>^Kíma/kokíma<v><imp>$^!/!<sent>$^./.<sent>$</code>

*Support for intensifying suffixes (-a becomes -áká to denote certainty (infinitive, future) or insistence (imperative))
**Example sentence 1: "Azalákí kopésáká" (He was really distributing...)
**Old output: <code>^Azalákí/kozala<v><urp><p3><sg><aa>/kozala<vaux><urp><p3><sg><aa>$ ^kopésáká/*kopésáká$^./.<sent>$</code>
**New output <code>^Azalákí/kozala<v><urp><p3><sg><aa>/kozala<vaux><urp><p3><sg><aa>$ ^kopésáká/kopésa<v><inf><intns>$^./.<sent>$</code>
**Example sentence 2: "Akopésáká" (He will really give...)
**Old output: <code>^Akopésáká/*Akopésáká$^./.<sent>$</code>
**New output <code>^Akopésáká/kopésa<v><intns><fut><p3><sg><aa>$^./.<sent>$</code>
**Example sentence 3: "Pésáká" (Now do give...)
**Old output: <code>^Pésáká/*Pésáká$^./.<sent>$</code>
**New output <code>^Pésáká/kopésa<v><imp><intns>$^./.<sent>$</code>

*Support for more than 1 radical extension
**Example sentence: "Nabángisamákí na Pierre" (I was scared by Pierre, or more literally, I was made, by Pierre, to fear something/someone)
**Old output: <code>^Nabángisamákí/*Nabángisamákí$ ^na/na<pr>/na<cngcoo>$ ^Pierre/Pierre<n><prop>$^./.<sent>$</code>
**New output: <code>^Nabángisamákí/kobánga<v><caus><pass><urp><p1><sg><aa>$ ^na/na<pr>/na<cngcoo>$ ^Pierre/Pierre<n><prop>$^./.<sent>$</code>

*Support for numerals
**Example sentence: "zómi" (ten)
**Old output: <code>^zómi/*zómi$^./.<sent>$</code>
**New output: <code>^zómi/zómi<num><ord>/zómi<num><card>$^./.<sent>$</code>

*Support for quantifiers
**Example sentence: "mbelí nyónso" (each knife)
**Old output: <code>^mbelí/mbelí<n><cl9><sg>/mbelí<n><cl9><pl>$ ^nyónso/*nyónso$^./.<sent>$</code>
**New output: <code>^mbelí/mbelí<n><cl9><sg>/mbelí<n><cl9><pl>$ ^nyónso/nyónso<det><qnt>$^./.<sent>$</code>

====Additional Constraint Grammar Rules====
*If I am an adjective and a quantifier but I do not agree in number with the preceding noun, I must be a quantifier
**Example sentence: "bilóko moké" (A few things, ''not'' small things)
**Old output (below)
<pre>
"<bilóko>"
"elóko" n cl7 pl
"<moké>"
"moké" det qnt
"moké" adj sg
"<.>"
"." sent
</pre>
:*New output (below)
<pre>
"<bilóko>"
"elóko" n cl7 pl
"<moké>"
"moké" det qnt
; "moké" adj sg REMOVE:21
"<.>"
"." sent
</pre>

*If I am a noun that can be singular or plural and am followed by a singular adjective, I cannot be plural
**Example sentence: "nzelá molaí" (a long road)
**Old output (below)
<pre>
"<nzelá>"
"nzelá" n cl9 sg
"nzelá" n cl9 pl
"<molaí>"
"molaí" adj sg
"<.>"
"." sent
</pre>
:*New output (below)
<pre>
"<nzelá>"
"nzelá" n cl9 sg
; "nzelá" n cl9 pl REMOVE:25
"<molaí>"
"molaí" adj sg
"<.>"
"." sent
</pre>

*If I am a noun that can be singular or plural and am followed by a plural adjective, I cannot be singular
**Example sentence: "nzelá milaí" (long roads)
**Old output (below)
<pre>
"<nzelá>"
"nzelá" n cl9 sg
"nzelá" n cl9 pl
"<milaí>"
"molaí" adj pl
"<.>"
"." sent
</pre>
:*New output (below)
<pre>
"<nzelá>"
"nzelá" n cl9 pl
; "nzelá" n cl9 sg REMOVE:28
"<milaí>"
"molaí" adj pl
"<.>"
"." sent
</pre>

*If I can be an ordinal and a cardinal and am modifying a plural noun, I am not an ordinal. (One is never both an ordinal and cardinal, so this is not a problem.)
**Example sentence: "bilóko míbalé" (two things, ''not'' the second thing)
**Old output (below)
<pre>
"<bilóko>"
"elóko" n cl7 pl
"<míbalé>"
"míbalé" num ord
"míbalé" num card
"<.>"
"." sent

</pre>
:*New output (below)
<pre>
"<bilóko>"
"elóko" n cl7 pl
"<míbalé>"
"míbalé" num card
; "míbalé" num ord REMOVE:31
"<.>"
"." sent
</pre>

*If I can be and ordinal and a cardinal and am modifying a a singular noun, I am not a cardinal. (One is never both an ordinal and cardinal, so this is not a problem.)
**Example sentence: "elóko mibalé" (the second thing, ''not'' two things)
**Old output (below)
<pre>
"<elóko>"
"elóko" n cl7 sg
"<míbalé>"
"míbalé" num ord
"míbalé" num card
"<.>"
"." sent
</pre>
:*New output (below)
<pre>
"<elóko>"
"elóko" n cl7 sg
"<míbalé>"
"míbalé" num ord
; "míbalé" num card REMOVE:34
"<.>"
"." sent
</pre>

*If I can be a demonstrative nominal and a relative pronoun and am proceeded by a noun, I cannot be a nominal.
**Example sentence: "Mobali óyo amonaki ngáí" (The man who saw me...)
**Old output (below)
<pre>
"<Mobali>"
"mobali" n cl1 sg
"<óyo>"
"óyo" prn rel
"óyo" det dem
"óyo" prn dem sg
"<amonaki>"
"komóna" v urp p3 sg aa
"<ngáí>"
"ngáí" prn pers p1 sg aa
"<.>"
"." sent
</pre>
:*New output (below)
<pre>
"<Mobali>"
"mobali" n cl1 sg
"<óyo>"
"óyo" prn rel
"óyo" det dem
; "óyo" prn dem sg REMOVE:37
"<amonaki>"
"komóna" v urp p3 sg aa
"<ngáí>"
"ngáí" prn pers p1 sg aa
"<.>"
"." sent
</pre>

====Additional Transfer Rules====
*Enhance existing basic verb transfer rule to work with newly implemented imperative
**Example sentence: "ina" (sing)
**Old output
***Tagger: <code>^in<v><imp><p1><sg>$^.<sent>$</code>
***Biltrans: <code>^in<v><imp><p1><sg>/koyemba<v><imp><p1><sg>$^.<sent>/.<sent>$</code>
***Chunker: <code>^verb<verb>{^koyemba<v><p1><sg><aa>$}$^sent<SENT>{^.<sent>$}$</code>
***Interchunk: <code>^verb<verb>{^koyemba<v><p1><sg><aa>$}$^sent<SENT>{^.<sent>$}$</code>
***Postchunk: <code>^koyemba<v><p1><sg><aa>$^.<sent>$</code>
***Complete translation: <code>#koyemba</code>
**New output
***Tagger:<code>^in<v><imp><p1><sg>$^.<sent>$</code>
***Biltrans:<code>^in<v><imp><p1><sg>/koyemba<v><imp><p1><sg>$^.<sent>/.<sent>$</code>
***Chunker:<code>^verb<verb>{^koyemba<v><imp>$}$^sent<SENT>{^.<sent>$}$</code>
***Interchunk:<code>^verb<verb>{^koyemba<v><imp>$}$^sent<SENT>{^.<sent>$}$</code>
***Postchunk: <code>^koyemba<v><imp>$^.<sent>$</code>
***Complete translation: <code>yemba</code>

*Map Kikuyu associative to Lingala connective by removing extra agreement tags found in Kikuyu
**Example sentence: "wa"
**Old output
***Tagger: <code>^a<assoc><cl1><sg>$^.<sent>$</code>
***Biltrans: <code>^a<assoc><cl1><sg>/ya<conn><cl1><sg>$^.<sent>/.<sent>$</code>
***Chunker: <code>^default<default>{^ya<conn><cl1><sg>$}$^sent<SENT>{^.<sent>$}$</code>
***Interchunk: <code>^default<default>{^ya<conn><cl1><sg>$}$^sent<SENT>{^.<sent>$}$</code>
***Postchunk: <code>^ya<conn><cl1><sg>$^.<sent>$</code>
***Complete translation: <code>#ya</code>
**New output
***Tagger: <code>^a<assoc><cl1><sg>$^.<sent>$</code>
***Biltrans: <code>^a<assoc><cl1><sg>/ya<conn><cl1><sg>$^.<sent>/.<sent>$</code>
***Chunker: <code>^conn_chunk<conn>{^ya<conn>$}$^sent<SENT>{^.<sent>$}$</code>
***Interchunk: <code>^conn_chunk<conn>{^ya<conn>$}$^sent<SENT>{^.<sent>$}$</code>
***Postchunk: <code>^ya<conn>$^.<sent>$</code>
***Complete translation: <code>ya</code>

*Basic adjective transfer: keep number markings if a variable adjective in Lingala, discard otherwise
**Example sentence 1: anene (big)
**Old output
***Tagger: <code>^nene<adj><cl1><pl>$^.<sent>$</code>
***Biltrans: <code>^nene<adj><cl1><pl>/monéne<adj><cl1><pl>$^.<sent>/.<sent>$</code>
***Chunker: <code>^default<default>{^monéne<adj><cl1><pl>$}$^sent<SENT>{^.<sent>$}$</code>
***Interchunk: <code>^default<default>{^monéne<adj><cl1><pl>$}$^sent<SENT>{^.<sent>$}$</code>
***Postchunk: <code>^monéne<adj><cl1><pl>$^.<sent>$</code>
***Complete translation: <code>#monéne</code>
**New output
***Tagger: <code>^nene<adj><cl1><pl>$^.<sent>$</code>
***Biltrans: <code>^nene<adj><cl1><pl>/monéne<adj><cl1><pl>$^.<sent>/.<sent>$</code>
***Chunker: <code>^adjective<adj>{^monéne<adj><pl>$}$^sent<SENT>{^.<sent>$}$</code>
***Interchunk: <code>^adjective<adj>{^monéne<adj><pl>$}$^sent<SENT>{^.<sent>$}$</code>
***Postchunk: <code>^monéne<adj><pl>$^.<sent>$</code>
***Complete translation: <code>minéne</code>
**Example sentence 2: mata (sour)
**Old output
***Tagger: <code>^mata<adj><cl7><pl>$^.<sent>$</code>
***Biltrans: <code>^mata<adj><cl7><pl>/bololo<adj><cl7><pl>$^.<sent>/.<sent>$</code>
***Chunker: <code>^default<default>{^bololo<adj><cl7><pl>$}$^sent<SENT>{^.<sent>$}$</code>
***Interchunk: <code>^default<default>{^bololo<adj><cl7><pl>$}$^sent<SENT>{^.<sent>$}$</code>
***Postchunk: <code>^bololo<adj><cl7><pl>$^.<sent>$</code>
***Complete translation: <code>#bololo</code>
**New output
***Tagger: <code>^mata<adj><cl7><pl>$^.<sent>$</code>
***Biltrans: <code>^mata<adj><cl7><pl>/bololo<adj><cl7><pl>$^.<sent>/.<sent>$</code>
***Chunker: <code>^adjective<adj>{^bololo<adj>$}$^sent<SENT>{^.<sent>$}$</code>
***Interchunk: <code>^adjective<adj>{^bololo<adj>$}$^sent<SENT>{^.<sent>$}$</code>
***Postchunk: <code>^bololo<adj>$^.<sent>$</code>
***Complete translation: <code>bololo</code>

===lin Transducer Evaluation===
*Precision (annotated corpus): 80.53%
*Recall (annotated corpus): 55.64%
*Coverage (large corpus): 62.08%
*Number of words (large corpus): 275,546 (actual), 304,425 (tokenized)
*Number of stems in transducer: 273

===kik Transducer Evaluation===
*Precision (annotated corpus):
*Recall (annotated corpus):
*Coverage (large corpus):
*Number of words (large corpus):
*Number of stems in transducer:

===lin → kik MT Evaluation===
*WER (longer corpus):
*PER (longer corpus):
*Percentage unknown words:
*Percentage known words:
*Trimmed coverage over longer corpus [lin-kik.automorf.bin applied to lin.corpus.large.txt]: 48.88%
*Trimmed coverage over large corpus [lin-kik.automorf.bin applied to lin.longer.txt]: 88.57%
*Number of stems in longer corpus: 275,546 (actual), 304,425 (tokenized)
*Number of stems in large corpus: 117 (actual) 140 (tokenized, perhaps including punctuation)

===kik → lin MT Evaluation===
*WER (longer corpus):
*PER (longer corpus):
*Percentage unknown words:
*Percentage known words:
*Trimmed coverage over longer corpus [kik-lin.automorf.bin applied to kik.corpus.larger.txt]:
*Trimmed coverage over large corpus [kik-lin.automorf.bin applied to kik.longer.txt]:
*Number of stems in longer corpus:
*Number of stems in large corpus:

[[Category:Sp17_TranslationPairs]]
[[Category:Lingala]]
[[Category:Kikuyu]]