Lingala/Transducer

A transducer for Lingala is available on Github.

Generator Evaluation

Initial Evaluation of Morphological Generation

Using the -csi flag as in the analysis assignment, there are currently 119 tests that pass and 14 tests that fail, for a total of 133 tests. The corpus currently has a coverage of 35.22% (with 1485 tokenized words).

Using the -cl flag to test morphological generation, there are initially 113 tests that pass and 31 tests that fail, for a total of 144 tests. Note that 6 of these failing tests are spellrelax tests, which should not be considered for the purposes of morphological generation (since the wrong form will never be generated).

Final Evaluation of Morphological Generation

There are now 115 passing tests and 18 failing tests, for a total of 133. Note that 6 of the failing tests are spellrelax tests, and so should not be considered for the purposes of this assignment. This represents 86% passing tests if including the spellrelax tests and 90% passing tests if not.

For this part of the assignment, I added a single twol rule. This is to deal with the present tense verb suffix. Almost all verbs take the suffix -í in the present tense, but those with a stem ending in -w- take the suffix -é. I set up an archiphoneme for the present tense endings that by default maps to í, but the twol rule changes it to é with preceded by a w.

I also modified one existing rule in my twoc file, the inverse rule for prefix tagging. This rule checks the number of square bracket tags in the suffix (typically denoted with a +), and for each one, ensures there is a matching square bracket tag in the prefix (typically denoted with a -). All forms without a matching prefix tag are suppressed. I modified the rule by adding the suffix versions of the singular and plural tags to the matched sets, which is necessary for adjective generation to work correctly since invariable adjectives do not have any square-bracket suffix tags and should not be tagged for number.

I have one additional rule in the twoc file, the main rule for prefix tagging that checks that there is a square bracket tag in the suffix for every square bracket tag in the prefix, that required no modification for this portion of the assignment.

As part of the assignment, the binary used for generation had to be changed to hfst-proc, from lt-proc. This was done with the assistance of Francis Tyers, who also added support for periods and commas to the lexc file for debugging purposes. As a result of these changes, and after all additional work with twol rules, corpus coverage increased to 41.50%, with 1665 tokenized words being analyzed.

Analyzer Evaluation

Evaluation

Coverage

After adding four common words to the transducer, there are 63 stems, giving a coverage of 35.22% of the corpus. The improvement gained by adding additional common words to the transducer is detailed in the Notes section.

Top Unknown Words

The top unknown words are as follows:

• nsɔ (21)
• nyɔ (14)
• mi (12)
• ete (11)
• lya (10)
• wa (10)
• mwa (10)
• nkema (9)
• Nzambe (9)
• li (8)
• té (7)
• ma (7)
• likolo (7)
• bwa (7)
• esalemaki (7)
• ba (7)
• Nkolo (7)
• banzelu (6)
• tata (6)
• likoki (6)

Tests

In the main yaml file (lin.yaml), 119 tests pass and 14 fail, for a total of 133 tests. In the common words yaml file, 4 tests pass and 13 fail, for a total of 17 tests (spellrelax caused some tests to be viewed as duplicates, even though aq-covtest reported them as distinct words).

Notes

Addition of Common Words

The unknown words added to the corpus include:

• o<pr> ↔ o, a preposition whose addition brought the (initial coverage) from 26.06% to 29.23%
• mpé<cngcoo> ↔ mpé, a conjunction (often translated "and"), which increased the coverage from 29.23% to 32.93%
• to<cngcoo> ↔ to, a conjunction (translated "or"), which increased coverage from 32.93% to 33.80%
• te<neg> ↔ te, the Lingala word for "no" which is used generally to indicate negation, which increased coverage from 33.80% to 35.22%

Failing Tests

• mokonzi<n><pl><cl1><abs><inst> ↔ bakimokonzi fails because I only implemented a single level of prefix stacking. I get the sense that it is quite rare to have more than two prefix (with the grammar book only mentioning the possibility in passing). Implementing another layer of prefix stacking would require a substantial number of additional suffix entries in the lexicon due to the way Apertium handles prefixes. It seemed unproductive to clutter the lexicon in this way for such a rarely used morphological form.
• Tests for the pronoun baýo wâna and óyo wâna fail because each pronoun essentially requires its own stem and suffix lexicon. A cursory review of Lingala literature suggests that use of this demonstrative is rare (with óyo being preferred), so it seemed unproductive to clutter the lexicon to support this, at least initially.
• Most of the tests for verb mood fail. This is mainly because support for the subjunctive mood has not yet been implemented. Because of the way prefixes are handled, adding an additional verb prefix possibility requires doubling the number of verb suffix possibilities, of which there are already an extremely large number. Since the subjunctive is denoted by tone only, which is not typically marked in literature, it seemed unproductive to add all of the extra suffix possibilities since the form would likely never be analyzed.
• Two of the verb present tense tests past. This is because the verb stem in question, kokámwa, takes an irregular ending in the present tense, é instead of í. For the initial version of the transducer, I have limited endings to the present tense to the most commonly found, í. If it turns out there are a large number of verbs that end with é in the present tense, support can be added in the future.