Shan/Transducer

Repository

Link to the Shan transducer repository:https://github.swarthmore.edu/Ling073-sp23/ling073-shn

Total Number of Stems in the Transducer

Lexicons: 30

Lexicon entries: 230

Patterns: 1

Pattern entries: 15

Counts for individual lexicons:

Noun: 25

Numeral: 25

Punctuation: 22

Classifier: 5

Pronoun: 20

AuxiliaryAfter: 4

AuxiliaryBefore: 4

Conjunction: 5

Adjective: 5

Adverb: 14

Interjection: 5

Preposition: 5

Verb: 64

QuestionParticle: 11

Determiner: 2

All anonymous lexicons: 14

Initial Evaluation

Coverage after running the analyzer is 9/574

After looking at the common words, we added three more words into our transducer. This increased the coverage to 13/574.

Current Coverage

13 / 574 (~2%)

Current list of top unknowns

ဢၼ်လီ<unk>

ၵိၼ်မႃး<unk>

ႁဵတ်းသင်<unk>

ႁဵတ်းတိၵ်းတိၵ်း<unk>

ၽူၼ်မႃးၼႆ<unk>

ၵေႃႉဢၼ်မႃးယူႇ<unk>

ၵမ်ၸိုၼ်း<unk>

ဢမ်ႇ<unk>

လႃႈလီႁဵတ်း<unk>

ယေႈၶႅၼ်းလီ<unk>

Tests

All morph and gen tests pass

Notes

Because there is a lack of morphology, all tests pass for the morph and gen tests after adding the vocabulary to the lexd file.

Evaluation with tokeniser

Coverage after running the analyzer with the tokensier is 129 / 1589

Current Coverage

129 / 1589 (~8%)

Current list of top unknowns

း<unk>

ႈ<unk>

ႇ<unk>

ၵ<unk>

ႉ<unk>

ႁ<unk>

မၼ<unk>

လ<unk>

ပ<unk>

််<unk>

ၶ<unk>

တ<unk>

ဢမ<unk>

မ<unk>

သ<unk>

ယ<unk>

ူ<unk>

ႆႆ<unk>

ႃႇ<unk>

ၼ<unk>

Tests

All morph and gen tests pass

Notes

The tokeniser is not working very well. It think that some letters are words as we can see from the unknown words. These are all just letters. However, the tokensier does tokenise some words correctly which is why the coverage of the basic corpus increased with the tokeniser on.

Spellrelax

Diacritics are notoriously hard to deal with and different interfaces display different diacritics in many different ways. So we have added spell relax rules to let the transducer accept diacritics that come before and after the vowel to be the same thing to try and alleviate that problem.

Large Corpus Evaluation

The large corpus we will be testing on is all wikipedia pages written in the Shan language. The corpus has been cleaned before using.

No tokeniser, No spellrelax

Coverage

224754 / 883263 (~25%)

Top Unknown Words

24775 ^ဝၼ်း။/*ဝၼ်း။$

10055 ^သေ/*သေ$

9591 ^ယဝ်ႉ။/*ယဝ်ႉ။$

8944 ^၊/*၊$

8491 ^ၵေႃႉ/*ၵေႃႉ$

6278 ^ပဵၼ်/*ပဵၼ်$

5388 ^မီးဝႆႉ/*မီးဝႆႉ$

4668 ^ႁူဝ်ၼပ်ႉၵူၼ်း/*ႁူဝ်ၼပ်ႉၵူၼ်း$

4579 ^ၵေႃႉယဝ်ႉ။/*ၵေႃႉယဝ်ႉ။$

Tokensier, No spellrelax

Coverage

475688 / 1307850 (~36%)

Top Unknown Words

24774 ^ဝၼ်း။/*ဝၼ်း။$

23412 ^လ/*လ$

20324 ^ႇ/*ႇ$

19396 ^း/*း$

13200 ^ႈ/*ႈ$

10622 ^သေ/*သေ$

9855 ^၊/*၊$

9468 ^ယဝ်ႉ။/*ယဝ်ႉ။$

No tokeniser, Spellrelax

Coverage

224752 / 883263 (~25%)

Top Unknown Words

24775 ^ဝၼ်း။/*ဝၼ်း။$

10055 ^သေ/*သေ$

9591 ^ယဝ်ႉ။/*ယဝ်ႉ။$

8944 ^၊/*၊$

8491 ^ၵေႃႉ/*ၵေႃႉ$

6278 ^ပဵၼ်/*ပဵၼ်$

5388 ^မီးဝႆႉ/*မီးဝႆႉ$

Tokensier, Spellrelax

Coverage

713556 / 1486937 (~47%)

Top Unknown Words

29640 ^။/*။$

24774 ^ဝၼ်း။/*ဝၼ်း။$

14277 ^သေ/*သေ$

12722 ^၊/*၊$

10429 ^မီးဝႆႉ/*မီးဝႆႉ$

10281 ^ယဝ်ႉ/*ယဝ်ႉ$

9473 ^ယဝ်ႉ။/*ယဝ်ႉ။$

Notes

As we can see there is a big improvement in coverage with and without the tokeniser. Also, the spellrelax did not seem to do anything without the tokeniser but it worked very well with the tokeniser. I'm guessing this is because even though the spellrelax is working as intended, without tokenising, the spell corrections don't create recognizable words regardless so it didn't increase the ocverage without the tokeniser.

Precision and Recall

Precision: 74.4%

Recall: 30.9%