Latest revision as of 23:36, 8 May 2023

Git page for our Japhug transducer.

Git page for our corpus.

Wiki page to the list of resources.

Analyser evaluation

Evaluation

Total number of stems in transducer

Lexicons: 22
Lexicon entries: 99
Patterns: 10
Pattern entries: 22

Counts for individual lexicons:
Punctuation: 22
V-Negative: 2
N-Possessive: 12
V-Participles: 3
V-Stems: 21
V-Transitivity: 2
V-Causative: 1
V-Reflexive: 1
V-Passive: 1
N-Stems: 13
N-Collective: 1
Gender: 2
Numeral: 10
All anonymous lexicons: 8

Current coverage

Updated on Mar 20, 2023 We wrote a script and obtained over 6000 word roots from the Japhug dictionary by Guillaume Jacques [1] As we added the roots to the lexd file, the coverage becomes nearly 75%.

coverage: 2500 / 3341 (~0.74827895839568991320)
remaining unknown forms: 841
ambiguity: 3924 / 3341 (~1.17449865309787488776)

The current list of top unknown words returned by aq-covtest

TOP UNKNOWN WORDS:
     27 ^pɣɤtɕɯ/*pɣɤtɕɯ$
     10 ^tɕu/*tɕu$
      8 ^rɟɤlpu/*rɟɤlpu$
      8 ^rcanɯ/*rcanɯ$
      7 ^ɯrkɯ/*ɯrkɯ$
      7 ^tɤwa/*tɤwa$
      7 ^tɤsno/*tɤsno$
      7 ^tɕɤn/*tɕɤn$
      7 ^ki/*ki$
      6 ^ɯpi/*ɯpi$
      6 ^ɣɯ/*ɣɯ$
      6 ^tɤtɕɯ/*tɤtɕɯ$
      6 ^qhe/*qhe$
      6 ^pɯme/*pɯme$
      6 ^pjɤɕti/*pjɤɕti$
      6 ^pjɤŋu/*pjɤŋu$
      5 ^ɯloʁ/*ɯloʁ$
      5 ^tɯɣjɤn/*tɯɣjɤn$
      5 ^tonɯrdoʁ/*tonɯrdoʁ$
      5 ^pɯŋu/*pɯŋu$

Notes

Tests that completely fail:

• analysis and generation (Irrelevant tests- they are the "house/houses" example by Jonathan)

• Reduplicated Collective-gen and morph (we avoided reduplication and sandhi for this lab)

• Gerund-gen and morph (gerund contains partial reduplication, so we didn't code it)

• Sigmatic Causative-gen (0/11 test match gold)

Tests that partially fail (We need to check that we're able to generate all the gold standards in our terminal. Not sure why tests below failing):

• Possessive prefixes on nouns-gen (2/25 test not passing) and morph (2/20 test not passing)

• Negation on Verbs-gen (2/4 test do not match gold)

• Sigmatic Causative-morph (2/11 test match gold)

• Passive-morph (1/3 test not passing)

Additional top word(s) added:

• lɯlu (We already have lɯlu as 'lɯlu:lɯlɤ', the right part is the sandhi. But to improve coverage, we added 'lɯlu:lɯlu')

• βʑɯ ("mouse")

• tɕendɤre ("then")

• ɕti (affirmative "be")

Resulting change in coverage:

Coverage went from 24.23% to 31.26%

Generator evaluation

Initial evaluation of morphological generation

includes commonwords, excludes first two irreverent tests

number of passing morphological analysis tests: 16

number of failing morphological analysis tests: 5

number of gold matches for the generation test: 43

Current coverage: 31.26%

Final evaluation of morphological generation

number of passing morphological analysis tests: 17

number of failing morphological analysis tests: 4

number of twol rules that you added: 6

New coverage number: 31.26%

Precision and Recall

Precision: 100% (so 100% of the returned analyses were correct)

Recall: 65.09% (only 65.09% of the correct analyses were returned)

We ran precisionRecall jya.eval.test.txt jya.eval.gold.txt.