Difference between revisions of "Japhug/Transducer"
From LING073
(→Evaluation) |
|||
| (34 intermediate revisions by 2 users not shown) | |||
| Line 1: | Line 1: | ||
| − | + | Git page for our [https://github.swarthmore.edu/Ling073-sp23/ling073-jya Japhug transducer]. | |
| − | [https://github.swarthmore.edu/Ling073-sp23/ling073-jya | + | Git page for our [https://github.swarthmore.edu/Ling073-sp23/ling073-jya-corpus corpus]. |
| + | [https://wikis.swarthmore.edu/ling073/Category:Japhug Wiki page] to the list of resources. | ||
| + | =Analyser evaluation= | ||
==Evaluation== | ==Evaluation== | ||
===Total number of stems in transducer=== | ===Total number of stems in transducer=== | ||
| + | <pre> | ||
| + | Lexicons: 22 | ||
| + | Lexicon entries: 99 | ||
| + | Patterns: 10 | ||
| + | Pattern entries: 22 | ||
| − | + | Counts for individual lexicons: | |
| + | Punctuation: 22 | ||
| + | V-Negative: 2 | ||
| + | N-Possessive: 12 | ||
| + | V-Participles: 3 | ||
| + | V-Stems: 21 | ||
| + | V-Transitivity: 2 | ||
| + | V-Causative: 1 | ||
| + | V-Reflexive: 1 | ||
| + | V-Passive: 1 | ||
| + | N-Stems: 13 | ||
| + | N-Collective: 1 | ||
| + | Gender: 2 | ||
| + | Numeral: 10 | ||
| + | All anonymous lexicons: 8 | ||
| + | </pre> | ||
| + | |||
| + | ===Current coverage=== | ||
| + | |||
| + | Updated on Mar 20, 2023 | ||
| + | We wrote a script and obtained over 6000 word roots from the Japhug dictionary by Guillaume Jacques [https://hal.inria.fr/halshs-01244860/] | ||
| + | As we added the roots to the lexd file, the coverage becomes nearly 75%. | ||
| − | + | <pre> | |
| + | coverage: 2500 / 3341 (~0.74827895839568991320) | ||
| + | remaining unknown forms: 841 | ||
| + | ambiguity: 3924 / 3341 (~1.17449865309787488776) | ||
| − | |||
| − | + | </pre> | |
| + | ===The current list of top unknown words returned by aq-covtest=== | ||
| + | <pre> | ||
| + | TOP UNKNOWN WORDS: | ||
| + | 27 ^pɣɤtɕɯ/*pɣɤtɕɯ$ | ||
| + | 10 ^tɕu/*tɕu$ | ||
| + | 8 ^rɟɤlpu/*rɟɤlpu$ | ||
| + | 8 ^rcanɯ/*rcanɯ$ | ||
| + | 7 ^ɯrkɯ/*ɯrkɯ$ | ||
| + | 7 ^tɤwa/*tɤwa$ | ||
| + | 7 ^tɤsno/*tɤsno$ | ||
| + | 7 ^tɕɤn/*tɕɤn$ | ||
| + | 7 ^ki/*ki$ | ||
| + | 6 ^ɯpi/*ɯpi$ | ||
| + | 6 ^ɣɯ/*ɣɯ$ | ||
| + | 6 ^tɤtɕɯ/*tɤtɕɯ$ | ||
| + | 6 ^qhe/*qhe$ | ||
| + | 6 ^pɯme/*pɯme$ | ||
| + | 6 ^pjɤɕti/*pjɤɕti$ | ||
| + | 6 ^pjɤŋu/*pjɤŋu$ | ||
| + | 5 ^ɯloʁ/*ɯloʁ$ | ||
| + | 5 ^tɯɣjɤn/*tɯɣjɤn$ | ||
| + | 5 ^tonɯrdoʁ/*tonɯrdoʁ$ | ||
| + | 5 ^pɯŋu/*pɯŋu$ | ||
| + | </pre> | ||
| − | + | ==Notes== | |
| + | '''Tests that completely fail: ''' | ||
| + | |||
| + | *analysis and generation (Irrelevant tests- they are the "house/houses" example by Jonathan) | ||
| + | |||
| + | *Reduplicated Collective-gen and morph (we avoided reduplication and sandhi for this lab) | ||
| + | |||
| + | *Gerund-gen and morph (gerund contains partial reduplication, so we didn't code it) | ||
| + | |||
| + | *Sigmatic Causative-gen (0/11 test match gold) | ||
| + | |||
| + | |||
| + | '''Tests that partially fail''' (We need to check that we're able to generate all the gold standards in our terminal. Not sure why tests below failing): | ||
| + | |||
| + | *Possessive prefixes on nouns-gen (2/25 test not passing) and morph (2/20 test not passing) | ||
| + | |||
| + | *Negation on Verbs-gen (2/4 test do not match gold) | ||
| + | |||
| + | *Sigmatic Causative-morph (2/11 test match gold) | ||
| + | |||
| + | *Passive-morph (1/3 test not passing) | ||
| + | |||
| + | |||
| + | '''Additional top word(s) added: ''' | ||
| + | *lɯlu (We already have lɯlu as 'lɯlu:lɯlɤ', the right part is the sandhi. But to improve coverage, we added 'lɯlu:lɯlu') | ||
| + | |||
| + | *βʑɯ ("mouse") | ||
| + | |||
| + | *tɕendɤre ("then") | ||
| − | + | *ɕti (affirmative "be") | |
| + | |||
| − | + | '''Resulting change in coverage:''' | |
| − | + | Coverage went from 24.23% to 31.26% | |
| − | + | =Generator evaluation= | |
| − | + | ==Initial evaluation of morphological generation== | |
| − | + | ''includes commonwords, excludes first two irreverent tests'' | |
| − | + | number of passing morphological analysis tests: 16 | |
| − | + | number of failing morphological analysis tests: 5 | |
| − | + | number of gold matches for the generation test: 43 | |
| − | + | '''Current coverage''': 31.26% | |
| − | + | ==Final evaluation of morphological generation== | |
| − | + | number of passing morphological analysis tests: 17 | |
| − | + | number of failing morphological analysis tests: 4 | |
| − | + | number of twol rules that you added: 6 | |
| + | '''New coverage number''': 31.26% | ||
| − | == | + | ==Precision and Recall== |
| − | + | Precision: 100% (so 100% of the returned analyses were correct) | |
| − | + | Recall: 65.09% (only 65.09% of the correct analyses were returned) | |
| − | + | We ran <code>precisionRecall jya.eval.test.txt jya.eval.gold.txt</code>. | |
| − | |||
| − | |||
[[Category:Sp23_Transducers]][[Category:Japhug]][[Category:Transducers]] | [[Category:Sp23_Transducers]][[Category:Japhug]][[Category:Transducers]] | ||
Latest revision as of 23:36, 8 May 2023
Git page for our Japhug transducer.
Git page for our corpus.
Wiki page to the list of resources.
Analyser evaluation
Evaluation
Total number of stems in transducer
Lexicons: 22 Lexicon entries: 99 Patterns: 10 Pattern entries: 22 Counts for individual lexicons: Punctuation: 22 V-Negative: 2 N-Possessive: 12 V-Participles: 3 V-Stems: 21 V-Transitivity: 2 V-Causative: 1 V-Reflexive: 1 V-Passive: 1 N-Stems: 13 N-Collective: 1 Gender: 2 Numeral: 10 All anonymous lexicons: 8
Current coverage
Updated on Mar 20, 2023 We wrote a script and obtained over 6000 word roots from the Japhug dictionary by Guillaume Jacques [1] As we added the roots to the lexd file, the coverage becomes nearly 75%.
coverage: 2500 / 3341 (~0.74827895839568991320) remaining unknown forms: 841 ambiguity: 3924 / 3341 (~1.17449865309787488776)
The current list of top unknown words returned by aq-covtest
TOP UNKNOWN WORDS:
27 ^pɣɤtɕɯ/*pɣɤtɕɯ$
10 ^tɕu/*tɕu$
8 ^rɟɤlpu/*rɟɤlpu$
8 ^rcanɯ/*rcanɯ$
7 ^ɯrkɯ/*ɯrkɯ$
7 ^tɤwa/*tɤwa$
7 ^tɤsno/*tɤsno$
7 ^tɕɤn/*tɕɤn$
7 ^ki/*ki$
6 ^ɯpi/*ɯpi$
6 ^ɣɯ/*ɣɯ$
6 ^tɤtɕɯ/*tɤtɕɯ$
6 ^qhe/*qhe$
6 ^pɯme/*pɯme$
6 ^pjɤɕti/*pjɤɕti$
6 ^pjɤŋu/*pjɤŋu$
5 ^ɯloʁ/*ɯloʁ$
5 ^tɯɣjɤn/*tɯɣjɤn$
5 ^tonɯrdoʁ/*tonɯrdoʁ$
5 ^pɯŋu/*pɯŋu$
Notes
Tests that completely fail:
- analysis and generation (Irrelevant tests- they are the "house/houses" example by Jonathan)
- Reduplicated Collective-gen and morph (we avoided reduplication and sandhi for this lab)
- Gerund-gen and morph (gerund contains partial reduplication, so we didn't code it)
- Sigmatic Causative-gen (0/11 test match gold)
Tests that partially fail (We need to check that we're able to generate all the gold standards in our terminal. Not sure why tests below failing):
- Possessive prefixes on nouns-gen (2/25 test not passing) and morph (2/20 test not passing)
- Negation on Verbs-gen (2/4 test do not match gold)
- Sigmatic Causative-morph (2/11 test match gold)
- Passive-morph (1/3 test not passing)
Additional top word(s) added:
- lɯlu (We already have lɯlu as 'lɯlu:lɯlɤ', the right part is the sandhi. But to improve coverage, we added 'lɯlu:lɯlu')
- βʑɯ ("mouse")
- tɕendɤre ("then")
- ɕti (affirmative "be")
Resulting change in coverage:
Coverage went from 24.23% to 31.26%
Generator evaluation
Initial evaluation of morphological generation
includes commonwords, excludes first two irreverent tests
number of passing morphological analysis tests: 16
number of failing morphological analysis tests: 5
number of gold matches for the generation test: 43
Current coverage: 31.26%
Final evaluation of morphological generation
number of passing morphological analysis tests: 17
number of failing morphological analysis tests: 4
number of twol rules that you added: 6
New coverage number: 31.26%
Precision and Recall
Precision: 100% (so 100% of the returned analyses were correct)
Recall: 65.09% (only 65.09% of the correct analyses were returned)
We ran precisionRecall jya.eval.test.txt jya.eval.gold.txt.
