Difference between revisions of "Japhug/Transducer"
From LING073
(→The current list of top unknown words returned by aq-covtest) |
|||
| (30 intermediate revisions by 2 users not shown) | |||
| Line 1: | Line 1: | ||
| − | + | Git page for our [https://github.swarthmore.edu/Ling073-sp23/ling073-jya Japhug transducer]. | |
| + | Git page for our [https://github.swarthmore.edu/Ling073-sp23/ling073-jya-corpus corpus]. | ||
| + | |||
| + | [https://wikis.swarthmore.edu/ling073/Category:Japhug Wiki page] to the list of resources. | ||
| + | =Analyser evaluation= | ||
==Evaluation== | ==Evaluation== | ||
===Total number of stems in transducer=== | ===Total number of stems in transducer=== | ||
| − | + | <pre> | |
| − | Lexicons: | + | Lexicons: 22 |
| − | + | Lexicon entries: 99 | |
| − | Lexicon entries: | + | Patterns: 10 |
| − | + | Pattern entries: 22 | |
| − | Patterns: | ||
| − | |||
| − | Pattern entries: | ||
| − | |||
Counts for individual lexicons: | Counts for individual lexicons: | ||
| − | |||
Punctuation: 22 | Punctuation: 22 | ||
| − | |||
V-Negative: 2 | V-Negative: 2 | ||
| − | |||
N-Possessive: 12 | N-Possessive: 12 | ||
| − | |||
V-Participles: 3 | V-Participles: 3 | ||
| − | |||
V-Stems: 21 | V-Stems: 21 | ||
| − | |||
V-Transitivity: 2 | V-Transitivity: 2 | ||
| − | |||
V-Causative: 1 | V-Causative: 1 | ||
| − | |||
V-Reflexive: 1 | V-Reflexive: 1 | ||
| − | |||
V-Passive: 1 | V-Passive: 1 | ||
| − | + | N-Stems: 13 | |
| − | N-Stems: | + | N-Collective: 1 |
| − | |||
Gender: 2 | Gender: 2 | ||
| + | Numeral: 10 | ||
| + | All anonymous lexicons: 8 | ||
| + | </pre> | ||
| − | + | ===Current coverage=== | |
| − | + | Updated on Mar 20, 2023 | |
| + | We wrote a script and obtained over 6000 word roots from the Japhug dictionary by Guillaume Jacques [https://hal.inria.fr/halshs-01244860/] | ||
| + | As we added the roots to the lexd file, the coverage becomes nearly 75%. | ||
| − | + | <pre> | |
| + | coverage: 2500 / 3341 (~0.74827895839568991320) | ||
| + | remaining unknown forms: 841 | ||
| + | ambiguity: 3924 / 3341 (~1.17449865309787488776) | ||
| − | + | </pre> | |
===The current list of top unknown words returned by aq-covtest=== | ===The current list of top unknown words returned by aq-covtest=== | ||
<pre> | <pre> | ||
| − | + | TOP UNKNOWN WORDS: | |
| − | + | 27 ^pɣɤtɕɯ/*pɣɤtɕɯ$ | |
| − | + | 10 ^tɕu/*tɕu$ | |
| − | + | 8 ^rɟɤlpu/*rɟɤlpu$ | |
| − | + | 8 ^rcanɯ/*rcanɯ$ | |
| − | + | 7 ^ɯrkɯ/*ɯrkɯ$ | |
| − | + | 7 ^tɤwa/*tɤwa$ | |
| − | + | 7 ^tɤsno/*tɤsno$ | |
| − | + | 7 ^tɕɤn/*tɕɤn$ | |
| − | + | 7 ^ki/*ki$ | |
| − | + | 6 ^ɯpi/*ɯpi$ | |
| − | + | 6 ^ɣɯ/*ɣɯ$ | |
| − | + | 6 ^tɤtɕɯ/*tɤtɕɯ$ | |
| − | + | 6 ^qhe/*qhe$ | |
| − | + | 6 ^pɯme/*pɯme$ | |
| − | + | 6 ^pjɤɕti/*pjɤɕti$ | |
| − | + | 6 ^pjɤŋu/*pjɤŋu$ | |
| − | + | 5 ^ɯloʁ/*ɯloʁ$ | |
| − | + | 5 ^tɯɣjɤn/*tɯɣjɤn$ | |
| − | + | 5 ^tonɯrdoʁ/*tonɯrdoʁ$ | |
| − | + | 5 ^pɯŋu/*pɯŋu$ | |
| − | |||
| − | |||
</pre> | </pre> | ||
| − | |||
| − | |||
==Notes== | ==Notes== | ||
| − | Tests that completely fail: | + | '''Tests that completely fail: ''' |
*analysis and generation (Irrelevant tests- they are the "house/houses" example by Jonathan) | *analysis and generation (Irrelevant tests- they are the "house/houses" example by Jonathan) | ||
| Line 89: | Line 83: | ||
| − | Tests that partially fail (We're able to generate all the gold standards in our terminal. Not sure why failing): | + | '''Tests that partially fail''' (We need to check that we're able to generate all the gold standards in our terminal. Not sure why tests below failing): |
*Possessive prefixes on nouns-gen (2/25 test not passing) and morph (2/20 test not passing) | *Possessive prefixes on nouns-gen (2/25 test not passing) and morph (2/20 test not passing) | ||
| Line 98: | Line 92: | ||
*Passive-morph (1/3 test not passing) | *Passive-morph (1/3 test not passing) | ||
| + | |||
| + | '''Additional top word(s) added: ''' | ||
| + | *lɯlu (We already have lɯlu as 'lɯlu:lɯlɤ', the right part is the sandhi. But to improve coverage, we added 'lɯlu:lɯlu') | ||
| + | |||
| + | *βʑɯ ("mouse") | ||
| + | |||
| + | *tɕendɤre ("then") | ||
| + | |||
| + | *ɕti (affirmative "be") | ||
| + | |||
| + | |||
| + | '''Resulting change in coverage:''' | ||
| + | |||
| + | Coverage went from 24.23% to 31.26% | ||
| + | |||
| + | =Generator evaluation= | ||
| + | |||
| + | ==Initial evaluation of morphological generation== | ||
| + | |||
| + | ''includes commonwords, excludes first two irreverent tests'' | ||
| + | |||
| + | number of passing morphological analysis tests: 16 | ||
| + | |||
| + | number of failing morphological analysis tests: 5 | ||
| + | |||
| + | number of gold matches for the generation test: 43 | ||
| + | |||
| + | '''Current coverage''': 31.26% | ||
| + | |||
| + | ==Final evaluation of morphological generation== | ||
| + | |||
| + | number of passing morphological analysis tests: 17 | ||
| + | |||
| + | number of failing morphological analysis tests: 4 | ||
| + | |||
| + | number of twol rules that you added: 6 | ||
| + | |||
| + | '''New coverage number''': 31.26% | ||
| + | |||
| + | ==Precision and Recall== | ||
| + | |||
| + | Precision: 100% (so 100% of the returned analyses were correct) | ||
| + | Recall: 65.09% (only 65.09% of the correct analyses were returned) | ||
| + | We ran <code>precisionRecall jya.eval.test.txt jya.eval.gold.txt</code>. | ||
[[Category:Sp23_Transducers]][[Category:Japhug]][[Category:Transducers]] | [[Category:Sp23_Transducers]][[Category:Japhug]][[Category:Transducers]] | ||
Latest revision as of 23:36, 8 May 2023
Git page for our Japhug transducer.
Git page for our corpus.
Wiki page to the list of resources.
Analyser evaluation
Evaluation
Total number of stems in transducer
Lexicons: 22 Lexicon entries: 99 Patterns: 10 Pattern entries: 22 Counts for individual lexicons: Punctuation: 22 V-Negative: 2 N-Possessive: 12 V-Participles: 3 V-Stems: 21 V-Transitivity: 2 V-Causative: 1 V-Reflexive: 1 V-Passive: 1 N-Stems: 13 N-Collective: 1 Gender: 2 Numeral: 10 All anonymous lexicons: 8
Current coverage
Updated on Mar 20, 2023 We wrote a script and obtained over 6000 word roots from the Japhug dictionary by Guillaume Jacques [1] As we added the roots to the lexd file, the coverage becomes nearly 75%.
coverage: 2500 / 3341 (~0.74827895839568991320) remaining unknown forms: 841 ambiguity: 3924 / 3341 (~1.17449865309787488776)
The current list of top unknown words returned by aq-covtest
TOP UNKNOWN WORDS:
27 ^pɣɤtɕɯ/*pɣɤtɕɯ$
10 ^tɕu/*tɕu$
8 ^rɟɤlpu/*rɟɤlpu$
8 ^rcanɯ/*rcanɯ$
7 ^ɯrkɯ/*ɯrkɯ$
7 ^tɤwa/*tɤwa$
7 ^tɤsno/*tɤsno$
7 ^tɕɤn/*tɕɤn$
7 ^ki/*ki$
6 ^ɯpi/*ɯpi$
6 ^ɣɯ/*ɣɯ$
6 ^tɤtɕɯ/*tɤtɕɯ$
6 ^qhe/*qhe$
6 ^pɯme/*pɯme$
6 ^pjɤɕti/*pjɤɕti$
6 ^pjɤŋu/*pjɤŋu$
5 ^ɯloʁ/*ɯloʁ$
5 ^tɯɣjɤn/*tɯɣjɤn$
5 ^tonɯrdoʁ/*tonɯrdoʁ$
5 ^pɯŋu/*pɯŋu$
Notes
Tests that completely fail:
- analysis and generation (Irrelevant tests- they are the "house/houses" example by Jonathan)
- Reduplicated Collective-gen and morph (we avoided reduplication and sandhi for this lab)
- Gerund-gen and morph (gerund contains partial reduplication, so we didn't code it)
- Sigmatic Causative-gen (0/11 test match gold)
Tests that partially fail (We need to check that we're able to generate all the gold standards in our terminal. Not sure why tests below failing):
- Possessive prefixes on nouns-gen (2/25 test not passing) and morph (2/20 test not passing)
- Negation on Verbs-gen (2/4 test do not match gold)
- Sigmatic Causative-morph (2/11 test match gold)
- Passive-morph (1/3 test not passing)
Additional top word(s) added:
- lɯlu (We already have lɯlu as 'lɯlu:lɯlɤ', the right part is the sandhi. But to improve coverage, we added 'lɯlu:lɯlu')
- βʑɯ ("mouse")
- tɕendɤre ("then")
- ɕti (affirmative "be")
Resulting change in coverage:
Coverage went from 24.23% to 31.26%
Generator evaluation
Initial evaluation of morphological generation
includes commonwords, excludes first two irreverent tests
number of passing morphological analysis tests: 16
number of failing morphological analysis tests: 5
number of gold matches for the generation test: 43
Current coverage: 31.26%
Final evaluation of morphological generation
number of passing morphological analysis tests: 17
number of failing morphological analysis tests: 4
number of twol rules that you added: 6
New coverage number: 31.26%
Precision and Recall
Precision: 100% (so 100% of the returned analyses were correct)
Recall: 65.09% (only 65.09% of the correct analyses were returned)
We ran precisionRecall jya.eval.test.txt jya.eval.gold.txt.
