Difference between revisions of "Japhug/Transducer"
(→Initial evaluation of morphological generation) |
|||
| (10 intermediate revisions by 2 users not shown) | |||
| Line 1: | Line 1: | ||
| − | + | Git page for our [https://github.swarthmore.edu/Ling073-sp23/ling073-jya Japhug transducer]. | |
| + | |||
| + | Git page for our [https://github.swarthmore.edu/Ling073-sp23/ling073-jya-corpus corpus]. | ||
[https://wikis.swarthmore.edu/ling073/Category:Japhug Wiki page] to the list of resources. | [https://wikis.swarthmore.edu/ling073/Category:Japhug Wiki page] to the list of resources. | ||
| Line 31: | Line 33: | ||
===Current coverage=== | ===Current coverage=== | ||
| + | |||
| + | Updated on Mar 20, 2023 | ||
| + | We wrote a script and obtained over 6000 word roots from the Japhug dictionary by Guillaume Jacques [https://hal.inria.fr/halshs-01244860/] | ||
| + | As we added the roots to the lexd file, the coverage becomes nearly 75%. | ||
| + | |||
<pre> | <pre> | ||
| − | + | coverage: 2500 / 3341 (~0.74827895839568991320) | |
| − | + | remaining unknown forms: 841 | |
| + | ambiguity: 3924 / 3341 (~1.17449865309787488776) | ||
| + | |||
| + | |||
</pre> | </pre> | ||
===The current list of top unknown words returned by aq-covtest=== | ===The current list of top unknown words returned by aq-covtest=== | ||
<pre> | <pre> | ||
| − | + | TOP UNKNOWN WORDS: | |
| − | + | 27 ^pɣɤtɕɯ/*pɣɤtɕɯ$ | |
| − | + | 10 ^tɕu/*tɕu$ | |
| − | + | 8 ^rɟɤlpu/*rɟɤlpu$ | |
| − | + | 8 ^rcanɯ/*rcanɯ$ | |
| − | + | 7 ^ɯrkɯ/*ɯrkɯ$ | |
| − | + | 7 ^tɤwa/*tɤwa$ | |
| − | + | 7 ^tɤsno/*tɤsno$ | |
| − | + | 7 ^tɕɤn/*tɕɤn$ | |
| − | + | 7 ^ki/*ki$ | |
| − | + | 6 ^ɯpi/*ɯpi$ | |
| − | + | 6 ^ɣɯ/*ɣɯ$ | |
| − | + | 6 ^tɤtɕɯ/*tɤtɕɯ$ | |
| − | + | 6 ^qhe/*qhe$ | |
| − | + | 6 ^pɯme/*pɯme$ | |
| − | + | 6 ^pjɤɕti/*pjɤɕti$ | |
| − | + | 6 ^pjɤŋu/*pjɤŋu$ | |
| − | + | 5 ^ɯloʁ/*ɯloʁ$ | |
| − | + | 5 ^tɯɣjɤn/*tɯɣjɤn$ | |
| − | + | 5 ^tonɯrdoʁ/*tonɯrdoʁ$ | |
| − | + | 5 ^pɯŋu/*pɯŋu$ | |
</pre> | </pre> | ||
| Line 114: | Line 124: | ||
==Final evaluation of morphological generation== | ==Final evaluation of morphological generation== | ||
| − | number of passing morphological analysis tests: | + | number of passing morphological analysis tests: 17 |
| + | |||
| + | number of failing morphological analysis tests: 4 | ||
| + | |||
| + | number of twol rules that you added: 6 | ||
| + | |||
| + | '''New coverage number''': 31.26% | ||
| + | |||
| + | ==Precision and Recall== | ||
| + | |||
| + | Precision: 100% (so 100% of the returned analyses were correct) | ||
| − | + | Recall: 65.09% (only 65.09% of the correct analyses were returned) | |
| − | + | We ran <code>precisionRecall jya.eval.test.txt jya.eval.gold.txt</code>. | |
| − | |||
[[Category:Sp23_Transducers]][[Category:Japhug]][[Category:Transducers]] | [[Category:Sp23_Transducers]][[Category:Japhug]][[Category:Transducers]] | ||
Latest revision as of 23:36, 8 May 2023
Git page for our Japhug transducer.
Git page for our corpus.
Wiki page to the list of resources.
Analyser evaluation
Evaluation
Total number of stems in transducer
Lexicons: 22 Lexicon entries: 99 Patterns: 10 Pattern entries: 22 Counts for individual lexicons: Punctuation: 22 V-Negative: 2 N-Possessive: 12 V-Participles: 3 V-Stems: 21 V-Transitivity: 2 V-Causative: 1 V-Reflexive: 1 V-Passive: 1 N-Stems: 13 N-Collective: 1 Gender: 2 Numeral: 10 All anonymous lexicons: 8
Current coverage
Updated on Mar 20, 2023 We wrote a script and obtained over 6000 word roots from the Japhug dictionary by Guillaume Jacques [1] As we added the roots to the lexd file, the coverage becomes nearly 75%.
coverage: 2500 / 3341 (~0.74827895839568991320) remaining unknown forms: 841 ambiguity: 3924 / 3341 (~1.17449865309787488776)
The current list of top unknown words returned by aq-covtest
TOP UNKNOWN WORDS:
27 ^pɣɤtɕɯ/*pɣɤtɕɯ$
10 ^tɕu/*tɕu$
8 ^rɟɤlpu/*rɟɤlpu$
8 ^rcanɯ/*rcanɯ$
7 ^ɯrkɯ/*ɯrkɯ$
7 ^tɤwa/*tɤwa$
7 ^tɤsno/*tɤsno$
7 ^tɕɤn/*tɕɤn$
7 ^ki/*ki$
6 ^ɯpi/*ɯpi$
6 ^ɣɯ/*ɣɯ$
6 ^tɤtɕɯ/*tɤtɕɯ$
6 ^qhe/*qhe$
6 ^pɯme/*pɯme$
6 ^pjɤɕti/*pjɤɕti$
6 ^pjɤŋu/*pjɤŋu$
5 ^ɯloʁ/*ɯloʁ$
5 ^tɯɣjɤn/*tɯɣjɤn$
5 ^tonɯrdoʁ/*tonɯrdoʁ$
5 ^pɯŋu/*pɯŋu$
Notes
Tests that completely fail:
- analysis and generation (Irrelevant tests- they are the "house/houses" example by Jonathan)
- Reduplicated Collective-gen and morph (we avoided reduplication and sandhi for this lab)
- Gerund-gen and morph (gerund contains partial reduplication, so we didn't code it)
- Sigmatic Causative-gen (0/11 test match gold)
Tests that partially fail (We need to check that we're able to generate all the gold standards in our terminal. Not sure why tests below failing):
- Possessive prefixes on nouns-gen (2/25 test not passing) and morph (2/20 test not passing)
- Negation on Verbs-gen (2/4 test do not match gold)
- Sigmatic Causative-morph (2/11 test match gold)
- Passive-morph (1/3 test not passing)
Additional top word(s) added:
- lɯlu (We already have lɯlu as 'lɯlu:lɯlɤ', the right part is the sandhi. But to improve coverage, we added 'lɯlu:lɯlu')
- βʑɯ ("mouse")
- tɕendɤre ("then")
- ɕti (affirmative "be")
Resulting change in coverage:
Coverage went from 24.23% to 31.26%
Generator evaluation
Initial evaluation of morphological generation
includes commonwords, excludes first two irreverent tests
number of passing morphological analysis tests: 16
number of failing morphological analysis tests: 5
number of gold matches for the generation test: 43
Current coverage: 31.26%
Final evaluation of morphological generation
number of passing morphological analysis tests: 17
number of failing morphological analysis tests: 4
number of twol rules that you added: 6
New coverage number: 31.26%
Precision and Recall
Precision: 100% (so 100% of the returned analyses were correct)
Recall: 65.09% (only 65.09% of the correct analyses were returned)
We ran precisionRecall jya.eval.test.txt jya.eval.gold.txt.
