Difference between revisions of "Okinawan/Transducer"
(Created page with "== Evaluation == My current coverage is horrendous (0.48%) due to a tokenization error, I believe. It reports many single-character words, but my corpus does not seem to have...") |
|||
| (2 intermediate revisions by the same user not shown) | |||
| Line 1: | Line 1: | ||
| + | = Analyser = | ||
| + | |||
== Evaluation == | == Evaluation == | ||
| − | My current coverage is horrendous (0.48%) due to a tokenization error, I believe. It reports many single-character words, but my corpus does not seem to have the ones that aq-covtest claims to see. However, there are several | + | My current coverage is horrendous (0.48%) due to a tokenization error, I believe. It reports many single-character words (such as な, え, お, and friends) but my corpus does not seem to have the ones that aq-covtest claims to see. However, there are several real words that it is (correctly) identifying as commonly unknown. These tend to be discourse markers, such as |
| − | real words that it is (correctly) identifying as commonly unknown. These tend to be discourse markers, such as | ||
なあ, また, as a lot of my corpora are transcripts of conversations. Adding these analyses to the transducer increases coverage to 4.43%. | なあ, また, as a lot of my corpora are transcripts of conversations. Adding these analyses to the transducer increases coverage to 4.43%. | ||
| + | There are currently 27 stems, according to lexccounter. | ||
| + | |||
| + | Number of tokenised words in the corpus: 835 | ||
| + | Coverage: 4.43% | ||
| + | Top unknown words in the corpus: | ||
| + | About 28/50 tests currently pass. | ||
| + | |||
| + | 37 え | ||
| + | 17 ち | ||
| + | 16 ん | ||
| + | 14 う | ||
| + | 14 さ | ||
| + | 13 い | ||
| + | 11 ら | ||
| + | 11 とぅ | ||
| + | 10 うり | ||
| + | 10 あ | ||
| + | 10 や | ||
| + | 10 お | ||
| + | 9 ぬ | ||
| + | 8 か | ||
| + | 7 み | ||
| + | 7 やい | ||
| + | 7 うまんか | ||
| + | 7 ちゅ | ||
| + | 6 あんち | ||
| + | 6 さい | ||
| + | Translation time: 0.004895448684692383 seconds | ||
== Notes == | == Notes == | ||
The tests that don't yet pass are either some extra nominal suffixive tests that I haven't gotten around to yet, or involve verbs. Verbs are a bit more complicated than the nominals, since they retain the verb stem and overwrite the rest. | The tests that don't yet pass are either some extra nominal suffixive tests that I haven't gotten around to yet, or involve verbs. Verbs are a bit more complicated than the nominals, since they retain the verb stem and overwrite the rest. | ||
| + | |||
| + | = Generator = | ||
| + | |||
| + | == Initial evaluation of morphological generator == | ||
| + | |||
| + | 27 passes; 51 fails. | ||
| + | Coverage: 4.43% | ||
| + | |||
| + | == Twol rules == | ||
| + | |||
| + | I added four rules (technically five, but the "jun" one is irrelevant). | ||
| + | |||
| + | * One that makes -u endings for group2 verbs work. | ||
| + | * One that makes -a endings for group2 verbs work. | ||
| + | * One that makes -i endings for group2 verbs work. | ||
| + | * One that makes multicharacter group2 verb stems work (ch and j). | ||
| + | |||
| + | == Post evaluation == | ||
| + | |||
| + | 22 Passes, 56 fails. (This is because my tests were obsoleted by new grammatical info, not because it became worse. | ||
| + | Coverage: 15.17% | ||
| + | |||
[[Category:Sp17_Transducers]] | [[Category:Sp17_Transducers]] | ||
Latest revision as of 19:21, 11 May 2017
Contents
Analyser
Evaluation
My current coverage is horrendous (0.48%) due to a tokenization error, I believe. It reports many single-character words (such as な, え, お, and friends) but my corpus does not seem to have the ones that aq-covtest claims to see. However, there are several real words that it is (correctly) identifying as commonly unknown. These tend to be discourse markers, such as なあ, また, as a lot of my corpora are transcripts of conversations. Adding these analyses to the transducer increases coverage to 4.43%.
There are currently 27 stems, according to lexccounter.
Number of tokenised words in the corpus: 835 Coverage: 4.43% Top unknown words in the corpus: About 28/50 tests currently pass.
37 え 17 ち 16 ん 14 う 14 さ 13 い 11 ら 11 とぅ 10 うり 10 あ 10 や 10 お 9 ぬ 8 か 7 み 7 やい 7 うまんか 7 ちゅ 6 あんち 6 さい Translation time: 0.004895448684692383 seconds
Notes
The tests that don't yet pass are either some extra nominal suffixive tests that I haven't gotten around to yet, or involve verbs. Verbs are a bit more complicated than the nominals, since they retain the verb stem and overwrite the rest.
Generator
Initial evaluation of morphological generator
27 passes; 51 fails. Coverage: 4.43%
Twol rules
I added four rules (technically five, but the "jun" one is irrelevant).
- One that makes -u endings for group2 verbs work.
- One that makes -a endings for group2 verbs work.
- One that makes -i endings for group2 verbs work.
- One that makes multicharacter group2 verb stems work (ch and j).
Post evaluation
22 Passes, 56 fails. (This is because my tests were obsoleted by new grammatical info, not because it became worse. Coverage: 15.17%
