Difference between revisions of "Ainu/Transducer"
(→Notes) |
|||
| (3 intermediate revisions by the same user not shown) | |||
| Line 1: | Line 1: | ||
| − | == Evaluation == | + | == Generator == |
| + | |||
| + | === Initial evaluation of morphological generation === | ||
| + | |||
| + | There are 79 stems in the transducer. 75/115 ain.yaml tests pass (40 fail). 2/20 commonwords.yaml tests pass (18 fail). Generation test has 77/164 ain.yaml tests pass (87 fail). | ||
| + | |||
| + | === Final evaluation of morphological generation === | ||
| + | |||
| + | After changing the grammar dramatically, 78/110 ain.yaml tests pass (32 fail). I added 9 new .twol rules, although only 7 are working. I did not have any before (I only had prefix/suffix tag-matching .twoc rules). Somehow my corpus coverage has gone down to 18.27% of the 925 tokenised words in the corpus. | ||
| + | |||
| + | === Notes === | ||
| + | |||
| + | Adding the .twoc prefix fix for getting rid of paths with odd numbers of feature tags fixed a lot. 77/133 ain.yaml tests pass, 56 fail. | ||
| + | |||
| + | |||
| + | == Analyser == | ||
| + | |||
| + | === Evaluation === | ||
The most common unknown words originally were | The most common unknown words originally were | ||
| Line 69: | Line 86: | ||
| − | Original coverage of these words is 11.19%. Now the coverage is | + | Original coverage of these words is 11.19%. Now the coverage is 22.76%. |
| − | There are 79 stems in the transducer. | + | There are 79 stems in the transducer. 75/115 ain.yaml tests pass (40 fail). 2/20 commonwords.yaml tests pass (18 fail). |
Now the most common words are | Now the most common words are | ||
| Line 141: | Line 158: | ||
|} | |} | ||
| − | == Notes == | + | === Notes === |
After adding "ネ<vbser> ↔ ネ" and "コ<cnjsub> ↔ コ" to the transducer, coverage went from 11.19% to 21.21%. | After adding "ネ<vbser> ↔ ネ" and "コ<cnjsub> ↔ コ" to the transducer, coverage went from 11.19% to 21.21%. | ||
The tests I am struggling with are for adjectivity-- I need a new grammar book to confirm that there are really so many differing adjective endings because the current book I am using as a reference was written over 100 years ago. I have not yet had time to add pronoun or noun affinity tests. The remaining failing tests are irregular cases at which I will have to look more closely. | The tests I am struggling with are for adjectivity-- I need a new grammar book to confirm that there are really so many differing adjective endings because the current book I am using as a reference was written over 100 years ago. I have not yet had time to add pronoun or noun affinity tests. The remaining failing tests are irregular cases at which I will have to look more closely. | ||
| − | |||
| − | |||
| − | |||
| − | |||
| − | == Transducer == | + | === Transducer === |
Ainu transducer available on [https://github.swarthmore.edu/dwurtel1/ling073-ain my GitHub]. | Ainu transducer available on [https://github.swarthmore.edu/dwurtel1/ling073-ain my GitHub]. | ||
[[Category:Sp17_Transducers]] | [[Category:Sp17_Transducers]] | ||
Latest revision as of 02:57, 27 February 2017
Contents
Generator
Initial evaluation of morphological generation
There are 79 stems in the transducer. 75/115 ain.yaml tests pass (40 fail). 2/20 commonwords.yaml tests pass (18 fail). Generation test has 77/164 ain.yaml tests pass (87 fail).
Final evaluation of morphological generation
After changing the grammar dramatically, 78/110 ain.yaml tests pass (32 fail). I added 9 new .twol rules, although only 7 are working. I did not have any before (I only had prefix/suffix tag-matching .twoc rules). Somehow my corpus coverage has gone down to 18.27% of the 925 tokenised words in the corpus.
Notes
Adding the .twoc prefix fix for getting rid of paths with odd numbers of feature tags fixed a lot. 77/133 ain.yaml tests pass, 56 fail.
Analyser
Evaluation
The most common unknown words originally were
| Word | Times Encountered |
|---|---|
| ネ | 60 |
| カ | 50 |
| アン | 44 |
| タ | 30 |
| ワ | 29 |
| コ | 25 |
| イ | 20 |
| ヒ | 18 |
| ピ | 16 |
| ルウェ | 16 |
| パ | 16 |
| ㇺ | 14 |
| ㇷ | 14 |
| ウタ | 14 |
| セコ | 13 |
| ハウェアン | 13 |
| キ | 12 |
| クニ | 10 |
| ニ | 9 |
| ヤㇰ | 9 |
Original coverage of these words is 11.19%. Now the coverage is 22.76%.
There are 79 stems in the transducer. 75/115 ain.yaml tests pass (40 fail). 2/20 commonwords.yaml tests pass (18 fail).
Now the most common words are
| Word | Times Encountered |
|---|---|
| カ | 50 |
| アン | 44 |
| タ | 30 |
| ワ | 29 |
| イ | 20 |
| ヒ | 18 |
| ピ | 16 |
| ルウェ | 16 |
| パ | 16 |
| ㇺ | 14 |
| ㇷ | 14 |
| ウタ | 14 |
| セコ | 13 |
| ハウェアン | 13 |
| キ | 12 |
| クニ | 10 |
| ニ | 9 |
| ヤㇰ | 9 |
| チュㇷ | 8 |
| カノ | 8 |
Notes
After adding "ネ<vbser> ↔ ネ" and "コ<cnjsub> ↔ コ" to the transducer, coverage went from 11.19% to 21.21%.
The tests I am struggling with are for adjectivity-- I need a new grammar book to confirm that there are really so many differing adjective endings because the current book I am using as a reference was written over 100 years ago. I have not yet had time to add pronoun or noun affinity tests. The remaining failing tests are irregular cases at which I will have to look more closely.
Transducer
Ainu transducer available on my GitHub.
