Difference between revisions of "Kaingang/Transducer"
From LING073
| Line 1: | Line 1: | ||
[https://github.swarthmore.edu/Ling073-sp19/ling073-kgp] Link to Kaingang Transducer | [https://github.swarthmore.edu/Ling073-sp19/ling073-kgp] Link to Kaingang Transducer | ||
| − | == | + | == First Evaluation == |
*First evaluation of coverage 2/21/19: | *First evaluation of coverage 2/21/19: | ||
$ aq-covtest ling073-kgp-corpus/kgp.corpus.basic.txt ling073-kgp/kgp.automorf.bin | $ aq-covtest ling073-kgp-corpus/kgp.corpus.basic.txt ling073-kgp/kgp.automorf.bin | ||
| Line 28: | Line 28: | ||
11 ũ | 11 ũ | ||
Translation time: 0.005640268325805664 seconds | Translation time: 0.005640268325805664 seconds | ||
| − | |||
| − | |||
*Many of these top unknown words are included in our wiki and even in our morph tests. They just happen to be markers, which are added to nouns, verbs, or pronouns to indicate something about them. | *Many of these top unknown words are included in our wiki and even in our morph tests. They just happen to be markers, which are added to nouns, verbs, or pronouns to indicate something about them. | ||
| − | + | == Classification of First Top Unknown Words == | |
{|class="wikitable" | {|class="wikitable" | ||
|- | |- | ||
| Line 81: | Line 79: | ||
|} | |} | ||
| − | + | == Second Evaluation == | |
| − | * | + | *Added words that are subject indicators (tóg, tỹ, mỹ, vỹ, jé) with {{tag|subj}}. The second evaluation of coverage, now with added subject indicators, 2/22/19: |
$ aq-covtest ling073-kgp-corpus/kgp.corpus.basic.txt ling073-kgp/kgp.automorf.bin | $ aq-covtest ling073-kgp-corpus/kgp.corpus.basic.txt ling073-kgp/kgp.automorf.bin | ||
Number of tokenised words in the corpus: 1303 | Number of tokenised words in the corpus: 1303 | ||
| Line 108: | Line 106: | ||
8 José | 8 José | ||
Translation time: 0.0059816837310791016 seconds | Translation time: 0.0059816837310791016 seconds | ||
| − | |||
| − | |||
*How much each word added to the coverage (percentage each individual word added): | *How much each word added to the coverage (percentage each individual word added): | ||
** tóg -> 32.16% - 26.94% = 5.22% | ** tóg -> 32.16% - 26.94% = 5.22% | ||
| Line 126: | Line 122: | ||
**No tests in kgp.yaml fail. | **No tests in kgp.yaml fail. | ||
**5 out of 20 of the tests in commonwords.yaml pass. | **5 out of 20 of the tests in commonwords.yaml pass. | ||
| − | |||
== Experiment == | == Experiment == | ||
| + | Tried testing coverage on a text file containing all of unparsed bible: | ||
$ aq-covtest ling073-kgp-corpus/bible_kaingang.txt ling073-kgp/kgp.automorf.bin | $ aq-covtest ling073-kgp-corpus/bible_kaingang.txt ling073-kgp/kgp.automorf.bin | ||
Number of tokenised words in the corpus: 388887 | Number of tokenised words in the corpus: 388887 | ||
| − | Coverage: | + | Coverage: 42.44% |
Top unknown words in the corpus: | Top unknown words in the corpus: | ||
| − | |||
| − | |||
16033 mũ | 16033 mũ | ||
10740 ke | 10740 ke | ||
9523 nĩ | 9523 nĩ | ||
| − | |||
6238 ẽg | 6238 ẽg | ||
5604 Topẽ | 5604 Topẽ | ||
| Line 147: | Line 140: | ||
4428 ẽn | 4428 ẽn | ||
4389 ũ | 4389 ũ | ||
| − | |||
3985 vĩ | 3985 vĩ | ||
3569 h | 3569 h | ||
| Line 153: | Line 145: | ||
3006 nỹtĩ | 3006 nỹtĩ | ||
2907 vẽnh | 2907 vẽnh | ||
| − | Translation time: 0. | + | 2769 kar |
| + | 2722 mré | ||
| + | 2508 t | ||
| + | 2407 sóg | ||
| + | Translation time: 0.7014105319976807 seconds | ||
| + | *Coverage looks a lot similar to our combined corpus file! | ||
[[Category:Sp18_Transducers]] | [[Category:Sp18_Transducers]] | ||
[[Category:Kaingang]] | [[Category:Kaingang]] | ||
Revision as of 21:49, 22 February 2019
[1] Link to Kaingang Transducer
Contents
First Evaluation
- First evaluation of coverage 2/21/19:
$ aq-covtest ling073-kgp-corpus/kgp.corpus.basic.txt ling073-kgp/kgp.automorf.bin Number of tokenised words in the corpus: 1303 Coverage: 26.94% Top unknown words in the corpus: 68 tóg 49 tỹ 44 mũ 42 nĩ 38 mỹ 36 ẽg 33 tĩ 24 vỹ 21 ki 20 han 19 ke 19 to 17 Topẽ 16 jé 16 kar 15 vẽnh 14 h 13 vĩ 12 vẽ 11 ũ Translation time: 0.005640268325805664 seconds
- Many of these top unknown words are included in our wiki and even in our morph tests. They just happen to be markers, which are added to nouns, verbs, or pronouns to indicate something about them.
Classification of First Top Unknown Words
| Kaingang | Approximate Meaning | Classification |
|---|---|---|
| tóg | subject is agent | subject indicator |
| tỹ | agent is ergative, topic marker | subject indicator |
| mũ | doing, an action is done repeatedly | might be verb to do? / verb marker |
| nĩ | in the situation of doing the action, | situational/continuous marker |
| mỹ | subject in a yes-or-no question | subject indicator |
| ẽg | ? | ? |
| tĩ | habitually | verb habitual marker |
| vỹ | subject is topic | subject indicator |
| ki | in, on, at | circumstance marker |
| han | to get better | verb |
| ke | leftovers, remaining | noun |
| to | to, in the direction of | circumstance marker |
| Topẽ | God, anything sacred, to pray | noun, verb |
| jé | subject expects the action, speaker desires the action | subject indicator |
| tĩ | habitually | verb habitual marker |
| kar | all | noun |
| vẽnh | of oneself | reflexive pronoun |
| h | ? | ? |
| vĩ | word, speech, to speak | noun, verb |
| vẽ | is, was, ergative | aspect marker |
| ũ | makes indefinite noun | indefinite noun marker |
Second Evaluation
- Added words that are subject indicators (tóg, tỹ, mỹ, vỹ, jé) with <subj>. The second evaluation of coverage, now with added subject indicators, 2/22/19:
$ aq-covtest ling073-kgp-corpus/kgp.corpus.basic.txt ling073-kgp/kgp.automorf.bin Number of tokenised words in the corpus: 1303 Coverage: 41.98% Top unknown words in the corpus: 44 mũ 42 nĩ 36 ẽg 33 tĩ 21 ki 20 han 19 ke 19 to 17 Topẽ 16 kar 15 vẽnh 14 h 13 vĩ 12 vẽ 11 ũ 11 nén 9 ẽn 9 Ẽg 8 Jesus 8 José Translation time: 0.0059816837310791016 seconds
- How much each word added to the coverage (percentage each individual word added):
- tóg -> 32.16% - 26.94% = 5.22%
- tỹ -> 30.78% - 26.94% = 3.84%
- mỹ -> 29.85% - 26.94% = 2.91%
- vỹ -> 28.78% - 26.94% = 1.84%
- jé -> 28.17% - 26.94% = 1.53%
Notes
- Total number of stems in the transducer:
$ lexccounter apertium-kgp.kgp.lexc Unique entries: 105
- Current coverage over combined corpus: 41.98%
- Current list of unknown words returned by aq-covtest: mũ (44), nĩ (42), ẽg (36), tĩ (33), ki (21), han (20), ke (19), to (17), Topẽ (16), kar (15), vẽnh (14), h (13), vĩ (12), vẽ (11), ũ (11), nén (9), ẽn (9), Ẽg (8), Jesus José (8)
- Tests:
- No tests in kgp.yaml fail.
- 5 out of 20 of the tests in commonwords.yaml pass.
Experiment
Tried testing coverage on a text file containing all of unparsed bible:
$ aq-covtest ling073-kgp-corpus/bible_kaingang.txt ling073-kgp/kgp.automorf.bin Number of tokenised words in the corpus: 388887 Coverage: 42.44% Top unknown words in the corpus: 16033 mũ 10740 ke 9523 nĩ 6238 ẽg 5604 Topẽ 5401 ki 5244 han 5013 tĩ 4944 to 4428 ẽn 4389 ũ 3985 vĩ 3569 h 3440 Jesus 3006 nỹtĩ 2907 vẽnh 2769 kar 2722 mré 2508 t 2407 sóg Translation time: 0.7014105319976807 seconds
- Coverage looks a lot similar to our combined corpus file!
