RGyalrong (Situ)/Transducer

Generator Evaluation

Initial Evaluation of Morphological Generation

Gold Tests:

$ apertium-regtest test -c .-gen
Corpus 1 of 7: Dual Enclitics-gen
  5/5 (100.0%) tests pass (3/5 (60.0%) match gold)

Corpus 2 of 7: Plural Enclitics-gen
  5/5 (100.0%) tests pass (3/5 (60.0%) match gold)

Corpus 3 of 7: Contrast, Indefinite, and Prominence Marker-gen
  20/20 (100.0%) tests pass (12/20 (60.0%) match gold)

Corpus 4 of 7: First Person Pronouns-gen
  6/6 (100.0%) tests pass (6/6 (100.0%) match gold)

Corpus 5 of 7: Third Person Pronouns-gen
  10/10 (100.0%) tests pass (10/10 (100.0%) match gold)

Corpus 6 of 7: Nouns and Noun Phrases-gen
  14/14 (100.0%) tests pass (12/14 (85.71%) match gold)

Corpus 7 of 7: Nominalisation of Verbs-gen
  14/14 (100.0%) tests pass (12/14 (85.71%) match gold)

All tests pass. 

Corpus Tests:

$ aq-covtest ../ling073-jya/corpus/new-basic-corpus.txt jya.automorf.bin
Number of tokenised words in the corpus: 1412
Coverage: 16.78%
Top unknown words in the corpus:
88       nə
81       kə
67       də
53       na
39       məʔ
32       jo
29       da
24       wu
22       bə
22       jə
20       ti
19       ma
17       txi
16       shə
15       dxe
15       ka
14       ta
14       shnə
14       va
12       do

Total Test Count:

 $ wc -l test/*/*-morph-input.txt
 20 test/Contrast, Indefinite, and Prominence Marker/Contrast, Indefinite, and Prominence Marker-morph-input.txt
  5 test/Dual Enclitics/Dual Enclitics-morph-input.txt
  6 test/First Person Pronouns/First Person Pronouns-morph-input.txt
 14 test/Nominalisation of Verbs/Nominalisation of Verbs-morph-input.txt
 14 test/Nouns and Noun Phrases/Nouns and Noun Phrases-morph-input.txt
  5 test/Plural Enclitics/Plural Enclitics-morph-input.txt
 10 test/Third Person Pronouns/Third Person Pronouns-morph-input.txt
 74 total 

Analyzer Evaluation

Gold Tests:

 Corpus 1 of 7: Dual Enclitics-morph
  5/5 (100.0%) tests pass (3/5 (60.0%) match gold)

Corpus 2 of 7: Plural Enclitics-morph
  5/5 (100.0%) tests pass (3/5 (60.0%) match gold)

Corpus 3 of 7: Contrast, Indefinite, and Prominence Marker-morph
  20/20 (100.0%) tests pass (11/20 (55.0%) match gold)

Corpus 4 of 7: First Person Pronouns-morph
  6/6 (100.0%) tests pass (6/6 (100.0%) match gold)

Corpus 5 of 7: Third Person Pronouns-morph
  8/8 (100.0%) tests pass (0/8 (0.0%) match gold)

Corpus 6 of 7: Nouns and Noun Phrases-morph
  14/14 (100.0%) tests pass (3/14 (21.43%) match gold)

Corpus 7 of 7: Nominalisation of Verbs-morph
  14/14 (100.0%) tests pass (7/14 (50.0%) match gold)

All tests pass. 

Corpus Tests:

$ aq-covtest ../ling073-jya/corpus/215622887517957257.txt jya.automorf.bin
Number of tokenised words in the corpus: 104
Coverage: 12.50%
Top unknown words in the corpus:
19       ŋo
10       nə
7        ne
6        tsə
6        zi
4        wu
4        le
3        tʰa
3        ta
3        ŋə
2        səⁱ
2        si
2        ŋe
2        tsʰoŋ
2        mə
2        coŋ
1        tʰi
1        ti
1        tə
1        de
Translation time: 0.007706165313720703 seconds 

Total Test Count:

 $ wc -l test/*/*-morph-input.txt
 20 test/Contrast, Indefinite, and Prominence Marker/Contrast, Indefinite, and Prominence Marker-morph-input.txt
  5 test/Dual Enclitics/Dual Enclitics-morph-input.txt
  6 test/First Person Pronouns/First Person Pronouns-morph-input.txt
 14 test/Nominalisation of Verbs/Nominalisation of Verbs-morph-input.txt
 14 test/Nouns and Noun Phrases/Nouns and Noun Phrases-morph-input.txt
  5 test/Plural Enclitics/Plural Enclitics-morph-input.txt
 10 test/Third Person Pronouns/Third Person Pronouns-morph-input.txt
 74 total 

Notes:

1. At this point, the corpus was using a different writing style (IPA) than the romanization that we later developed. Therefore, many of the misses (and hence the low accuracy) are due to the fact that our corpus hasn't been fully transcribed in our romanization yet.

Repo link

GitHub