Difference between revisions of "Urum/Transducer"

From LING073
Jump to: navigation, search
(→‎Evaluation)
 
(26 intermediate revisions by 2 users not shown)
Line 1: Line 1:
 
This is the page for our [https://github.swarthmore.edu/Ling073-sp23/ling073-uum Urum transducer].
 
This is the page for our [https://github.swarthmore.edu/Ling073-sp23/ling073-uum Urum transducer].
  
−
== Evaluation ==
+
= Analyzer Evaluation =
 
'''Total number of stems in the transducer'''
 
'''Total number of stems in the transducer'''
 +
 +
**Lexicons: 17
 +
**Lexicon entries: 107
 +
**Patterns: 7
 +
**Pattern entries: 14
 +
 +
*Counts for individual lexicons:
 +
**Punctuation: 22
 +
**V-Root: 10
 +
**TransitivityTag: 2
 +
**VPres: 6
 +
**VPresNeg: 6
 +
**VPast: 6
 +
**N-Root: 27
 +
**Plurality: 1
 +
**Possession: 6
 +
**Cases: 6
 +
**Numeral: 10
 +
**All anonymous lexicons: 5
  
 
'''Current coverage over your combined corpus'''
 
'''Current coverage over your combined corpus'''
 +
 +
*coverage: 333 / 1178 (~0.28268251273344651952)
 +
 +
*remaining unknown forms: 845
 +
 +
*ambiguity: 1341 / 1178 (~1.13837011884550084890)
  
 
'''The current list of top unknown words returned by aq-covtest'''
 
'''The current list of top unknown words returned by aq-covtest'''
Line 29: Line 54:
 
       4 ^эт/*эт$
 
       4 ^эт/*эт$
 
       4 ^эди/*эди$
 
       4 ^эди/*эди$
−
coverage: 333 / 1178 (~0.28268251273344651952)
 
−
remaining unknown forms: 845
 
−
ambiguity: 1341 / 1178 (~1.13837011884550084890)
 
  
 +
'''Number of tests that pass from regtests in test/'''
 +
Corpus 1 of 10: Plurality-morph
 +
  8/8 (100.0%) tests pass (8/8 (100.0%) match gold)
 +
 +
Corpus 2 of 10: Verbs Present Tense-morph
 +
  10/11 (90.91%) tests pass (10/10 (100.0%) match gold)
 +
 +
Corpus 3 of 10: Possession-morph
 +
  5/7 (71.43%) tests pass (5/5 (100.0%) match gold)
 +
 +
Corpus 4 of 10: Accusative Case-morph
 +
  3/3 (100.0%) tests pass (3/3 (100.0%) match gold)
 +
 +
Corpus 5 of 10: Dative Case-morph
 +
  4/4 (100.0%) tests pass (4/4 (100.0%) match gold)
 +
 +
Corpus 6 of 10: Locative Case-morph
 +
  4/6 (66.67%) tests pass (2/4 (50.0%) match gold)
  
−
'''Number of tests that pass in each yaml file'''
+
Corpus 7 of 10: Genitive Case-morph
 +
  3/3 (100.0%) tests pass (3/3 (100.0%) match gold)
 +
 
 +
Corpus 8 of 10: Ablative Case-morph
 +
  4/4 (100.0%) tests pass (4/4 (100.0%) match gold)
 +
 
 +
Corpus 9 of 10: Past Tense-morph
 +
  6/6 (100.0%) tests pass (6/6 (100.0%) match gold)
 +
 
 +
Corpus 10 of 10: Verb Negation-morph
 +
  4/4 (100.0%) tests pass (4/4 (100.0%) match gold)
 +
 
 +
 
 +
All tests pass.
  
 
== Notes ==
 
== Notes ==
  
 
'''What tests still don't work and why'''
 
'''What tests still don't work and why'''
−
* '''Accusative Case''', we still have no idea why they are not working correctly. When we tried to check for as to why the error might be occuring, we checked each of the individual generations per the hfst-expand greping for the correct output, and it was able to be found for ever single instance. As for why this does not register as "gold" by the apertium-regtest, is still unclear. Perhaps there could be an issue with the unicode of the tests on the wiki (will check later).
+
* '''Locative Case ''', For locative case, there are some additional steps that need to account for the dropping of [м] from possession. This will need to be ironed out later.
−
* '''Locative Case '''
+
* '''Posessiveness ''', Similar to the issue in the Locative Case, these are actually generating perfectly to match the gold, however, for some reason they do not actually register when looking at them through the tests. This does not make sense, grepping should provide an accurate snapshot of what is actually being generated.
 +
 
 +
'''To be revised and other comments (might resolve the issues mentioned above!)'''
 +
* Verb negation <neg> should be just a -м{A} suffix that comes before the verb ending. I just added notes to lexd on this, waiting for approval from Sasha/JNW. Will have to edit lexd accordingly, and change order of tags in Urum/Grammar. (from pres-neg-person to pres-person-neg)
 +
* maybe something is not working because we redefined {A} in twol in the process. It used to be {A}:0 by default, now it's the "first type of vowel harmony".
 +
* Alina is currently reviewing and editing Urum/Grammar. There are some issues with Locative Case + Possession that I would like to ask JNW about.
 +
 
 +
 
 +
= Initial Generator Evaluation =
 +
 
 +
'''The current list of top unknown words returned by aq-covtest'''
 +
 
 +
Number of tokenised words in the corpus: 3030
 +
Coverage: 21.52%
 +
 
 +
TOP UNKNOWN WORDS:
 +
    116 т
 +
      56 и
 +
      46 ве
 +
      31 бир
 +
      29 не
 +
      27 дер
 +
      25 да
 +
      23 иди
 +
      22 ки
 +
      18 Ол
 +
      17 де
 +
      17 ол
 +
      17 бу
 +
      15 Анд
 +
      14 ер
 +
      14 даа
 +
      13 илен
 +
      13 Михаил
 +
      12 ичӱн
 +
      10 ем
 +
 
 +
'''Number of tests that pass from regtests in test/'''
 +
Corpus 1 of 10: Plurality-gen
 +
  0/8 (0.0%) tests pass
 +
 
 +
Corpus 2 of 10: Verbs Present Tense-gen
 +
  8/12 (66.67%) tests pass (0/8 (0.0%) match gold)
 +
 
 +
Corpus 3 of 10: Possession-gen
 +
  4/7 (57.14%) tests pass (0/4 (0.0%) match gold)
 +
 
 +
Corpus 4 of 10: Accusative Case-gen
 +
  1/3 (33.33%) tests pass (0/1 (0.0%) match gold)
 +
 
 +
Corpus 5 of 10: Dative Case-gen
 +
  0/4 (0.0%) tests pass
 +
 
 +
Corpus 6 of 10: Locative Case-gen
 +
  0/6 (0.0%) tests pass
 +
 
 +
Corpus 7 of 10: Genitive Case-gen
 +
  3/3 (100.0%) tests pass (0/3 (0.0%) match gold)
 +
 
 +
Corpus 8 of 10: Ablative Case-gen
 +
  0/4 (0.0%) tests pass
 +
 
 +
Corpus 9 of 10: Past Tense-gen
 +
  4/6 (66.67%) tests pass (0/4 (0.0%) match gold)
 +
 
 +
Corpus 10 of 10: Verb Negation-gen
 +
  2/4 (50.0%) tests pass (2/2 (100.0%) match gold)
 +
 
 +
= Final Generator Evaluation =
 +
Corpus 1 of 10: Plurality-morph
 +
  8/8 (100.0%) tests pass (8/8 (100.0%) match gold)
 +
 
 +
Corpus 2 of 10: Verbs Present Tense-morph
 +
  10/11 (90.91%) tests pass (10/10 (100.0%) match gold)
 +
 
 +
Corpus 3 of 10: Possession-morph
 +
  5/7 (71.43%) tests pass (5/5 (100.0%) match gold)
 +
 
 +
Corpus 4 of 10: Accusative Case-morph
 +
  3/3 (100.0%) tests pass (3/3 (100.0%) match gold)
 +
 
 +
Corpus 5 of 10: Dative Case-morph
 +
  4/4 (100.0%) tests pass (4/4 (100.0%) match gold)
 +
 
 +
Corpus 6 of 10: Locative Case-morph
 +
  2/6 (33.33%) tests pass (2/2 (100.0%) match gold)
 +
 
 +
Corpus 7 of 10: Genitive Case-morph
 +
  3/3 (100.0%) tests pass (3/3 (100.0%) match gold)
 +
 
 +
Corpus 8 of 10: Ablative Case-morph
 +
  4/4 (100.0%) tests pass (4/4 (100.0%) match gold)
 +
 
 +
Corpus 9 of 10: Past Tense-morph
 +
  6/6 (100.0%) tests pass (6/6 (100.0%) match gold)
 +
 
 +
Corpus 10 of 10: Verb Negation-morph
 +
  4/4 (100.0%) tests pass (4/4 (100.0%) match gold)
 +
 
 +
'''Total Number of twol Rules'''
 +
TOTAL: 16 Rules
 +
 
 +
'''Corpus Coverage'''
 +
 
 +
*Number of tokenised words in the corpus: 3030
  
 +
*Coverage: 21.39%
  
 +
Top unknown words in the corpus:
 +
    116 т
 +
      56 и
 +
      46 ве
 +
      31 бир
 +
      29 не
 +
      27 дер
 +
      25 да
 +
      23 иди
 +
      22 ки
 +
      18 Ол
 +
      17 де
 +
      17 ол
 +
      17 бу
 +
      15 Анд
 +
      14 ер
 +
      14 даа
 +
      13 илен
 +
      13 Михаил
 +
      12 ичӱн
 +
      10 ем
 +
Translation time: 0.05725860595703125 seconds
  
 +
= Precision and Recall Statistics =
  
 +
Totals: 137 forms, 106 tp, 31 fp, 0 tn, 106 fn
 +
* '''Precision''': 77.37226%
 +
* '''Recall''': 50.00000%
  
 
[[Category:Sp23_Transducers]][[Category:Urum]]
 
[[Category:Sp23_Transducers]][[Category:Urum]]

Latest revision as of 00:16, 1 April 2023

This is the page for our Urum transducer.

Analyzer Evaluation

Total number of stems in the transducer

    • Lexicons: 17
    • Lexicon entries: 107
    • Patterns: 7
    • Pattern entries: 14
  • Counts for individual lexicons:
    • Punctuation: 22
    • V-Root: 10
    • TransitivityTag: 2
    • VPres: 6
    • VPresNeg: 6
    • VPast: 6
    • N-Root: 27
    • Plurality: 1
    • Possession: 6
    • Cases: 6
    • Numeral: 10
    • All anonymous lexicons: 5

Current coverage over your combined corpus

  • coverage: 333 / 1178 (~0.28268251273344651952)
  • remaining unknown forms: 845
  • ambiguity: 1341 / 1178 (~1.13837011884550084890)

The current list of top unknown words returned by aq-covtest

TOP UNKNOWN WORDS: 

    13 *
    11 ^д/*д$
    11 ^ве/*ве$
    10 ^дэ/*дэ$
    10 ^да/*да$
     8 ^эй/*эй$
     7 ^бен/*бен$
     6 ^олсун/*олсун$
     6 ^махлуклар/*махлуклар$
     6 ^йар/*йар$
     6 ^и/*и$
     6 ^бир/*бир$
     5 ^эм/*эм$
     5 ^т`и/*т`и$
     5 ^илен/*илен$
     5 ^доғду/*доғду$
     5 ^бу/*бу$
     5 ^Христос/*Христос$
     4 ^эт/*эт$
     4 ^эди/*эди$

Number of tests that pass from regtests in test/ Corpus 1 of 10: Plurality-morph 

 8/8 (100.0%) tests pass (8/8 (100.0%) match gold)

Corpus 2 of 10: Verbs Present Tense-morph 

 10/11 (90.91%) tests pass (10/10 (100.0%) match gold)

Corpus 3 of 10: Possession-morph 

 5/7 (71.43%) tests pass (5/5 (100.0%) match gold)

Corpus 4 of 10: Accusative Case-morph 

 3/3 (100.0%) tests pass (3/3 (100.0%) match gold)

Corpus 5 of 10: Dative Case-morph 

 4/4 (100.0%) tests pass (4/4 (100.0%) match gold)

Corpus 6 of 10: Locative Case-morph 

 4/6 (66.67%) tests pass (2/4 (50.0%) match gold)

Corpus 7 of 10: Genitive Case-morph 

 3/3 (100.0%) tests pass (3/3 (100.0%) match gold)

Corpus 8 of 10: Ablative Case-morph 

 4/4 (100.0%) tests pass (4/4 (100.0%) match gold)

Corpus 9 of 10: Past Tense-morph 

 6/6 (100.0%) tests pass (6/6 (100.0%) match gold)

Corpus 10 of 10: Verb Negation-morph 

 4/4 (100.0%) tests pass (4/4 (100.0%) match gold)


All tests pass.

Notes

What tests still don't work and why

  • Locative Case , For locative case, there are some additional steps that need to account for the dropping of [м] from possession. This will need to be ironed out later.
  • Posessiveness , Similar to the issue in the Locative Case, these are actually generating perfectly to match the gold, however, for some reason they do not actually register when looking at them through the tests. This does not make sense, grepping should provide an accurate snapshot of what is actually being generated.

To be revised and other comments (might resolve the issues mentioned above!)

  • Verb negation <neg> should be just a -м{A} suffix that comes before the verb ending. I just added notes to lexd on this, waiting for approval from Sasha/JNW. Will have to edit lexd accordingly, and change order of tags in Urum/Grammar. (from pres-neg-person to pres-person-neg)
  • maybe something is not working because we redefined {A} in twol in the process. It used to be {A}:0 by default, now it's the "first type of vowel harmony".
  • Alina is currently reviewing and editing Urum/Grammar. There are some issues with Locative Case + Possession that I would like to ask JNW about.


Initial Generator Evaluation

The current list of top unknown words returned by aq-covtest

Number of tokenised words in the corpus: 3030 Coverage: 21.52%

TOP UNKNOWN WORDS: 

    116	 т
     56	 и
     46	 ве
     31	 бир
     29	 не
     27	 дер
     25	 да
     23	 иди
     22	 ки
     18	 Ол
     17	 де
     17	 ол
     17	 бу
     15	 Анд
     14	 ер
     14	 даа
     13	 илен
     13	 Михаил
     12	 ичӱн
     10	 ем

Number of tests that pass from regtests in test/ Corpus 1 of 10: Plurality-gen 

 0/8 (0.0%) tests pass

Corpus 2 of 10: Verbs Present Tense-gen 

 8/12 (66.67%) tests pass (0/8 (0.0%) match gold)

Corpus 3 of 10: Possession-gen 

 4/7 (57.14%) tests pass (0/4 (0.0%) match gold)

Corpus 4 of 10: Accusative Case-gen 

 1/3 (33.33%) tests pass (0/1 (0.0%) match gold)

Corpus 5 of 10: Dative Case-gen 

 0/4 (0.0%) tests pass

Corpus 6 of 10: Locative Case-gen 

 0/6 (0.0%) tests pass

Corpus 7 of 10: Genitive Case-gen 

 3/3 (100.0%) tests pass (0/3 (0.0%) match gold)

Corpus 8 of 10: Ablative Case-gen 

 0/4 (0.0%) tests pass

Corpus 9 of 10: Past Tense-gen 

 4/6 (66.67%) tests pass (0/4 (0.0%) match gold)

Corpus 10 of 10: Verb Negation-gen 

 2/4 (50.0%) tests pass (2/2 (100.0%) match gold)

Final Generator Evaluation

Corpus 1 of 10: Plurality-morph 

 8/8 (100.0%) tests pass (8/8 (100.0%) match gold)

Corpus 2 of 10: Verbs Present Tense-morph 

 10/11 (90.91%) tests pass (10/10 (100.0%) match gold)

Corpus 3 of 10: Possession-morph 

 5/7 (71.43%) tests pass (5/5 (100.0%) match gold)

Corpus 4 of 10: Accusative Case-morph 

 3/3 (100.0%) tests pass (3/3 (100.0%) match gold)

Corpus 5 of 10: Dative Case-morph 

 4/4 (100.0%) tests pass (4/4 (100.0%) match gold)

Corpus 6 of 10: Locative Case-morph 

 2/6 (33.33%) tests pass (2/2 (100.0%) match gold)

Corpus 7 of 10: Genitive Case-morph 

 3/3 (100.0%) tests pass (3/3 (100.0%) match gold)

Corpus 8 of 10: Ablative Case-morph 

 4/4 (100.0%) tests pass (4/4 (100.0%) match gold)

Corpus 9 of 10: Past Tense-morph 

 6/6 (100.0%) tests pass (6/6 (100.0%) match gold)

Corpus 10 of 10: Verb Negation-morph 

 4/4 (100.0%) tests pass (4/4 (100.0%) match gold)

Total Number of twol Rules TOTAL: 16 Rules

Corpus Coverage

  • Number of tokenised words in the corpus: 3030
  • Coverage: 21.39%

Top unknown words in the corpus: 

    116	 т
     56	 и
     46	 ве
     31	 бир
     29	 не
     27	 дер
     25	 да
     23	 иди
     22	 ки
     18	 Ол
     17	 де
     17	 ол
     17	 бу
     15	 Анд
     14	 ер
     14	 даа
     13	 илен
     13	 Михаил
     12	 ичӱн
     10	 ем

Translation time: 0.05725860595703125 seconds

Precision and Recall Statistics

Totals: 137 forms, 106 tp, 31 fp, 0 tn, 106 fn
  • Precision: 77.37226%
  • Recall: 50.00000%
Retrieved from "http://wikis.swarthmore.edu/ling073/index.php?title=Urum/Transducer&oldid=20766"