View source for User:Tjones5/Final project

==Improvements Made to the Transducer==
Final version:
https://github.com/tkjones117/ling073-wbp


Commit: https://github.swarthmore.edu/tjones5/ling073-wbp/commit/125bb63f33f5f50caf682b746b972c549b4da7a0
*.lexc:
**optional dashes throughout (verb suffixes, pronouns)
**improved organization of pronoun analysis: separation of 1st person from 2nd/3rd person
**added several noun stems
*.twol:
**added %{U%} archiphoneme to appear in suffixes on nouns in dative case

Commit: https://github.swarthmore.edu/tjones5/ling073-wbp/commit/f8841c64db1c248815bc5c5a3ce4a23b8d30f641
*.lexc:
**added present verb analysis (ex: nyina = present form of "to be", not piyanimi)
**added verb tense and suffix analysis (apparently sometimes verbs can object suffixes; previously I thought it was only auxiliaries that could do this)
**added "kirra" as another possible allative object ending
**added clitic analysis and several clitics -- something needs to be changed so that they increase coverage
**added 6 noun stems, ~55 verb stems, 3 adverb stems
*.twol:
**eliminated bi-directionality of some rules

Commit: https://github.swarthmore.edu/tjones5/ling073-wbp/commit/2857cffb76e0014b8ea90b21f40e4cc957ae280f
*.lexc:
**added "yungu" as a variation of the auxiliary "yunga"
**added more optional dashes in the auxiliary suffixes
**added ability to append another object suffixes to auxiliary verb
**added auxiliary suffixes to adverbs

Commit: https://github.swarthmore.edu/tjones5/ling073-wbp/commit/8cd682e70a81378184b6a90f381f16e6d130459c
*.lexc:
**added a few more stems

==Improvements Made to Disambiguation==
Commit: https://github.swarthmore.edu/tjones5/ling073-wbp/commit/6edc1d225ba43f369468448b83d009616a9c6c41

*Ambiguity before disambiguation: 	                ~1.41077302631578947368
*Ambiguity after disambiguation (Pre-final project): 	~1.35831766917293233083
*Ambiguity after disambiguation (Final):                ~1.34727443609022556391

 # If there is a transitive verb two words to the left, I don't take a subject.
 REMOVE (n sg prf) IF (-2 Vtv) ;

 # If there is a transitive verb two words to the left, I'm in the absolutive case.
 SELECT (n sg abs) IF (-2 Vtv) ;

 # If there is an intransitive verb two words to the left, I'm the subject.
 SELECT (n sg abs) IF (-2 Viv) ;

 # If there is an auxiliary to the right or left, I cannot be a numeral.
 REMOVE (num) IF ((1 Aux) OR (-1 Aux)) ;

 # If there is an auxiliary to the left, I cannot be an auxiliary.
 REMOVE (vaux) IF (-1 Aux) ;

==Final Evaluation==
Number of stems: ~340

Final coverage: 61.48%

Against new_corpus.txt:
*Precision: 84.21053%
*Recall: 32.98969%

==Presentation==
<!--What will be expected for this assignment is a '''10 minute''' presentation in which you share the following with your classmates:
* A '''description of the problem''' addressed by your project. For some projects this may require thinking about it in a somewhat new light.  Whether or not the problem is something we've dealt with in class, provide a brief but complete introduction to the problem.--> 
===Description of the problem===
* Word structure is analyzed by composition of morphemes (the smallest units with grammatical meaning)
* A morphological transducer connects forms with analyses. 
* Surface level -- represents the concatenation of letters which make up the actual spelling of the word, e.g. wolves
* Lexical level -- represents the concatenation of morphemes making up a word, e.g. NOUN-wolf + PLURAL-es 
* Morphological parsing -- the problem of recognizing that a word breaks down into component morphemes, as above with "wolves"

[[Image:Simple transducer.png|1000px]]

===Previous work===
<!--* Any '''previous work''' you've found that also addresses this problem or similar problems, and how it relates to what you're doing.-->
*Many languages have working [http://wiki.apertium.org/wiki/Languages Apertium transducers]: 
*Warlpiri doesn't have a working transducer -- in fact, no languages from the Pama–Nyungan family have a transducer

===Benefits of this project===
<!--* Your thoughts on who might '''benefit from your project''' and in what way.  E.g., might a language community be able to find a use for what you're doing, or maybe [computational] linguists working on a language or issue?-->
*Why is a transducer useful to a community of Warlpiri speakers?
**Search engine
**Spell checker
**Machine Translation, especially to/from English (the primary language of most Warlpiri speakers). Many are trying to revive the use of Warlpiri, so there are many potentially useful applications.
***eng → wbp MT system could be used to look up how to say/write a certain word or phrase
***wbp → eng MT system could be used to analyze (and then translate) untranslated Warlpiri stories, or give alternate translations
**All of these applications might also be helpful in starting an analyzer for other indigenous languages that have the same orthography, are highly agglutinative, and have other grammatical similarities 

===Approaches===
<!--* '''How you're approaching the solution''' to the overall problem, including how you're implementing the solution. You can talk here about smaller individual issues that have arisen as well.-->

*Mostly have worked within lexc and twol files

*Pronouns (optional dash not included in table):
{|class="wikitable sortable"
! Meaning !! Word !! Form
|-
! I, me
|  ngaju(lu) || {{morphTest|ngaju{{tag|prn}}{{tag|pers}}{{tag|p1}}{{tag|sg}}|ngaju}}<br />{{morphTest|ngaju{{tag|prn}}{{tag|pers}}{{tag|p1}}{{tag|sg}}|ngajulu}}
|-
! you
|  nyuntu(lu) || {{morphTest|nyuntu{{tag|prn}}{{tag|pers}}{{tag|p2}}{{tag|sg}}|nyuntu}} <br />{{morphTest|nyuntu{{tag|prn}}{{tag|pers}}{{tag|p2}}{{tag|sg}}|nyuntulu}}
|-
! he/she/it; to him/her/it
|  nyanungu || {{morphTest|nyanungu{{tag|prn}}{{tag|pers}}{{tag|p3}}{{tag|sg}}|nyanungu}}
|-
! we (you & me)
|  ngali(jarra) || {{morphTest|ngali{{tag|prn}}{{tag|pers}}{{tag|p1}}{{tag|du}}{{tag|incl}}|ngali}} <br /> {{morphTest|ngali{{tag|prn}}{{tag|pers}}{{tag|p1}}{{tag|du}}{{tag|incl}}|ngalijarra}}
|-
! we (him/her/it & me)
|  ngajarra || {{morphTest|ngajarra{{tag|prn}}{{tag|pers}}{{tag|p1}}{{tag|du}}{{tag|excl}}|ngajarra}}
|-
! we (you & me & other(s))
|  ngalipa || {{morphTest|ngalipa{{tag|prn}}{{tag|pers}}{{tag|p1}}{{tag|pl}}{{tag|incl}}|ngalipa}}
|-
! we (them & me)
|  nganimpa || {{morphTest|nganimpa{{tag|prn}}{{tag|pers}}{{tag|p1}}{{tag|pl}}{{tag|excl}}|nganimpa}}
|-
! you (both/two)
|  nyumpala || {{morphTest|nyumpala{{tag|prn}}{{tag|pers}}{{tag|p2}}{{tag|du}}|nyumpala}}
|-
! you (more than 2)
|  nyurrarla || {{morphTest|nyurrarla{{tag|prn}}{{tag|pers}}{{tag|p2}}{{tag|pl}}|nyurrarla}}
|-
! they/them (both/two)
|  nyanungu-jarra || {{morphTest|nyanungu-jarra{{tag|prn}}{{tag|pers}}{{tag|p3}}{{tag|du}}|nyanungu-jarra}}
|-
! they/them (more than 2)
|  nyanungu-rra/nyanungu-patu || {{morphTest|nyanungu{{tag|prn}}{{tag|pers}}{{tag|p3}}{{tag|pl}}|nyanungu-rra}} <br /> {{morphTest|nyanungu{{tag|prn}}{{tag|pers}}{{tag|p3}}{{tag|pl}}|nyanungu-patu}}
|-
|}

*pirli (a noun with just 2 vowels) has different endings than warlkurru (which has more than two vowels)
**however when suffixes such as -jarra are added, pirli gets the endings of nouns with >2 vowels: pirli-ngku → pirli-jarra-rlu 

{|class="wikitable sortable"
! case name !! ~meaning !! tag !! possible forms !! pirli "rock" (2 vowels) !! warlkurru (>2 vowels) "axe"
|-
! absolutive
| subject of intransitive verbs and object of transitive verbs || {{tag|abs}} || — || {{morphTest|pirli{{tag|n}}{{tag|sg}}{{tag|abs}}|pirli}} || {{morphTest|warlkurru{{tag|n}}{{tag|sg}}{{tag|abs}}|warlkurru}}
|-
! dative
| "to" || {{tag|dat}} || -ku || {{morphTest|pirli{{tag|n}}{{tag|sg}}{{tag|dat}}|pirli-ku}} || {{morphTest|warlkurru{{tag|n}}{{tag|sg}}{{tag|dat}}|warlkurru-ku}}
|-
! ergative
|"agent of a transitive verb" || {{tag|erg}} || style="background-color: teal;" | -ngku, -rlu || {{morphTest|pirli{{tag|n}}{{tag|sg}}{{tag|erg}}|pirli-ngku}} || {{morphTest|warlkurru{{tag|n}}{{tag|sg}}{{tag|erg}}|warlkurru-rlu}}
|-
! allative
| "onto" || {{tag|all}} || -kurra || {{morphTest|pirli{{tag|n}}{{tag|sg}}{{tag|all}}|pirli-kurra}} || {{morphTest|warlkurru{{tag|n}}{{tag|sg}}{{tag|all}}|warlkurru-kurra}}
|-
! comitative
| "with" || {{tag|com}} || -ngkajinta, -rlajinta || {{morphTest|pirli{{tag|n}}{{tag|sg}}{{tag|com}}|pirli-ngkajinta}} || {{morphTest|warlkurru{{tag|n}}{{tag|sg}}{{tag|com}}|warlkurru-rlajinta}}
|-
! elative
| "out of" || {{tag|ela}} || -ngurlu || {{morphTest|pirli{{tag|n}}{{tag|sg}}{{tag|ela}}|pirli-ngurlu}} || {{morphTest|warlkurru{{tag|n}}{{tag|sg}}{{tag|ela}}|warlkurru-ngurlu}}
|-
! locative
| "at, in on" || {{tag|loc}} || -ngka, -rla || {{morphTest|pirli{{tag|n}}{{tag|sg}}{{tag|loc}}|pirli-ngka}} || {{morphTest|warlkurru{{tag|n}}{{tag|sg}}{{tag|loc}}|warlkurru-rla}}
|}


<!-- *adding (optional) dashes between everything; selecting only one analysis by using ! Dir/LR -->

*Disambiguation is a challenge because of flexible word order
 echo "rdaka" | apertium -d . wbp-morph: ^rdaka/rdaka<num>/rdaka<n><sg><abs>$^./.<sent>$

 echo "rdaka-pala" | apertium -d . wbp-morph: ^rdaka-pala/rdaka<num>$^./.<sent>$

*Trying to write the following disambiguation rule: # If there is a noun to the right or left, I cannot be a numeral-like noun.
*Examples of numeral-like nouns, sometimes distinguished by adding the number marker -pala.
**mirdi = "four" or "elbow"
**rdaka = "five" or "hand"
**wirlki = "seven" or "boomerang"
**narntirnki = "nine" or "curled"


*Next to implement: perhaps reduplication?
 echo "kurdu" | apertium -d . wbp-morph: ^kurdu/kurdu<n><sg><abs>$^./.<sent>$

 echo "kurdu-kurdu" | apertium -d . wbp-morph: ^kurdu/kurdu<n><pl><abs>$^./.<sent>$

*Need to write scraper to test on a larger corpus (current corpus is ~13,000 words)


===Evaluation===
<!--* '''How you're evaluating the effectiveness''' of the solution, and some preliminary look at results of the evaluation. We'll understand that the project isn't yet complete, so the evaluation may show that the project is entirely ineffective and useless :-P -->
* Coverage over a large corpus 
* Precision and recall against hand-annotated randomly selected forms

{|class="wikitable"
|-
! status !! description !! stems !! coverage
|-
!style="background-color: red;"| '''prototype'''
| language module that has not received heavy development
| <1,000
| <60% 
|-
!style="background-color: orange;"| '''development'''
| language module under development 
| ≥1,000 
| ≥60% 
|-
!style="background-color: yellow;"| '''working'''
| language module with near-production-quality performance 
| ≥8,000
| ≥80% 
|-
!style="background-color: green;"| '''production''' 
| language module used in a released pair
| ≥10,000
| ≥90%
|}

*Table Source: http://wiki.apertium.org/wiki/Languages
*Background on analyzers: http://www.phil.uu.nl/tst/2012/Slides/SLP_Lecture2.pdf


[[Category:sp17_FinalProjects]]