Difference between revisions of "User:Tjones5/Final project"

Latest revision as of 00:40, 11 May 2017

Improvements Made to the Transducer

Final version: https://github.com/tkjones117/ling073-wbp

Commit: https://github.swarthmore.edu/tjones5/ling073-wbp/commit/125bb63f33f5f50caf682b746b972c549b4da7a0

• .lexc:
  • optional dashes throughout (verb suffixes, pronouns)
  • improved organization of pronoun analysis: separation of 1st person from 2nd/3rd person
  • added several noun stems
• .twol:
  • added %{U%} archiphoneme to appear in suffixes on nouns in dative case

Commit: https://github.swarthmore.edu/tjones5/ling073-wbp/commit/f8841c64db1c248815bc5c5a3ce4a23b8d30f641

• .lexc:
  • added present verb analysis (ex: nyina = present form of "to be", not piyanimi)
  • added verb tense and suffix analysis (apparently sometimes verbs can object suffixes; previously I thought it was only auxiliaries that could do this)
  • added "kirra" as another possible allative object ending
  • added clitic analysis and several clitics -- something needs to be changed so that they increase coverage
  • added 6 noun stems, ~55 verb stems, 3 adverb stems
• .twol:
  • eliminated bi-directionality of some rules

Commit: https://github.swarthmore.edu/tjones5/ling073-wbp/commit/2857cffb76e0014b8ea90b21f40e4cc957ae280f

• .lexc:
  • added "yungu" as a variation of the auxiliary "yunga"
  • added more optional dashes in the auxiliary suffixes
  • added ability to append another object suffixes to auxiliary verb
  • added auxiliary suffixes to adverbs

Commit: https://github.swarthmore.edu/tjones5/ling073-wbp/commit/8cd682e70a81378184b6a90f381f16e6d130459c

• .lexc:
  • added a few more stems

Improvements Made to Disambiguation

Commit: https://github.swarthmore.edu/tjones5/ling073-wbp/commit/6edc1d225ba43f369468448b83d009616a9c6c41

• Ambiguity before disambiguation: ~1.41077302631578947368
• Ambiguity after disambiguation (Pre-final project): ~1.35831766917293233083
• Ambiguity after disambiguation (Final): ~1.34727443609022556391

# If there is a transitive verb two words to the left, I don't take a subject.
REMOVE (n sg prf) IF (-2 Vtv) ;

# If there is a transitive verb two words to the left, I'm in the absolutive case.
SELECT (n sg abs) IF (-2 Vtv) ;

# If there is an intransitive verb two words to the left, I'm the subject.
SELECT (n sg abs) IF (-2 Viv) ;

# If there is an auxiliary to the right or left, I cannot be a numeral.
REMOVE (num) IF ((1 Aux) OR (-1 Aux)) ;

# If there is an auxiliary to the left, I cannot be an auxiliary.
REMOVE (vaux) IF (-1 Aux) ;

Final Evaluation

Number of stems: ~340

Final coverage: 61.48%

Against new_corpus.txt:

• Precision: 84.21053%
• Recall: 32.98969%

Presentation

Description of the problem

• Word structure is analyzed by composition of morphemes (the smallest units with grammatical meaning)
• A morphological transducer connects forms with analyses.
• Surface level -- represents the concatenation of letters which make up the actual spelling of the word, e.g. wolves
• Lexical level -- represents the concatenation of morphemes making up a word, e.g. NOUN-wolf + PLURAL-es
• Morphological parsing -- the problem of recognizing that a word breaks down into component morphemes, as above with "wolves"

Previous work

• Many languages have working Apertium transducers:
• Warlpiri doesn't have a working transducer -- in fact, no languages from the Pama–Nyungan family have a transducer

Benefits of this project

• Why is a transducer useful to a community of Warlpiri speakers?
  • Search engine
  • Spell checker
  • Machine Translation, especially to/from English (the primary language of most Warlpiri speakers). Many are trying to revive the use of Warlpiri, so there are many potentially useful applications.
    • eng → wbp MT system could be used to look up how to say/write a certain word or phrase
    • wbp → eng MT system could be used to analyze (and then translate) untranslated Warlpiri stories, or give alternate translations
  • All of these applications might also be helpful in starting an analyzer for other indigenous languages that have the same orthography, are highly agglutinative, and have other grammatical similarities

Approaches

• Mostly have worked within lexc and twol files

• Pronouns (optional dash not included in table):

• pirli (a noun with just 2 vowels) has different endings than warlkurru (which has more than two vowels)
  • however when suffixes such as -jarra are added, pirli gets the endings of nouns with >2 vowels: pirli-ngku → pirli-jarra-rlu

• Disambiguation is a challenge because of flexible word order

echo "rdaka" | apertium -d . wbp-morph: ^rdaka/rdaka<num>/rdaka<n><sg><abs>$^./.<sent>$

echo "rdaka-pala" | apertium -d . wbp-morph: ^rdaka-pala/rdaka<num>$^./.<sent>$

• Trying to write the following disambiguation rule: # If there is a noun to the right or left, I cannot be a numeral-like noun.
• Examples of numeral-like nouns, sometimes distinguished by adding the number marker -pala.
  • mirdi = "four" or "elbow"
  • rdaka = "five" or "hand"
  • wirlki = "seven" or "boomerang"
  • narntirnki = "nine" or "curled"

• Next to implement: perhaps reduplication?

echo "kurdu" | apertium -d . wbp-morph: ^kurdu/kurdu<n><sg><abs>$^./.<sent>$

echo "kurdu-kurdu" | apertium -d . wbp-morph: ^kurdu/kurdu<n><pl><abs>$^./.<sent>$

• Need to write scraper to test on a larger corpus (current corpus is ~13,000 words)

Evaluation

• Coverage over a large corpus
• Precision and recall against hand-annotated randomly selected forms

• Table Source: http://wiki.apertium.org/wiki/Languages
• Background on analyzers: http://www.phil.uu.nl/tst/2012/Slides/SLP_Lecture2.pdf