User:Tjones5/Final project

Placeholder page for final project

Notes

• To implement:
  • Reduplication

• Get a big corpus:
  • Build a scraper? to get corpus up to 25K
  • Use regular expressions to clean up scraped (or copied/pasted) text

• For evaluation:
  • Go back and look at coverage

• Other ling to do: UD pipeline

Presentation

What will be expected for this assignment is a 10 minute presentation in which you share the following with your classmates:

• A description of the problem addressed by your project. For some projects this may require thinking about it in a somewhat new light. Whether or not the problem is something we've dealt with in class, provide a brief but complete introduction to the problem.

Description of the problem

• What is a transducer? What does it do?
• Word structure is analyzed by composition of morphemes (the smallest units with grammatical meaning)
• A morphological transducer connects forms with analyses.
• Surface level -- represents the concatenation of letters which make up the actual spelling of the word, e.g. foxes
• Lexical level -- represents the concatenation of morphemes making up a word, e.g. NOUN-fox + PLURAL-es
• Morphological parsing -- the problem of recognizing that a word breaks down into component morphemes, as above with "foxes"

Previous work

• Any previous work you've found that also addresses this problem or similar problems, and how it relates to what you're doing.
  • Many languages have working Apertium transducers:
  • Warlpiri doesn't have a working transducer -- in fact, no languages from the Pama–Nyungan family have a transducer

Benefits of this project

• Your thoughts on who might benefit from your project and in what way. E.g., might a language community be able to find a use for what you're doing, or maybe [computational] linguists working on a language or issue?
  • Why is a transducer useful to a community of Warlpiri speakers?
  • Search engine
  • Spell checker
  • Machine Translation, especially to/from English (the primary language of most Warlpiri speakers). Many are trying to revive the use of Warlpiri, so there are many potentially useful applications.
    • A eng → wbp MT system could be used to look up how to say/write a certain word or phrase
    • A wbp → eng MT system could be used to analyze (and then translate) untranslated Warlpiri phrases, or give alternate translations

Approaches

• How you're approaching the solution to the overall problem, including how you're implementing the solution. You can talk here about smaller individual issues that have arisen as well.

• Mostly have worked within lexc and twol files
  • Examples??
• Disambiguation is a challenge because of flexible word order
  • Numerals are one thing to deal with
• Need to build a scraper to test on a larger corpus

Evaluation

• How you're evaluating the effectiveness of the solution, and some preliminary look at results of the evaluation. We'll understand that the project isn't yet complete, so the evaluation may show that the project is entirely ineffective and useless :-P
• Coverage over a large corpus
• Precision and recall against a hand-annotated randomly selected forms

status	description	stems	coverage
prototype	language module that has not received heavy development	<1,000	<60%
development	language module under development	≥1,000	≥60%
working	language module with near-production-quality performance	≥8,000	≥80%
production	language module used in a released pair	≥10,000	≥90%

• Table Source: http://wiki.apertium.org/wiki/Languages
• Background on analyzers: http://www.phil.uu.nl/tst/2012/Slides/SLP_Lecture2.pdf