User:Dwurtel1/Final project
Contents
Project Details
What I did
For my final project, I continued work on my Ainu transducer to get near ~85% coverage on a large corpus. Although I had been working toward my final project earlier than two weeks ago, many changes are lost within earlier commits. This project is important because Ainu is lacking in linguistic tools and this is the first step to a MT system between Japanese and Ainu.
Corpus
I copied nearly half of all free available Ainu articles on from [1]. Although they have some English and Kanji thrown in by the author, this seemed like the most realistic and honest look at modern Ainu. The author specializes in Esperanto and writes about many different governmental issues and health practices, which make for a lot of one-off lexical stems. The total word count is 7362 with 25,335 characters (although these numbers vary greatly based on what editor's word/char count tools I use).
Transducer
I added 50-60 words, very few of which were in dictionary form. This required some analysis of what the root of the unidentified word was before I could look it up (and even then there were many that had no answer. This took much time, but I would get as far as I could fixing the current iteration of the top twenty unknown words, and then restart the process. I did this four times. Most words required looking at the context in order to analyze properly, so there was some inherent disambiguation although no disambiguation rules were added themselves. I also added rules for verb prefixing with respect to colloquial and some classical subject/object pronoun incorporation. It was challenging to separate different affixes and their meanings. For instance, the prefix ア is often used to describe indefiniteness but from context it is sometimes still unclear whether this should be viewed as an indicator of first person, fourth person, or something else entirely.
Further Research
As I do plan to continue working on the transducer, my first goal is to fix spellrelax issues with twol rules. For instance, there are some hacky fixes for ウタㇽ vs ウタラ wherein a small {ra, ri, ru, re, ro} are all equivalent to a large {ra, ri, ru, re, ro} at the end of the word, but what happens with different spellings of 'good', for instance: ピㇽカ/ピㇼカ/ピリカ? In addition, I often stumbled upon possible noun compounding written with spaces sometimes アイヌ ウタラ and without, others アイヌイタㇻ. One avenue of approach is to completely change the orthography. Nearly every site that has modified katakana for Ainu also has the latin letter Ainu, and things like -ay could only ever be interpreted as such instead of katakana's アイ vs. アィ.
Repositories
Ainu Transducer: public
Note: I don't think I am allowed to make a public repo with these materials
Ainu Corpus: private
Evaluation
- Original coverage over the colloquial corpus: 52.06%
- Final coverage over the colloquial corpus: 76.81%
Due to timing I am still working on precision and recall.
Post-class Evaluation
- All available Ainu Times articles (colloquial corpus) added: ~14000 words
- All Ainu Folklore transcriptions added: ~20000 words
- Coverage over full colloquial corpus: 72.72% (16,256 tokenized words)
- Coverage over classical corpus: 64.62% (30,129 tokenized words)
- Coverage over entire combined corpus: 67.45% (46382 tokenized words)
