Revision as of 07:27, 4 May 2023

What I Did

Throughout this semester, I was introduced to several tools we used in class to develop language technologies like hfst, lexd, and Apertium. I did not understand what was happening behind the simple files we created in class and how they were converted to output, and using them felt difficult or not intuitive. I decided to use Python to build my own analyser for Amis, the language I worked with all semester long. By building a tool myself in Python, I have full control over all parts of the tool and its functionality.

I wanted a program that, like the .lexd file I created earlier in the semester, would use Amis grammar patterns to put together morphemes to form words and show tags.

The first step was to assemble a list of morphemes. Luckily, our lexd file already had lots of morphemes listed out for me to use. Entries in lexd file look something like this:

aku:aku     <pers><sg><p1>:
isu:isu     <pers><sg><p2>:
ingra:ingra <pers><sg><p3>:
ira:ira     <pers><sg><p3>: # Possible fix to pronoun spelling issue
ita:ita     <pers><pl><p1><incl>:
ami:ami     <pers><pl><p1><excl>:
amu:amu     <pers><pl><p2>:
angra:angra <pers><pl><p3>:

Each line has multiple instances of the morpheme, several tags, and sometimes even comments. This is unfortunately not so useful for me. I need to be able to identify the actual Amis roots, the tags, and get rid of any comments. I wrote a Python program, text_formatter.py, which takes care of this for me. The resulting text looks something like this:

aku,pers,sg,p1
isu,pers,sg,p2
ingra,pers,sg,p3
ira,pers,sg,p3
ita,pers,pl,p1,incl
ami,pers,pl,p1,excl
amu,pers,pl,p2
angra,pers,pl,p3

We've gotten rid of any extra white space, extra characters, comments, and we ended up with a more predictable format where it is clear that the Amis text comes first, followed by comma separated tags. This format will be much easier to use to populate a Python data structure.

Next, I created a data structure in Python that would be able to organize this text. My data structure separately stores the bit of Amis at the front, a list of tags, and it keeps track of the root so we can add prefixes and suffixes. I stored several lists of these data structures as variables in a dictionary, with the keys being category of morphemes, such as noun roots, verb tenses, or case markers. Finally, I implemented patterns to stitch morphemes together, while also concatenating any associated tags. The basic architecture of a pattern is as follows:

1. List the desired roots on their own.
2. Identify which types of morphemes are optional and compile a list of them.
3. For each of these options, append to all items in the first list a copy of itself with the new morpheme attached.

Pseudo Code:

forms = [all of the roots]
options = [several dictionary keys]
for each option{
    for each form{
        append to forms: a copies of the form with options attached
    }
}

The innermost loop adds more items to the list of forms, so the number of iterations of the outermost loop dynamically increases.

Links to my Code

https://github.com/lchien1/chien-ami-analysis

Results