Difference between revisions of "Rbuckle1 Arobey1 Cstuartroe/Final Project"

Revision as of 18:58, 15 May 2018

The Problem

Adding dictionary entries to a bilingual Apertium dictionary is mind numbing but incredibly important. We thought that after all the structural transfer and other rules had been written, perhaps we could automate a way to fill out the dictionary with word entries.

Our Idea

We would take two parallel texts, one in each language, and use to find word pairs. The idea was that we would translate one from one language to the other, and then analyze the second with a monolingual transducer, and compare the two outputs. We compared word lemmas and parts of speech and used this to remove as many accurately translated pairs as possible. We then used a thesaurus to remove word pairs that had been translated slightly differently then expected. Examples of this include, grandma and grandmother, or adorable and sweet. The remaining words we considered candidates for translation.

Building Entries

Word pairs that were candidates for translation were fed into a function that outputted a correctly formatted dictionary entry to the command line, users of the program then had the ability to either approve or deny the entry. Approved entries are written to the dictionary file directly above the first existing entry with the same part of speech, and denied entries are discarded.

Additional Tools

We also built an interface for adding dictionary entries from the command line. Stored in "interface.py" this tool allows you to input to monolingual Apertium directories containing transducers, as well as a dictionary and a few other parameters, and then add entries from the command line. It will ask you for a word in each language, it will then find analysis of these files via the apertium transducers, and suggest a dictionary entry. If you approve the dictionary entry the will write the entry to the dictionary in the same way as the above tool. It will then ask you for two new words and repeat the process.

Further Work

Our entry builder/approval system is fully operational and has not produced an invalidly formatted dictionary entry yet. Our algorithm for finding translation candidates could be dramatically approved though. We believe that this tool could be very powerful for ling073 students and computational linguists alike and could also be very valuable to language communities. If someone could improve our word matching algorithm and take advantage of our interface that could be very valuable. One idea we have for improving our word matching algorithm is to use the principles of Dynamic Programming to build a 2D area with one text on each axis and to iterate through the area scoring each possible word pair. The word pair scores could then be used to determine the best candidates for translation.

Our Code

Our code can be found here: https://github.com/ryebuckley/ApertiumDictionaryBuilder.git
It is licensed under an open source license and we encourage anyone and everyone to use it and improve it.

Category:sp18_FinalProjects