Difference between revisions of "User:Aconant1/Final project"

Latest revision as of 08:39, 14 May 2017

Here I detail the beginnings of a user-friendly pipeline for learned lexical selection rules. While the architecture already exists, it is unwieldy and non-trivial to set up. Although my project does not solve every problem with the process, it makes it more approachable, and I hope to expand on it in the future. The code can be found here, on GitHub. Additionally, a great deal of the project's time was spent researching the process, including the theory behind it and how to get the current implementation to turn out anything interesting.

Designing the configuration

The desire was for a single input file to govern the runtime of the program. This meant designing a (very) simply yaml format in which users could input their configuration information. The example file that I included with my submission can be seen below:

This can be contrasted with the hard-coded parameters in the current Makefile architecture by noting the absence of the SCRIPTS parameter. Because this parameter varied computer-to-computer rather than pair-to-pair, it made sense to separate the two.

Main Features

My program can be divided into logical, discrete units:

Dependency validation

First off, one of the most difficult-to-manage parts of the Makefile architecture for lexical selection rule learning is its dependency on dependencies, so to speak. Therefore, the first order of business is to ensure that the executables that the pipeline eventually relies on being there are actually there. This means checking for apertium, apertium-destxt, apertium-pretransfer, multitrans, lrx-comp, among others, as well as the various scripts normally found in apertium-lex-tools that would be tragic to miss.

Dealing with yaml

Next, the program reads the user-supplied yaml file, checking both its integrity and the sanity of its contents. This means that the user must pass a valid source language corpus name, translation pair direction, path to a language pair, name of a bilingual dictionary binary, and the path to a target language binary language model (compiled with IRSTLM).

Makefile generation

Once everything is properly validated, the program generates a Makefile that can be run to generate lexical selection rules.

Evaluation

Because this project is more qualitative than quantitative, evaluation is a difficult thing to talk about. One way to do so might be to compare monolearner with what we had before and with what we might still desire. First and foremost, because it operates at a higher layer of abstraction than the Makefiles, monolearner effectively acts as pass-through. This means that as long as it provably generates syntactically valid Makefiles, it cannot fundamentally alter the already-existing functionality (for better or worse)Although this project does not make it *easier* to set up all of the dependencies, it does make it much clearer what needs to be done/configured/set up without consulting a wiki page. Additionally, the program will fail gracefully in most (all?) cases, explaining clearly what failed and why. Additionally, rather than attempting to reinvent the wheel, the program interfaces with already-known functional code and instead opts to generate the input to the next segment of the pipeline. Another area where monolearner could be expanded is its implicit reliance on SRILM. Because SRILM contains ngram-count, the program that is used to convert corpora into binary language models, the user is currently expected to know about this beforehand. However, this could easily be automated.

Previous work

I want to quickly mention some resources that I relied on for this project:

• A non-native Python library known as PyYAML trivializes parsing yaml files.
• English and Spanish corpora were borrowed from here.
• The bidirectional English/Spanish dictionary was borrowed from Apertium's trunk.