User:Aconant1/Final project - Revision history

Aconant1 at 08:39, 14 May 2017

2017-05-14T08:39:59Z

Aconant1 at 08:42, 11 May 2017

2017-05-11T08:42:54Z

Aconant1: Created page with "Here I detail the beginnings of a [http://wiki.apertium.org/wiki/Ideas_for_Google_Summer_of_Code/User-friendly_lexical_selection_training user-friendly pipeline for learned le..."

2017-05-11T08:34:08Z

Created page with "Here I detail the beginnings of a [http://wiki.apertium.org/wiki/Ideas_for_Google_Summer_of_Code/User-friendly_lexical_selection_training user-friendly pipeline for learned le..."

New page

Here I detail the beginnings of a [http://wiki.apertium.org/wiki/Ideas_for_Google_Summer_of_Code/User-friendly_lexical_selection_training user-friendly pipeline for learned lexical selection rules]. While the architecture already exists, it is unwieldy and non-trivial to set up. Although my project does not solve every problem with the process, it makes it more approachable, and I hope to expand on it in the future. The code can be found [https://github.swarthmore.edu/aconant1/ling073-monolearner here], on GitHub.

== Designing the configuration ==

The desire was for a single input file to govern the runtime of the program. This meant designing a (very) simply yaml format in which users could input their configuration information. The example file that I included with my submission can be seen below:

[[File:C80de69219ddfefefb328d651b0959d6.png]]

This can be contrasted with the hard-coded parameters in the current Makefile architecture by noting the absence of the SCRIPTS parameter. Because this parameter varied computer-to-computer rather than pair-to-pair, it made sense to separate the two.

== Main Features ==

My program can be divided into logical, discrete units:

=== Dependency validation ===

First off, one of the most difficult-to-manage parts of the Makefile architecture for lexical selection rule learning is its dependency on dependencies, so to speak. Therefore, the first order of business is to ensure that the executables that the pipeline eventually relies on being there are actually there. This means checking for <code>apertium, apertium-destxt, apertium-pretransfer, multitrans, lrx-comp</code>, among others, as well as the various scripts normally found in <code>apertium-lex-tools</code> that would be tragic to miss.

=== Dealing with yaml ===

Next, the program reads the user-supplied yaml file, checking both its integrity and the sanity of its contents. This means that the user must pass a valid source language corpus name, translation pair direction, path to a language pair, name of a bilingual dictionary binary, and the path to a target language binary language model (compiled with IRSTLM).

== Evaluation ==

Because this project is more qualitative than quantitative, evaluation is a difficult thing to talk about. One way to do so might be to compare <code>monolearner</code> with what we had before and with what we might still desire. Although this project does not make it *easier* to

== Previous work ==

I want to quickly mention some resources that I relied on for this project:

* A non-native Python library known as PyYAML trivializes parsing yaml files.
* English and Spanish corpora were borrowed from [http://www.statmt.org/europarl/ here].
* The bidirectional English/Spanish dictionary was borrowed from Apertium's trunk.

[[Category:sp17_FinalProjects]]

← Older revision		Revision as of 08:39, 14 May 2017
Line 1:		Line 1:
−	Here I detail the beginnings of a [http://wiki.apertium.org/wiki/Ideas_for_Google_Summer_of_Code/User-friendly_lexical_selection_training user-friendly pipeline for learned lexical selection rules]. While the architecture already exists, it is unwieldy and non-trivial to set up. Although my project does not solve every problem with the process, it makes it more approachable, and I hope to expand on it in the future. The code can be found [https://github.~~swarthmore.edu~~/~~aconant1~~/ling073-monolearner here], on GitHub. Additionally, a great deal of the project's time was spent researching the process, including the theory behind it and how to get the current implementation to turn out anything interesting.	+	Here I detail the beginnings of a [http://wiki.apertium.org/wiki/Ideas_for_Google_Summer_of_Code/User-friendly_lexical_selection_training user-friendly pipeline for learned lexical selection rules]. While the architecture already exists, it is unwieldy and non-trivial to set up. Although my project does not solve every problem with the process, it makes it more approachable, and I hope to expand on it in the future. The code can be found [https://github.com/AWConant/ling073-monolearner here], on GitHub. Additionally, a great deal of the project's time was spent researching the process, including the theory behind it and how to get the current implementation to turn out anything interesting.

	== Designing the configuration ==		== Designing the configuration ==

@@ Line 1: / Line 1: @@
-Here I detail the beginnings of a [http://wiki.apertium.org/wiki/Ideas_for_Google_Summer_of_Code/User-friendly_lexical_selection_training user-friendly pipeline for learned lexical selection rules]. While the architecture already exists, it is unwieldy and non-trivial to set up. Although my project does not solve every problem with the process, it makes it more approachable, and I hope to expand on it in the future. The code can be found [https://github.swarthmore.edu/aconant1/ling073-monolearner here], on GitHub.
+Here I detail the beginnings of a [http://wiki.apertium.org/wiki/Ideas_for_Google_Summer_of_Code/User-friendly_lexical_selection_training user-friendly pipeline for learned lexical selection rules]. While the architecture already exists, it is unwieldy and non-trivial to set up. Although my project does not solve every problem with the process, it makes it more approachable, and I hope to expand on it in the future. The code can be found [https://github.swarthmore.edu/aconant1/ling073-monolearner here], on GitHub. Additionally, a great deal of the project's time was spent researching the process, including the theory behind it and how to get the current implementation to turn out anything interesting.
 == Designing the configuration ==
@@ Line 20: / Line 20: @@
 Next, the program reads the user-supplied yaml file, checking both its integrity and the sanity of its contents. This means that the user must pass a valid source language corpus name, translation pair direction, path to a language pair, name of a bilingual dictionary binary, and the path to a target language binary language model (compiled with IRSTLM).
 == Evaluation ==
-Because this project is more qualitative than quantitative, evaluation is a difficult thing to talk about. One way to do so might be to compare <code>monolearner</code> with what we had before and with what we might still desire. Although this project does not make it *easier* to
+Because this project is more qualitative than quantitative, evaluation is a difficult thing to talk about. One way to do so might be to compare <code>monolearner</code> with what we had before and with what we might still desire. First and foremost, because it operates at a higher layer of abstraction than the Makefiles, <code>monolearner</code> effectively acts as pass-through. This means that as long as it provably generates syntactically valid Makefiles, it cannot fundamentally alter the already-existing functionality (for better or worse)Although this project does not make it *easier* to set up all of the dependencies, it does make it much clearer what needs to be done/configured/set up without consulting a wiki page. Additionally, the program will fail gracefully in most (all?) cases, explaining clearly what failed and why. Additionally, rather than attempting to reinvent the wheel, the program interfaces with already-known functional code and instead opts to generate the input to the next segment of the pipeline. Another area where <code>monolearner</code> could be expanded is its implicit reliance on SRILM. Because SRILM contains <code>ngram-count</code>, the program that is used to convert corpora into binary language models, the user is currently expected to know about this beforehand. However, this could easily be automated.
 == Previous work ==