Difference between revisions of "Spring 2018/Lexical transfer"

From LING073
Jump to: navigation, search
Line 5: Line 5:
 
# Initialise it (in the new directory) with the following command: <code>./auotgen --with-lang1=/path/to/ling073-xyz --with-lang1=/path/to/ling073-abc</code>
 
# Initialise it (in the new directory) with the following command: <code>./auotgen --with-lang1=/path/to/ling073-xyz --with-lang1=/path/to/ling073-abc</code>
 
# Compile with <code>make</code> as always.
 
# Compile with <code>make</code> as always.
 +
 +
== Adding to the dictionary ==
 +
You can add entries to the <code>apertium-xyz-abc.xyz-abc.dix</code> file in the "main" section.  This dictionary is in XML format, the basics of which you can learn about in the first chapter of [https://www.sitepoint.com/really-good-introduction-xml/ this introduction to XML].
 +
 +
An entry might look like this, with a translation from the "left side" (in <code>&lt;l&gt;</code> tags) to the right side (in <code>&lt;r&gt;</code> tags):
 +
<pre>
 +
<e><p><l>үй<s n="n"/></l>  <r>house<s n="n"/></r></p></e>
 +
</pre>
 +
 +
You only want to include the stem and the category and subcategory tags, not any grammatical tags (i.e., no singular or plural; no cases; etc.).
 +
 +
Then:
 +
# Make sure each of these stems is in the respective monolingual dictionaries (lexc files)
 +
# Compile (<code>make</code>) and check for errors.
 +
# Test, e.g. <code>echo "үй" | apertium -d . xyz-abc</code> and debug (see [[#Testing]] for more details)
 +
# Commit!
 +
 +
== Testing ==
 +
  
 
== The assignment ==
 
== The assignment ==

Revision as of 08:38, 16 March 2017

Bootstrapping a language pair

  1. To bootstrap a language pair whose primary function is to translate from language xyz to language abc, do the following: apertium-init -a1 hfst -a2 hfst xyz-abc
  2. Rename the directory to ling073-xyz-abc.
  3. Commit the entire thing to git the same way you did your transducer after bootstrapping it (unless you did it wrong).
  4. Initialise it (in the new directory) with the following command: ./auotgen --with-lang1=/path/to/ling073-xyz --with-lang1=/path/to/ling073-abc
  5. Compile with make as always.

Adding to the dictionary

You can add entries to the apertium-xyz-abc.xyz-abc.dix file in the "main" section. This dictionary is in XML format, the basics of which you can learn about in the first chapter of this introduction to XML.

An entry might look like this, with a translation from the "left side" (in <l> tags) to the right side (in <r> tags):

 <e><p><l>үй<s n="n"/></l>  <r>house<s n="n"/></r></p></e>

You only want to include the stem and the category and subcategory tags, not any grammatical tags (i.e., no singular or plural; no cases; etc.).

Then:

  1. Make sure each of these stems is in the respective monolingual dictionaries (lexc files)
  2. Compile (make) and check for errors.
  3. Test, e.g. echo "үй" | apertium -d . xyz-abc and debug (see #Testing for more details)
  4. Commit!

Testing

The assignment

  1. Bootstrap a language pair. (The person you're working with will bootstrap the reverse pair)
  2. Add all the words for transfer tests to dix.
    • And make sure each analyser can analyse all sentences correctly.
  3. Make some decisions to bring your tagsets closer together, and add mappings to dix.
    • You may have used any of <v>, <vb>, <vblex>, <verb> for verbs, and if it's not the same one, it doesn't matter.
    • If one of you marked transitivity on verbs and one of you didn't, ________
    • You probably will want to bring your grammatical information tags closer together, e.g., <nz>/<nmz> and <ger> could potentially be combined.
  4. Find more words (examine dictionaries, texts) so that there's a total of at least 50 words, with at least 10 nouns and 10 verbs.
    • Make sure all the words are in monolingual dictionaries.
    • Find at least one case in each direction of a one-to-many mapping.
  5. Evaluate in the following ways:
    • WER, PER on tests (probably 0%)
    • number of stems translated correctly on tests (hopefully close to 100%)
    • number of forms in corpus with translated stems

way to scrape and store tests