Spring 2018/Lexical transfer
From LING073
Bootstrapping a language pair
- To bootstrap a language pair whose primary function is to translate from language xyz to language abc, do the following:
apertium-init -a1 hfst -a2 hfst xyz-abc - Rename the directory to
ling073-xyz-abc. - Commit the entire thing to git the same way you did your transducer after bootstrapping it (unless you did it wrong).
- Initialise it (in the new directory) with the following command:
./auotgen --with-lang1=/path/to/ling073-xyz --with-lang1=/path/to/ling073-abc - Compile with
makeas always.
Adding to the dictionary
You can add entries to the apertium-xyz-abc.xyz-abc.dix file in the "main" section. This dictionary is in XML format, the basics of which you can learn about in the first chapter of this introduction to XML.
An entry might look like this, with a translation from the "left side" (in <l> tags) to the right side (in <r> tags):
<e><p><l>үй<s n="n"/></l> <r>house<s n="n"/></r></p></e>
You only want to include the stem and the category and subcategory tags, not any grammatical tags (i.e., no singular or plural; no cases; etc.).
Then:
- Make sure each of these stems is in the respective monolingual dictionaries (lexc files)
- Compile (
make) and check for errors. - Test, e.g.
echo "үй" | apertium -d . xyz-abcand debug (see #Testing for more details) - Commit!
Testing
The assignment
- Bootstrap a language pair. (The person you're working with will bootstrap the reverse pair)
- Add all the words for transfer tests to dix.
- And make sure each analyser can analyse all sentences correctly.
- Make some decisions to bring your tagsets closer together, and add mappings to dix.
- You may have used any of <v>, <vb>, <vblex>, <verb> for verbs, and if it's not the same one, it doesn't matter.
- If one of you marked transitivity on verbs and one of you didn't, ________
- You probably will want to bring your grammatical information tags closer together, e.g., <nz>/<nmz> and <ger> could potentially be combined.
- Find more words (examine dictionaries, texts) so that there's a total of at least 50 words, with at least 10 nouns and 10 verbs.
- Make sure all the words are in monolingual dictionaries.
- Find at least one case in each direction of a one-to-many mapping.
- Evaluate in the following ways:
- WER, PER on tests (probably 0%)
- number of stems translated correctly on tests (hopefully close to 100%)
- number of forms in corpus with translated stems
way to scrape and store tests