Difference between revisions of "Morphological generator"

Revision as of 08:03, 16 February 2017

The basics

Twol rules are written to change the output of the lexc level of the transducer on a character-by-character basis. The basic structure of a twol rule is a mapping of lexc-level symbols to orthographic form-level symbols in a particular environment. A basic rule mapping lexc-level (or "morphophonological") %{x%} to form-level (or "phonetic") x when it's before form-level y would be the following:

%{x%}:x <=> _ :y ;

The mapping is lexc-level character, a colon, and form-level character, and the environment is a series of characters separated by spaces. These characters can consist of multiple symbols, and also use : to separate lexc level from form level. An underscore (_) is used to show the exact position among these symbols where the change should take place.

Some useful symbols

The basic stuff:

• : separates the lexc side of the transducer (left) from the output side (right); e.g. %>:0
• 0 on the left side means empty output
• _ is the location in which the rule is applied
• <=> separates what gets mapped from the environment, and can be read as "where"
• ; represents the end of an environment

Some more advanced stuff:

• \ means "not" the thing after it
• () means that the thing inside it is optional
• [] groups blocks of things
• | means "or"
• * after a symbol (e.g., x:y* or [ … ]* means 0 to infinite occurrences of that thing
• / optionally interleaves the symbol after it into all possible positions in the set of symbols before it, e.g. [ :a :b :c ]/:x should be equivalent to [ (:x) :a (:x) :b (:x) :c (:x) ].

Some more tricks

You can use list multiple environments

x:y <=> _ :a ;
        :b _ ;

The rule above maps x to y before an output a and after an output b.

You can also exclude certain environments using the word except:

x:y <=> _ :a ;
    except
        :b _ ;

The rule above maps x to y before an output a unless it's after an output b.

You can do a whole bunch of pairs of symbols in the same environment using a single rule, like this:

%{A%}:Cy <=> :Cy _ ;
      where Cx in ( a e o )
            Cy in ( a e o ) ;
      matched ;

This rule turns %{A%} into either a, e, or o to match the character before it.

A similar rule:

Cx:Cy <=> %{x%}:0 _ ;
      where Cx in ( a e o )
            Cy in ( ə i u ) ;
      matched ;

This rule says that a becomes ə, e becomes i, and o becomes u after an archiphoneme {x} that is deleted.

Multiple correct forms

Sometimes there are multiple correct forms for a given analysis in a language. In cases like this, you should analyse all of them, but only generate one. You can use Dir/LR in the comment section of a form to allow only form-to-analysis mapping (and not analysis-to-form mapping). You may also want to add both forms to your tests file.

An example of this is the English word "dwarf", the plural of which can be "dwarves" or "dwarfs". Since the former is considered correct by more people (or simply, is more commonly used), we'll consider it correct. You'll want something like this in your lexc file:

dwarf:dwar%{F%} NounInfl ;
dwarf:dwarf NounInfl ; ! Dir/LR

If you want to have both output forms in your yaml tests file (good for analysis tests, not for generation tests), you can have an entry like the following:

dwarf<n><pl> : {dwarves, dwarfs}

Common pitfalls

• Problem: Error messages about "conflicts"

  Cause: Rules that apply to the same left side cannot have overlapping environments. If the environments do overlap, it won't know which right side you want.

  Solution: Make sure the contexts for the rules are different. If necessary, put the context for one of the rules under an "except" in the other rule.

Useful commands

See what lexc outputs

echo "life<n><pl>" | hfst-lookup .deps/eng.LR.lexc.hfst 

There should be a line like this:

> life<n><pl>	li{f}e>{e}s	0.000000

This shows that lexc part of the transducer maps house<n><> to house>{e}s.

Check what's being generated by the full transducer

echo "^life<n><pl>$" | apertium -d . -f none eng-gener

will output something like

lifes/lifees/lives/livees

The assignment

This assignment will be due at the beginning of the Thursday class during the 6th week (this semester: 11:20am on Thursday, February 23, 2017).

Initial evaluation

1. If you're still working on your analyser, finish that first.
2. Commit and push your code at this point, and then add a tag "analyser":
   • git tag analyser
   • Then push the tag to the repo: git push origin analyser
3. Make a subsection on your Transducer wiki page for "initial evaluation of morphological generation" and note the number of passing and failing morphological analysis tests at this point using the morphology test method used for your analyser assignment.
4. Also rerun the coverage test on your corpus, and add the current coverage number.
5. Run a morph test again on your tests file, but this time with the -l option (to test morphological generation) instead of with -s (morphological analysis tests). You also don't need -i—the full command you'll probably want is morph-test -cl xyz.yaml | most.) Put the number of passes and fails for the generation test in the same subsection on the wiki page.

The hard stuff

Continue development of your transducer, focussing on twol rules (in the twoc file as well!), so that two-thirds of your [minimum 50] tests pass.

• Note that if every test passes analysis but has multiple forms generated, exactly one-half of your tests will pass—this is simply due to the way the tests are counted (i.e., each form will be counted as both passing and failing). In such a situation, each form which subsequently is made to have only a correct form generated will reduce the number of failing tests by one, but not increase the number of passing tests. Hence, if all your tests analyse correctly, then you only need to write twol rules to make half of them generate correctly.
• In another situation, half (or a little more) of your tests might be passing, and these same tests might generate correctly because you don't have any need for twol rules. If you do not need at least one twol rule (in twol or twoc files), then you should do extra work on your lexc file so that at least 85% of your tests pass.
• If you already had some necessary twol rules implemented and didn't need to modify even one of them for this part of the assignment, then you only need to aim for 75% passing generation tests.
• Make sure that your code compiles with no errors, and preferably with no warnings!

Final evaluation

When you're done with the above, add a new subsection for "final evaluation of morphological generation" and add the following:

• the number of passing and failing tests now,
• the number of twol rules that you added (and a separate number for any that you might have already had that you modified) for this part of the assignment (even if zero because they were already all working). You can use git to compare your work since you started work on this using the "analyser" tag you made at the beginning:
  • to compare all your work: git diff analyser
  • to compare only a given file (e.g., your twol file): git diff analyser apertium-xyz.xyz.twol
• Again run the coverage test on your corpus, and add the new coverage number.

Additional resources

• There's some documentation on how to deal with twol on the Apertium wiki:
  • apertium:twol
  • apertium:Working with twol
  • apertium:Starting a new language with HFST
• A number of the stable Apertium language modules have twol files, some of which are pretty extensive. You can browse those for examples or inspiration.
• There's some really in-depth documentation of twol available too.