Difference between revisions of "Morphological analyser"

From LING073
Jump to: navigation, search
(In-class exercise)
(In-class exercise)
Line 222: Line 222:
See [[Morphological analyser/Exercises]].
See [[Morphological analyser/Exercises]].
The work we did on this in class is available on Swarthmore's github at [https://github.swarthmore.edu/Ling073-sp17/ling073-eng ling073-sp18/ling073-eng].
The work we did on this in class is available on Swarthmore's github at [https://github.swarthmore.edu/Ling073-sp18/ling073-eng ling073-sp18/ling073-eng].
== Evaluation ==
== Evaluation ==

Revision as of 12:25, 20 February 2018

Morphological transducers

A morphological transducer is just a directed graph. It consists of nodes (numbered below) and arcs (with labels), with a starting node (0 below) and an ending node (16 below).

Simple transducer.png

You follow the arcs that are available from your input. The only acceptable paths are ones that start from starting node and end at the ending node. You may match your input to either side of the arc's label (separated by : above), and the other side is returned as output.

In the transducer above, the left side is the form and the right side is the analysis. If you match your input to the left side (the form), then your output will be the right side (the analysis)—this is morphological analysis. Likewise, if you follow the transducer by matching your input to the right side (the analysis) and output the left side (the form), then you are performing morphological generation.

An example of a complete path is w:w o:o l:l v:f e:<n> s:<pl>. The left/form side of this spells wolves and the right/analysis side of this spells wolf<n><pl>. Mapping between one and the other is as simple as taking one as input and following the path—by outputting the other side of each arc, you will get the other as output!

Question: What are all the possible paths provided by this transducer?

The formalism we use (lexc)

Transducers are pretty cool, and quite efficient... for computers. Following paths by hand is tedious, and drawing a transducer for anything more complex than the example above is torture. See the transducer below for Tuvan.

Tuvan transducer.png

This transducer provides the combinations of about 8 case marker, 5 possessive morphemes, and the plural marker for three Tuvan nouns.

An example is өг>{L}{A}р>{i}м>{D}{A}н mapping to өг<n><pl><px1sg><abl>, meaning "from my houses". The analysis side is clear to anyone familiar with tags (and knowing that "өг" means "house"). The form side is actually something that will get fixed by morphophonology, which we'll worry about later (for now: letters like {L} can be realised in a variety of ways, and > is used as a morpheme boundary); the actual orthographic form is өглеримден.

Question: How can we quantify the complexity of this graph?

Fortunately, we don't have to draw this graph by hand. We can simply define the various sections of it and link them together with a straightforward formalism called lexc. A section of a lexc file that corresponds (mostly) to the graph above looks like the following:


%<gen%>:%>%{N%}{I%}ң # ;
%<acc%>:%>%{N%}%{I%} # ;
%<dat%>:%>%{G%}%{A%} # ;
%<loc%>:%>%{D%}%{A%} CLITICS-COPULA ;
%<abl%>:%>%{D%}%{A%}н # ;
%<all%>:%>%{J%}е # ;
%<all%>:%>%{D%}%{I%}в%{A%} # ; ! Dir/LR


%<px1sg%>:%>%{i%}м CASES ;
%<px2sg%>:%>%{i%}ң CASES ;
%<px3sp%>:%>%{z%}%{I%}%{n%} CASES ;
%<px1pl%>:%>%{i%}в%{I%}с CASES ;
%<px2pl%>:%>%{i%}ң%{A%}р CASES ;




%<pl%>:%>%{L%}%{A%}р N-INFL-COMMON ;


%<n%>%<attr%>: # ;
%<n%>: SUBST ;


өг:өг N1 ; ! "yurt"
аът:аът N1 ; ! "horse"
ном:ном N1 ; ! "book"


  • What is % doing?
  • What is ! doing?
  • What is : doing?
  • How are the continuation lexica (LEXICONs) connected?
  • What is ; doing?
  • What is # doing?
  • What is mentioned in this code that isn't in the graph above?
  • What is not mentioned in this code that is in the graph above?
  • Can you match sections of the graph to sections of the code?

Additional nuances


The symbols like {L} above will need to be realised as different characters in different context.

For any symbols in your language that will be realised in different ways in different environments, you'll want to set up such an "archiphoneme". Use an uppercase letter for something that just has different forms, and use a lowercase letter for something that is inserted or deleted (i.e., is sometimes realised as nothing).

For now, it will suffice to define all the ways in which each archiphoneme surfaces by making a list in your twol file. This essentially allows all of the options to surface, which means you will be able to analyse incorrect forms as well as correct ones. Later, when you make a generator, you'll write rules to constrain where each of the symbols can occur.

Defining symbols

Don't forget to define all your symbols (archiphonemes like {L}, and tags like <pl>) in the lexc file! And define your archiphoneme symbols in the twol file, each with all its possible outputs.

So your twol file should contain an Alphabet section, which lists all the characters of the alphabet, and then all the archiphonemes with all their realisations. You will also want the > morpheme separator and some punctuation marks, all escaped. A condensed example for Tuvan follows:


   А Б В Г Д Е Ё Ж З И Й К Л М Н Ң О Ө П Р С Т У Ү Ф Х Ц Ч Ш Щ Ъ Ы Ь Э Ю Я
   а б в г д е ё ж з и й к л м н ң о ө п р с т у ү ф х ц ч ш щ ъ ы ь э ю я

   %{A%}:а %{A%}:е
   %{L%}:л %{L%}:н
   %{i%}:0 %{i%}:ы %{i%}:и %{i%}:у %{i%}:ү



Starting point

You'll need a Root lexicon in your lexc file. Bootstrapping a new language module per the instructions will create this for you, but don't forget that it's a thing!

Morphology that isn't suffixes

You may have noticed that analyses are generally in the form stem + POS tags + subcategory tags + function tags. What if some of your functional morphology occurs before the stem?

You can certainly implement that in lexc, but there's a problem: your tags will occur in the middle of the analysis. So instead of something like do<v><tv><rep><prc> ↔ redoing, you'd get something like <rep>do<v><tv><prc> ↔ redoing. This is undesirable.

Currently, the best way to handle this is documented in two places on the Apertium wiki: apertium:Replacement for flag diacritics and apertium:Morphotactic constraints with twol. The rules go in a new file, apertium-xyz.xyz.twoc. You will also need to modify your Makefile.am to look more like the Makefile for Chukchi in terms of the twoc stuff (replacing ckt with the code for your language). You will then have to reconfigure your module (./autogen.sh) before recompiling (make).

A slightly more generalised version of this solution: The twoc file should include all your tags (both in angle brackets and square brackets) in the alphabet. Then add a set named e.g. Features with all square-bracket tags. You can then add a Rule that just removes any path without features that match. I.e., you only get the forms that have both a plus and minus version of a given feature. A short example is provided below:


  %[%-nt%]:0 %[%-m%]:0 %[%-f%]:0 %[%-pl%]:0
  %[%+nt%]:0 %[%+m%]:0 %[%+f%]:0 %[%+pl%]:0



"Remove paths without matching suffix feature"
Fx:0 /<= _ ;
       _ :* Fy:0 ;
   where Fx in ( %[%-nt%] %[%-m%] %[%-f%] %[%-pl%] )
         Fy in ( %[%+nt%] %[%+m%] %[%+f%] %[%+pl%] )
   matched ;

If you can ever have forms with an odd number of feature tags output from lexc (e.g., a path where there's only a %[%+m%] form with no %- feature of any sort before it), you'll need another rule to get rid those paths too, something like a reverse of the above rule.

"Remove paths without matching prefix feature"
Fx:0 /<= _ ;
       Fy:0 :* _ ;
   where Fy in ( %[%-m%] )
         Fx in ( %[%+m%] )
matched ; 

A matching lexc file, using gender circumfixes in Avar, might look like this:


! etc.


Prefixes ;


%<aor%>%<nt%>%[%+nt%]:уна # ;
%<aor%>%<m%>%[%+m%]:уна # ;
%<aor%>%<f%>%[%+f%]:уна # ;
%<aor%>%<pl%>%[%+pl%]:уна # ;

LEXICON Prefixes

%[%-nt%]:б%> Verbs ;
%[%-m%]:в%> Verbs ;
%[%-f%]:й%> Verbs ;
%[%-pl%]:р%> Verbs ;


бицине%<v%>%<tv%>:иц AOR ; ! "говорить"

The output analyses would be the following:


In-class exercise

See Morphological analyser/Exercises.

The work we did on this in class is available on Swarthmore's github at ling073-sp18/ling073-eng.


Individual forms

To test whether/how your analyser is analysing a form, you can run the following:

echo "form" | apertium -d /path/to/analyser/ xyz-morph

An example might be the following:

apertium-tyv$ echo өглеримден | apertium -d . tyv-morph

This output means that for the form өглеримден there is one analysis: өг<n><pl><px1sg><abl>. A form with multiple analysis would have them separated by /, like the following:


A form with no analyses in the transducer will just return the form with an * before it, like the following:


A long list of forms with known analyses

To test whether your analyser is analysing forms correctly, you can put your analyses into a yaml file and use morph-test or aq-morftest:

morph-test -csi xyz.yaml | most


aq-morftest -csi xyz.yaml | most

Coverage over a corpus

To test coverage over a corpus, you can use aq-covtest:

aq-covtest xyz.corpus.basic.txt /path/to/xyz.automorf.bin

Generating forms

If you need to test how a form generates, you can do something like the following:

echo "^house<n><pl>$" | apertium -d . -f none xyz-gener

This will return all forms currently being generated, e.g. houses/housees

The assignment

This assignment will be due on Thursday of the 5th week of class before class starts (this semester: 11:20am on Thursday, February 16th, 2017).

This assignment is to develop a morphological analyser that implements a good deal of the basic morphology of your language.

Getting set up

  1. Bootstrap a transducer for your language.
  2. Initialise the module (./autogen.sh), and compile it (make).
    • If this is successful, you should have several "modes" available; run apertium -d . -l to see.
    • One mode should be an xyz-morph mode; this is your analyser. Check it by running echo "houses" | apertium -d . xyz-morph , which should give you a morphological analysis of the word "houses".
  3. Integrate any comments I've provided to you on your grammar documentation page so that all of your morphTests are in good order. See the sanity checks at Grammar documentation#Sanity checks to check the main things.
  4. Add all of the tags you came up with during the Grammar documentation assignment to the Multichar_Symbols section of the apertium-xyz.xyz.lexc file. Provide a symbol, and a brief comment explaining what the symbol means.
  5. Add all the characters of your language's orthography to the Alphabet section of the apertium-xyz.xyz.twol file. You may need to add archiphonemes later.
  6. Use the morphTests2yaml script to create a yaml test file in a subdirectory called tests/. Commit this file to the git repo. (You can remove blank sections if you like, and if they appear in the file.) There should be at least 50 tests in this file—make sure you have enough.

The hard stuff

  1. Build your morphological transducer, adding all of the stems from your Grammar documentation assignment, categorised correctly, so that at least half of your tests pass. You'll need to build up the morphotactics too.
    • If too many of your grammar points are too hard to implement at this point (e.g., require some rules to change some characters to other characters), then you can skip one or two of them and instead add more "easy" forms to your transducer.
    • Alternatively "hard-code" some forms, but add a comment in the lexc file near the relevant forms indicating that they need further work, and mention it on the wiki page (see below).
    • Also don't forget to clean up your grammar page. Don't forget to rerun the scraping script to get a fresh version of the tests file. If you're happy with your grammar page and don't expect to change it much, feel free to "clean up" your tests file manually.
  2. Create a page on the wiki Language/Transducer that links to the code and has Evaluation and Notes sections.


When you've finished getting half of your tests to pass.

Evaluate coverage on your corpus and add the one of the most frequent unanalysed words:

  1. Use aq-covtest to see how many forms in your basic corpus are analysed, and what the top unknown forms are.
    • Make note of the coverage at this point
  2. Make a new yaml file in your tests directory with the top unanalysed words, and name it something like commonwords.yaml. For the analysis side, just put an <unk> tag (for "unknown") after each form. Don't forget to commit this to your git repository.
  3. Figure out what the analyses of at least three of these words should be, using the resources you have available (grammar books, etc.), and update the analysis side of your yaml file accordingly.
  4. Add at least one of these analyses to your transducer so that the test passes.
  5. Rerun aq-covtest to see by how much your coverage improved.
    • Add a note to the notes section of the additional top word(s) you added, and the resulting change in coverage (e.g., «by adding "and<cnjcoo> ↔ and" to the transducer, coverage went from 9.76% to 12.32%»)

In the Evaluation section on the wiki page, add the following:

  • Total number of stems in the transducer. You can use the lexccounter script, or count the stems manually. (For languages with non-suffixational morphology, you'll probably need to count the stems manually.)
  • Current coverage over your combined corpus
  • The current list of top unknown words returned by aq-covtest
  • Number of tests that pass in each yaml file
    • The main yaml file should have at least half of the tests passing
    • The commonwords.yaml file should have at least 1 passing test


  1. Add yourself to the AUTHORS file.
  2. Make sure the COPYING file contains an open-source license to your liking (default should be GPL3).
  3. Add links to the transducer repo and wiki page to the list of resources you developed for your language on the language's page on this wiki.

Sanity checks before submitting

  1. Did you commit just the initial files created by bootstrap before you initialised or compiled the module? If not, start over with bootstrapping, being sure copy over any files you've changed. Or use this method.
  2. Did you commit your updates to lexc and twol files? And the yaml test file?
  3. Do you have at least 50 tests in the main tests file? Do at least half of them pass a morph-test?
  4. Did you add everything asked for to the wiki page (evaluation, etc.) and your repo (e.g., both yaml files).
  5. If you have trouble analysing or compiling, are all your tags and symbols defined in both lexc and twol files?