View source for Morphological disambiguator

== Why we need disambiguation ==
Imagine you have a word that has two different tagsets, e.g.
 ^houses/house<n><pl>/house<v><tv><p3><sg>$
 ^this/this<det><dem><sg>/this<pron><dem><sg>$

Normally your analyser will randomly choose one of these.  Now imagine you have a sentence where the wrong analysis is chosen (in this case, twice!):
 ^The/The<det><def>$ ^motel/motel<n><sg>$ ^houses/house<n><pl>$ ^this/this<pron><dem><sg>$ ^dog/dog<n><sg>$^./.<sent>$

The goal of a disambiguator is to choose the correct analysis based on the surrounding words.

== Using Constraint Grammar to disambiguate ==
Constraint Grammar (CG) is a formalism that can be used to formalise context sensitive constraints to select or remove analyses from the list of possible analyses.

=== The structure of a CG file ===

At the top you define delimiters.  You can then add lists (which are kind of like sets in twol).  After that you can have any number of constraints.  You can also put lists and constraints in sections. In the original specification for constraint grammar, all the constraints in each section are applied simultaneously, but the sections are applied in order and lists are always global. In the version we use (VISL CG3), the constraints are applied in strict order within the sections.

=== Constraints ===

Think of constraints as a series of heuristics to whittle down the number of analyses until you have as little ambiguity as possible.

The most commonly used constraints are to either '''remove''' a reading under a certain condition or '''select''' a reading under a certain condition.  The syntax is e.g., <code>REMOVE <target> [contextual tests] ;</code>.

Targets can be defined as list names—like <code>Nominals</code>—or individual tag names, in parentheses—e.g., <code>(n pl)</code>.  This target identifies the reading to select or remove.

Contextual tests come together in ()s.  All tests must match in order for the constraint to apply.  Inside the parentheses, the first definition is a position: <code>0</code> means the current cohort, <code>-1</code> means the previous cohort, etc.  The rest of the test is what should be matched.  If you want to match any member of the <code>Nominals</code> set in the previous position, it would be <code>(-1 Nominals)</code>  You can match tag sequences by adding an extra set of parentheses, e.g. <code>(1 (n pl))</code> would match a plural noun in the following position.  You can match specific baseforms with quotation marks, e.g. <code>(1 ("house" n))</code> would match any form of the noun "house" in the following position.

'''Classroom activity''': implement example at top of page.  [https://github.swarthmore.edu/Ling073-sp22/ling073-eng/blob/master/apertium-eng.eng.rlx Solution].

=== Useful commands ===

The file that your disambiguation constraints are defined in is the <code>apertium-xyz.xyz.rlx</code> file.

==== Getting output before disambiguation ====
example command:
 echo "The motel houses this dog" | apertium -d . eng-morph
example output:
 ^The/the<det><def>$ ^motel/motel<n><sg>$ ^houses/house<n><pl>/house<v><tv><p3><sg>$ ^this/this<det><dem><sg>/this<prn><dem><sg>$ ^dog/dog<n><sg>$^./.<sent>$

==== Getting output after disambiguation ====
example command:
 echo "The motel houses this dog" | apertium -d . eng-tagger
example output:
 ^The<det><def>$ ^motel<n><sg>$ ^house<v><tv><p3><sg>$ ^this<det><dem><sg>$ ^dog<n><sg>$^.<sent>$

==== Seeing which constraints are doing what ====
example command:
 echo "The motel houses this dog" | apertium -d . eng-disam
example output:
<pre>
"<The>"
	"the" det def
"<motel>"
	"motel" n sg
"<houses>"
	"house" v tv p3 sg
;	"house" n pl REMOVE:26
"<this>"
	"this" det dem sg
;	"this" prn dem sg REMOVE:29
"<dog>"
	"dog" n sg
"<.>"
	"." sent
</pre>


==== Calculating ambiguity ====
''Note: `lt-proc` and `.bin` can be replaced with `hfst-proc` and `.hfst` throughout if there are problems with tokenisation or similar''

The ambiguity of a corpus is the average number of analyses provided per token.
You can get the number of tokens like this:
 cat corpus.txt | lt-proc /path/to/xyz.automorf.bin | sed 's/$\W*\^/$\n^/g' | wc -l
You can get the total number of analyses in your corpus like this:
 cat corpus.txt | lt-proc /path/to/xyz.automorf.bin | sed 's/$\W*\^/$\n^/g' | cut -f2- -d'/' | sed 's/\//\n/g' | wc -l
You can get the total number of analyses in your corpus after disambiguation like this:
 cat corpus.txt | lt-proc /path/to/xyz.automorf.bin | cg-proc /path/to/xyz.rlx.bin | sed 's/$\W*\^/$\n^/g' | cut -f2- -d'/' | sed 's/\//\n/g' | wc -l


Automating it a little further, you can put something like the following in a file and run it as a shell script.  The first three lines need to be replaced with the path to your corpus, your compiled analyser, and your compiled disambiguator, respectively.
<pre>
CORPUS=/path/to/xyz.corpus.txt
MORPH=/path/to/xyz.automorf.bin
RLX=/path/to/xyz.rlx.bin # you might need to remove ".bin"
TOKENS=`apertium-destxt ${CORPUS} | lt-proc ${MORPH} | apertium-retxt | sed 's/$\W*\^/$\n^/g' | wc -l`
ANALYSES=`apertium-destxt ${CORPUS} | lt-proc ${MORPH} | apertium-retxt | sed 's/$\W*\^/$\n^/g' | cut -f2- -d'/' | sed 's/\//\n/g' | wc -l`
DISAMB=`apertium-destxt ${CORPUS} | lt-proc ${MORPH} | cg-proc ${RLX} | apertium-retxt | sed 's/$\W*\^/$\n^/g' | cut -f2- -d'/' | sed 's/\//\n/g' | wc -l`
AMBIGPRE=`calc $ANALYSES/$TOKENS`
AMBIGPOST=`calc $DISAMB/$TOKENS`
echo "Ambiguity before disambiguation: ${AMBIGPRE}"
echo "Ambiguity after disambiguation: ${AMBIGPOST}"
</pre>

It's recommended that you put this script in <code>tests/</code> and name it <code>disambiguation-test.sh</code>.  Don't forget to <code>git add</code> it, <code>git commit</code>, and <code>git push</code>.

==== Checking what forms are ambiguous in the corpus and transducer ====
You can get all of the words in the corpus along with their analyses (or lack thereof) like this:
 cat corpus.txt | lt-proc /path/to/xyz.automorf.bin | sed 's/$\W*\^/$\n^/g'
You can get all of the words in the corpus that have more than one analysis like this:
 cat corpus.txt | lt-proc /path/to/xyz.automorf.bin | sed 's/$\W*\^/$\n^/g' | grep '\/.*\/'

You can find all forms in your transducer like this:
 hfst-expand xyz.automorf.hfst

You can count ambiguity in your transducer by comparing the number of all forms to the number of unique forms.

Number of all forms:
 hfst-expand xyz.automorf.hfst | wc -l

Number of unique forms:
 hfst-expand xyz.automorf.hfst | cut -f1 -d':' | sort -u | wc -l

If the number's different you'll want to compare the lists:
<pre>
hfst-expand xyz.automorf.hfst | cut -f1 -d':' | sort > /tmp/totalforms
hfst-expand xyz.automorf.hfst | cut -f1 -d':' | sort -u > /tmp/uniqforms
diff /tmp/totalforms /tmp/uniqforms
</pre>

Then you'll want to see what the multiple analyses are of any forms that show up there.

== The assignment ==
This assignment is due at the end of the day on Friday of the 7th week of class (this semester, '''23:59, Friday, March 4th, 2022''').  During our class meeting on the following Thursday ('''March 17th'''), you will also be expected to '''present a brief [[midterm overview]]''' to the class.

=== Getting set up ===
# If you're still working on your [[morphological generator|generator]], finish that first.
# Commit and push your code at this point, and then '''add a tag''' "generator":
#* <code>git tag generator</code>
#* Then push the tag to the repo: <code>git push origin generator</code>
# Make a <code>Language/Disambiguation</code> page on the wiki, linked to from your main Language page, with a section titled something like "initial evaluation of ambiguity".
#* Add the page to the category [[:Category:Sp22_Disambiguation]] and the one for your language.
# Find the "xyz-tagger" section of your modes.xml file.  Change <code>cg-proc -w</code> to <code>cg-proc -1 -n -w</code> (but not in the "xyz-disam" section!).  Recompile, commit, push.
<!-- # If you haven't yet, add some punctuation to your transducer.
#* You can look at the Punctuation lexicon in an existing transducer, e.g. the [https://github.com/apertium/apertium-tuk/blob/master/apertium-tuk.tuk.lexc Turkmen lexc file].
#* Make sure to define your new multicharacter symbols (i.e., the new tags being used).
# Find the "xyz-tagger" section of your modes.xml file.  Change <code>cg-proc -w</code> to <code>cg-proc -1 -n -w</code> (but not in the "xyz-disam" section!).  Recompile, commit, push.
-->

=== Identify ambiguity ===
# Add some more of the most common stems identified by a coverage test.
# Measure the level of ambiguity in your corpus (see [[#Checking what forms are ambiguous in the corpus and transducer|above]]).
#* If ambiguity is 1 (i.e., there is no ambiguity), then find any form in the language that can have more than one analysis.  Check a grammar book and dictionary and think creatively.  For example, demonstratives are ambiguous in many languages between determiners and pronouns, colour and material terms can often be ambiguous between adjectives and nouns, etc.
#** Implement this into your transducer.
#** Recompile, and measure level of ambiguity again.
#* If ambiguity is greater than 1 (i.e., there are at least some ambiguous forms), then figure out what form is ambiguous.
# List the level of ambiguity into the "initial evaluation of ambiguity" section on the disambig eval wiki page.
# Find two sentences for two possible readings of '''any ambiguous term'''.  Make sure all the words are in your transducer with correct analyses.  Put them on the wiki page, highlighting the correct reading of the ambiguous form in each sentence.  For example, if the word was "this", examples might be:
#* <code>^This/this<det><dem><sg>/'''this<prn><dem><sg>'''$ ^is/be<v><iv><pres><p3><sg>$ ^a/a<det><ind>$ ^dog/dog<n><sg>$^./.<sent>$</code>
#* <code>^I/I<prn><pers><p1><sg><sbj>$ ^saw/see<v><tv><past>$ ^this/'''this<det><dem><sg>'''/this<prn><dem><sg>$ ^dog/dog<n><sg>$^./.<sent>$</code>
#* Note that it's best to use real sentences, e.g. from your corpus (or any of the sources that went into the corpus).  If you absolutely must, you may create examples, but make note of this!

=== Write constraints ===
# Once you have identified examples of ambiguity, come up with '''prose forms of the constraints''' you'll need to write to get the correct form in each of the two sentences.
#* Be as '''linguistically sound''' as you can, so that the constraints will apply in the broadest possible range of scenarios.
# Put the prose forms of the constraints as '''comments in your code'''.
# Using them as guidance, '''implement CG versions of the constraints'''.
#* Be sure to define any sets/lists you might need.

=== Finish up ===
# Once you get your disambiguator working as expected, commit your code
# Also, add a link to the git repository on the disambiguator wiki page, and vice versa (i.e., a link to the wiki page as a comment in the <code>.rlx</code> file in the git repo)
# Measure ambiguity again and add that number to the wiki page under something like "final evaluation of ambiguity".

== More resources ==
* '''[[:apertium:Constraint Grammar|Constraint Grammar]]''' on the Apertium wiki
* [[:apertium:Apertium and Constraint Grammar|Apertium and Constraint Grammar]] on the Apertium wiki
* [http://visl.sdu.dk/cg3/single/ Constraint Grammar manual]

[[Category:Assignments]]
[[Category:Tutorials]]
[[Category:Disambiguation]]