Difference between revisions of "Morphological disambiguator"
(→Constraints) |
(→Useful commands) |
||
| Line 32: | Line 32: | ||
==== Getting output before disambiguation ==== | ==== Getting output before disambiguation ==== | ||
| + | example command: | ||
| + | echo "The motel houses this dog" | apertium -d . eng-morph | ||
| + | example output: | ||
| + | ^The/the<det><def>$ ^motel/motel<n><sg>$ ^houses/house<n><pl>/house<v><tv><p3><sg>$ ^this/this<det><dem><sg>/this<prn><dem><sg>$ ^dog/dog<n><sg>$^./.<sent>$ | ||
==== Getting output after disambiguation ==== | ==== Getting output after disambiguation ==== | ||
| + | example command: | ||
| + | echo "The motel houses this dog" | apertium -d . eng-tagger | ||
| + | example output: | ||
| + | ^The<det><def>$ ^motel<n><sg>$ ^house<v><tv><p3><sg>$ ^this<det><dem><sg>$ ^dog<n><sg>$^.<sent>$ | ||
| + | |||
| + | ==== Seeing which constraints are doing what ==== | ||
| + | example command: | ||
| + | echo "The motel houses this dog" | apertium -d . eng-disam | ||
| + | example output: | ||
| + | <pre> | ||
| + | "<The>" | ||
| + | "the" det def | ||
| + | "<motel>" | ||
| + | "motel" n sg | ||
| + | "<houses>" | ||
| + | "house" v tv p3 sg | ||
| + | ; "house" n pl REMOVE:26 | ||
| + | "<this>" | ||
| + | "this" det dem sg | ||
| + | ; "this" prn dem sg REMOVE:29 | ||
| + | "<dog>" | ||
| + | "dog" n sg | ||
| + | "<.>" | ||
| + | "." sent | ||
| + | </pre> | ||
| + | |||
==== Calculating ambiguity ==== | ==== Calculating ambiguity ==== | ||
| + | The ambiguity of a corpus is the average number of analyses provided per token. | ||
| + | You can get the number of tokens like this: | ||
| + | cat corpus.txt | lt-proc /path/to/xyz.automorf.bin | sed 's/$\W*\^/$\n^/g' | wc -l | ||
| + | You can get the number of analyses in your corpus like this: | ||
| + | cat corpus.txt | lt-proc /path/to/xyz.automorf.bin | sed 's/$\W*\^/$\n^/g' | cut -f2- -d'/' | sed 's/\//\n/g' | wc -l | ||
| − | ==== | + | Automating it a little further, you can put something like this in a file and run it as a shell script: |
| + | <pre> | ||
| + | CORPUS=/path/to/corpus.txt | ||
| + | BIN=/path/to/xyz.automorf.bin | ||
| + | TOKENS=`cat ${CORPUS} | lt-proc ${BIN} | sed 's/$\W*\^/$\n^/g' | wc -l` | ||
| + | ANALYSES=`cat ${CORPUS} | lt-proc ${BIN} | sed 's/$\W*\^/$\n^/g' | cut -f2- -d'/' | sed 's/\//\n/g' | wc -l` | ||
| + | calc $ANALYSES/$TOKENS | ||
| + | </pre> | ||
== The assignment == | == The assignment == | ||
Revision as of 07:46, 23 February 2017
Contents
Why we need disambiguation
Imagine you have a word that has two different tagsets, e.g.
^houses/house<n><pl>/house<v><tv><p3><sg>$ ^this/this<det><dem><sg>/this<pron><dem><sg>$
Normally your analyser will randomly choose one of these. Now imagine you have a sentence where the wrong analysis is chosen (in this case, twice!):
^The/The<det><def>$ ^motel/motel<n><sg>$ ^houses/house<n><pl>$ ^this/this<pron><dem><sg>$ ^dog/dog<n><sg>$^./.<sent>$
The goal of a disambiguator is to choose the correct analysis based on the surrounding words.
Using Constraint Grammar to disambiguate
Constraint Grammar (CG) is a formalism that can be used to formalise context sensitive constraints to select or remove analyses from the list of possible analyses.
The structure of a CG file
At the top you define delimiters. You can then add lists (which are kind of like sets in twol). After that you can have any number of constraints. You can also put lists and constraints in sections: all the constraints in each section are applied simultaneously, but the sections are applied in order and lists are always global.
Constraints
Think of constraints as a series of heuristics to whittle down the number of analyses until you have as little ambiguity as possible.
The most commonly used constraints are to either remove a reading under a certain condition or select a reading under a certain condition. The syntax is e.g., REMOVE <target> [contextual tests] ;.
Targets can be defined as list names—like Nominals—or individual tag names, in parentheses—e.g., (n pl). This target identifies the reading to select or remove.
Contextual tests come together in ()s. All tests must match in order for the constraint to apply. Inside the parentheses, the first definition is a position: 0 means the current cohort, -1 means the previous cohort, etc. The rest of the test is what should be matched. If you want to match any member of the Nominals set in the previous position, it would be (-1 Nominals) You can match tag sequences by adding an extra set of parentheses, e.g. (1 (n pl)) would match a plural noun in the following position. You can match specific baseforms with quotation marks, e.g. (1 ("house" n)) would match any form of the noun "house" in the following position.
Example above
Useful commands
Getting output before disambiguation
example command:
echo "The motel houses this dog" | apertium -d . eng-morph
example output:
^The/the<det><def>$ ^motel/motel<n><sg>$ ^houses/house<n><pl>/house<v><tv><p3><sg>$ ^this/this<det><dem><sg>/this<prn><dem><sg>$ ^dog/dog<n><sg>$^./.<sent>$
Getting output after disambiguation
example command:
echo "The motel houses this dog" | apertium -d . eng-tagger
example output:
^The<det><def>$ ^motel<n><sg>$ ^house<v><tv><p3><sg>$ ^this<det><dem><sg>$ ^dog<n><sg>$^.<sent>$
Seeing which constraints are doing what
example command:
echo "The motel houses this dog" | apertium -d . eng-disam
example output:
"<The>" "the" det def "<motel>" "motel" n sg "<houses>" "house" v tv p3 sg ; "house" n pl REMOVE:26 "<this>" "this" det dem sg ; "this" prn dem sg REMOVE:29 "<dog>" "dog" n sg "<.>" "." sent
Calculating ambiguity
The ambiguity of a corpus is the average number of analyses provided per token. You can get the number of tokens like this:
cat corpus.txt | lt-proc /path/to/xyz.automorf.bin | sed 's/$\W*\^/$\n^/g' | wc -l
You can get the number of analyses in your corpus like this:
cat corpus.txt | lt-proc /path/to/xyz.automorf.bin | sed 's/$\W*\^/$\n^/g' | cut -f2- -d'/' | sed 's/\//\n/g' | wc -l
Automating it a little further, you can put something like this in a file and run it as a shell script:
CORPUS=/path/to/corpus.txt
BIN=/path/to/xyz.automorf.bin
TOKENS=`cat ${CORPUS} | lt-proc ${BIN} | sed 's/$\W*\^/$\n^/g' | wc -l`
ANALYSES=`cat ${CORPUS} | lt-proc ${BIN} | sed 's/$\W*\^/$\n^/g' | cut -f2- -d'/' | sed 's/\//\n/g' | wc -l`
calc $ANALYSES/$TOKENS
The assignment
- Add words
- Measure ambiguity
- Identify ambiguous form
- Write rule
- Measure ambiguity
More resources
- Constraint Grammar on the Apertium wiki
- Apertium and Constraint Grammar on the Apertium wiki
