LING073 - User contributions [en]

Dependency syntax

2017-04-18T18:00:47Z

FTyers1-guest: /* Converting between conllu and CG formats */

Dependency syntax is a framework for syntax which considers relations between words to be "dependencies". It happens to be easy to deal with for various computational linguistics applications: any given token has a "head" (what it depends on) and a "relation" to that head. E.g., in "a house", a determiner/article depends on a noun (the noun is the head to the determiner), and the relation is one of <code>det</code> (depending on the annotation standard used).

[http://universaldependencies.org/ Universal Dependencies] (UD) is a project aimed at designing a standardised set of dependency syntax annotation principles for use across languages, with community-contributed language-specific annotation guidelines. It's become pretty popular, and recently released its 2nd version of annotation principles.

== Guidance ==
=== Annotation of a single sentence ===

You start with a sentence:
Everyone in this class works on a different language.

You run it through your disambiguator and manually disambiguate as needed so that each word has only one analysis (in CG) format:
<pre>
"<Everyone>"
"everyone" prn ind mf sg
"<in>"
"in" pr
"<this>"
"this" det dem sg
"<class>"
"class" n sg
"<works>"
"work" vblex pres p3 sg
"<on>"
"on" adv
"<a>"
"a" det ind sg
"<different>"
"different" adj
"<language>"
"language" n sg
"<.>"
"." sent
</pre>

Then you consider each word and what other word it depends on and how. E.g., "Everyone" has an <code>nsubj</code> dependency on "works" (i.e., it's its nominal subject). You then encode these relationships in the sentence:
<pre>
"<Everyone>"
"everyone" prn ind mf sg @nsubj #1->5
"<in>"
"in" pr @case #2->4
"<this>"
"this" det dem sg @det #3->4
"<class>"
"class" n sg @nmod #4->1
"<works>"
"work" vblex pres p3 sg @root #5->0
"<on>"
"on" adv @compound:prt #6->5
"<a>"
"a" det ind sg @det #7->9
"<different>"
"different" adj @amod #8->9
"<language>"
"language" n sg @obj #9->5
"<.>"
"." sent @punct #10->5
</pre>

When plotted as a tree, this sentence looks like the following:

[[File:UD tree example.png|800px|Dependency tree of "Everyone in this class works on a different language.]]

=== Training a model ===
'''[https://ufal.mff.cuni.cz/udpipe UDPipe]''' is a Free / Open Source tool (should be on the virtual machine) that can train a parser for (read: make a model for how to best guess) UD relations. It will use whatever information you give it (including lemmas, morphological tags, etc.), and can be trained to guess these as well.

The following command trains a UD parser on <code>corpus.conllu</code> and outputs the model to <code>abc.model.udpipe</code>:
cat corpus.conllu | udpipe --tagger=none --tokenizer=none --train abc.model.udpipe

=== Testing a model ===
The following command parses <code>corpus.conllu</code> using the previously trained <code>abc.model.udpipe</code>:
udpipe --parse abc.model.udpipe corpus.conllu

With the additional switch <code>--accuracy</code>, it will return measures of its accuracy:
udpipe --accuracy --parse abc.model.udpipe corpus.conllu

The two main metrics it returns are the following:
* Label Attachment Score ('''LAS'''): the percentage of tokens that the parser assigned the right dependency head ''and'' relation to.
* Unlabeled Attachment Score ('''UAS'''): the percentage of tokens that the parser assigned the right dependency head to.

=== Removing morphological information ===
In conllu format, column 3 is for lemmas, column 5 is for POS tags, and column 6 is for other morphological tags. All values in these columns need to be blanked to <code>_</code> to create a corpus with no morphological information.

The following command will output a corpus with columns 3, 5, and 6 replaced with <code>_</code>:
sed -r 's/[^\t]+/_/3 ; s/[^\t]+/_/5 ; s/[^\t]+/_/6' corpus.conllu

To output it to a new file, just use <code>></code>:
sed -r 's/[^\t]+/_/3 ; s/[^\t]+/_/5 ; s/[^\t]+/_/6' corpus.conllu > new.corpus.conllu

=== Converting between conllu and CG formats ===
Try the scripts available in Fran's [https://github.com/ftyers/ud-scripts ud-scripts] repo, in particular <code>vislcg3-to-conllu.py</code> and <code>conllu-to-vislcg.py</code>. (And let someone know and/or file a bug report if you notice any problems.)

To convert to conllu format from CG (where "prefix" is just an arbitrary code to prefix to sentence ids):
cat corpus.txt | vislcg3-to-conllu.py prefix > corpus.conllu

To convert to CG format from conllu:
cat corpus.conllu | conllu-to-vislcg.py > corpus.txt

The conversion may be lossy so don't expect a 1:1 roundtrip.

=== Useful links ===
* [[Dependency_syntax/examples|annotated examples from class]]
* Jonathan's [http://jonorthwash.github.io/visualise.html UD visualiser tool].
* Fran's [[:apertium:Udpipe|UDPipe]] notes.
* All 37 [http://universaldependencies.org/u/dep/index.html UD relations] (and documentation and examples of each).

== The assignment ==
This assignment is due before the Tuesday class of the last week of classes (this semester, '''11:20am on Tuesday, 25 April 2017''')

=== Prepare your corpora ===
# Copy your <code>abc.annotated.basic.txt</code> corpus (from the previous assignment) to <code>abc.annotated.ud.txt</code>, and '''hand-disambiguate''' it. In other words, remove all but the correct analysis for each form (i.e., the one that's right in that sentence). Add, commit, and push this corpus.
# Create a '''second corpus''' of at least '''500 characters''' (or 250 for syllabic writing systems), one sentence per line, and call it <code>abc.annotated2.raw.txt</code>. Run it through your tagger and convert to CG format as before. You can skip the "all and only all possible analyses" step that you did for precision and recall, and just correct it to have the correct analysis for the given sentence. Call this new file <code>abc.annotated2.ud.txt</code>, and add, commit, and push these two files.

=== Annotate your corpora for dependencies and post-process ===
# Add '''dependency annotation''' (heads and relations) to both <code>abc.annotated.ud.txt</code> and <code>abc.annotated2.ud.txt</code>. Commit and push.
# '''Convert these to conllu''' format, in files with <code>.conllu</code> extensions instead of <code>.txt</code> extensions. Commit and push.
# '''Remove all morphological information'''' from <code>abc.annotated.ud.conllu</code> and call it <code>abc.annotated.nomorph.conllu</code>. Add, commit, and push.

=== Train and evaluate a parser ===
# '''Train two parsers''': one on <code>abc.annotated.ud.conllu</code> and one on <code>abc.annotated.nomorph.conllu</code>. Call the models <code>abc.withmorph.udpipe</code> and <code>abc.nomorph.udpipe</code>, respectively. Put them in the <code>dev/</code> folder of your repository and commit them.
# Try '''parsing''' both <code>abc.annotated.ud.txt</code> and <code>abc.annotated2.ud.txt</code> with each of the models you made. Make note of the UAS and LAS scores.

=== Documentation ===
# Make a page on the wiki called <code>Language/Universal_Dependencies</code>, add it to the category [[:Category:sp17_UD|sp17_UD]] and link to it as a resource from your main wiki page for the language.
# Add an "Evaluation" section, and make a grid that includes the UAS and LAS scores from your four parsing attempts above, as well as the number of forms (and sentences) in each corpus.
# Add a "Dependency relations" section, and make a subsection for each of five dependency relations that you used at least twice in your annotation. For each relation, provide:
#* A description of the relation, noting various ways it might be used in the language,
#* Two examples of the relation from your corpus, preferably illustrating what you described.

[[Category:Assignments]]
[[Category:Tutorials]]
[[Category:Dependency syntax]]

Hfst-minimise trick

2017-03-30T18:03:51Z

FTyers1-guest:

There's a "trick" using <code>hfst-minimise</code> that will get around some issues in converting hfst-format transducers to lttoolbox-format transducers. That is, if <code>hfst-proc xyz.automorf.hfst</code> and <code>lt-proc xyz.automorf.bin</code> seem to be acting differently, try this trick.

== The trick ==
In your <code>Makefile.am</code> file:
* In the rule for making <code>$(LANG1).automorf.hfst</code>, pipe the output of <code>hfst-invert</code> into <code>hfst-minimise</code> before piping the output of that into <code>hfst-fst2fst</code>. So the relevant part of this line should now look something like this:
*: ...<code>| hfst-invert | hfst-minimise | hfst-fst2fst -O -o $</code>
* In the rule for <code>$(LANG1).autogen.hfst</code>, first run <code>hfst-minimise</code> on the input before piping that into <code>hfst-fst2fst</code>. That rule should now look something like this:
*: <code>hfst-minimise $< | hfst-fst2fst -O -o $@</code>
* Run <code>make clean</code>, reinitialise your module (<code>./autogen.sh</code>), and run <code>make</code> to recompile.
* Test the output of <code>lt-proc</code> and <code>hfst-proc</code> again.

== The reason ==

This happens because there are two fairly minor bugs in the software we use that no-one has got around to fixing yet (partly because there are workarounds):

* In <code>hfst-proc</code> there is a problem with longest-match left-to-right tokenisation, see the bug description [http://sourceforge.net/tracker/?func=detail&aid=3383731&group_id=224521&atid=1061990 here].
* In <code>lt-comp</code> (the ATT compiler) there is a problem with compiling entries where one of the sides begins with an epsilon from the start state but the other doesn't.

[[Category:Tutorials]]

Hfst-minimise trick

2017-03-30T17:59:55Z

FTyers1-guest:

Morphological disambiguator

2017-02-23T09:15:49Z

FTyers1-guest: clarifying behaviour

== Why we need disambiguation ==
Imagine you have a word that has two different tagsets, e.g.
^houses/house<n><pl>/house<v><tv><p3><sg>$
^this/this<det><dem><sg>/this<pron><dem><sg>$

Normally your analyser will randomly choose one of these. Now imagine you have a sentence where the wrong analysis is chosen (in this case, twice!):
^The/The<det><def>$ ^motel/motel<n><sg>$ ^houses/house<n><pl>$ ^this/this<pron><dem><sg>$ ^dog/dog<n><sg>$^./.<sent>$

The goal of a disambiguator is to choose the correct analysis based on the surrounding words.

== Using Constraint Grammar to disambiguate ==
Constraint Grammar (CG) is a formalism that can be used to formalise context sensitive constraints to select or remove analyses from the list of possible analyses.

=== The structure of a CG file ===

At the top you define delimiters. You can then add lists (which are kind of like sets in twol). After that you can have any number of constraints. You can also put lists and constraints in sections. In the original specification for constraint grammar, all the constraints in each section are applied simultaneously, but the sections are applied in order and lists are always global. In the version we use (VISL CG3), the constraints are applied in strict order within the sections.

=== Constraints ===

Think of constraints as a series of heuristics to whittle down the number of analyses until you have as little ambiguity as possible.

The most commonly used constraints are to either '''remove''' a reading under a certain condition or '''select''' a reading under a certain condition. The syntax is e.g., <code>REMOVE <target> [contextual tests] ;</code>.

Targets can be defined as list names—like <code>Nominals</code>—or individual tag names, in parentheses—e.g., <code>(n pl)</code>. This target identifies the reading to select or remove.

Contextual tests come together in ()s. All tests must match in order for the constraint to apply. Inside the parentheses, the first definition is a position: <code>0</code> means the current cohort, <code>-1</code> means the previous cohort, etc. The rest of the test is what should be matched. If you want to match any member of the <code>Nominals</code> set in the previous position, it would be <code>(-1 Nominals)</code> You can match tag sequences by adding an extra set of parentheses, e.g. <code>(1 (n pl))</code> would match a plural noun in the following position. You can match specific baseforms with quotation marks, e.g. <code>(1 ("house" n))</code> would match any form of the noun "house" in the following position.

'''Classroom activity''': implement example at top of page.

=== Useful commands ===

The file that your disambiguation constraints are defined in is the <code>apertium-xyz.xyz.rlx</code> file.

==== Getting output before disambiguation ====
example command:
echo "The motel houses this dog" | apertium -d . eng-morph
example output:
^The/the<det><def>$ ^motel/motel<n><sg>$ ^houses/house<n><pl>/house<v><tv><p3><sg>$ ^this/this<det><dem><sg>/this<prn><dem><sg>$ ^dog/dog<n><sg>$^./.<sent>$

==== Getting output after disambiguation ====
example command:
echo "The motel houses this dog" | apertium -d . eng-tagger
example output:
^The<det><def>$ ^motel<n><sg>$ ^house<v><tv><p3><sg>$ ^this<det><dem><sg>$ ^dog<n><sg>$^.<sent>$

==== Seeing which constraints are doing what ====
example command:
echo "The motel houses this dog" | apertium -d . eng-disam
example output:
<pre>
"<The>"
"the" det def
"<motel>"
"motel" n sg
"<houses>"
"house" v tv p3 sg
; "house" n pl REMOVE:26
"<this>"
"this" det dem sg
; "this" prn dem sg REMOVE:29
"<dog>"
"dog" n sg
"<.>"
"." sent
</pre>

==== Calculating ambiguity ====
The ambiguity of a corpus is the average number of analyses provided per token.
You can get the number of tokens like this:
cat corpus.txt | lt-proc /path/to/xyz.automorf.bin | sed 's/$\W*\^/$\n^/g' | wc -l
You can get the number of analyses in your corpus like this:
cat corpus.txt | lt-proc /path/to/xyz.automorf.bin | sed 's/$\W*\^/$\n^/g' | cut -f2- -d'/' | sed 's/\//\n/g' | wc -l

Automating it a little further, you can put something like this in a file and run it as a shell script:
<pre>
CORPUS=/path/to/corpus.txt
BIN=/path/to/xyz.automorf.bin
TOKENS=`cat ${CORPUS} | lt-proc ${BIN} | sed 's/$\W*\^/$\n^/g' | wc -l`
ANALYSES=`cat ${CORPUS} | lt-proc ${BIN} | sed 's/$\W*\^/$\n^/g' | cut -f2- -d'/' | sed 's/\//\n/g' | wc -l`
calc $ANALYSES/$TOKENS
</pre>

== The assignment ==
=== Getting set up ===
# If you're still working on your [[morphological generator|generator]], finish that first.
# Commit and push your code at this point, and then '''add a tag''' "analyser":
#* <code>git tag generator</code>
#* Then push the tag to the repo: <code>git push origin generator</code>
# Make a <code>Language/Disambiguation</code> page on the wiki, linked to from your main Language page, with a section titled something like "initial evaluation of ambiguity".
#* Add the page to the category [[:Category:Sp17_Disambiguation]].
# If you haven't yet, add some punctuation to your transducer.
#* You can look at the Punctuation lexicon an existing the [http://svn.code.sf.net/p/apertium/svn/languages/apertium-tuk/apertium-tuk.tuk.lexc Turkmen lexc file].
#* Make sure to define your new multicharacter symbols (i.e., the new tags being used).
# Find the "xyz-tagger" section of your modes.xml file. Change <code>cg-proc -w</code> to <code>cg-proc -1 -n -w</code> (but not in the "xyz-disam" section!). Recompile, commit, push.

=== Identify ambiguity ===
# Add some more of the most common stems identified by a coverage test.
# Measure the level of ambiguity in your corpus.
#* If ambiguity is 1 (i.e., there is no ambiguity), then find any form in the language that can have more than one analysis. Check a grammar book and dictionary and think creatively.
#** Implement this into your transducer.
#** Recompile, and measure level of ambiguity again.
#* If ambiguity is greater than 1 (i.e., there are at least some ambiguous forms), then figure out what form is ambiguous.
# List the level of ambiguity into the "initial evaluation of ambiguity" section on the disambig eval wiki page.
# Find two sentences for two possible readings of the ambiguous term, highlighting the correct reading of the ambiguous form. For example, if the word was "this", examples might be:
#* <code>^This/this<det><dem><sg>/'''this<prn><dem><sg>'''$ ^is/be<v><iv><pres><p3><sg>$ ^a/a<det><ind>$ ^dog/dog<n><sg>$^./.<sent>$</code>
#* <code>^I/I<prn><pers><p1><sg><sbj>$ ^saw/see<v><tv><past>$ ^this/'''this<det><dem><sg>'''/this<prn><dem><sg>$ ^dog/dog<n><sg>$^./.<sent>$</code>
#* Note that it's best to use real sentences, e.g. from your corpus (or any of the sources that went into the corpus). If you absolutely must, you may create examples, but make note of this!

=== Write constraints ===
# Once you have identified examples of ambiguity, come up with '''prose forms of the constraints''' you'll need to write to get the correct form in each of the two sentences.
#* Be as '''linguistically sound''' as you can, so that the constraints will apply in the broadest possible range of scenarios.
# Using the prose forms of the constraints as guidance (and e.g., providing them as comments in the code), '''implement CG versions of the constraints'''.
#* Be sure to define any sets/lists you might need.

=== Finish up ===
# Once you get your disambiguator working as exapected, commit your code
# Measure ambiguity again and add that number to the wiki page.

== More resources ==
* [[:apertium:Constraint Grammar|Constraint Grammar]] on the Apertium wiki
* [[:apertium:Apertium and Constraint Grammar|Apertium and Constraint Grammar]] on the Apertium wiki
* [http://visl.sdu.dk/cg3/single/ Constraint Grammar manual]

[[Category:Assignments]]
[[Category:Tutorials]]
[[Category:Disambiguation]]