Dealing with multi-word expressions

There are a number of considerations and strategies when dealing with multi-word expressions (MWEs).

Monolingual dictionary

Imagine you wanted to have analyses for single tokens that include spaces, such as ^fur coats/fur coat<n><pl>$ or even just ^a lot/a lot<adv>. In lexd, you can simply add these to the appropriate lexicon, being sure to escape the space (i.e., \ ):

LEXICON NounStems

...
coat:coat             # "пальто"
fur:fur               # "жүн"
fur\ coat:fur\ coat   # "тон"
...

Any stems like spaces will act like any other stem, and get any morphology specified by the patterns that reference the NounStems lexicon.

Now imagine that you have examples like ^took away/take away<v><past>$, ^shows up/show up<v><pres><p3><sg>$. These are harder to implement in lexd, because if you used the above approach, you'd get verbal morphology at the end (on the second word) instead of on the first word.

You could technically make individual lexicons for each verb (take, show, throw, go, be, etc.) and each set of things each one patterns with (for take, a simplified list might include: after, apart, away, back, down, in, it, off, on, out, over, to, up). However, that would get really unwieldy really fast.

A better approach for this kind of MWE is to use apertium-separable.

monodix approach

While lexd fails at being able to handle this nicely, monodix format (transducers written in .dix format) offers a straightforward solution.

...

Bilingual dictionary

Use  for spaces in entries.

 <e><p><l>break<b/>a<b/>leg<s n="vblex"/></l>  <r>tener<b/>suerte<s n="vblex"/></r></p></e>

However, this doesn't fix the problem that the form on the right side needs to be implemented in the transducer in a way where the first elements is conjugated.

In combination with monodix approach

If you use the monodix approach discussed above, you can use <g/> for the space after tener instead of  to indicate that everything after that is fixed.

Apertium-separable

Apertium-separable allows you to treat MWEs as single entries in a bilingual dictionary, but as individual components at the monolingual level.

Some use cases follow.

MWEs as translations of a single concept to complex concepts

In the example of "fur coat", this is a single concept and word in some languages (Kyrgyz) but a complex concept using multiple words in others (English). Other common examples include "younger sister", "older sister", etc., or even just expressions like "a lot" in English which happen to take up two apparent words but could be thought of as one.

For these cases it is often easiest to include a multi-word entry in your monolingual dictionary that's just used for translation. However, it's also possible to use apertium-separable to separate these expressions out into two words when translating into the language where they're two words (e.g., "younger sister" in English), or to combine them into a single expression when translating from the language where they're two words. In this case, you would want the following:

separate words in the monolingual dictionary ("young" and "sister")
a single translation in the bilingual dictionary
<e><l>youngersister<s n="n"/></l> <r>сиңди<s n="n"/></r></e>

an entry in ….xyz-abc.lsx, where xyz is the language with the MWE:

   <e lm="younger sister" c="сиңди">
      <p>
         <l>young<s n="adj"/><s n="sint"/><s n="cmp"/><d/>sister<s n="n"/></l>
         <r>younger<b/>sister<s n="n"/></r>
      </p>
      <i><t/><d/></i>
   </e>

This will result in a translation chain like the following if translating from English to Kyrgyz (xyz-abc):

input text	analysis	retokenisation	lexical transfer	generation
younger sisters	`^young<adj><sint><cmp>$ ^sister<n><pl>$`	`^younger sister<n><pl>$`	`^сиңди<n><pl>$`	сиңдилер

And the following translation chain if translating from Kyrgyz to English (abc-xyz):

input text	analysis	lexical transfer	retokenisation	generation
сиңдилер	`^сиңди<n><pl>$`	`^younger sister<n><pl>$`	`^young<adj><sint><cmp>$ ^sister<n><pl>$`	younger sisters

MWEs where inflection applies differently than if it were a single element

In the case of "take away", "show up", "get over", etc., the main challenge is that the first element is inflected, and the second element isn't a simple set list that can combine with all (or even some) verbs.

...

MWEs where the parts can be separated

An additional challenge with English verbs like "take away" "throw out", "send up", etc. is that the final element doesn't always immediately follow the first. Often a noun phrase (which might also be a pronoun) intervenes between the elements. E.g., "takes the toy away", "threw it out", "sending several big weather balloons up".

Fortunately, apertium-separable was designed exactly for this sort of situation.

You will need (with example of "throw out" = "ташта"):

an entry in ….xyz-abc.lsx, where xyz is the language with the MWE:

                <e lm="throw out" c="ташта">
                        <p>
                                <l>throw<s n="vblex"/></l>
                                <r>throw<b/>out<s n="vblex"/></r>
                        </p>
                        <i><t/><d/></i>
                        <par n="SN"/>
                        <p>
                                <l>out<s n="adv"/><d/></l>
                                <r></r>
                        </p>
                </e>

the "SN" paradigm stands in for "noun phrase" (SN = "sintagma nominal")

Any paradigms will have to be defined in the <pardefs>...</pardefs> section of the file:

                <pardef n="SN">
                        <e><par n="prn"/></e>
                        <e><par n="n"/></e>
                        <e><par n="adj"/><par n="n"/></e>
                        <e><par n="adj"/><par n="adj"/><par n="n"/></e>
                        <e><par n="det"/><par n="n"/></e>
                        <e><par n="det"/><par n="adj"/><par n="n"/></e>
                        <e><par n="det"/><par n="adj"/><par n="adj"/><par n="n"/></e>
                </pardef>

In this case, the SN is defined as a pronoun, a noun, an adjective + a noun, two adjectives plus a noun, a determiner + a noun, a determiner + an adjective + a noun, or a determiner + an adjective + an adjective + a noun

Any paradigms used in these paradigms will also need to be defined, e.g.:
```
 <pardef n="adj">
 <e><w/><s n="adj"/><d/></e>
 <e><w/><s n="adj"/><t/><d/></e>
 </pardef>
```
In this case, the adj paradigm is defined as a lemma (<w/>) followed by an adjective tag followed by a word boundary (<d/>, or a lemma (<w/>) followed by an adjective tag followed by another tag (<t/>, such as comparative) followed by a word boundary (<d/>.
The tags will also have to be defined in the <sdefs>...</sdefs> section:
```
 <sdef n="adj"/>
```
A single translation in your bilingual dictionary:
<e><l>throwout<s n="vblex"/></l> <r>ташта<s n="v"/><s n="tv"/></r></e>
Your monolingual dictionary (for the MWE language) will have the various components as separate words
- In this case, "throw" will be a verb (throw<vblex>) and "out" will be an adverb (out<adv>).

This will result in a translation chain like the following if translating from English to Kyrgyz:

input text	analysis	retokenisation	lexical transfer	structural transfer	generation
throws the toys out	`^throw<vblex><pres><p3><sg>$ ^the<det><def><sp>$ ^toy<n><pl>$ ^out<adv>`	`^throw out<vblex><pres><p3><sg>$ ^the<det><def><sp>$ ^toy<n><pl>$`	`^ташта<v><tv><pres><p3><sg>$ ^$ ^оюнчук<n><pl>$`	`^оюнчук<n><pl><acc>$ ^ташта<v><tv><aor><p3><sg>$`	оюнчукту таштайт

And the following translation chain if translating from Kyrgyz to English:

input text	analysis	lexical transfer	structural transfer	retokenisation	generation
оюнчукту таштайт	`^оюнчук<n><pl><acc>$ ^ташта<v><tv><aor><p3><sg>$`	`^toy<n><pl><acc>$ ^throw out<vblex><aor><p3><sg>$`	`^throw out<vblex><pres><p3><sg>$ ^the<det><def><sp>$ ^toy<n><pl>$`	`^throw<vblex><pres><p3><sg>$ ^the<det><def><sp>$ ^toy<n><pl>$ ^out<adv>`	throws the toy out

Augmenting a pair to use apertium-separable

You will want:

Two vanilla lsx files:
- apertium-PAIR.xyz-abc.lsx
- apertium-PAIR.abc-xyz.lsx

Compilation rules in your Makefile.am:

Add to TARGETS_COMMON:

        $(PREFIX1).autoseq.bin  \
        $(PREFIX1).revautoseq.bin       \
        $(PREFIX2).autoseq.bin  \
        $(PREFIX2).revautoseq.bin \

Add further down somewhere:

###############################################################################
## Lexical sequences
###############################################################################

$(PREFIX1).autoseq.bin: $(BASENAME).$(PREFIX1).lsx
        lsx-comp lr $< $@

$(PREFIX2).autoseq.bin: $(BASENAME).$(PREFIX2).lsx
        lsx-comp lr $< $@

$(PREFIX1).revautoseq.bin: $(BASENAME).$(PREFIX1).lsx
        lsx-comp rl $< $@

$(PREFIX2).revautoseq.bin: $(BASENAME).$(PREFIX2).lsx
        lsx-comp rl $< $@

Add to your pipeline in modes.xml:

In your xyz-abc direction,

right before pretransfer add:

      <program name="lsx-proc">
        <file name="xyz-abc.autoseq.bin"/>
      </program>

right before autogen add:

      <program name="lsx-proc" debug-suff="revautoseq">
        <file name="abc-xyz.revautoseq.bin"/>
      </program>

In your abc-xyz direction,

right before pretransfer add:

      <program name="lsx-proc">
        <file name="abc-xyz.autoseq.bin"/>
      </program>

right before autogen add:

      <program name="lsx-proc" debug-suff="revautoseq">
        <file name="xyz-abc.revautoseq.bin"/>
      </program>

Dealing with multi-word expressions

Contents

Monolingual dictionary

monodix approach

Bilingual dictionary

In combination with monodix approach

Apertium-separable

MWEs as translations of a single concept to complex concepts

MWEs where inflection applies differently than if it were a single element

MWEs where the parts can be separated

Augmenting a pair to use apertium-separable

Navigation menu

Personal tools

Namespaces

Variants

Views

More

Search

Navigation

Tools