View source for Dealing with multi-word expressions

There are a number of considerations and strategies when dealing with multi-word expressions (MWEs).

== Monolingual dictionary ==

Imagine you wanted to have analyses for single tokens that include spaces, such as <code>^fur coats/fur coat{{tag|n}}{{tag|pl}}$</code> or even just <code>^a lot/a lot{{tag|adv}}</code>.  In <tt>lexd</tt>, you can simply add these to the appropriate lexicon, being sure to escape the space (i.e., <code>\ </code>):
<pre>
LEXICON NounStems

...
coat:coat             # "пальто"
fur:fur               # "жүн"
fur\ coat:fur\ coat   # "тон"
...
</pre>

Any stems like spaces will act like any other stem, and get any morphology specified by the patterns that reference the <code>NounStems</code> lexicon.

Now imagine that you have examples like <code>^took away/take away{{tag|v}}{{tag|past}}$</code>, <code>^shows up/show up{{tag|v}}{{tag|pres}}{{tag|p3}}{{tag|sg}}$</code>.  These are harder to implement in <tt>lexd</tt>, because if you used the above approach, you'd get verbal morphology at the end (on the second word) instead of on the first word.

You could technically make individual lexicons for each verb (take, show, throw, go, be, etc.) and each set of things each one patterns with (for take, a simplified list might include: after, apart, away, back, down, in, it, off, on, out, over, to, up).  However, that would get really unwieldy really fast.

A better approach for this kind of MWE is to use [[#Apertium-separable|apertium-separable]].

=== monodix approach ===
While <tt>lexd</tt> fails at being able to handle this nicely, <tt>monodix</tt> format (transducers written in <code>.dix</code> format) offers a straightforward solution.

...

== Bilingual dictionary ==

Use {{tag|b/}} for spaces in entries.

<pre>
 <e><p><l>break<b/>a<b/>leg<s n="vblex"/></l>  <r>tener<b/>suerte<s n="vblex"/></r></p></e>
</pre>

However, this doesn't fix the problem that the form on the right side needs to be implemented in the transducer in a way where the first elements is conjugated.

=== In combination with monodix approach ===
If you use the [[#monodix approach|monodix approach]] discussed above, you can use {{tag|g/}} for the space after <code>tener</code> instead of {{tag|b/}} to indicate that everything after that is fixed.

== Apertium-separable ==

Apertium-separable allows you to treat MWEs as single entries in a bilingual dictionary, but as individual components at the monolingual level.

Some use cases follow.

=== MWEs as translations of a single concept to complex concepts ===
In the example of "fur coat", this is a single concept and word in some languages (Kyrgyz) but a complex concept using multiple words in others (English).  Other common examples include "younger sister", "older sister", etc., or even just expressions like "a lot" in English which happen to take up two apparent words but could be thought of as one.

For these cases it is often easiest to include a multi-word entry in your monolingual dictionary that's just used for translation.  However, it's also possible to use <tt>apertium-separable</tt> to separate these expressions out into two words when translating ''into'' the language where they're two words (e.g., "younger sister" in English), or to combine them into a single expression when translating ''from'' the language where they're two words.  In this case, you would want the following:
# separate words in the monolingual dictionary ("young" and "sister")
# a single translation in the bilingual dictionary
#: <code>{{tag|e}}{{tag|p}}{{tag|l}}younger{{tag|b/}}sister&lt;s n="n"/&gt;{{tag|/l}}  {{tag|r}}сиңди&lt;s n="n"/&gt;{{tag|/r}}{{tag|/p}}{{tag|/e}}</code>
# an entry in <code>….{{rainbow|xyz}}-{{rainbow|abc}}.lsx</code>, where <code>{{rainbow|xyz}}</code> is the language with the MWE:
#: <pre>   <e lm="younger sister" c="сиңди">&#10;      <p>&#10;         <l>young<s n="adj"/><s n="sint"/><s n="cmp"/><d/>sister<s n="n"/></l>&#10;         <r>younger<b/>sister<s n="n"/></r>&#10;      </p>&#10;      <i><t/><d/></i>&#10;   </e></pre>

This will result in a translation chain like the following if translating from English to Kyrgyz ({{rainbow|xyz}}-{{rainbow|abc}}):
{|class="wikitable"
! input text !! analysis !! retokenisation !! lexical transfer !! generation
|-
| younger sisters || <code>^young{{tag|adj}}{{tag|sint}}{{tag|cmp}}$ ^sister{{tag|n}}{{tag|pl}}$</code> || <code>^younger sister{{tag|n}}{{tag|pl}}$</code> || <code>^сиңди{{tag|n}}{{tag|pl}}$</code> || сиңдилер
|}

And the following translation chain if translating from Kyrgyz to English ({{rainbow|abc}}-{{rainbow|xyz}}):
{|class="wikitable"
! input text !! analysis !! lexical transfer !! retokenisation !! generation
|-
| сиңдилер || <code>^сиңди{{tag|n}}{{tag|pl}}$</code> || <code>^younger sister{{tag|n}}{{tag|pl}}$</code> || <code>^young{{tag|adj}}{{tag|sint}}{{tag|cmp}}$ ^sister{{tag|n}}{{tag|pl}}$</code> || younger sisters
|}

=== MWEs where inflection applies differently than if it were a single element ===
In the case of "take away", "show up", "get over", etc., the main challenge is that the first element is inflected, and the second element isn't a simple set list that can combine with all (or even some) verbs.

...

=== MWEs where the parts can be separated ===
An additional challenge with English verbs like "take away" "throw out", "send up", etc. is that the final element doesn't always immediately follow the first.  Often a noun phrase (which might also be a pronoun) intervenes between the elements.  E.g., "takes the toy away", "threw it out", "sending several big weather balloons up".

Fortunately, apertium-separable was designed exactly for this sort of situation.

You will need (with example of "throw out" = "ташта"):
# an entry in <code>….{{rainbow|xyz}}-{{rainbow|abc}}.lsx</code>, where <code>{{rainbow|xyz}}</code> is the language with the MWE:
#: <pre>                <e lm="throw out" c="ташта">&#10;                        <p>&#10;                                <l>throw<s n="vblex"/></l>&#10;                                <r>throw<b/>out<s n="vblex"/></r>&#10;                        </p>&#10;                        <i><t/><d/></i>&#10;                        <par n="SN"/>&#10;                        <p>&#10;                                <l>out<s n="adv"/><d/></l>&#10;                                <r></r>&#10;                        </p>&#10;                </e>&#10;</pre>
#: the "<code>SN</code>" paradigm stands in for "noun phrase" (SN = "sintagma nominal")
# Any paradigms will have to be defined in the <code>{{tag|pardefs}}...{{tag|/pardefs}}</code> section of the file:
#: <pre>                <pardef n="SN">&#10;                        <e><par n="prn"/></e>&#10;                        <e><par n="n"/></e>&#10;                        <e><par n="adj"/><par n="n"/></e>&#10;                        <e><par n="adj"/><par n="adj"/><par n="n"/></e>&#10;                        <e><par n="det"/><par n="n"/></e>&#10;                        <e><par n="det"/><par n="adj"/><par n="n"/></e>&#10;                        <e><par n="det"/><par n="adj"/><par n="adj"/><par n="n"/></e>&#10;                </pardef></pre>
#: In this case, the <code>SN</code> is defined as a pronoun, a noun, an adjective + a noun, two adjectives plus a noun, a determiner + a noun, a determiner + an adjective + a noun, or a determiner + an adjective + an adjective + a noun
# Any paradigms used in these paradigms will also need to be defined, e.g.:<pre>                <pardef n="adj">&#10;                        <e><i><w/><s n="adj"/><d/></i></e>&#10;                        <e><i><w/><s n="adj"/><t/><d/></i></e>&#10;                </pardef></pre>
#: In this case, the <code>adj</code> paradigm is defined as a lemma ({{tag|w/}}) followed by an adjective tag followed by a word boundary ({{tag|d/}}, or a lemma ({{tag|w/}}) followed by an adjective tag followed by another tag ({{tag|t/}}, such as comparative) followed by a word boundary ({{tag|d/}}.
# The tags will also have to be defined in the <code>{{tag|sdefs}}...{{tag|/sdefs}}</code> section:<pre>                <sdef n="adj"/></pre>
# A single translation in your bilingual dictionary:
#: <code>{{tag|e}}{{tag|p}}{{tag|l}}throw{{tag|b/}}out&lt;s n="vblex"/&gt;{{tag|/l}}  {{tag|r}}ташта&lt;s n="v"/&gt;&lt;s n="tv"/&gt;{{tag|/r}}{{tag|/p}}{{tag|/e}}</code>
# Your monolingual dictionary (for the MWE language) will have the various components as separate words
#* In this case, "throw" will be a verb (<code>throw{{tag|vblex}}</code>) and "out" will be an adverb (<code>out{{tag|adv}}</code>).

This will result in a translation chain like the following if translating from English to Kyrgyz:
{|class="wikitable"
! input text !! analysis !! retokenisation !! lexical transfer !! structural transfer !! generation
|-
| throws the toys out || <code>^throw{{tag|vblex}}{{tag|pres}}{{tag|p3}}{{tag|sg}}$ ^the{{tag|det}}{{tag|def}}{{tag|sp}}$ ^toy{{tag|n}}{{tag|pl}}$ ^out{{tag|adv}}</code> || <code>^throw out{{tag|vblex}}{{tag|pres}}{{tag|p3}}{{tag|sg}}$ ^the{{tag|det}}{{tag|def}}{{tag|sp}}$ ^toy{{tag|n}}{{tag|pl}}$</code> || <code>^ташта{{tag|v}}{{tag|tv}}{{tag|pres}}{{tag|p3}}{{tag|sg}}$ ^$ ^оюнчук{{tag|n}}{{tag|pl}}$</code> || <code>^оюнчук{{tag|n}}{{tag|pl}}{{tag|acc}}$ ^ташта{{tag|v}}{{tag|tv}}{{tag|aor}}{{tag|p3}}{{tag|sg}}$</code> || оюнчукту таштайт
|}

And the following translation chain if translating from Kyrgyz to English:
{|class="wikitable"
! input text !! analysis !! lexical transfer !! structural transfer !! retokenisation !! generation
|-
| оюнчукту таштайт || <code>^оюнчук{{tag|n}}{{tag|pl}}{{tag|acc}}$ ^ташта{{tag|v}}{{tag|tv}}{{tag|aor}}{{tag|p3}}{{tag|sg}}$</code> || <code>^toy{{tag|n}}{{tag|pl}}{{tag|acc}}$ ^throw out{{tag|vblex}}{{tag|aor}}{{tag|p3}}{{tag|sg}}$</code> || <code>^throw out{{tag|vblex}}{{tag|pres}}{{tag|p3}}{{tag|sg}}$ ^the{{tag|det}}{{tag|def}}{{tag|sp}}$ ^toy{{tag|n}}{{tag|pl}}$</code> || <code>^throw{{tag|vblex}}{{tag|pres}}{{tag|p3}}{{tag|sg}}$ ^the{{tag|det}}{{tag|def}}{{tag|sp}}$ ^toy{{tag|n}}{{tag|pl}}$ ^out{{tag|adv}}</code> || throws the toy out
|}

=== Augmenting a pair to use apertium-separable ===
You will want:
* Two vanilla lsx files: 
** <code>apertium-{{rainbow|PAIR}}.{{rainbow|xyz}}-{{rainbow|abc}}.lsx</code>
** <code>apertium-{{rainbow|PAIR}}.{{rainbow|abc}}-{{rainbow|xyz}}.lsx</code>
* Compilation rules in your <code>Makefile.am</code>:
*# Add to <code>TARGETS_COMMON</code>:
*#: <pre>        $(PREFIX1).autoseq.bin  \&#10;        $(PREFIX1).revautoseq.bin       \&#10;        $(PREFIX2).autoseq.bin  \&#10;        $(PREFIX2).revautoseq.bin \</pre>
*# Add further down somewhere:
*#: <pre>###############################################################################&#10;## Lexical sequences&#10;###############################################################################&#10;&#10;$(PREFIX1).autoseq.bin: $(BASENAME).$(PREFIX1).lsx&#10;        lsx-comp lr $< $@&#10;&#10;$(PREFIX2).autoseq.bin: $(BASENAME).$(PREFIX2).lsx&#10;        lsx-comp lr $< $@&#10;&#10;$(PREFIX1).revautoseq.bin: $(BASENAME).$(PREFIX1).lsx&#10;        lsx-comp rl $< $@&#10;&#10;$(PREFIX2).revautoseq.bin: $(BASENAME).$(PREFIX2).lsx&#10;        lsx-comp rl $< $@</pre>
* Add to your pipeline in <code>modes.xml</code>:
** In your {{rainbow|xyz}}-{{rainbow|abc}} direction,
**: right before pretransfer add: <pre>      <program name="lsx-proc">&#10;        <file name="xyz-abc.autoseq.bin"/>&#10;      </program></pre>
**: right before autogen add: <pre>      <program name="lsx-proc" debug-suff="revautoseq">&#10;        <file name="abc-xyz.revautoseq.bin"/>&#10;      </program></pre>
** In your {{rainbow|abc}}-{{rainbow|xyz}} direction,
**: right before pretransfer add: <pre>      <program name="lsx-proc">&#10;        <file name="abc-xyz.autoseq.bin"/>&#10;      </program></pre>
**: right before autogen add: <pre>      <program name="lsx-proc" debug-suff="revautoseq">&#10;        <file name="xyz-abc.revautoseq.bin"/>&#10;      </program></pre>