View source for Hfst-minimise trick

There's a "trick" using <code>hfst-minimise</code> that will get around some issues in converting hfst-format transducers to lttoolbox-format transducers.  That is, if <code>hfst-proc xyz.automorf.hfst</code> and <code>lt-proc xyz.automorf.bin</code> seem to be acting differently, try this trick.

== The trick ==
In your <code>Makefile.am</code> file:
* In the rule for making <code>$(LANG1).automorf.hfst</code>, pipe the output of <code>hfst-invert</code> into <code>hfst-minimise</code> before piping the output of that into <code>hfst-fst2fst</code>.  So the relevant part of this line should now look something like this:
*: ...<code>| hfst-invert | hfst-minimise | hfst-fst2fst -O -o $</code>
* In the rule for <code>$(LANG1).autogen.hfst</code>, first run <code>hfst-minimise</code> on the input before piping that into <code>hfst-fst2fst</code>.  That rule should now look something like this:
*: <code>hfst-minimise $< | hfst-fst2fst -O -o $@</code>
* Run <code>make clean</code>, reinitialise your module (<code>./autogen.sh</code>), and run <code>make</code> to recompile.
* Test the output of <code>lt-proc</code> and <code>hfst-proc</code> again.

The relevant section should end up looking something like this (modulo any spellrelax or twoc elements):
<pre>
$(LANG1).autogen.hfst: .deps/$(LANG1).RL.hfst
	hfst-minimise $< | hfst-fst2fst -O -o $@

$(LANG1).automorf.hfst: .deps/$(LANG1).LR.hfst
	hfst-invert $< | hfst-minimise | hfst-fst2fst -O -o $@

$(LANG1).autogen.att.gz: $(LANG1).autogen.hfst
	hfst-fst2txt $< | gzip -9 -c -n > $@

$(LANG1).automorf.att.gz: $(LANG1).automorf.hfst
	hfst-fst2txt $< | gzip -9 -c -n > $@
</pre>


== The reason ==

This happens because there are two fairly minor bugs in the software we use that no-one has got around to fixing yet (partly because there are workarounds):

* In <code>hfst-proc</code> there is a problem with longest-match left-to-right tokenisation, see the bug description [http://sourceforge.net/tracker/?func=detail&aid=3383731&group_id=224521&atid=1061990 here].  For this reason we convert the compiled hfst transducer to lttoolbox format, which doesn't have this bug in processing the transducer.
* In <code>lt-comp</code> (the ATT compiler) there is a problem with compiling entries where one of the sides begins with an epsilon from the start state but the other doesn't.  One work-around is the hfst-minimise trick; another is to just make sure none of your start states are empty on one side and not on the other.


[[Category:Tutorials]]