Hfst-minimise trick

There's a "trick" using hfst-minimise that will get around some issues in converting hfst-format transducers to lttoolbox-format transducers. That is, if hfst-proc xyz.automorf.hfst and lt-proc xyz.automorf.bin seem to be acting differently, try this trick.

The trick

In your Makefile.am file:

• In the rule for making $(LANG1).automorf.hfst, pipe the output of hfst-invert into hfst-minimise before piping the output of that into hfst-fst2fst. So the relevant part of this line should now look something like this:

  ...| hfst-invert | hfst-minimise | hfst-fst2fst -O -o $

• In the rule for $(LANG1).autogen.hfst, first run hfst-minimise on the input before piping that into hfst-fst2fst. That rule should now look something like this:

  hfst-minimise $< | hfst-fst2fst -O -o $@

• Run make clean, reinitialise your module (./autogen.sh), and run make to recompile.
• Test the output of lt-proc and hfst-proc again.

The reason

This happens because there are two fairly minor bugs in the software we use that no-one has got around to fixing yet (partly because there are workarounds):

• In hfst-proc there is a problem with longest-match left-to-right tokenisation, see the bug description here. For this reason we convert the compiled hfst transducer to lttoolbox format, which doesn't have this bug in processing the transducer.
• In lt-comp (the ATT compiler) there is a problem with compiling entries where one of the sides begins with an epsilon from the start state but the other doesn't. One work-around is the hfst-minimise trick; another is to just make sure none of your start states are empty on one side and not on the other.