Difference between revisions of "Hfst-minimise trick"

Revision as of 18:03, 30 March 2017

There's a "trick" using hfst-minimise that will get around some issues in converting hfst-format transducers to lttoolbox-format transducers. That is, if hfst-proc xyz.automorf.hfst and lt-proc xyz.automorf.bin seem to be acting differently, try this trick.

The trick

In your Makefile.am file:

• In the rule for making $(LANG1).automorf.hfst, pipe the output of hfst-invert into hfst-minimise before piping the output of that into hfst-fst2fst. So the relevant part of this line should now look something like this:

  ...| hfst-invert | hfst-minimise | hfst-fst2fst -O -o $

• In the rule for $(LANG1).autogen.hfst, first run hfst-minimise on the input before piping that into hfst-fst2fst. That rule should now look something like this:

  hfst-minimise $< | hfst-fst2fst -O -o $@

• Run make clean, reinitialise your module (./autogen.sh), and run make to recompile.
• Test the output of lt-proc and hfst-proc again.

The reason

This happens because there are two fairly minor bugs in the software we use that no-one has got around to fixing yet (partly because there are workarounds):

• In hfst-proc there is a problem with longest-match left-to-right tokenisation, see the bug description here.
• In lt-comp (the ATT compiler) there is a problem with compiling entries where one of the sides begins with an epsilon from the start state but the other doesn't.