Difference between revisions of "Using flag diacritics for faster compilation"

From LING073
Jump to: navigation, search
(→‎Running tests)
(→‎Running tests)
 
(8 intermediate revisions by the same user not shown)
Line 1: Line 1:
−
`lexd` is a bit slow to compile dictionaries that make heavy use of filters.
+
<code>lexd</code> is a bit slow to compile dictionaries that make use of prefixes with large lexicons (or similar).
  
 
To speed this up, you can enable compiling to flag diacritics.
 
To speed this up, you can enable compiling to flag diacritics.
  
 
== Implementing the change ==
 
== Implementing the change ==
−
First thing you want to do is edit your `Makefile.am`:
+
First thing you want to do is edit your <code>Makefile.am</code>:
−
* remove `autogen.bin` and `automorf.bin` lines from the <code>TARGETS_COMMON</code> list at the top
+
* remove <code>autogen.bin</code> and <code>automorf.bin</code> lines from the <code>TARGETS_COMMON</code> list at the top
−
* add `-f` as a switch to the `lexd` commmands
+
* add <code>-f</code> as a switch to the <code>lexd</code> commmands
−
* add `-F` as a switch to the `hfst-compose` command in the the `automorf.hfst` line
+
* add <code>-F</code> as a switch to the <code>hfst-compose</code> command in the the <code>automorf.hfst</code> line
  
−
This will all look something like this:
+
Be sure to rerun <code>./autogen.sh</code> when you're done.
 +
 
 +
Also make sure to do all of this the same way in your multilingual pair.
 +
 
 +
The changes will look something like this:
 
<pre>
 
<pre>
 
  TARGETS_COMMON =                      \
 
  TARGETS_COMMON =                      \
Line 43: Line 47:
 
=== Updating modes.xml ===
 
=== Updating modes.xml ===
 
You'll want to change all the instances of <code>lt-proc</code> and <code>auto*.bin</code> in your <code>modes.xml</code> file to <code>hfst-proc</code> and <code>auto*.hfst</code>, respectively.
 
You'll want to change all the instances of <code>lt-proc</code> and <code>auto*.bin</code> in your <code>modes.xml</code> file to <code>hfst-proc</code> and <code>auto*.hfst</code>, respectively.
 +
 +
Make sure to do the same in your multilingual pair.
  
 
=== Running hfst-expand and hfst-lookup ===
 
=== Running hfst-expand and hfst-lookup ===
−
You'll also need to add <code>-X obey-flags</code> to <code>hfst-expand</code> or <code>hfst-lookup</code>.
+
You'll also need to add <code>-X obey-flags</code> to <code>hfst-expand</code> or <code>hfst-lookup</code> any time you run those commands.
  
 
=== Running tests ===
 
=== Running tests ===
 
In theory you shouldn't have to do anything extra?
 
In theory you shouldn't have to do anything extra?
 +
 +
But it's a good idea to run regression tests after implementing these changes!
  
 
[[Category:Documentation]]
 
[[Category:Documentation]]
 
[[Category:Tutorials]]
 
[[Category:Tutorials]]

Latest revision as of 00:16, 23 March 2023

lexd is a bit slow to compile dictionaries that make use of prefixes with large lexicons (or similar).

To speed this up, you can enable compiling to flag diacritics.

Implementing the change

First thing you want to do is edit your Makefile.am:

  • remove autogen.bin and automorf.bin lines from the TARGETS_COMMON list at the top
  • add -f as a switch to the lexd commmands
  • add -F as a switch to the hfst-compose command in the the automorf.hfst line

Be sure to rerun ./autogen.sh when you're done.

Also make sure to do all of this the same way in your multilingual pair.

The changes will look something like this: 

 TARGETS_COMMON =                       \
        $(LANG1).automorf.hfst          \
-       $(LANG1).automorf.bin           \
        $(LANG1).automorf.att.gz        \
        $(LANG1).autogen.hfst           \
-       $(LANG1).autogen.bin            \
        $(LANG1).autogen.att.gz         \
        $(LANG1).autopgen.bin           \
        $(LANG1).rlx.bin            \
        $(LANG1).zhfst


 .deps/$(LANG1).RL.lexd.hfst: .deps/$(LANG1).RL.lexd
-       lexd $< .deps/$(LANG1).RL.lexd.att
+       lexd -f $< .deps/$(LANG1).RL.lexd.att
        hfst-txt2fst .deps/$(LANG1).RL.lexd.att -o $@
 
 .deps/$(LANG1).LR.lexd.hfst: .deps/$(LANG1).LR.lexd
-       lexd $< .deps/$(LANG1).LR.lexd.att
+       lexd -f $< .deps/$(LANG1).LR.lexd.att
        hfst-txt2fst .deps/$(LANG1).LR.lexd.att -o $@

 $(LANG1).automorf.hfst: .deps/$(LANG1).LR.hfst .deps/$(LANG1).spellrelax.hfst
-       hfst-compose -1 $< -2 .deps/$(LANG1).spellrelax.hfst | hfst-invert | hfst-fst2fst -O -o $@
+       hfst-compose -F -1 $< -2 .deps/$(LANG1).spellrelax.hfst | hfst-invert | hfst-fst2fst -O -o $@

Additional steps

Updating modes.xml

You'll want to change all the instances of lt-proc and auto*.bin in your modes.xml file to hfst-proc and auto*.hfst, respectively.

Make sure to do the same in your multilingual pair.

Running hfst-expand and hfst-lookup

You'll also need to add -X obey-flags to hfst-expand or hfst-lookup any time you run those commands.

Running tests

In theory you shouldn't have to do anything extra?

But it's a good idea to run regression tests after implementing these changes!

Retrieved from "http://wikis.swarthmore.edu/ling073/index.php?title=Using_flag_diacritics_for_faster_compilation&oldid=20587"