Apertium-regtest

The tool apertium-regtest is a "regression test" tool that helps to test large numbers of inputs simultaneously and to keep track of what has changed and what is still wrong.

General premise and structure

Apertium-regtest runs as a web client, a command-line interface, or simply checks tests on the command line.

When run from a transducer or translation directory, it checks the test/ directory for tests.

Each test consists of a list of inputs and a list of "gold" (i.e., correct) forms provided by the user, as well as a list of output forms generated by the software upon request and a list of expected forms, generated as the user interacts with the software

Commands

• To run apertium-regtest to access in a web browser:

  $ apertium-regtest web

• To set up an ssh tunnel to a lab machine, to allow you to access the web interface remotely:

  $ ssh -NL 3000:localhost:3000 machinename.cs.swarthmore.edu (with the name of the machine you're running apertium-regtest on in place of machinename)

  • Then you should be able to access http://localhost:3000 from the computer this command is run on.
• To check the number of morph tests matching gold:

  $ apertium-regtest test -c .-morph

Using the web interface

When you first run the web interface:

1. First you'll want to click the button to run all tests.
2. Then you want to accept all tests as expected (even if incorrect / not matching gold). You'll probably have to click on the button for each test.
3. Then you can filter for e.g. tests not matching gold.

When you update something in your transducer or translator and recompile, you'll want to:

1. Rerun all tests.
2. The software will show you what's changed. You'll want to accept the updated forms as expected (again, even if incorrect / not matching gold).

Notes:

• For working on the analyser, you'll mostly just want "-morph" tests. For working on the generator, you'll mostly want just "-gen" tests. Feel free to filter on these
• Apertium-regtest can also be used to work on translation.
• There are a few known issues with the web interface.
• Apertium-regtest is fairly new and the lore for how to use it hasn't been well established. So you get to help with that!

Manually adding tests

1. Identify the right file:
   • To add an analysis test, find the correct *-morph-input.txt file.
   • To add a generation test, find the correct *-gen-input.txt file.
   • If you add a file instead of using an existing one, you'll need to add it to the test/tests.json file.
2. Simply add a line to this file with the input (for analysis it should be a form; for generation it should be an analysis)
3. Load the regtest web interface (apertium-regtest web); there should be a notification in red that tests were added. Click the button to accept all.
4. Now rerun tests ("All Corpora", or just the one that you modified)
5. Filter by "no gold"
6. Add gold
   • If the output of the test is correct, just click "Add as gold"; if not, click "Add manual gold" and add the correct output manually
7. Then go back through with no filters and click the "Accept: morph" (for example) on these tests