User:Yli11/Final Project

Improving Interpretability and Maintainability in Apertium Regression Testing

Authors: User:Kpeng1 and User:Yli11

Our Focus

We noted that Apertium-regtest helps Apertium developers compare current analyzer/generator outputs against expected (gold) outputs. But before these changes, some information was hard to interpret or manage:

CLI (command line) summaries showed per-corpus gold match counts, but not a total. CLI output gave numbers, but not the actual forms behind those numbers. Web UI allowed users to add gold outputs, but not remove mistaken golds. This matters because language-pair development often involves many small forms, hashes, and output files. If the tool only gives aggregate numbers, developers have to manually trace through test files to understand what changed.

We looked for issues from [1] and resolved some of them:

#28 Total gold match count

CLI showed gold match counts per corpus, but no final total across all corpora. ->Added a final summary line such as 21/96 (21.88%) tests match gold.

Before:

Corpus 1: 3/13 match gold

Corpus 2: 3/20 match gold

...

All tests pass.

(*This is what we currently get in our analysis)

After:

Adds an extra line:

21/96 (21.88%) tests match gold.

All tests pass.

Implementation:

• Added a total_gold counter in static_test().
• Accumulated gold matches across all corpora.
• Printed a final total after all corpus summaries.

Benefit:

Users can quickly understand the overall gold coverage/result quality without manually adding per-corpus numbers.

#29 Show forms behind test counts

apertium-regtest test showed only numbers, not which forms matched or failed gold/expected. ->Added repeatable --show options: gold, not-gold, expected, unexpected.

Added:

apertium-regtest test --show gold

apertium-regtest test --show not-gold

apertium-regtest test --show expected

apertium-regtest test --show unexpected

These commends will allow users to list out all words of one of the descriptions.

Example:

Forms not matching gold:

 [az3b5Gk9edGE#3] ^okometh<n><aa><px1sg>$
   generator output: nokometha/nokomtha
   generator gold: nokometha

Implementation:

Added --show as a repeatable CLI argument. Added helper functions to print the input hash, input form, current output, expected output, and gold output. Kept default test behavior unchanged unless --show is used.

Benefit:

Developers can immediately see which forms correspond to the summary numbers. This reduces manual searching through expected/gold files.

#23 Remove golds in web UI

Users could add gold values in the web interface, but could not remove mistaken ones. ->Added a Remove button next to each gold value and backend support for deleting empty gold entries.

Before:

Users could add or replace gold values.

If a gold value was wrong, they had to manually edit gold files.

After:

Each gold value in the web UI has a Remove button.

Removing the last gold value deletes the gold entry instead of saving an empty block.

Implementation:

Updated static/regtest.js to render each gold value with a Remove button.

Added a click handler that removes the selected value and sends the remaining list through the existing gold callback.

Updated Corpus.set_gold() so an empty gold list deletes the entry.

Added small CSS changes to align gold values and remove buttons cleanly.

Benefit:

The web UI is more complete and safer to use.

Users can correct mistakes without manually editing files with hashes.

Continuing Work: possible alternative gold/acceptance filter #34

We’re also working on a possible alternative design for the web UI filters. The current interface has a Show/Hide Unchanged Results toggle that can be confusing, because it mixes different concepts together. Issue #34 proposes separating the filters into two clearer dimensions:

Gold filter: All, No Gold, Matches Gold, Doesn't Match Gold

Acceptance filter: All, Matches Accepted, Doesn't Match Accepted

This makes it easier for users to ask more specific questions, such as:

Which results have no gold form?

Which results match gold?

Which results differ from the accepted/expected output?

Which accepted results still do not match gold?

We have started implementing this by replacing the old mixed filter controls with two separate button groups and adding row-level classes for gold status and acceptance status. The goal is to make the web interface easier to understand and closer to the way users actually reason about regression test results.

Previous Work

This project builds on existing apertium-regtest functionality:

Existing CLI mode already reported pass/fail and per-corpus gold matches.

Existing web UI already supported adding and replacing golds.

The issue discussion also mentions tools like morph-test / aq-morphtest, which show more detailed form-level information. The new --show options bring some of that transparency into apertium-regtest test.

Benefits

Apertium language-pair developers.

Computational linguists debugging analyzers and generators.

Students or researchers working on low-resource language technology.

Language communities indirectly, because better testing tools make language technology easier to maintain.

Evaluation

We evaluated the changes in several ways:

• Ran syntax checks:
• Tested on the ling073-kic server data.
• Confirmed that --show not-gold printed the exact forms responsible for gold mismatches.
• Smoke-tested gold deletion to confirm that removing the final gold value deletes the entry from the saved gold file.
• Verified that default CLI behavior remains mostly unchanged unless optional flags are used.
• Two issues: #28, #29 has already been accepted and updated by the author