User:Rkamal1/Final project - Revision history

Rkamal1 at 18:52, 20 May 2021

2021-05-20T18:52:43Z

Rkamal1 at 03:45, 20 May 2021

2021-05-20T03:45:56Z

Rkamal1: /* Spell Checker */

2021-05-20T03:45:30Z

‎Spell Checker

Rkamal1 at 03:43, 20 May 2021

2021-05-20T03:43:08Z

Rkamal1: /* Evaluation */

2021-05-19T17:49:36Z

‎Evaluation

Rkamal1: /* Evaluation */

2021-05-19T17:33:01Z

‎Evaluation

Rkamal1: /* Transducer */

2021-05-19T17:32:39Z

‎Transducer

Rkamal1: /* Transducer */

2021-05-19T17:32:22Z

‎Transducer

Rkamal1: /* Evaluation */

2021-05-19T17:29:51Z

‎Evaluation

Rkamal1: Created page with "For my final project, I expanded my Sorani Kurdish (ckb) finite-state transducer and boosted coverage on a large corpus. I also enabled spell-checking on my package by adding..."

2021-05-19T17:27:33Z

Created page with "For my final project, I expanded my Sorani Kurdish (ckb) finite-state transducer and boosted coverage on a large corpus. I also enabled spell-checking on my package by adding..."

New page

For my final project, I expanded my Sorani Kurdish (ckb) finite-state transducer and boosted coverage on a large corpus. I also enabled spell-checking on my package by adding a speller directory with a list of character pairs and weights. I licensed my work under the MIT License. This Wikipedia page outlines the work I did and provides different evaluation metrics. For more details, see my [https://github.com/rezKamal/ckb-spellchecker/blob/main/Project_Poster.pdf project poster]

== Code ==

[https://github.com/rezKamal/ckb-spellchecker Public GitHub Repo]

== Methods ==

My transducer was written in lexd format. It contained nearly 2000 stems, the majority of which were nouns. Most of the stems in the transducer were obtained by copying Kurdish lemmas from Apertium's bilingual Sorani-English transducer. Much of the language's basic morphology has been implemented, including definiteness markers, pluralization, subject pronoun suffixes, past and non-past conjugations, and some tenses of the subjunctive mood. Given Sorani's agglutinative nature and the way different word endings affect the way certain suffixes are added, many twol rules are used.

Among other things, the speller directory contains .txt files that contain single- and multi-character mappings for possible misspelling pairs. Each mapping comes with its respective weight. Lower weights correspond to higher likelihoods of a character mapping being brought up when suggesting corrections for a misspelled word.

== Evaluation ==

The transducer currently has:
* Coverage
** '''63% coverage over Sorani Kurdish Wikipedia''' (~2 million words)
** 73% coverage over Sorani Kurdish Bible excerpt (~1000 words)

There are currently 1916 stems in the transducer, with 1235 noun stems.

@@ Line 25: / Line 25: @@
 === Spell Checker ===
-The spell checker correctly outputs 'C' for correct spellings that exist in the transducer and 'W' for incorrect spellings. Because I do not have a great deal of twol narrowing rules, the checker currently fails to flag certain misspellings.
+The spell checker correctly outputs 'C' for correct spellings that exist in the transducer, 'W' for incorrect spellings, and 'S' for suggestions. Because I do not have a great deal of twol narrowing rules, the checker currently fails to flag certain misspellings.
 [[Category: sp21_FinalProjects]] [[Category: Central_Kurdish]]

← Older revision		Revision as of 03:45, 20 May 2021
Line 1:		Line 1:
−	For my final project, I expanded my Sorani Kurdish (ckb) finite-state transducer and boosted coverage on a large corpus. I also enabled spell-checking on my package by adding a speller directory with a list of character pairs and weights. I licensed my work under the MIT License. This Wikipedia page outlines the work I did and provides different evaluation metrics. For more details, see my [https://github.com/rezKamal/ckb-spellchecker/blob/~~main~~/~~Project_Poster~~.pdf project poster]	+	For my final project, I expanded my Sorani Kurdish (ckb) finite-state transducer and boosted coverage on a large corpus. I also enabled spell-checking on my package by adding a speller directory with a list of character pairs and weights. I licensed my work under the MIT License. This Wikipedia page outlines the work I did and provides different evaluation metrics. For more details, see my [https://github.com/rezKamal/ckb-spellchecker/blob/master/Final_Project.pdf project poster]

	== Code ==		== Code ==

@@ Line 25: / Line 25: @@
 === Spell Checker ===
-The spell checker correctly outputs 'C' for correct spellings that exist in the transducer and 'W' for incorrect spellings. Because I do not have a great deal of twol narrowing rules, the checker currently fails to flag certain misspellings. Because of unknown issues for which debugging is beyond my capabilities, the spell checker currently does not output a list of suggestions for every misspelling.
+The spell checker correctly outputs 'C' for correct spellings that exist in the transducer and 'W' for incorrect spellings. Because I do not have a great deal of twol narrowing rules, the checker currently fails to flag certain misspellings.
 [[Category: sp21_FinalProjects]] [[Category: Central_Kurdish]]

← Older revision		Revision as of 03:43, 20 May 2021
Line 26:		Line 26:

	The spell checker correctly outputs 'C' for correct spellings that exist in the transducer and 'W' for incorrect spellings. Because I do not have a great deal of twol narrowing rules, the checker currently fails to flag certain misspellings. Because of unknown issues for which debugging is beyond my capabilities, the spell checker currently does not output a list of suggestions for every misspelling.		The spell checker correctly outputs 'C' for correct spellings that exist in the transducer and 'W' for incorrect spellings. Because I do not have a great deal of twol narrowing rules, the checker currently fails to flag certain misspellings. Because of unknown issues for which debugging is beyond my capabilities, the spell checker currently does not output a list of suggestions for every misspelling.
		+
		+	[[Category: sp21_FinalProjects]] [[Category: Central_Kurdish]]

@@ Line 17: / Line 17: @@
 * 73% coverage over Sorani Kurdish Bible excerpt (~1000 words)
-=== Precision and recall ===
+=== Precision & Recall ===
 * 98.1% precision
 * 68.2% recall

@@ Line 13: / Line 13: @@
 == Evaluation ==
-=== Transducer ===
+=== Coverage ===
 * 63% coverage over Sorani Kurdish Wikipedia (~2 million words)
 * 73% coverage over Sorani Kurdish Bible excerpt (~1000 words)
-'''Precision and recall'''
+=== Precision and recall ===
 * 98.1% precision
 * 68.2% recall

← Older revision		Revision as of 17:32, 19 May 2021
Line 24:		Line 24:

	There are currently 1916 stems in the transducer, with 1235 noun stems.		There are currently 1916 stems in the transducer, with 1235 noun stems.
		+
		+	=== Spell Checker ===

	The spell checker correctly outputs 'C' for correct spellings that exist in the transducer and 'W' for incorrect spellings. Because I do not have a great deal of twol narrowing rules, the checker currently fails to flag certain misspellings. Because of unknown issues for which debugging is beyond my capabilities, the spell checker currently does not output a list of suggestions for every misspelling.		The spell checker correctly outputs 'C' for correct spellings that exist in the transducer and 'W' for incorrect spellings. Because I do not have a great deal of twol narrowing rules, the checker currently fails to flag certain misspellings. Because of unknown issues for which debugging is beyond my capabilities, the spell checker currently does not output a list of suggestions for every misspelling.