User:Yzhang4/Final project - Revision history

Yzhang4: /* Implementation Method */

2023-05-09T00:08:15Z

‎Implementation Method

2023-05-09T00:03:08Z

‎Future Improvements

2023-05-08T23:10:42Z

‎Evaluation

2023-05-08T23:10:15Z

‎Evaluation

2023-05-07T21:19:34Z

‎Evaluation

2023-05-07T19:55:21Z

‎Table

2023-05-07T19:55:00Z

‎Table

2023-05-06T21:29:03Z

‎Evaluation

2023-05-06T21:14:41Z

‎External Resources

2023-05-06T21:13:55Z

‎Evaluation

@@ Line 229: / Line 229: @@
 We built a mini-transducer based on apertium to implement IPA to Tibetan Script transcriber. Specifically, we hard coded the rules for transcription in lexd and twol files. Moreover, because there are ambiguation in transcribing consonant clusters, we also used an xfst file to deal with that kind of ambiguation.
-Currently, we have about 8 lexicon categories with about 170 lexicon entires in the lexd file, 5 twol rules and 61 disambiguation rules in the xfst file.
+Currently, we have about 17 patterns with 164 lexicons in the lexd file, 1 twol rule and 63 disambiguation rules in the xfst file.
 =Installation=

← Older revision		Revision as of 00:03, 9 May 2023
Line 302:		Line 302:

	For some Japhug expressions with multiple consonants between vowels, there still exists ambiguity. For example, the word "paχɕi" (apple) can be transcribed as either "པཱ༌ཧ༹ཤི" when treating the consonant χ as the first consonant of the second syllable, or "པཱག༌ཤི" if treating χ as the last consonant of the first syllable. We'll try to figure out some ways to standardize that.		For some Japhug expressions with multiple consonants between vowels, there still exists ambiguity. For example, the word "paχɕi" (apple) can be transcribed as either "པཱ༌ཧ༹ཤི" when treating the consonant χ as the first consonant of the second syllable, or "པཱག༌ཤི" if treating χ as the last consonant of the first syllable. We'll try to figure out some ways to standardize that.
		+
		+	We need to develop a keyboard layout for the Japhug Tibetan Script, which is slightly different than that for Central Tibetan.
		+
		+	Also, we need to hear more feedback from the Japhug community, and promote the Japhug Tibetan Script Among Japhug speakers.

	=External Resources=		=External Resources=

@@ Line 272: / Line 272: @@
 We also tested the coverage of the multiorthographical transducer in the corpus:
 <pre>
 TOP UNKNOWN WORDS:
 ^ཉྀ༌ངུ།/*ཉྀ༌ངུ།$
@@ Line 302: / Line 297: @@
 ambiguity: 5416 / 2656 (~2.03915662650602409639)
 jya.corpus.basic.tibt Mon May 8 07:10:04 PM EDT 2023    2344/2656       ~0.88253012048192771084 \|      5416/2656       ~2.03915662650602409639
 =Future Improvements=

@@ Line 269: / Line 269: @@
 Word Error Rate = %30.04394726328406
 </pre>
 =Future Improvements=

← Older revision		Revision as of 21:19, 7 May 2023
Line 262:		Line 262:

	Most of the mis-matches have something to do with consonant clusters. For example, the word "ɲɯɤznɤkharnɯ" should be expected to have transcription "ཉྀ༌འ༌འས་ཁཱ་རྣྀ་", but it generates incorrect form "ཉྀ༌འ༌ཟའཀ༌ཧཱ྇༌རནྀ༌" in our transcriber, in which prenasalization "zn" is not transcribed correctly, and aspiration consonant "ཁ(kh)" is divided into two parts and becomes "ཀ(k)" and "ཧ྇(h)" separately.		Most of the mis-matches have something to do with consonant clusters. For example, the word "ɲɯɤznɤkharnɯ" should be expected to have transcription "ཉྀ༌འ༌འས་ཁཱ་རྣྀ་", but it generates incorrect form "ཉྀ༌འ༌ཟའཀ༌ཧཱ྇༌རནྀ༌" in our transcriber, in which prenasalization "zn" is not transcribed correctly, and aspiration consonant "ཁ(kh)" is divided into two parts and becomes "ཀ(k)" and "ཧ྇(h)" separately.
		+	When transcribe back from IPA to Tibetan Script, the WER is much higher:
		+	<pre>
		+	Total tokens = 2503
		+	Matched tokens = 1751
		+	Not Matched tokens = 752
		+	Word Error Rate = %30.04394726328406
		+	</pre>

	=Future Improvements=		=Future Improvements=

@@ Line 167: / Line 167: @@
+|
+|
-!style="background:beige;"|ɬ
+|ɬ
-!style="background:beige;"|ལྷ
+|ལྷ
 |s
 |ས

@@ Line 260: / Line 260: @@
 Word Error Rate = %0.9988014382740711
 </pre>
 =Future Improvements=

← Older revision		Revision as of 21:14, 6 May 2023
Line 266:		Line 266:

	=External Resources=		=External Resources=
		+	[https://github.swarthmore.edu/Ling073-sp23/ling073-jya-corpus Japhug Corpus and Relating Tools]
		+
	[https://wikis.swarthmore.edu/ling073/Japhug Japhug Grammar resources]		[https://wikis.swarthmore.edu/ling073/Japhug Japhug Grammar resources]

@@ Line 250: / Line 250: @@
 =Evaluation=
-Since there are not many existing texts of Japhug in this transcription, we'll just transcribe a sample corpus ([https://github.swarthmore.edu/Ling073-sp23/ling073-jya-corpus/blob/main/jya.corpus.basic.txt jya.corpus.basic.txt]) from IPA to Tibetan script. After than, we manually change each transcribed word into the most correct form, to make the target transcription file.
+Since there are not many existing texts of Japhug in this transcription, we'll just transcribe a sample corpus ([https://github.swarthmore.edu/Ling073-sp23/ling073-jya-corpus/blob/main/jya.corpus.basic.txt jya.corpus.basic.txt]) from IPA to Tibetan script. After than, we manually change each transcribed word into the most correct form, to make the target transcription file ([https://github.swarthmore.edu/Ling073-sp23/ling073-jya-corpus/blob/main/jya.corpus.basic.tibt.target.txt jya.corpus.basic.tibt.target.txt]).
-Finally, we directly check the word error rate of the transcriber by comparing the target transcription with the transcription generated by the transcriber. The results are shown below:
+Finally, we directly check the word error rate of the transcriber by comparing the target transcription with the transcription generated by the transcriber using [https://github.swarthmore.edu/Ling073-sp23/ling073-jya-corpus/blob/main/WER_calculator.py this script]. The results are shown below:
 <pre>