User:Doldham1/Final project - Revision history

Doldham1: /* Corpus */

2017-05-13T04:04:59Z

‎Corpus

Doldham1: /* What I did not do */

2017-05-13T04:01:37Z

‎What I did not do

Doldham1: /* Evaluation */

2017-05-13T03:41:26Z

‎Evaluation

Doldham1: /* Evaluation */

2017-05-13T03:33:11Z

‎Evaluation

Doldham1: /* What I did not do */

2017-05-13T03:30:11Z

‎What I did not do

Doldham1: /* Evaluation */

2017-05-13T03:29:49Z

‎Evaluation

Doldham1: /* What I did */

2017-05-13T03:24:49Z

‎What I did

Doldham1: /* What I did not do */

2017-05-13T03:21:12Z

‎What I did not do

Doldham1: /* Corpus */

2017-05-13T03:19:55Z

‎Corpus

Doldham1: /* Corpus */

2017-05-13T03:19:39Z

‎Corpus

@@ Line 4: / Line 4: @@
 ====Corpus====
-I selected four pages from the Japanese Wikipedia ([https://ja.wikipedia.org/wiki/%E5%BA%83%E5%B3%B6%E5%B9%B3%E5%92%8C%E8%A8%98%E5%BF%B5%E5%85%AC%E5%9C%92 Hiroshima Peace Park], [https://ja.wikipedia.org/wiki/%E3%82%B0%E3%83%AC%E3%83%BC%E3%83%97%E3%83%95%E3%83%AB%E3%83%BC%E3%83%84 Grapefruit], [https://ja.wikipedia.org/wiki/%E7%B5%9E%E3%82%8A Shibori], and [https://ja.wikipedia.org/wiki/%E3%82%AA%E3%83%BC%E3%82%AF_%28%E3%83%88%E3%83%BC%E3%83%AB%E3%82%AD%E3%83%B3%29 Orcs]) and copied most of the articles.  I excluded things like information in tables and sentences with a lot of English.  These articles were intended to from very different domains, and thus represent a wide range of the Japanese lexicon.
+I selected four pages from the Japanese Wikipedia ([https://ja.wikipedia.org/wiki/%E5%BA%83%E5%B3%B6%E5%B9%B3%E5%92%8C%E8%A8%98%E5%BF%B5%E5%85%AC%E5%9C%92 Hiroshima Peace Park], [https://ja.wikipedia.org/wiki/%E3%82%B0%E3%83%AC%E3%83%BC%E3%83%97%E3%83%95%E3%83%AB%E3%83%BC%E3%83%84 Grapefruit], [https://ja.wikipedia.org/wiki/%E7%B5%9E%E3%82%8A Shibori], and [https://ja.wikipedia.org/wiki/%E3%82%AA%E3%83%BC%E3%82%AF_%28%E3%83%88%E3%83%BC%E3%83%AB%E3%82%AD%E3%83%B3%29 Orcs]) and copied most of the articles.  I excluded things like information in tables and sentences with a lot of English.  These articles were intended to from very different domains, and thus represent a wide range of the Japanese lexicon.  The total word count is 767.
 ====Transducer====

@@ Line 36: / Line 36: @@
 *There are many more kanji which need to be added to the twol alphabet.  They should also be organised better.
 *Roman characters and some number of English words would need to be added in order to significantly increase the coverage of the Japanese Wikipedia.
 ==Repositories==

@@ Line 48: / Line 48: @@
 *Coverage over the orc article: 23.28%
 *Coverage over the first three combined: 38.15% (I was using this as my primary progress tracker.)
-*Coverage over all articles:
+*Coverage over all articles: 32.28%
 Precision and recall would also be useful metrics, but I did not have time to work on them.

@@ Line 47: / Line 47: @@
 *Coverage over the shibori article: 39.33%
 *Coverage over the orc article: 23.28%
-*Coverage over the first three combined: 38.20% (I was using this as my primary progress tracker.)
+*Coverage over the first three combined: 38.15% (I was using this as my primary progress tracker.)
 *Coverage over all articles:

@@ Line 21: / Line 21: @@
 ===What I did not do===
-*I did not automate the process of corpus tokenisation- the corpus files were all hand-tokenised.
+*I did not automate the process of corpus tokenisation- the corpus files were all hand-tokenised, and I have no doubt that I was inconsistent in doing so.
 *I did not end up creating yaml files to test the morphology.  This would have been useful.
 *I did not touch prefixes.  There are not a large number of them in Japanese, but they do exist, like 毎 (every), and お and ご for honorific, etc.  While words with 毎 could be lexicalised, that seems less practical for the honorifics.

@@ Line 43: / Line 43: @@
 ==Evaluation==
-*Coverage over the Hiroshima article:
+*Coverage over the Hiroshima article: 41.88%
-*Coverage over the grapefruit article:
+*Coverage over the grapefruit article: 31.76%
-*Coverage over the shibori article:
+*Coverage over the shibori article: 39.33%
-*Coverage over the orc article:
+*Coverage over the orc article: 23.28%
 *Coverage over all articles:

@@ Line 1: / Line 1: @@
 ==Project Details==
 ===What I did===
 ====Corpus====
 I selected four pages from the Japanese Wikipedia ([https://ja.wikipedia.org/wiki/%E5%BA%83%E5%B3%B6%E5%B9%B3%E5%92%8C%E8%A8%98%E5%BF%B5%E5%85%AC%E5%9C%92 Hiroshima Peace Park], [https://ja.wikipedia.org/wiki/%E3%82%B0%E3%83%AC%E3%83%BC%E3%83%97%E3%83%95%E3%83%AB%E3%83%BC%E3%83%84 Grapefruit], [https://ja.wikipedia.org/wiki/%E7%B5%9E%E3%82%8A Shibori], and [https://ja.wikipedia.org/wiki/%E3%82%AA%E3%83%BC%E3%82%AF_%28%E3%83%88%E3%83%BC%E3%83%AB%E3%82%AD%E3%83%B3%29 Orcs]) and copied most of the articles.  I excluded things like information in tables and sentences with a lot of English.  These articles were intended to from very different domains, and thus represent a wide range of the Japanese lexicon.