Initial corpus assembly

Your first corpus is just a list of text. You need enough authentic text to fill approximately one page—or about 5000 characters.

The text can be drawn from different sources, all ideally allowing for redistribution under a free license. If all the texts you have available are copyrighted, that's fine—just make sure your repo is private and shared only with me (and the grader).

The assignment

This assignment is due at the beginning of class on the Thursday of the third week of class (this semester 11:20am on February 2, 2017).

For this assignment, you should create a private repository in your GitHub account called ling073-xyz-corpus, and share it with the professor. You'll build a corpus of authentic text here. The corpus will be built from materials as described at Building Corpora#Places to look for corpora. You may have to type up a certain amount of text from print sources using your new keyboard layout.

1. To start with, add any raw corpora you find to this repository. For example, if you find a bible translation or newspaper site online, strip out all the text (ignoring notices, menus, etc.) as best you can (there are tools for this if copy-paste is giving sub-optimal results), and put it in a single file (per source) in this directory. Name these files something like xyz.type.label.txt. Replace the word type with the type of source, e.g., bible, phrases, wikipedia, news, tweets. For label, use either a publisher (e.g., IBT for a bible translation from IBT); or date you got the content from a site that changes, like wikipedia or a news site (e.g., 2017-01-28); or author (e.g., smith if they're all from a book by Smith). You can combine these too if you need, using an underscore (_).
2. Add a brief description of each of these files in a separate MANIFEST file. Ideally all of your content will be redistributable (e.g., licensed under one of the Creative Commons licenses), but chances are that you will have to include non-redistributable content as well. So also include mention of what license each of the corpora is covered under in the MANIFEST file (or if the original source doesn't say; if it's a print book, though, chances are that it's "all rights reserved"). You can format each line something like the following:

   xyz.news.WN_2017-01-28.txt - contains news from WorldNews™ website, acquired on 2017-01-28, copyright WorldNews Eurasia, Inc.

3. Take sentences from a range of these raw files, and add them to a file xyz.corpus.basic.txt. This file should consist of one sentence per line, and should contain at least 5000 characters. Do everything you can to make sure all of the material you include in this new file is in the same orthography. You may have to manually convert sentences as you copy them over from the other files.

   Note: you can count how many characters are in your file with the following command: wc -c xyz.corpus.basic.txt

   From within vim, you can issue the command :!wc -c %

   If you are using gedit, you can check this in Document Tools → Document Statistics.

4. You should enter English glosses for at least a quarter of the sentences in your basic corpus in an xyz-eng.corpus.basic.txt file, with available glosses on corresponding line numbers. This may result in a number of blank lines throughout the file. You will probably have trouble finding English glosses for a lot of the text (prioritise diversity of sources of text in this file as well), but bible translations should be pretty straightforward to find English equivalents for, and phrase books and grammars almost always provide glosses.
5. Add a README file to address anything else you think is important, such as issues that came up during corpus assembly that you had to make decisions on.