Rgoel1/Final Project

An LLM-Based Approach to Generalized Linguistic Data Scraping

For my final project, I developed a (preliminary) general-use web scraper that uses a Large Language Model (LLM) to analyze the content and pagination structure of any given website containing linguistic text, and then automatically traverses the site while structuring/exporting the data as specified.

The project aims to take the burden of having to manually write bespoke scrapers for different websites off of both academic linguists and communities looking to construct reliable corpora of their language's resources.

- Bible.is (sentences from Genesis, Magyar): extracted 99.74% of all sentences without error, 4/1533 of sentences were either missing or contained a typographical error; used [existing scraper](https://github.swarthmore.edu/Ling073-sp25/web-scrapers/blob/master/bible_is_scraper.py) as ground truth/reference for accuracy calculations
- Webonary.org (all entries of dictionary, Rungus): since no bespoke/manually written scraper existed, I had to manually construct from the site a sample of 109 dictionary entries; of these, the scraper accurately extracted 97.24%, with 3/109 entries either missing or containing a typographical error
- Wiktionary.org (lemmas, Adyghe): extracted 89.55% of all lemmas without error, 626/5624 of lemmas were either missing or contained a typographical error (see why in Future Work section); used [[existing scraper] https://github.swarthmore.edu/Ling073-sp25/web-scrapers/blob/master/wiktionary2bidix.py] as ground truth/reference for accuracy calculations

Overall, the scraper performed quite well, handling 3 wholly different kinds of sites with high accuracy
The worst performing site was Wiktionary.org, likely due to to the Cyrillic script of the target language; the model had trouble extracting some lemmas/characters with high-fidelity, indicating a potential lack in the LLM's tokenizer vocabulary/exposure to non-Western content in training data; future work could entail finetuning an OSS model with expanded vocabulary
Potential future features include: allowing user to dictate what the structure of outputs so more information can be scraped according to the content on a given site (trivial)