Normalization of Dutch User-Generated Content

TitleNormalization of Dutch User-Generated Content
Publication TypeConference Paper
Year of Publication2013
AuthorsDe Clercq, O, Schulz, S, Desmet, B, Lefever, E, Hoste, V
Conference Name9th International Conference on Recent Advances in Natural Language Processing (RANLP 2013)
Date Published09/2013
Conference LocationHissar, Bulgaria
Abstract

This paper describes a phrase-based machine translation approach to normalize Dutch user-generated content (UGC). We compiled a corpus of three different social media genres (text messages, message board posts and tweets) to have a sample of this recent domain. We describe the various characteristics of this noisy text material and explain how it has been manually normalized using newly developed guidelines. For the automatic normalization task we focus on text messages, and find that a cascaded SMT system where a token-based module is followed by a translation at the character level gives the best word error rate reduction. After these initial experiments, we investigate the system’s robustness on the complete domain of UGC by testing it on the other two social media genres, and find that the cascaded approach performs best on these genres as well. To our knowledge, we deliver the first proof-of-concept system for Dutch UGC normalization, which can serve as a baseline for future work.

File: