Unsupervised and Transfer Learning for Words segmentation in Korean Social Media

社交媒体几乎已经渗透到我们生活的方方面面。人们经常以多种方式使用社交媒体：社交网络、生产力、教育、导航以及介于两者之间的一切。处理**多语言社交媒体是非常具有挑战性的，因为我们必须处理大量可能**迅速变得不堪重负的语言。**UQAM的团队在Sadat Fatiha教授的指导下，与**工业合作伙伴Advanced Symbol Inc.共同完成了一个NSERC Engage项目，题为“在社交网络中桥接语言和**用于紧凑表示的半监督学习”，该项目的目标是对中文和**日语社交媒体进行预处理。开发的语言工具在准确率和**F度量方面表现良好，日语切分高达4.80分，中文切分高达**2.69分。**目前的NSERC Engage Plus项目旨在推进初始项目，在ASI的亚洲语言人工智能流水线中添加并纳入韩语**作为第三种语言。**在该项目中，我们主要关注的是对Twitter上的韩语短消息(140**字符限制)的语言预处理，这些短消息是非结构化和高噪声的。由于我们没有针对**嘈杂的韩国社交媒体数据集的注释数据，因此将探索**和调查非监督和迁移学习技术。如此强大的工具和技术的构建将有助于在ASI设计一条完整的亚洲语言数据挖掘和自然语言处理管道，因为该公司在以自动方式解决此类问题方面**专业知识非常有限。

Social media has pervaded virtually every aspect of our lives. People use social media in many ways on a**regular basis: networking, productivity, education, navigation and everything in between. Dealing with**multilingual social media is very challenging, as we have to deal with a certain amount of languages that can**quickly become overwhelming .**UQAM's team, under the direction of Prof. Sadat Fatiha, has completed a NSERC Engage project with the**industrial partner, Advanced Symbolic Inc., entitled 'Bridging Languages in Social Networks and**Semi-Supervised Learning for a Compact Representation', which goal was pre-processing Chinese and**Japanese social media. The developed linguistic tools have shown good performances in terms of accuracy and**F-measure, outperforming other segmentation tools with up to 4.80 pts for the Japanese segmentation and up to**+2.69 for the Chinese.**The current NSERC Engage Plus project aims at pursuing the initial project, by adding and involving Korean**as a third language in ASI's Asian languages Artificial Intelligence pipeline.**In this project, our main concern is the linguistic pre-processing of South Korean short messages (140**characters limit) of Twitter that are unstructured and highly noisy. As we do not have annotated data for the**noisy set of South Korean social media data, unsupervised and transfer learning techniques will be explored**and investigated. The construction of such powerful tool and technology will help design a complete data**mining and Natural Language Processing pipeline for Asian languages at ASI, as the company has very little**expertise in solving such problems, in an automatic way.