Unsupervised and Transfer Learning for Words segmentation in Korean Social Media

社交媒体几乎渗透到我们生活的方方面面。人们经常以多种方式使用社交媒体：网络、工作、教育、导航以及介于两者之间的一切。处理多语言社交媒体是非常具有挑战性的，因为我们必须处理一定数量的语言，这些语言可能很快就会变得势不可挡。**UQAM的团队在Sadat Fatiha教授的指导下，与**工业合作伙伴Advanced Symbolic Inc.完成了一项NSERC Engage项目，名为“社交网络中的桥梁语言和面向紧凑表示的**半监督学习”，目标是对**中文和**日语社交媒体进行预处理。所开发的语言工具在准确性和**F-measure方面表现良好，日语分词高达4.80分，汉语分词高达**+2.69分，优于其他分词工具。**目前的NSERC Engage Plus项目旨在通过在ASI的亚洲语言人工智能管道中添加韩语**作为第三语言来实现最初的项目。**在这个项目中，我们主要关注的是Twitter的非结构化和高噪声的韩国短消息（140**字符限制）的语言预处理。由于我们没有针对**嘈杂的韩国社交媒体数据集的注释数据，我们将探索和研究无监督学习和迁移学习技术**。这种强大的工具和技术的构建将有助于在ASI设计一个完整的亚洲语言数据挖掘和自然语言处理管道，因为该公司在解决此类问题方面几乎没有专业知识，以自动的方式。

Social media has pervaded virtually every aspect of our lives. People use social media in many ways on a**regular basis: networking, productivity, education, navigation and everything in between. Dealing with**multilingual social media is very challenging, as we have to deal with a certain amount of languages that can**quickly become overwhelming .**UQAM's team, under the direction of Prof. Sadat Fatiha, has completed a NSERC Engage project with the**industrial partner, Advanced Symbolic Inc., entitled 'Bridging Languages in Social Networks and**Semi-Supervised Learning for a Compact Representation', which goal was pre-processing Chinese and**Japanese social media. The developed linguistic tools have shown good performances in terms of accuracy and**F-measure, outperforming other segmentation tools with up to 4.80 pts for the Japanese segmentation and up to**+2.69 for the Chinese.**The current NSERC Engage Plus project aims at pursuing the initial project, by adding and involving Korean**as a third language in ASI's Asian languages Artificial Intelligence pipeline.**In this project, our main concern is the linguistic pre-processing of South Korean short messages (140**characters limit) of Twitter that are unstructured and highly noisy. As we do not have annotated data for the**noisy set of South Korean social media data, unsupervised and transfer learning techniques will be explored**and investigated. The construction of such powerful tool and technology will help design a complete data**mining and Natural Language Processing pipeline for Asian languages at ASI, as the company has very little**expertise in solving such problems, in an automatic way.