Collaborative Research: Information Theory for the Rare Events Regime

这项研究涉及文本文档自动处理的改进方法的开发。这项研究特别关注两个具体的文本处理问题：无损数据压缩和分类。无损数据压缩对于减少大型文本文档的存储和传输要求非常有用。这也是通过密码学安全发送文本的重要一步。分类问题经常出现在网络搜索问题和取证问题中，其目的是确定匿名文档的作者。方法是通过使用信息论中的一种新的渐近机制来获得对这些问题的新见解，该机制比经典模型更准确地模拟真实的文本来源。具体地说，研究人员考虑了一种数据集和源字母表的大小相对较大的制度。可以看出，大多数经典的信息论技术在这种“罕见事件”制度下都是失败的。尽管如此，可以开发出适合这一制度的新技术，从而产生新的算法和见解。

This research involves the development of improved methods forautomatic processing of text documents. The research focusesin particular on two specific text-handling problems: losslessdata compression and classification. Lossless data compressionis useful for reducing storage and transmission requirementsfor large text documents. It is also an important step insending text securely via cryptography. Classification arisesfrequently in web search problems and also in forensics, wherethe goal is to determine the authorship of an anonymous document.The approach is to draw new insights into these problems byusing a novel asymptotic regime in information theory, whichmore accurately models real text sources than classicalmodels do. Specfically, the investigators consider a regime in which the size of the data set and the source alphabet are comparably large. It can be shown that most classical information theory techniquesfail in this "rare-events" regime. Nonetheless, new techniques can be developed that are tailored to this regime that yield new algorithms and insights.