Collaborative Research: Information Theory of the Rare Event Regime

点击翻译按钮获取中文摘要

This research involves the development of improved methods forautomatic processing of text documents. The research focusesin particular on two specific text-handling problems: losslessdata compression and classification. Lossless data compressionis useful for reducing storage and transmission requirementsfor large text documents. It is also an important step insending text securely via cryptography. Classification arisesfrequently in web search problems and also in forensics, wherethe goal is to determine the authorship of an anonymous document.The approach is to draw new insights into these problems byusing a novel asymptotic regime in information theory, whichmore accurately models real text sources than classicalmodels do. Specfically, the investigators consider a regime in which the size of the data set and the source alphabet are comparably large. It can be shown that most classical information theory techniquesfail in this "rare-events" regime. Nonetheless, new techniques can be developed that are tailored to this regime that yield new algorithms and insights.