NSF-BSF: CIF: Small: Self-adapting Code Generation in Rate-distortion Theory, Machine Learning, and Channel Coding

NSF-BSF：CIF：小型：率失真理论、机器学习和信道编码中的自适应代码生成

• 批准号: 1909423
• 负责人: Kenneth Rose
• 金额: $ 49.99万
• 依托单位: University of California-Santa Barbara
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-07-01 至 2023-06-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1909423&HistoricalAwards=false
• 关键词: NSF; BSF; CIF; Small; Self

This project builds on the investigators' early information-theoretic results, establishing a mechanism called 'natural type selection' for source coding, which adapts a randomly generated code and is shown to be asymptotically optimal. Fundamental expansions of this framework will be pursued to develop universally applicable methodologies, and thereby yield contributions to information theory itself alongside powerful learning techniques in other application fields, including wireless communications, content delivery, social media, artificial intelligence, and others. From the educational perspective, the project offers a training opportunity for graduate students to experience, first-hand, an international and interdisciplinary research collaboration, which combines theoretical depth with practical impact. It further offers opportunities for extensive curriculum enrichment, and to produce accomplished researchers and practitioners with capacities and skills that are in high demand.This project will develop novel approaches to learning, which employ universal self-adapting mechanisms for random code generation, designed to asymptotically achieve optimality for unknown source distributions. Research will be pursued, in terms of both theoretical analysis of performance bounds and powerful optimization approaches, along three main thrusts: i) Extension of the natural type selection framework to encompass continuous spaces and sources with memory, leveraging the concept of "parametric type" for continuous alphabets, which would expand applicability to virtually all practical scenarios of interest. ii) Applications in machine learning, where supervised learning (e.g., classification, regression) is reformulated as the rate-distortion problem of seeking the minimal amount of information to be learned from a source such that a desired output at the prescribed fidelity can be read from a random codebook; and, on the unsupervised learning side, where the "information bottleneck" method is reformulated universally in a self-adapting codebook generation setting. Both will leverage the optimization framework of deterministic annealing. iii) Applications in communications where stochastic mechanisms are developed for optimal channel input adaptation, including an important extension to multi-user communications which requires the development of a "distributed natural type selection" framework. This project is a collaborative effort between researchers in the US and Israel, with funding for Israeli researchers provided by the Bi-National Science Foundation (BSF).This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该项目建立在研究人员的早期信息理论结果的基础上，建立了一种称为“自然类型选择”的信源编码机制，该机制适应随机生成的代码，并被证明是渐近最优的。该框架的基本扩展将致力于开发普遍适用的方法，从而为信息理论本身以及其他应用领域的强大学习技术做出贡献，包括无线通信，内容交付，社交媒体，人工智能等。从教育的角度来看，该项目为研究生提供了一个培训机会，亲身体验国际和跨学科的研究合作，将理论深度与实际影响相结合。它进一步提供了广泛的课程丰富的机会，并产生有成就的研究人员和从业人员的能力和技能，在高需求。该项目将开发新的学习方法，采用通用的自适应机制随机代码生成，旨在渐近实现未知源分布的最优性。研究将追求，在性能边界和强大的优化方法，沿着三个主要的推力理论分析：i）自然类型选择框架的扩展，以包括连续空间和内存源，利用连续字母表的“参数类型”的概念，这将扩大适用性，几乎所有的实际情况下的利益。ii）机器学习中的应用，其中监督学习（例如，分类、回归）被重新表述为寻找要从源学习的最小量的信息的速率失真问题，使得可以从随机码本读取规定保真度的期望输出;并且，在无监督学习侧，其中“信息瓶颈”方法在自适应码本生成设置中被普遍地重新表述。两者都将利用确定性退火的优化框架。iii）在通信中的应用，其中随机机制被开发用于最佳信道输入自适应，包括对多用户通信的重要扩展，其需要开发“分布式自然类型选择”框架。 该项目是美国和以色列研究人员的合作成果，以色列研究人员的资金由两国科学基金会（BSF）提供。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Stochastic Codebook Regeneration for Sequential Compression of Continuous Alphabet Sources

连续字母源顺序压缩的随机码本再生

• DOI: 10.1109/isit45174.2021.9517877
• 发表时间: 2021
• 期刊: 2021 IEEE International Symposium on Information Theory (ISIT
• 作者: Elshafiy, Ahmed;Namazi, Mahmoud;Zamir, Ram;Rose, Kenneth
• 通讯作者: Rose, Kenneth

On Effective Stochastic Mechanisms for On-The-Fly Codebook Regeneration

动态码本再生的有效随机机制

• DOI: 10.1109/isit44484.2020.9174348
• 发表时间: 2020
• 期刊: 2020 IEEE International Symposium on Information Theory (ISIT
• 作者: Elshafiy, Ahmed;Namazi, Mahmoud;Rose, Kenneth
• 通讯作者: Rose, Kenneth

On Stochastic Codebook Generation for Markov Sources

马尔可夫源的随机码本生成

• DOI: 10.1109/dcc55655.2023.00039
• 发表时间: 2023
• 期刊: DCC
• 作者: Elshafiy, Ahmed;Rose, Kenneth
• 通讯作者: Rose, Kenneth

A Stochastic Rate-Distortion Approach to Supervised Learning Systems

监督学习系统的随机率失真方法

• DOI: 10.1109/isit54713.2023.10206453
• 发表时间: 2023
• 期刊: Proceedings
• 作者: Elshafiy, Ahmed;Namazi, Mahmoud;Rose, Kenneth
• 通讯作者: Rose, Kenneth

On-The-Fly Stochastic Codebook Re-generation for Sources with Memory

具有内存的源的动态随机码本重新生成

• DOI: 10.1109/itw46852.2021.9457666
• 发表时间: 2021
• 期刊: 2020 IEEE Information Theory Workshop
• 作者: Elshafiy, Ahmed;Namazi, Mahmoud;Zamir, Ram;Rose, Kenneth
• 通讯作者: Rose, Kenneth