NSF-BSF: CIF: Small: Self-adapting Code Generation in Rate-distortion Theory, Machine Learning, and Channel Coding

该项目建立在研究人员早期的信息论结果的基础上，为源编码建立了一种称为“自然类型选择”的机制，该机制适用于随机生成的代码，并被证明是渐近最优的。这一框架的基本扩展将致力于开发普遍适用的方法，从而为信息论本身以及其他应用领域（包括无线通信、内容交付、社交媒体、人工智能等）的强大学习技术做出贡献。从教育角度来看，该项目为研究生提供了一个亲身体验国际跨学科研究合作的培训机会，将理论深度与实际影响相结合。它还提供了广泛丰富课程的机会，并培养出具有高需求的能力和技能的有成就的研究人员和实践者。该项目将开发新的学习方法，采用通用自适应机制生成随机代码，旨在逐步实现未知源分布的最优性。在性能界限的理论分析和强大的优化方法方面，将沿着三个主要方向进行研究：i)扩展自然类型选择框架，以包含具有记忆的连续空间和源，利用连续字母的“参数类型”概念，这将扩大适用性，几乎适用于所有感兴趣的实际场景。ii)在机器学习中的应用，其中监督学习（例如，分类，回归）被重新表述为寻求从源学习的最小信息量的率失真问题，以便可以从随机码本中读取规定保真度的期望输出；并且，在无监督学习方面，“信息瓶颈”方法在自适应码本生成设置中被普遍重新制定。两者都将利用确定性退火的优化框架。iii)在通信中的应用，其中开发了随机机制以实现最佳信道输入适应，包括需要开发“分布式自然类型选择”框架的多用户通信的重要扩展。该项目是美国和以色列研究人员的合作成果，由两国科学基金会（BSF）为以色列研究人员提供资金。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

This project builds on the investigators' early information-theoretic results, establishing a mechanism called 'natural type selection' for source coding, which adapts a randomly generated code and is shown to be asymptotically optimal. Fundamental expansions of this framework will be pursued to develop universally applicable methodologies, and thereby yield contributions to information theory itself alongside powerful learning techniques in other application fields, including wireless communications, content delivery, social media, artificial intelligence, and others. From the educational perspective, the project offers a training opportunity for graduate students to experience, first-hand, an international and interdisciplinary research collaboration, which combines theoretical depth with practical impact. It further offers opportunities for extensive curriculum enrichment, and to produce accomplished researchers and practitioners with capacities and skills that are in high demand.This project will develop novel approaches to learning, which employ universal self-adapting mechanisms for random code generation, designed to asymptotically achieve optimality for unknown source distributions. Research will be pursued, in terms of both theoretical analysis of performance bounds and powerful optimization approaches, along three main thrusts: i) Extension of the natural type selection framework to encompass continuous spaces and sources with memory, leveraging the concept of "parametric type" for continuous alphabets, which would expand applicability to virtually all practical scenarios of interest. ii) Applications in machine learning, where supervised learning (e.g., classification, regression) is reformulated as the rate-distortion problem of seeking the minimal amount of information to be learned from a source such that a desired output at the prescribed fidelity can be read from a random codebook; and, on the unsupervised learning side, where the "information bottleneck" method is reformulated universally in a self-adapting codebook generation setting. Both will leverage the optimization framework of deterministic annealing. iii) Applications in communications where stochastic mechanisms are developed for optimal channel input adaptation, including an important extension to multi-user communications which requires the development of a "distributed natural type selection" framework. This project is a collaborative effort between researchers in the US and Israel, with funding for Israeli researchers provided by the Bi-National Science Foundation (BSF).This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.