ERI: Generative Adversarial Networks for Video Coding

ERI:用于视频编码的生成对抗网络

基本信息

  • 批准号:
    2138635
  • 负责人:
  • 金额:
    $ 19.62万
  • 依托单位:
  • 依托单位国家:
    美国
  • 项目类别:
    Standard Grant
  • 财政年份:
    2022
  • 资助国家:
    美国
  • 起止时间:
    2022-02-01 至 2025-01-31
  • 项目状态:
    未结题

项目摘要

This award is funded in whole or in part under the American Rescue Plan Act of 2021 (Public Law 117-2).Video coding is an important technology that compresses video signals to save transmission bandwidth and to provide Internet users with visually pleasing decoded videos. Inspired by recent breakthroughs in deep learning, convolutional neural networks have been increasingly exploited into video coding algorithms to provide significant coding gains compared to conventional approaches. Nevertheless, existing convolutional neural network-based video coding schemes tend to generate blurry decoded images which are inconsistent with human perception, and the high computational complexity of these schemes hinders their deployment on power-constrained and computation resource-limited devices, such as smart phones and tablets. Recently, the generative adversarial network demonstrated its capability of decoding sharp and photo-realistic images at low bit rates, but little research has investigated its potential for video compression. This project will develop generative adversarial network-based video coding systems to enhance the coding efficiency, meanwhile providing decoded videos with high perceptual quality. The project will also investigate low-complexity algorithms to reduce the power consumption and to accelerate the inference speed of the proposed video coding systems so that they are suitable for mobile and low-latency applications. The success of the project is expected to accelerate the economic growth of streaming video services to benefit people’s daily professional and entertainment activities. It will advance surveillance video services to enhance public safety in places such as airport, offices, highway, and road intersections. The research activities of the project will provide opportunities to train graduate and undergraduate students including minority and under-represented groups through theses research, senior design projects, as well as machine learning and artificial intelligence courses. The research results of the project will be showcased in a summer engineering seminar program to motivate high school students to pursue science and engineering majors in college.This project will address two problems: (1) How to leverage temporal correlations among video frames and explore scene dynamics in a generative adversarial network-based video coding architecture? Two approaches are proposed: a hierarchical predictive coding approach, and a spatial-temporal coding architecture based on 3-dimensional convolution. Since most existing generative adversarial network models are for still image compression, the success of this research will open the door to generative adversarial network-based coding systems for video coding professionals. (2) How to reduce the computational complexity of deep video coding networks? Despite the performance benefits of deep learning-based video coding tools, few of them are currently being adopted in real-world scenarios. This is due to the high computational complexity, slow inference speed and the large graphic processing unit memory requirements associated with deep network computation. To address this problem, the proposed research will develop algorithms to reduce the complexity, model size and model parameters of deep learning-based video coding models via separable convolution operations. The research results will accelerate the deployment of deep video coding models in real-world applications.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
该奖项的全部或部分资金来自《2021年美国救援计划法案》(公法117-2)。视频编码是一项重要的技术,它对视频信号进行压缩,以节省传输带宽,并为互联网用户提供视觉上令人愉快的解码视频。受最近深度学习突破的启发,卷积神经网络已越来越多地被用于视频编码算法,以提供与传统方法相比显著的编码增益。然而,现有的基于卷积神经网络的视频编码方案往往会产生与人类感知不一致的模糊解码图像,并且这些方案的高计算复杂性阻碍了它们在智能手机和平板电脑等功率受限和计算资源受限的设备上的部署。最近,生成性对抗性网络展示了它在低比特率下解码清晰和逼真的图像的能力,但很少有人研究它在视频压缩方面的潜力。该项目将开发基于生成性对抗性网络的视频编码系统,以提高编码效率,同时提供高感知质量的解码视频。该项目还将研究低复杂度算法,以降低功耗并加快所提出的视频编码系统的推理速度,使其适用于移动和低延迟应用。该项目的成功有望加速流媒体视频服务的经济增长,造福人们的日常专业和娱乐活动。它将推进监控视频服务,以加强机场、办公室、高速公路和道路交叉口等场所的公共安全。该项目的研究活动将通过论文研究、高级设计项目以及机器学习和人工智能课程,为包括少数族裔和代表性不足群体在内的研究生和本科生提供培训机会。该项目的研究成果将在暑期工程研讨会项目中展示,以激励高中生攻读大学理工科专业。该项目将解决两个问题:(1)如何利用视频帧之间的时间相关性,并在基于生成性对抗性网络的视频编码体系结构中探索场景动态?提出了两种编码方法:分层预测编码方法和基于三维卷积的空时编码结构。由于现有的生成性对抗网络模型大多是用于静态图像压缩,本研究的成功将为视频编码专业人员打开基于生成性对抗网络的编码系统的大门。(2)如何降低深度视频编码网络的计算复杂度?尽管基于深度学习的视频编码工具具有性能优势,但目前在真实场景中采用的视频编码工具很少。这是由于计算复杂性高、推理速度慢以及与深度网络计算相关的大图形处理单元内存需求。为了解决这个问题,提出的研究将通过可分离的卷积运算来开发算法来降低基于深度学习的视频编码模型的复杂度、模型规模和模型参数。这项研究成果将加速深度视频编码模型在现实世界应用中的部署。这一奖项反映了NSF的法定使命,并通过使用基金会的智力优势和更广泛的影响审查标准进行评估,被认为值得支持。

项目成果

期刊论文数量(6)
专著数量(0)
科研奖励数量(0)
会议论文数量(0)
专利数量(0)
A generative adversarial network for video compression
  • DOI:
    10.1117/12.2618714
  • 发表时间:
    2022-05
  • 期刊:
  • 影响因子:
    0
  • 作者:
    Pengli Du;Ying Liu;Nam Ling;Lingzhi Liu;Yongxiong Ren;M. Hsu
  • 通讯作者:
    Pengli Du;Ying Liu;Nam Ling;Lingzhi Liu;Yongxiong Ren;M. Hsu
Learned image compression with transformers
  • DOI:
    10.1117/12.2656516
  • 发表时间:
    2023-06
  • 期刊:
  • 影响因子:
    0
  • 作者:
    Tianma Shen;Y. Liu
  • 通讯作者:
    Tianma Shen;Y. Liu
Side Information Driven Image Coding for Machines
  • DOI:
    10.1109/pcs56426.2022.10018039
  • 发表时间:
    2022-12
  • 期刊:
  • 影响因子:
    0
  • 作者:
    Zhongpeng Zhang;Y. Liu
  • 通讯作者:
    Zhongpeng Zhang;Y. Liu
Generative Video Compression with a Transformer-Based Discriminator
  • DOI:
    10.1109/pcs56426.2022.10018030
  • 发表时间:
    2022-12
  • 期刊:
  • 影响因子:
    0
  • 作者:
    Pengli Du;Y. Liu;Nam Ling;Yongxiong Ren;Lingzhi Liu
  • 通讯作者:
    Pengli Du;Y. Liu;Nam Ling;Yongxiong Ren;Lingzhi Liu
A Survey of Efficient Deep Learning Models for Moving Object Segmentation
用于运动物体分割的高效深度学习模型综述
{{ item.title }}
{{ item.translation_title }}
  • DOI:
    {{ item.doi }}
  • 发表时间:
    {{ item.publish_year }}
  • 期刊:
  • 影响因子:
    {{ item.factor }}
  • 作者:
    {{ item.authors }}
  • 通讯作者:
    {{ item.author }}

数据更新时间:{{ journalArticles.updateTime }}

{{ item.title }}
  • 作者:
    {{ item.author }}

数据更新时间:{{ monograph.updateTime }}

{{ item.title }}
  • 作者:
    {{ item.author }}

数据更新时间:{{ sciAawards.updateTime }}

{{ item.title }}
  • 作者:
    {{ item.author }}

数据更新时间:{{ conferencePapers.updateTime }}

{{ item.title }}
  • 作者:
    {{ item.author }}

数据更新时间:{{ patent.updateTime }}

Ying Liu其他文献

Polymerization-Driven Self-Wrinkling on a Frozen Hydrogel Surface toward Ultra-Stretchable Polypyrrole-Based Supercapacitors
冷冻水凝胶表面聚合驱动的自皱,用于超可拉伸聚吡咯基超级电容器
  • DOI:
    10.1021/acsami.2c13829
  • 发表时间:
    2022
  • 期刊:
  • 影响因子:
    9.5
  • 作者:
    Yufeng Wang;Ying Liu;Zhengtao Wang;Dai Hai Nguyen;Chao Zhang;Tianxi Liu
  • 通讯作者:
    Tianxi Liu
Loop-Mediated Isothermal Amplification (LAMP): Potential Point-of-Care Testing for Vulvovaginal Candidiasis
环介导等温扩增 (LAMP):外阴阴道念珠菌病的潜在护理点测试
  • DOI:
  • 发表时间:
  • 期刊:
  • 影响因子:
    0
  • 作者:
    Meng Li;Xiangyu Jin;Qingyun Jiang;Hongbo Wei;Anni Deng;Zeyin Mao;Ying Wang;Zhen Zeng;Yifan Wu;Shuai Liu;Juhyun Kim;Xiaoqian Wang;Ying Liu;Jun Liu;Wenqi Lv;Leyang Huang;Q. Liao;Guoliang Huang;Lei Zhang
  • 通讯作者:
    Lei Zhang
p2 of RSV interacts with OsSGS3 and is a silencing suppressor
RSV 的 p2 与 OsSGS3 相互作用,是一种沉默抑制子
  • DOI:
  • 发表时间:
  • 期刊:
  • 影响因子:
    4.9
  • 作者:
    Jianguo Jia;Jianguo Wu;Zhao Han;Zhenguo Du;Liuyang Hu;Taiyun Wei;Qiying Lin;Ying Liu;Lianhui Xie;Zujian Wu;Donglai Xiao;Zhengjie Yuan
  • 通讯作者:
    Zhengjie Yuan
Innovation-oriented HRM, TMT reflexivity and organizational change in China: the moderated mediation effect of CEO leader mindfulness
中国创新导向的人力资源管理、TMT反身性与组织变革:CEO领导者正念的调节中介效应
  • DOI:
    10.1080/13602381.2022.2139058
  • 发表时间:
    2022-11
  • 期刊:
  • 影响因子:
    2.9
  • 作者:
    Lulu Zhou;Ying Liu;Tianshan Xue;Xiu Zhang
  • 通讯作者:
    Xiu Zhang
Postharvest UV‐C treatment of peach fruit: Changes in transcriptome profile focusing on genes involved in softening and senescence
桃果实采后 UV-C 处理:转录组谱的变化集中于参与软化和衰老的基因

Ying Liu的其他文献

{{ item.title }}
{{ item.translation_title }}
  • DOI:
    {{ item.doi }}
  • 发表时间:
    {{ item.publish_year }}
  • 期刊:
  • 影响因子:
    {{ item.factor }}
  • 作者:
    {{ item.authors }}
  • 通讯作者:
    {{ item.author }}

{{ truncateString('Ying Liu', 18)}}的其他基金

EAGER: Resolving the issue of pairing symmetry in Sr2RuO4
EAGER:解决 Sr2RuO4 中的配对对称性问题
  • 批准号:
    2312899
  • 财政年份:
    2023
  • 资助金额:
    $ 19.62万
  • 项目类别:
    Standard Grant
I-CORPS: Scalable Production of Polymeric Nanoparticles Encapsulating Hydrophobic Compounds
I-CORPS:封装疏水性化合物的聚合物纳米颗粒的可规模化生产
  • 批准号:
    1566113
  • 财政年份:
    2015
  • 资助金额:
    $ 19.62万
  • 项目类别:
    Standard Grant
CAREER: Understanding Nanoprecipitation - Scalable Production of Polymeric Nanoparticles Encapsulating Hydrophobic Compounds
职业:了解纳米沉淀 - 封装疏水性化合物的聚合物纳米颗粒的规模化生产
  • 批准号:
    1350731
  • 财政年份:
    2014
  • 资助金额:
    $ 19.62万
  • 项目类别:
    Standard Grant
Toroidal-spiral particles (TSPs) for co-delivery of multiple compounds of different sizes
用于共同递送多种不同尺寸化合物的环形螺旋颗粒 (TSP)
  • 批准号:
    1404884
  • 财政年份:
    2014
  • 资助金额:
    $ 19.62万
  • 项目类别:
    Standard Grant
EAGER: Preliminary Study on Novel self-assembled Toroidal-Spiral MicroParticles (TSMPs) for sustained release of therapeutic proteins and peptides: theory and experiments
EAGER:用于持续释放治疗性蛋白质和肽的新型自组装环形螺旋微粒(TSMP)的初步研究:理论和实验
  • 批准号:
    1039531
  • 财政年份:
    2010
  • 资助金额:
    $ 19.62万
  • 项目类别:
    Standard Grant
Materials World Network: Novel Physical Phenomena in Unusual Mesoscopic Superconductors
材料世界网络:异常介观超导体中的新物理现象
  • 批准号:
    0908700
  • 财政年份:
    2009
  • 资助金额:
    $ 19.62万
  • 项目类别:
    Standard Grant
US-France Cooperative Research: Search for Edge Currents and Domain Walls in SrRu0
美法合作研究:寻找SrRu0中的边缘电流和畴壁
  • 批准号:
    0340779
  • 财政年份:
    2004
  • 资助金额:
    $ 19.62万
  • 项目类别:
    Standard Grant
Experimental Studies of Nanoscopic Superconductors: Half-flux Quantum, Metallic State of Cooper Pairs, and the Berry's Phase
纳米超导体的实验研究:半通量量子、库珀对金属态和贝里相
  • 批准号:
    0202534
  • 财政年份:
    2002
  • 资助金额:
    $ 19.62万
  • 项目类别:
    Continuing Grant
Determination of the Exact Symmetry of the Pairing State in Sr2RuO4
Sr2RuO4 中配对态精确对称性的测定
  • 批准号:
    9974327
  • 财政年份:
    1999
  • 资助金额:
    $ 19.62万
  • 项目类别:
    Continuing Grant
CAREER: Mesoscopic Physics of Disordered Superconductors: An Arena for Research and Education
职业:无序超导体的介观物理:研究和教育的舞台
  • 批准号:
    9702661
  • 财政年份:
    1997
  • 资助金额:
    $ 19.62万
  • 项目类别:
    Continuing Grant

相似海外基金

SBIR Phase I: High Fidelity Climate Simulation Powered by Generative Adversarial Networks
SBIR 第一阶段:由生成对抗网络提供支持的高保真气候模拟
  • 批准号:
    2335370
  • 财政年份:
    2024
  • 资助金额:
    $ 19.62万
  • 项目类别:
    Standard Grant
Pure transformer encoder-based generative adversarial networks for molecular generation
用于分子生成的基于纯变压器编码器的生成对抗网络
  • 批准号:
    23KF0063
  • 财政年份:
    2023
  • 资助金额:
    $ 19.62万
  • 项目类别:
    Grant-in-Aid for JSPS Fellows
SOLARIS : Strengthening democratic engagement through value based generative adversarial networks
SOLARIS:通过基于价值的生成对抗网络加强民主参与
  • 批准号:
    10046757
  • 财政年份:
    2023
  • 资助金额:
    $ 19.62万
  • 项目类别:
    EU-Funded
Generative adversarial networks for demographic inferences of nonmodel species from genomic data
根据基因组数据对非模型物种进行人口统计推断的生成对抗网络
  • 批准号:
    NE/X009637/1
  • 财政年份:
    2023
  • 资助金额:
    $ 19.62万
  • 项目类别:
    Research Grant
GANCAT: Generative Adversarial Networks for CATegorization
GANCAT:用于分类的生成对抗网络
  • 批准号:
    EP/Y026489/1
  • 财政年份:
    2023
  • 资助金额:
    $ 19.62万
  • 项目类别:
    Fellowship
RI: Small: Optimal Transport Generative Adversarial Networks: Theory, Algorithms, and Applications
RI:小型:最优传输生成对抗网络:理论、算法和应用
  • 批准号:
    2327113
  • 财政年份:
    2023
  • 资助金额:
    $ 19.62万
  • 项目类别:
    Continuing Grant
Generative Adversarial Networks for MRI-driven Radiation Therapy
用于 MRI 驱动放射治疗的生成对抗网络
  • 批准号:
    489415
  • 财政年份:
    2023
  • 资助金额:
    $ 19.62万
  • 项目类别:
    Operating Grants
Latent Space Search for Adversarial Generative Networks for Sensitivity Quantification of Skilled Inspectors
对抗性生成网络的潜在空间搜索,用于熟练检查员的灵敏度量化
  • 批准号:
    23K11283
  • 财政年份:
    2023
  • 资助金额:
    $ 19.62万
  • 项目类别:
    Grant-in-Aid for Scientific Research (C)
Constructing Highly Accurate Supervised Outlier Detection Method by Quasiconformal Extension and Generative Adversarial Networks
通过拟共形扩展和生成对抗网络构建高精度监督异常值检测方法
  • 批准号:
    22K12050
  • 财政年份:
    2022
  • 资助金额:
    $ 19.62万
  • 项目类别:
    Grant-in-Aid for Scientific Research (C)
Upsampling of low-resolution/large-volume 3D tomographic images using generative adversarial neural networks applied to biological anthropology, medical imaging, and evolutionary biology
使用应用于生物人类学、医学成像和进化生物学的生成对抗神经网络对低分辨率/大容量 3D 断层扫描图像进行上采样
  • 批准号:
    571519-2021
  • 财政年份:
    2022
  • 资助金额:
    $ 19.62万
  • 项目类别:
    Alliance Grants
{{ showInfoDetail.title }}

作者:{{ showInfoDetail.author }}

知道了