Frameless video system: emulating the human visual system

无框视频系统：模拟人类视觉系统

• 批准号: RGPIN-2019-05551
• 负责人: Nasiopoulos, Panos
• 金额: $ 2.4万
• 依托单位: University of British Columbia
• 依托单位国家: 加拿大
• 项目类别: Discovery Grants Program - Individual
• 财政年份: 2021
• 资助国家: 加拿大
• 起止时间: 2021-01-01 至 2022-12-31
• 项目状态: 已结题
• 来源: https://www.nserc-crsng.gc.ca/ase-oro/Details-Detailles_eng.asp?id=738363
• 关键词: Frameless; video; system; emulating; human

The real world is analog and continuous in the most practical sense. The sound we hear has an amplitude variation over time, in a continuous and analog fashion. The light we sense through our eyes is also continuous both in amplitude and time, as the eye's retina receives a continuous flow of photons, sending that information to the brain through a chemical reaction. However, conventional video cameras cannot capture light information the way the human sees it, as they are limited to capturing discrete frames over time. Movies and videos are clever and sophisticated inventions that give the illusion of motion and realism to the human eyes, yet fooling them to believe they are seeing a faithful replica of the real world. The challenge of capturing video in discrete frames goes back to 1892, when Edison and Dickson developed a device called the Kinetoscope to view the resulting motion picture by playing back about 15 frames per second. It was until later that the addition of sound forced the frames to increase to 24 per second. Recently, these rates have significantly increased, but we continue treating video as a discrete sequence of images, imposing an unnecessary constraint on pixel information, dynamic range and precision, causing motion artifacts, compression challenges and an overall limited quality of experience. More importantly, this stroboscopic viewing is definitely tiresome to humans, as the eye and the brain have to provide the temporal integration necessary to perceive continuous motion. We believe that, considering the recent advances in video capturing and processing technologies, the time is ripe for a paradigm shift in the way we capture, process and display visual information, closely emulating the human visual system and its ability to perceive the continuity of the real world. Our long-term objective is to embark on a better understanding of the continuous time video representation and develop the framework and necessary technologies that will address visual quality, storage, delivery, and display requirements for frameless video, thus arriving at a more accurate rendition of the real world that is authentic and more pleasing to the eye. The implication to the digital media world is very significant. If frameless video shows promise, the broadcasting and entertainment industry, as well as the consumer electronics industry will have to embrace the opportunity to migrate to new ways to deliver entertainment, news and social media content to consumers. We suspect a similar reaction will have to occur for the computer vision community, driving significant new market opportunities across the digital media industries which represent some of the fastest growing sectors in the world, with a market growth rate of 25% and global sales forecast to hit US$215 billion by 2021.

现实世界在最实际的意义上是模拟的和连续的。我们听到的声音有一个振幅随时间的变化，以连续和模拟的方式。我们通过眼睛感受到的光在振幅和时间上也是连续的，因为眼睛的视网膜接收到连续的光子流，并通过化学反应将这些信息发送给大脑。然而，传统的摄像机不能像人类看到的那样捕捉光信息，因为它们被限制在随着时间的推移捕捉离散的帧。电影和录像是聪明而复杂的发明，它们给人眼带来运动和现实主义的错觉，但又欺骗他们相信他们看到的是真实世界的忠实复制品。捕捉离散帧视频的挑战可以追溯到1892年，当时爱迪生和迪克森发明了一种叫做活动放映机的设备，以每秒播放15帧的速度来观看生成的电影。直到后来，声音的加入才迫使帧数增加到每秒24帧。最近，这些速率显着增加，但我们继续将视频视为图像的离散序列，对像素信息，动态范围和精度施加不必要的限制，导致运动伪影，压缩挑战和整体体验质量有限。更重要的是，这种频闪观看对人类来说绝对是令人厌烦的，因为眼睛和大脑必须提供必要的时间整合来感知连续的运动。我们认为，考虑到视频捕捉和处理技术的最新进展，我们捕捉、处理和显示视觉信息的方式发生范式转变的时机已经成熟，我们可以密切模仿人类视觉系统及其感知现实世界连续性的能力。我们的长期目标是着手更好地理解连续时间视频表示，并开发框架和必要的技术，以解决无帧视频的视觉质量，存储，传输和显示要求，从而达到更准确的真实世界的再现，更真实，更赏心悦目。这对数字媒体世界的影响是非常重大的。如果无框视频显示出前景，广播和娱乐行业以及消费电子行业将不得不抓住机会，转向向消费者提供娱乐、新闻和社交媒体内容的新方式。我们怀疑计算机视觉社区也会出现类似的反应，这将为数字媒体行业带来重大的新市场机会，数字媒体行业是世界上增长最快的行业之一，市场增长率为25%，到2021年全球销售额预计将达到2150亿美元。