SMARTLENS: Closing the gap between real and virtual filming through smart high end lens calibration

SMARTLENS项目利用Mo-Sys在工作室质量虚拟制作方面的领导地位，为负担得起的高质量视频制作打开了广阔的新市场。视频无处不在，占所有互联网流量的82%，取代了广告、教育、新闻等领域的其他媒体形式；吸引了创意行业的专业人士探索和参与新形式的内容制作(GB 268 Bn)。虚拟制作技术使电影制作人能够将摄像机捕获的图像与计算机生成的元素相结合。SMARTLENS项目创建了一种新的自动化基本镜头校准方法，实现了显著的成本节约(和碳节约)，因为制作团队不再需要前往偏远地点拍摄困难的镜头。这是一个建立在与西蒙·朱利耶教授(UCL/TMI)长期合作基础上的合作项目，创新了基于人工智能的自动化镜头校准。所有的镜头都会以某种方式扭曲光路：改变焦距会改变镜头的视野，而现实世界中的直线在图像平面上看起来并不一定像直线。没有考虑到这种失真意味着虚拟图形与真实世界不对齐，并且任何将图形锚定在真实世界中的概念都将丢失。传统的镜头校准技术缓慢而繁琐，需要操作员手动重复测量目标和相机移动。SMARTLENS开发了一套算法和技术，将校准方法自动化。首先，选择合适的透镜参数模型。传统的计算机视觉模型不能描述诸如散焦图像的深度相关的径向失真之类的效果。因此，我们将从摄影测量文献中选择合适的模型。其次，我们将开发后端优化技术，使参数与合适的数据相匹配。第三，我们将调查各种校准目标的使用，包括点和线，以确定哪些在各种操作条件下最稳健。最后，我们将研究提供最佳性能的合适实践。Mo-Sys是当今最大的虚拟制作技术提供商，拥有大量的IP组合和成熟的高端客户基础(索尼、华纳兄弟、迪士尼、Netflix、BBC、NHK、福克斯、ESPN、天空、CNN)。从BBC的《每日一赛》到《地心引力》和《少年派的奇幻漂流》等电影，不一而足。通过这个项目，这个合作现在继续创新和扩展镜头校准和跟踪系统，

The SMARTLENS project leverages Mo-Sys' leadership in studio-quality virtual production and opens up broad new markets to affordable high-quality video production.Video is ubiquitous, comprising 82% of all Internet traffic and displacing other forms of media in advertising, in education, news and beyond; attracting professionals from across the creative industries (£268 Bn) to explore and participate in new forms of content production.'Virtual Production' technology allows filmmakers to combine images captured on camera with computer-generated elements. The SMARTLENS project creates a new automated method of essential lens calibration, allowing significant cost-saving (and carbon-saving), as production teams no longer need to travel to remote locations to capture difficult shots.This is a collaborative project building on long-time collaboration with Professor Simon Julier (UCL/TMI), innovating novel AI-based automated lens calibration. All lenses distort the path of light in some way: changing the focal length changes the field of view of the lens, and straight lines in the real world do not necessarily look like straight lines in the image plane. Failure to account for this distortion means that the virtual graphics do not line up with the real world, and any notion of the graphics being anchored in the real world is lost. Conventional lens calibration techniques are slow and cumbersome, requiring the operator to manually measure targets and camera movements repeatedly.SMARTLENS develops a set of algorithms and techniques which will automate the calibration approach. First, a suitable model of lens parameters will be chosen. Conventional computer vision models do not describe effects such as depth-dependent radial distortion for out of focus images. Therefore, we will chose a suitable model from the photogrammetry literature. Second, we will develop the back end optimisation techniques which will fit the parameters to suitable data. Third, we will investigate the use of various kinds of calibration targets, including points and lines to see which ones are most robust over the range of operating conditions. Finally, we will investigate suitable practices which give the best performance.Mo-Sys is the largest virtual production technology provider today, with a substantial IP portfolio and a sophisticated high-end customer base (Sony, Warner Bros, Disney, Netflix, BBC, NHK, Fox, ESPN, Sky, CNN). Credits range from BBC's Match of the Day to films such as Gravity and Life of Pi. Through this project, this collaboration now continues to innovate and extend lens calibration and tracking systems,