SMARTLENS: Closing the gap between real and virtual filming through smart high end lens calibration

SMARTLENS项目利用Mo-Sys在工作室质量虚拟制作方面的领导地位，为负担得起的高质量视频制作开辟了广阔的新市场。视频无处不在，占所有互联网流量的82%，并取代广告，教育，新闻等其他形式的媒体;吸引来自创意行业的专业人士（2680亿英镑）探索和参与新形式的内容制作。“虚拟制作”技术允许电影制片人将相机拍摄的图像与计算机生成的元素结合起来。SMARTLENS项目创建了一种新的自动化基本透镜校准方法，可以显著节省成本（和碳排放），因为制作团队不再需要前往偏远地区拍摄困难的镜头。这是一个与Simon Julier教授（UCL/TMI）长期合作的合作项目，创新了基于人工智能的新型自动透镜校准。所有的透镜都会以某种方式扭曲光的路径：改变焦距会改变透镜的视野，而真实的世界中的直线在像平面上看起来不一定是直线。不考虑这种失真意味着虚拟图形不与真实的世界对齐，并且图形被锚定在真实的世界中的任何概念都丢失了。传统的透镜校准技术缓慢而繁琐，需要操作员反复手动测量目标和相机运动。SMARTLENS开发了一套算法和技术，可以自动执行校准方法。首先，选择合适的透镜参数模型。传统的计算机视觉模型不描述诸如失焦图像的深度相关径向失真的效果。因此，我们将选择一个合适的模型从摄影测量文献。其次，我们将开发后端优化技术，使参数适合合适的数据。第三，我们将研究各种校准目标的使用，包括点和线，以确定哪些在操作条件范围内最鲁棒。Mo-Sys是当今最大的虚拟制作技术提供商，拥有丰富的IP组合和成熟的高端客户群（索尼，华纳兄弟，迪士尼，Netflix，BBC，NHK，Fox，ESPN，Sky，CNN）。从英国广播公司的《每日一赛》到《地心引力》和《少年派的奇幻漂流》等电影，都有获奖名单。通过这个项目，这种合作现在继续创新和扩展透镜校准和跟踪系统，

The SMARTLENS project leverages Mo-Sys' leadership in studio-quality virtual production and opens up broad new markets to affordable high-quality video production.Video is ubiquitous, comprising 82% of all Internet traffic and displacing other forms of media in advertising, in education, news and beyond; attracting professionals from across the creative industries (£268 Bn) to explore and participate in new forms of content production.'Virtual Production' technology allows filmmakers to combine images captured on camera with computer-generated elements. The SMARTLENS project creates a new automated method of essential lens calibration, allowing significant cost-saving (and carbon-saving), as production teams no longer need to travel to remote locations to capture difficult shots.This is a collaborative project building on long-time collaboration with Professor Simon Julier (UCL/TMI), innovating novel AI-based automated lens calibration. All lenses distort the path of light in some way: changing the focal length changes the field of view of the lens, and straight lines in the real world do not necessarily look like straight lines in the image plane. Failure to account for this distortion means that the virtual graphics do not line up with the real world, and any notion of the graphics being anchored in the real world is lost. Conventional lens calibration techniques are slow and cumbersome, requiring the operator to manually measure targets and camera movements repeatedly.SMARTLENS develops a set of algorithms and techniques which will automate the calibration approach. First, a suitable model of lens parameters will be chosen. Conventional computer vision models do not describe effects such as depth-dependent radial distortion for out of focus images. Therefore, we will chose a suitable model from the photogrammetry literature. Second, we will develop the back end optimisation techniques which will fit the parameters to suitable data. Third, we will investigate the use of various kinds of calibration targets, including points and lines to see which ones are most robust over the range of operating conditions. Finally, we will investigate suitable practices which give the best performance.Mo-Sys is the largest virtual production technology provider today, with a substantial IP portfolio and a sophisticated high-end customer base (Sony, Warner Bros, Disney, Netflix, BBC, NHK, Fox, ESPN, Sky, CNN). Credits range from BBC's Match of the Day to films such as Gravity and Life of Pi. Through this project, this collaboration now continues to innovate and extend lens calibration and tracking systems,