Telecommunications and Computer Vision Convergence Tools for Research Infrastructures (CONVERGE)

电信和计算机视觉已经发展成为独立的科学领域。这种情况预计将随着无线通信的出现而改变，无线电的特点是视距范围，可以从视觉数据中受益，以预测无线信道动态。在基于无线电成像的帮助下，计算机视觉应用也将变得更加强大。这个新的联合研究领域依赖于无线通信、计算机视觉、传感和机器学习，它具有很高的创新潜力，因为它可以实现大量的创新应用，并且在欧洲可以获得相关的专业知识。然而，只有在有足够的研究基础设施和工具的情况下，才能评估这一新领域的全部潜力。CONVERGE项目的主要目标是开发一套创新的工具集，该工具集遵循“从视图到通信，从通信到视图”的座右铭。该工具集是世界首创，包括视觉辅助大型智能表面、视觉辅助固定和移动基站、视觉无线电模拟器和3D环境建模器，以及用于无线电信号、视频流、RF传感和交通跟踪等多模态数据的机器学习算法。该工具集将部署到7个RIs中，主要与ESFRI的SLICES-RI保持一致，并提高其竞争力。CONVERGE还将为科学界提供利用RIs工具集获得的实验和模拟数据的开放数据集，通过解决相关的6G垂直领域来满足科学和工业要求，增强相关公司的竞争力，将欧洲的影响力扩展到世界公认的RIs，使创建新的RIs成为可能，为开发新的环境友好型工具做出贡献，并帮助欧盟应对其社会挑战。

Telecommunications and computer vision have evolved as separate scientific areas. This is envisioned to change with the advent of wireless communications with radios characterised by line-of-sight ranges which could benefit from visual data to predict the wireless channel dynamics. Computer vision applications will also become more robust if helped by radio-based imaging. This new joint research field relies on wireless communications, computer vision, sensing and machine learning, and it has a high innovation potential because of the large domain of innovative applications it enables and the relevant know-how available in Europe. However, the full potential of this new area can only be evaluated if adequate Research Infrastructures (RI) and tools are available. The main objective of the CONVERGE project is the development of an innovative toolset aligned with the motto “view-to-communicate and communicate-to-view”. This toolset is a world-first and consists of vision-aided large intelligentsurfaces, vision-aided fixed and mobile base stations, a vision-radio simulator and 3D environment modeler, and machine learning algorithms for multimodal data including radio signals, video streams, RF sensing, and traffic traces. This toolset will be deployed into 7 RIs mostly aligned with the ESFRI SLICES-RI and improve their competitiveness. CONVERGE will also provide the scientific community with open datasets of experimental and simulated data obtained with the toolset in the RIs, meet scientific and industrial requirements by addressing relevant 6G verticals, enhance the competitiveness of the involved companies, extend the European influence to world-wide recognised RIs, enable the creation of new RIs, contribute to the development of new environment-friendly tools, and help European Union to address its societal challenges.