I-Corps: Multi-Cue Facial Restoration (McFAR) for Recognition and Identification

这个i-Corps项目的更广泛的影响/商业潜力是开发一种多线索正脸恢复和识别系统，该系统可被安全当局利用或部署在监视设施上，以合成可疑肖像并利用有限的信息进行身份匹配。最近，在公共安全和安保领域越来越多地采用面部监控，这是一个重要因素，在市场上获得了显著的吸引力。随着国家边境、机场、海港和公共交通枢纽面临威胁，对先进的安全系统的需求很高。监控摄像头的视频通常是自动记录和分析的。由于强大的深度学习模型，视频人脸识别的性能得到了显著的提高，但在现实世界中，由于环境的限制，监控摄像头提供的分辨率有限，只能捕获目标嫌疑人的侧视。对低分辨率、非正面人脸的识别仍然是一个挑战。大多数现有的人脸识别算法都假设人脸是高分辨率的、接近正面的，但对于低分辨率的侧视图像却不能提供足够的性能。该技术利用多个方面的信息进行更忠实的人脸识别和识别，为安全应用提供了一种新的选择。该I-Corps项目基于一个多线索正脸恢复系统的开发，该系统提供了一个识别和识别框架，以从低分辨率、侧视人脸和目击者提供的叙事描述合成保持身份的高分辨率正脸图像。该技术通过一个超分辨率集成网络利用人脸图像的一系列极端姿势来合成高分辨率的正面化人脸。它的目标是识别由监控系统提供的多个线索以及目击者描述的低分辨率极端姿势人脸。为了提高学习表示的区分能力，引入了类内和类间约束来惩罚冗余特征。不使用朴素的融合方法，而是在生成模型中使用正交正则化来进行最优训练并学习更大跨度的综合表示。目击者可以提供面部的叙述性描述，该描述被馈送到语言编码器中并被转换为属性级表示。具有空间约束的描述引导的人脸编辑网络通过利用基于转换器的语言编码器进行图像翻译，使得能够编辑输入图像的属性级和几何内容。人脸编辑网络利用编码的特征并修改具有各种属性的正面化人脸，以生成用于高保真图像匹配的精细化正面人脸。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

The broader impact/commercial potential of this I-Corps project is the development of a multi-cue frontal face restoration and identification system that may be utilized by security authorities or deployed on surveillance facilities to synthesize suspect portraits and conduct identity matching with limited information. Recently, the rising adoption of facial surveillance in public safety and security is a vital factor gaining significant tractions in the market. With the threats to national borders, airports, seaports and public transportation hubs, advanced security systems are in high demand. Surveillance camera videos are often recorded and analyzed automatically. Video face recognition performance has been significantly boosted due to the powerful deep learning models but in the real-world, surveillance cameras provide limited resolution and only capture side-views of the target suspects due to limitations of the environment. Recognition with low-resolution, non-frontal faces remains a challenge. Most existing face recognition algorithms assume high-resolution, near-frontal faces yet cannot provide sufficient performance with low-resolution, side-view images. The proposed technology utilizes multiple aspects of information to conduct a more faithful face identification and recognition, which provides a novel option for security applications.This I-Corps project is based on the development of a multi-cue frontal face restoration system that provides a recognition and identification framework to synthesize identity-preserving, high-resolution frontal face images from low-resolution, side-view faces and a narrative description provided by eyewitnesses. The proposed technology utilizes a series of extreme poses of face images via a super-resolution integrated network to synthesize high-resolution frontalized faces. It aims to recognize low-resolution extreme-pose faces with multiple cues provided by the surveillance system as well as witness descriptions. To improve the discriminative ability of learning representation, intra- and inter-class constraints are imposed to penalize redundant features. Instead of employing naive fusion methods, orthogonal regularization is used in a generative model for optimal training and to learn a comprehensive representation of broader spans. Eyewitnesses may provide narrative description of the face that is fed into a language encoder and converted to attribute-level representations. A description-guided face editing network with spatial constraints enables the ability of editing both attribute-level and geometric contents of input images by leveraging transformer-based language encoder for image translation. The face editing network takes the encoded features and modifies the frontalized faces with various attributes to generate refined frontal faces for high fidelity image matching.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.