Multimodal, Interpretable, and Interactive Machine Learning for Multimedia

点击翻译按钮获取中文摘要

Despite the widespread adoption of ML in many domains, some underlying issues reduce the scope and ubiquity of ML adoption, especially for multimedia signal processing, where signals can take many forms. The proposed research program will attempt at solving two key issues: 1) ML for multimodal multimedia signals, and 2) increasing interpretability and interactivity of multimedia signal processing with ML. Multimodal signal is prevalent in many application areas of ML, such as action recognition (camera, depth, inertial sensors) and computer-aided diagnosis (Ultrasound, MRI, CT, PET). The rise of Deep Learning (DL) has resulted in impressive performance on learning from a single modality. However, challenges remain in how to combine these modalities efficiently. Another issue is the black box nature of ML, particularly DL, due to its complexity. This is critical in sensitive domains such as medical, where a thorough understanding of the model is necessary to earn trust of healthcare professionals. A closely related issue is interactivity, which enables domain knowledge integration in an ML model through user feedback. Genuine human questions such as interest and relevance are inherently tricky for ML models to capture. Together, interpretability and interactivity can help in increasing the trust of ML. The long-term objective of this proposal is to develop tools and techniques for multimodal, interpretable, and interactive ML that can serve as an enabling technology for a broad range of applications while training HQP for digital media and medical imaging industries. The short-term objectives are to create: 1. A Multi-level Multimodal learning method with Convolutional Neural Networks that can combine different levels of data abstraction and perform end-to-end training through novel moment-gated fusion layers that preserve discriminative information while reducing the feature space dimension, and associated loss functions  that force capture of discriminative and correlated multimodal information; 2. Interactive and Interpretable (I2ML) through model agnostic explanation and interaction, where we extend our recently proposed model agnostic method  to generate global explanations, and provide a human-in-the-loop mechanism for manipulation of the underlying features to capture domain expertise; 3. Applications of the frameworks where we apply and validate the proposed multimodal and I2ML frameworks in two domains: 1) gesture recognition in AR, 2) computer-aided diagnosis. The expected outcome of the program is new techniques and tools for the ubiquitous adoption of ML. With a rising digital economy and Canada's emerging role as a global technology hub, the proposed program will benefit Canada immensely, opening new means of integration between ML and multimedia for a broad range of applications, thus creating both new creative exploration and technological opportunities.