Voice and Video-based Service Provisioning on the Cloud

在美国，1.271亿用户在他们的汽车上使用语音控制。Voicebot.ai进行的调查显示，随着时间的推移，智能扬声器在英国和德国的使用频率相当高，这表明该技术已被明显采用。虽然深度学习极大地提高了自然语言理解、翻译和生成的水平，但在适应听者的认知能力、以情感和同情心回应、理解语境以及从非结构化和结构化的知识源中实时提供最佳回应方面，还需要做更多的工作来达到人类层面的沟通。我们建议构建一个语音和视频通信框架来提供云服务，并解决以下研究挑战：a)开发多流语音和视频数据融合，以提高机器认知；b)生成适应用户语音、语调、词汇和语速的语音响应，以便所有年龄组和多样性的用户更好地采用该技术；以及c)使用结合内存和存储的结构化和非结构化数据的分层知识库，在云上实现基于规则的实时和基于问答的信息检索。我们在IBM Watson语音和视频通信系统、基于对话和规则的问答(QA)系统、基于信息检索的三阶段实时问答系统以及使用内存图聚类技术进行分布式知识管理的分层多层知识管理系统方面进行了广泛的工作。然而，这些工作都没有解决基于分层混合知识库的云服务配置的实时语音和视频通信，这是我们打算在拟议的研究中解决的问题。云上的实时服务提供必须解决语音和视频数据和知识库的安全方面问题，根据用户请求执行第三方语音和视频处理工具和应用程序，使网络面临风险，最重要的是需要针对特定领域的语音、视频和文本数据培训强大的深度学习模型，以增强语言理解和响应生成。必须使用独立的云服务器来测试系统的实时可用性，并允许公众访问，而不会使定制的分布式知识管理系统危及学术网络。我们要求配备高内存GPU、防火墙和存储的计算服务器，以用于深度学习模型的强大培训，创建可公开访问的云服务器来解决安全方面的问题，并在真人和研究合作者的帮助下验证系统的学生支持、健康数据分析和知识服务。在疫情期间，在云上提供基于语音和视频的服务成为一种可行的替代方案，我们的研究将扩展这项技术，使其能够应用于加拿大的家庭护理和支持服务。

In US, 127.1 million users are using voice control in their vehicles. Surveys conducted by Voicebot.ai show that smart speaker use frequency over time in the UK and Germany is quite high indicating a clear adoption of the technology. Although deep learning has greatly advanced the state-of-the-art in natural language understanding, translation and generation, more work is needed to reach human level communication in terms of adapting to the listeners cognitive abilities, responding with emotion and compassion, understanding context, and deliver the best response from both unstructured and structured knowledge sources in real time. We propose to build a voice and video communication framework to provide cloud services and address the following research challenges: a) develop multi-stream voice and video data fusion to allow improved machine cognition, b) generate voice responses adapting to users' voice, tone, vocabulary, and speed of speech for effective communication and better adoption of the technology by users of all age group and diversity, and c) enable real time rule based and information retrieval based question answering on the cloud using a hierarchical knowledge base combining in-memory and stored structured and unstructured data. We did extensive work on the IBM Watson voice and video communication system, a dialogue and rule-based question answering (QA) system, an information retrieval based three-phase real time QA system, and a hierarchical multilayer knowledge management system using in-memory graph clustering techniques for distributed knowledge management. However, none of these work addressed voice and video communication in real time for cloud service provisioning based on a hierarchical hybrid knowledge base, which we intend to address in the proposed research. Real time service provisioning on the cloud must address security aspects of voice and video data and the knowledge base, execute 3rd party voice and video processing tools and applications putting the network at risk based on the users' request, and most importantly require robust deep learning models trained on domain specific voice, video, and text data to enhance language understanding and response generation. A stand-alone cloud server must be used to test the real time usability of the system and to enable public access without risking the academic network with a custom distributed knowledge management system. We request a compute server with high memory GPUs, firewall, and storage for robust training of deep learning models, creating a publicly accessible cloud server to address security aspects, and validating the system for student support, health data analytics and knowledge services with the help of real people and research collaborators. Voice and video-based service provisioning on the cloud became a viable alternative during the pandemic and our research will extend the technology to enable its applications to home care and support services in Canada.