Voice and Video-based Service Provisioning on the Cloud

在美国，有1.271亿用户在他们的车辆中使用语音控制。Voicebot进行的调查。ai显示，随着时间的推移，智能音箱在英国和德国的使用频率相当高，这表明该技术得到了明显的采用。尽管深度学习在自然语言理解、翻译和生成方面取得了很大的进步，但要达到人类水平的沟通，还需要做更多的工作，包括适应听众的认知能力、用情感和同情做出反应、理解上下文，以及实时地从非结构化和结构化的知识来源中提供最佳反应。我们建议构建一个语音和视频通信框架，提供云服务，并解决以下研究挑战：A)开发多流语音和视频数据融合，以提高机器认知能力；b)生成适应用户语音、语气、词汇和语速的语音响应，以实现有效沟通，并使所有年龄组和多样性的用户更好地采用该技术；c)使用结合内存和存储的结构化和非结构化数据的分层知识库，在云上实现基于规则和基于信息检索的实时问答。我们在IBM Watson语音和视频通信系统、基于对话和规则的问答（QA）系统、基于信息检索的三相实时问答系统以及使用内存图聚类技术进行分布式知识管理的分层多层知识管理系统上做了大量的工作。然而，这些工作都没有解决基于分层混合知识库的云服务提供的实时语音和视频通信，我们打算在拟议的研究中解决这个问题。云上的实时服务提供必须解决语音和视频数据和知识库的安全问题，执行基于用户请求的第三方语音和视频处理工具和应用程序，将网络置于风险之中，最重要的是需要经过特定领域语音、视频和文本数据训练的强大深度学习模型，以增强语言理解和响应生成。必须使用独立的云服务器来测试系统的实时可用性，并使公众能够访问，而不会因定制的分布式知识管理系统而危及学术网络。我们需要一个具有高内存gpu、防火墙和存储的计算服务器，用于深度学习模型的强大训练，创建一个可公开访问的云服务器来解决安全问题，并在真人和研究合作者的帮助下验证学生支持、健康数据分析和知识服务系统。在疫情期间，基于云的语音和视频服务供应成为一种可行的替代方案，我们的研究将扩展该技术，使其能够应用于加拿大的家庭护理和支持服务。

In US, 127.1 million users are using voice control in their vehicles. Surveys conducted by Voicebot.ai show that smart speaker use frequency over time in the UK and Germany is quite high indicating a clear adoption of the technology. Although deep learning has greatly advanced the state-of-the-art in natural language understanding, translation and generation, more work is needed to reach human level communication in terms of adapting to the listeners cognitive abilities, responding with emotion and compassion, understanding context, and deliver the best response from both unstructured and structured knowledge sources in real time. We propose to build a voice and video communication framework to provide cloud services and address the following research challenges: a) develop multi-stream voice and video data fusion to allow improved machine cognition, b) generate voice responses adapting to users' voice, tone, vocabulary, and speed of speech for effective communication and better adoption of the technology by users of all age group and diversity, and c) enable real time rule based and information retrieval based question answering on the cloud using a hierarchical knowledge base combining in-memory and stored structured and unstructured data. We did extensive work on the IBM Watson voice and video communication system, a dialogue and rule-based question answering (QA) system, an information retrieval based three-phase real time QA system, and a hierarchical multilayer knowledge management system using in-memory graph clustering techniques for distributed knowledge management. However, none of these work addressed voice and video communication in real time for cloud service provisioning based on a hierarchical hybrid knowledge base, which we intend to address in the proposed research. Real time service provisioning on the cloud must address security aspects of voice and video data and the knowledge base, execute 3rd party voice and video processing tools and applications putting the network at risk based on the users' request, and most importantly require robust deep learning models trained on domain specific voice, video, and text data to enhance language understanding and response generation. A stand-alone cloud server must be used to test the real time usability of the system and to enable public access without risking the academic network with a custom distributed knowledge management system. We request a compute server with high memory GPUs, firewall, and storage for robust training of deep learning models, creating a publicly accessible cloud server to address security aspects, and validating the system for student support, health data analytics and knowledge services with the help of real people and research collaborators. Voice and video-based service provisioning on the cloud became a viable alternative during the pandemic and our research will extend the technology to enable its applications to home care and support services in Canada.