Optimised Big Data Processing for a Pro-active Information System

摘要：在大数据时代，随着数据量的不断增加，信息生产与用户信息需求的时空分离变得更加复杂，具有动态和可扩展通信基础设施的动态大规模信息系统的必要性显而易见。这类系统的例子包括在线股票报价、互联网游戏、传感器网络、金融工作流程、流程自动化、运输、算法交易、数据中心平台和业务流程。Moshfeghi博士有一个新的激进想法，他的目标是开发主动信息系统。他提出的主动式信息系统将使这类系统能够预见用户的信息需求，并主动实现这些需求，即与之匹配的相关信息将在不提交任何查询的情况下提供给用户。在时间至关重要的情况下，这样的制度更是至关重要。由于用户必须处理的信息量如此之大，而且可能必须迅速做出决定，只有自主系统发起的信息检索过程(取代用户)才能提供时间关键的信息。主动信息系统是一种在用户的日常生活选择、与福利、能源、健康、交通、金融、贸易和安全等各个方面的挑战和问题方面具有几个深远优势的想法。这是一个根本性的进步。他的ACM-SIGIR 2016年的论文获得了信息检索领域最佳论文奖，他的ACM-WWW 2018和ACM-WWW 2019年的论文是数据库和信息系统领域最重要的会议，这证明了主动信息系统的潜力。这个博士项目是他未来十年正在进行的项目的一部分，也是他的校长奖学金计划的基础。背景：效率是这个计划的一个重要方面。我们正在通过优化性能来解决效率问题，例如通过使用进化算法。在进化算法方面有大量的研究，例如遗传算法、遗传、变异、选择等，它们在过去已经被提出用来生成优化问题的解决方案。并行处理组件负责适当地利用横向扩展/向上扩展系统的基础设施。目前在Hadoop、Spark和Mahout等大数据处理技术方面的进步将有助于满足此类系统固有的高可伸缩性要求。最后，系统需要以实时方式将最后一组信息提供给感兴趣的用户。目标：该项目将开发基础和核心技术，以提供高性能的处理基础设施，以优化情况识别、潜在信息需求和数据的高级匹配。在本项目的范围内，学生将研究高效大数据算法的开发，用于存储、处理、链接和分析项目中涉及的大型异类数据源。这一方向与数据到知识优先领域非常契合，与斯特拉斯克莱德大学在大数据和更广泛的数据科学方面的战略方向非常契合。目标：该项目有三个主要目标。与该模型相关的第一个目标是识别和评估适用于主动信息系统的(例如，进化的)算法。第二个目标是开发新颖和创新的大数据科学处理技术(例如，基于Spark/Mahout)，能够迅速处理此类系统中包含的海量数据。最终目标是调查与潜在信息需求匹配的数据是如何传递给用户的。这一点尤其重要，因为用户可能甚至不知道他们收到此类信息的原因。

Summary: In the era of Big Data with the ever-increasing volumes of data and the added complexity of the space-time decoupling of information production versus users with Information Needs, the necessity for dynamic large-scale information systems with dynamic and scalable communication infrastructures is apparent. Examples of such systems are online stock quotes, Internet games, sensor networks, financial workflows, process automation, transportation, algorithmic trading, datacentre platforms and business processes. Dr Moshfeghi has a new and radical idea where he aims to develop proactive information systems. His idea of proactive information systems will allow such systems to foresee users' information Needs and proactively realise them, in the sense that relevant information matching it will be delivered to users without any submitted queries. Such a system is even more crucial for situations where time is of the essence. As the amount of information which users must deal with can be so large, and decisions may have to be made promptly, only an autonomic system-initiated Information Retrieval process (replacing users) may provide the time-critical information.A pro-active information system is an idea with several far-reaching advantages in users' everyday life choices, challenges and problems relating to various aspects such as well-being, energy, health, transportation, finance, trading and safety, etc. This is a fundamental step forward. The potential of proactive information systems was demonstrated by his ACM-SIGIR 2016 paper which won the best paper award in the top-tear conference in Information Retrieval as well as his ACM-WWW 2018 and ACM-WWW 2019 papers which is the most important conference in Databases and Information Systems. This PhD project is part of his ongoing project for the next ten years and underpins his chancellor's fellowship proposal.Background: Efficiency is an important aspect of this proposal. We are addressing the efficiency problem by optimising performance, e.g. by using evolutionary algorithms. There is a wealth of research in evolutionary algorithms, e.g. genetic algorithms, inheritance, mutation, selection, etc. which has been proposed in the past to generate solutions to optimisation problems. The Parallel Processing Component is responsible for exploiting scaling-out/up system's infrastructure appropriately. Current advances in big data processing techniques such as Hadoop, Spark, and Mahout will facilitate meeting the high scalability requirements inherent in such a system. Finally, the system is required to deliver the final set of information to the interested users in a real-time fashion. Aims: This project will develop the underlying and core technology to provide a high-performing processing infrastructure to optimise the identification of situations, latent Information Needs, and the high-level matching of data. Within the scope of this project, the student will be investigating the development of efficient big data algorithms for storage, processing, linking, and analysing large heterogeneous data sources involved in the project. This direction fits well with the Data to Knowledge priority area, which fits perfectly with the University of Strathclyde's strategic directions on big data and more broadly data science. Objectives: This project has three main objectives. The first objective associated with this model is to identify and evaluate suitable (e.g., evolutionary) algorithms for a pro-active information system. The second objective is to develop novel and innovative big data science processing techniques (e.g., based on Spark/Mahout) that can process the vast amount of data incorporated in such systems promptly. The final objective is to investigate how the data, matching latent Information Need, are delivered to users. This is particularly important since users might not be even aware of the reason as to why they have received such information.