Design and implementation of big complex semantic data management system

在大数据时代，如何及时、高效地有效存储、管理、检索和分析如此大规模的复杂数据是一个重大挑战。本研究的主要目标是通过结合和扩展并行数据库系统、MapReduce计算平台以及我们之前在复杂语义数据库模型信息网络模型（Information Networking model， INM）和INM- dbms上的工作，设计和实现一个具有成本效益和高度可扩展的数据库管理系统，用于大型复杂语义数据的存储、查询和分析。并行数据库系统是在多处理器系统上实现的具有高度连通性的数据库管理系统（DBMS）。它的特点是使用定义良好的模式进行数据建模、具有高抽象级别的声明性查询语言、复杂的查询优化器和支持高效执行策略的丰富运行时环境。MapReduce计算范式由谷歌发起，由开源的Hadoop普及，是一种经济高效的分布式数据存储和处理系统，它建立在连接高带宽网络的低成本商用机器的大型集群上。近年来，它得到了工业界和研究界的广泛关注。INM-DBMS是一个复杂的语义数据库管理系统，具有层次和复合二进制和更高程度的关系及其组合，内置语义的一致性和完整性约束的各种关系，和丰富的演绎和活动规则。它是一种简洁而富有表现力的语言，由信息定义语言（IDL）、信息操作语言（IML）和信息查询语言（IQL）三部分组成。IDL和IML提供了强大的结构来表达与各种关系相关的丰富语义和完整性约束。声明性查询语言IQL有效地集成了数据库、XML和逻辑编程语言中的许多有用特性，如逻辑变量、隐式存在和非存在量化、显式通用量化、作为失败的否定、树表达式等。它可以探索对象的自然网络结构，以简洁、自然、紧凑的方式提取和构建有意义的结果。我们将首先对INM数据模型进行扩展，使其适应无模式和半结构化，然后在此基础上实现一个大型语义数据库管理系统。系统将由两层组成，以实现高并发性。操作层负责定义、操作和查询，而数据层负责数据存储。在操作层，通过散列函数将对象均匀地分配到存储节点，通过INM MapReduce库接口将查询任务分析解释为并行任务，实现分区均衡和分区并行。在数据层，将结构化的语义数据分区存储在各个节点中，保证了动态的可扩展性。在方法部分，划分了四种节点来执行不同的任务。节点之间相互独立，并与物料机分离，使得系统在物理资源分配方面具有很高的可扩展性和极强的容错能力。我们开发的大型复杂语义数据库管理系统可以为各种异构、半结构化和非结构化海量数据的有效管理、高效检索和及时分析做出重要贡献。它可用于语义搜索引擎、数据爆炸式增长的复杂社交网络服务、大规模数据分析和数据挖掘、知识图谱建立和知识发现等应用。

In the age of big data, how to effectively store, manage, retrieve, and analyze such large-scale complex data in a timely and efficient manner is a major challenge. The primary goal of the proposed research is to design and implement a cost effective and highly scalable database management system for big complex semantic data storage, query, and analytics by combining and extending ideas from parallel database systems, MapReduce computing platform, and our previous work on complex semantic database model Information Networking Model (INM) and INM-DBMS. A parallel database system is a database management system (DBMS) implemented on a multiprocessor system with high-degree connectivity. It features data modeling using well-defined schemas, declarative query languages with high levels of abstraction, sophisticated query optimizers, and a rich runtime environment that supports efficient execution strategies.MapReduce computing paradigm, started by Google and made popular by the open source Hadoop, is a cost-effective distributed data storage and processing systems on large clusters of low-cost commodity machines connected with high-bandwidth network. It has gained a lot of attention in recent years from industry and research.INM-DBMS is a complex semantic database management system that features hierarchical and composite binary and higher degree relationships and their combinations, built-in semantics for consistency and integrity constraints for various relationships, and rich deductive and active rules. It has a concise and compact but expressive language consisting of three parts, information definition language (IDL), information manipulation language (IML) and information query language (IQL). IDL and IML provide powerful constructs to express the rich semantics and integrity constraints associated with various relationships. The declarative query language IQL effectively incorporates many useful features found in database, XML and logic programming languages such as logical variables, implicit existential and non-existential quantification, explicit universal quantification, negation as failure, tree expressions, etc. It can explore the natural networking structure of objects to extract and construct meaningful results in a concise, natural, and compact way.We will first extend the INM data model to adapt to schema-free and semi-structured, and then implement a big semantic database management system based on it. The system will consist of two layers to achieve high concurrency. The manipulation layer is in charge of definitions, manipulations and queries while Data Layer takes care of data storage. In the manipulation layer, objects are evenly allocate to storage nodes by a hash function and query tasks are analysed and interpreted into parallel tasks via an INM MapReduce library interface to achieve partition balancing and partitioned parallelism. In the data layer, the structured semantic data is partitioned and stored in various nodes to ensure dynamic scalability. In the methodology part, four kinds of nodes are differentiated to execute different jobs. The nodes are independent of each other and separated from material machines, which makes the system highly extensible in physical resource allocation and extremely robust at fault-tolerance. Our developed big complex semantic DBMS can significantly contribute to the effective management, efficient retrieval, and timely analytics of various heterogeneous, semi-structured and unstructured massive data. It can be used in many applications such as semantic search engine, complex social network services with exploded increasing data, large-scale data analysis and data mining, knowledge graph establishing and knowledge discovery, etc.