Design and implementation of big complex semantic data management system

在大数据时代，如何及时高效地有效存储、管理、检索和分析如此大规模的复杂数据是一大挑战。本研究的主要目标是结合和扩展并行数据库系统、MapReduce计算平台的思想，结合我们之前在复杂语义数据库模型信息网络模型(INM)和INM-DBMS方面的工作，设计并实现一个高性价比和高度可扩展的大型复杂语义数据存储、查询和分析数据库管理系统。并行数据库系统是在具有高度连接的多处理器系统上实现的数据库管理系统(DBMS)。它的特点是使用定义良好的模式进行数据建模，具有高抽象级别的声明性查询语言，复杂的查询优化器，以及支持高效执行策略的丰富运行时环境。由Google发起并由开源Hadoop流行的MapReduce计算范式是一种高性价比的分布式数据存储和处理系统，位于连接高带宽网络的低成本商用机器的大型集群上。INM-DBMS是一个复杂的语义数据库管理系统，它具有层次化、复合化的二元和高阶关系及其组合，内置了对各种关系的一致性和完整性约束的语义，以及丰富的演绎和主动规则。它由信息定义语言(IDL)、信息操纵语言(IML)和信息查询语言(IQL)三部分组成。IDL和IML提供了强大的构造来表达与各种关系相关联的丰富语义和完整性约束。声明式查询语言IQL有效地结合了数据库、XML和逻辑编程语言中的许多有用特性，如逻辑变量、隐式存在和非存在量化、显式通用量化、失败否定、树表达式等。它可以探索对象的自然网络结构，以简洁、自然和紧凑的方式提取和构造有意义的结果。我们首先扩展INM数据模型以适应模式无关和半结构化，然后在此基础上实现一个大型语义数据库管理系统。该系统将由两层组成，以实现高并发。操作层负责定义、操作和查询，数据层负责数据存储。在操作层，通过哈希函数将对象均匀分配到存储节点，并通过INM MapReduce库接口将查询任务解析为并行任务，实现分区均衡和分区并行。在数据层，将结构化的语义数据划分并存储在各个节点上，以保证动态可伸缩性。在方法部分，区分四种不同的节点来执行不同的任务。节点之间相互独立，与物料机分离，使得系统在物理资源分配上具有高度的可扩展性和极强的容错性。我们开发的大型复杂语义数据库系统可以对各种异构、半结构化和非结构化的海量数据进行有效的管理、高效的检索和及时的分析。它可以应用于语义搜索引擎、数据呈爆炸式增长的复杂社交网络服务、大规模数据分析和数据挖掘、知识图建立和知识发现等领域。

In the age of big data, how to effectively store, manage, retrieve, and analyze such large-scale complex data in a timely and efficient manner is a major challenge. The primary goal of the proposed research is to design and implement a cost effective and highly scalable database management system for big complex semantic data storage, query, and analytics by combining and extending ideas from parallel database systems, MapReduce computing platform, and our previous work on complex semantic database model Information Networking Model (INM) and INM-DBMS. A parallel database system is a database management system (DBMS) implemented on a multiprocessor system with high-degree connectivity. It features data modeling using well-defined schemas, declarative query languages with high levels of abstraction, sophisticated query optimizers, and a rich runtime environment that supports efficient execution strategies.MapReduce computing paradigm, started by Google and made popular by the open source Hadoop, is a cost-effective distributed data storage and processing systems on large clusters of low-cost commodity machines connected with high-bandwidth network. It has gained a lot of attention in recent years from industry and research.INM-DBMS is a complex semantic database management system that features hierarchical and composite binary and higher degree relationships and their combinations, built-in semantics for consistency and integrity constraints for various relationships, and rich deductive and active rules. It has a concise and compact but expressive language consisting of three parts, information definition language (IDL), information manipulation language (IML) and information query language (IQL). IDL and IML provide powerful constructs to express the rich semantics and integrity constraints associated with various relationships. The declarative query language IQL effectively incorporates many useful features found in database, XML and logic programming languages such as logical variables, implicit existential and non-existential quantification, explicit universal quantification, negation as failure, tree expressions, etc. It can explore the natural networking structure of objects to extract and construct meaningful results in a concise, natural, and compact way.We will first extend the INM data model to adapt to schema-free and semi-structured, and then implement a big semantic database management system based on it. The system will consist of two layers to achieve high concurrency. The manipulation layer is in charge of definitions, manipulations and queries while Data Layer takes care of data storage. In the manipulation layer, objects are evenly allocate to storage nodes by a hash function and query tasks are analysed and interpreted into parallel tasks via an INM MapReduce library interface to achieve partition balancing and partitioned parallelism. In the data layer, the structured semantic data is partitioned and stored in various nodes to ensure dynamic scalability. In the methodology part, four kinds of nodes are differentiated to execute different jobs. The nodes are independent of each other and separated from material machines, which makes the system highly extensible in physical resource allocation and extremely robust at fault-tolerance. Our developed big complex semantic DBMS can significantly contribute to the effective management, efficient retrieval, and timely analytics of various heterogeneous, semi-structured and unstructured massive data. It can be used in many applications such as semantic search engine, complex social network services with exploded increasing data, large-scale data analysis and data mining, knowledge graph establishing and knowledge discovery, etc.