EAGER: An Efficient and Effective Distributed Information System

在数据密集型世界中，信息的巨大增长和新一波大数据浪潮为全球超大规模数据共享创造了广阔的前景，广泛分散的海量数据将在全球范围内汇集和共享。分布式数据密集型信息系统是实现这一未来的关键组成部分。该系统将允许用户高效和有效地搜索相似的数据。然而，前所未有的数据量，以及大规模的环境和参与者的自主性，对该系统的开发提出了高效率和有效性的挑战。这项研究将为南卡罗来纳州的教师、研究生和本科生以及K-12学生提供合作研究机会。对于开发一个高效和有效的信息搜索系统的需求不断增长，这一挑战代表了数据密集型计算面临的一个更强大的障碍。本建议旨在通过开发支持高效和有效的数据搜索的分布式信息系统来解决这一需求。该系统实现了高效率和高效率。效率意味着排序和搜索数据的速度和开销，而有效性意味着能够在系统中以更少的误报和误报找到所有匹配的数据。该系统将数据项转换为id，将数据项映射到分布式系统中的节点，并以分布式的方式进行相似性搜索。首先，以往的数据转换方法依赖于一个多维空间将一个数据项散列到一个索引，虽然效率很高，但由于数据降维中的维度诅咒，效率很低。以前的精确映射方法对数据项的每个关键字进行散列以进行数据搜索，这种方法非常有效，但效率低下。通过消除对多维空间的需求，该系统既高效又有效。其次，与以往一些依赖集中式或分层结构进行数据搜索的系统不同，该系统构建了分布式哈希表（DHT）结构，以分布式方式提供高效的数据搜索。与大多数传统的基于dht的数据共享只提供精确匹配服务不同，该系统提供相似度搜索。

The tremendous growth of information in the data-intensive world and a new wave of big data are creating a promising future for global ultra-large-scale data sharing, where widely-scattered massive data will be pooled and shared globally. A distributed data intensive information system is a critical component for realizing this future. The system will allow users to efficiently and effectively search similar data. However, the unprecedented amount of data, along with the large-scale environment and autonomous nature of participants pose high efficiency and effectiveness challenges to the development of such a system. This research will provide collaborative research opportunities for faculty, graduate and undergraduate students, as well as K-12 students in South Carolina.A growing need persists for developing an efficient and effective information searching system, and this challenge represents one of the more formidable hurdles facing data-intensive computing. This proposal is aimed at addressing this need through the development of a distributed information system supporting efficient and effective data searching. This system achieves both high efficiency and effectiveness. Efficiency means the speed and overhead of sorting and searching date, while effectiveness means the ability to find all matching data in the system with fewer false positives and false negatives. This system translates data items to IDs, maps the data items to nodes in a distributed system and enables the similarity searching in a distributed manner. First, previous data translation methods relying on a multi-dimensional space to hash a data item to one index achieve high efficiency but suffer from low effectiveness due to the curse of dimensionality in data dimension reduction. Previous exact mapping methods that hash each keyword of a data item for data search are highly effective but inefficient. By eliminating the need of a multi-dimensional space, this system is both highly efficient and effective. Second, unlike some previous systems relying on a centralized or hierarchical structure for data searching, this system builds a distributed hash table (DHT) structure, which provides highly efficient data searching in a distributed manner. Unlike most traditional DHT-based data sharing which provides only exact matching services, this system offers similarity searching.