EAGER: An Efficient and Effective Distributed Information System

在数据密集型世界中，信息的巨大增长和新一轮大数据浪潮正在为全球超大规模数据共享创造一个充满希望的未来，广泛分散的海量数据将在全球范围内汇集和共享。分布式数据密集型信息系统是实现这一未来的关键组成部分。该系统将使用户能够高效和有效地搜索类似数据。然而，前所未有的数据量，沿着的大规模环境和参与者的自主性，对这种系统的开发提出了高效率和有效性的挑战。这项研究将提供合作研究的机会，教师，研究生和本科生，以及K-12的学生在南卡罗来纳州。一个不断增长的需要持续发展一个高效和有效的信息搜索系统，这一挑战是数据密集型计算面临的更艰巨的障碍之一。本提案旨在通过开发一个支持高效率和高效益数据搜索的分布式信息系统来满足这一需求。该系统实现了高效率和有效性。效率意味着排序和搜索数据的速度和开销，而有效性意味着在系统中找到所有匹配数据的能力，同时减少误报和漏报。该系统将数据项转换为ID，将数据项映射到分布式系统中的节点，并以分布式方式实现相似性搜索。首先，以前的数据转换方法依赖于一个多维空间来散列一个数据项到一个索引实现高效率，但由于在数据降维灾难的有效性低。以前的精确映射方法，散列数据项的每个关键字的数据搜索是非常有效的，但效率低下。通过消除多维空间的需要，该系统既高效又有效。第二，与一些先前的系统依赖于集中式或分层结构进行数据搜索不同，该系统构建了分布式哈希表（DHT）结构，其以分布式方式提供高效的数据搜索。与大多数传统的基于DHT的数据共享仅提供精确匹配服务不同，该系统提供相似性搜索。

The tremendous growth of information in the data-intensive world and a new wave of big data are creating a promising future for global ultra-large-scale data sharing, where widely-scattered massive data will be pooled and shared globally. A distributed data intensive information system is a critical component for realizing this future. The system will allow users to efficiently and effectively search similar data. However, the unprecedented amount of data, along with the large-scale environment and autonomous nature of participants pose high efficiency and effectiveness challenges to the development of such a system. This research will provide collaborative research opportunities for faculty, graduate and undergraduate students, as well as K-12 students in South Carolina.A growing need persists for developing an efficient and effective information searching system, and this challenge represents one of the more formidable hurdles facing data-intensive computing. This proposal is aimed at addressing this need through the development of a distributed information system supporting efficient and effective data searching. This system achieves both high efficiency and effectiveness. Efficiency means the speed and overhead of sorting and searching date, while effectiveness means the ability to find all matching data in the system with fewer false positives and false negatives. This system translates data items to IDs, maps the data items to nodes in a distributed system and enables the similarity searching in a distributed manner. First, previous data translation methods relying on a multi-dimensional space to hash a data item to one index achieve high efficiency but suffer from low effectiveness due to the curse of dimensionality in data dimension reduction. Previous exact mapping methods that hash each keyword of a data item for data search are highly effective but inefficient. By eliminating the need of a multi-dimensional space, this system is both highly efficient and effective. Second, unlike some previous systems relying on a centralized or hierarchical structure for data searching, this system builds a distributed hash table (DHT) structure, which provides highly efficient data searching in a distributed manner. Unlike most traditional DHT-based data sharing which provides only exact matching services, this system offers similarity searching.