EAGER: An Efficient and Effective Distributed Information System

数据密集型世界中信息的巨大增长和新一轮大数据浪潮正在为全球超大规模数据共享创造一个充满希望的未来，其中广泛分散的海量数据将在全球范围内汇集和共享。分布式数据密集型信息系统是实现这一未来的关键组件。该系统将使用户能够高效和有效地搜索相似的数据。然而，前所未有的数据量，加上参与者的大规模环境和自治性质，给这样一个系统的发展带来了高效率和有效性的挑战。这项研究将为南卡罗来纳州的教职员工、研究生和本科生以及K-12学生提供合作研究的机会。开发高效和有效的信息搜索系统的需求持续增长，这一挑战是数据密集型计算面临的更大障碍之一。这项提议旨在通过开发一个支持高效率和有效数据搜索的分布式信息系统来满足这一需要。该系统达到了高效、高效的目的。效率意味着排序和搜索数据的速度和开销，而有效性意味着能够以较少的误报和漏报在系统中找到所有匹配的数据。该系统将数据项转换为ID，将数据项映射到分布式系统中的节点，并以分布式方式进行相似度搜索。首先，以前的数据转换方法依赖于多维空间将数据项散列到一个索引，虽然效率很高，但由于数据降维中的维度灾难，效率较低。以前对数据项的每个关键字进行散列以进行数据搜索的精确映射方法是非常有效的，但效率很低。通过消除对多维空间的需要，该系统既高效又有效。其次，不同于以往的一些系统依赖集中式或分层结构进行数据搜索，该系统构建了分布式哈希表(DHT)结构，以分布式的方式提供高效的数据搜索。与大多数传统的基于分布式哈希表的数据共享只提供精确匹配服务不同，该系统提供相似度搜索。

The tremendous growth of information in the data-intensive world and a new wave of big data are creating a promising future for global ultra-large-scale data sharing, where widely-scattered massive data will be pooled and shared globally. A distributed data intensive information system is a critical component for realizing this future. The system will allow users to efficiently and effectively search similar data. However, the unprecedented amount of data, along with the large-scale environment and autonomous nature of participants pose high efficiency and effectiveness challenges to the development of such a system. This research will provide collaborative research opportunities for faculty, graduate and undergraduate students, as well as K-12 students in South Carolina.A growing need persists for developing an efficient and effective information searching system, and this challenge represents one of the more formidable hurdles facing data-intensive computing. This proposal is aimed at addressing this need through the development of a distributed information system supporting efficient and effective data searching. This system achieves both high efficiency and effectiveness. Efficiency means the speed and overhead of sorting and searching date, while effectiveness means the ability to find all matching data in the system with fewer false positives and false negatives. This system translates data items to IDs, maps the data items to nodes in a distributed system and enables the similarity searching in a distributed manner. First, previous data translation methods relying on a multi-dimensional space to hash a data item to one index achieve high efficiency but suffer from low effectiveness due to the curse of dimensionality in data dimension reduction. Previous exact mapping methods that hash each keyword of a data item for data search are highly effective but inefficient. By eliminating the need of a multi-dimensional space, this system is both highly efficient and effective. Second, unlike some previous systems relying on a centralized or hierarchical structure for data searching, this system builds a distributed hash table (DHT) structure, which provides highly efficient data searching in a distributed manner. Unlike most traditional DHT-based data sharing which provides only exact matching services, this system offers similarity searching.