NeTS: Small: Incentivizing Internet-Scale Web Mining with Webcoin

四大搜索引擎在互联网的使用上占有独特的地位，因为它们使用专有索引将98%以上的互联网用户引导到他们所寻找的内容。虽然这些公司的贡献是不可否认的，但它们的设计必然受到它们的经济利益的影响，这些利益可能与用户的利益一致，也可能不一致，这引起了人们对它们对全球信息可用性的影响的担忧。阻碍大量搜索引擎扩散的关键障碍之一是令人生畏的，价值数十亿美元的爬行和索引eb级网络空间的任务。因此，这项任务是由少数能够投入大量资源进行24/7网络爬行和索引的玩家执行的。尽管之前曾努力使这一领域民主化，即促进使用分布式、客观和无偏见的索引，但这种分布式系统的使用并没有变得普遍。这主要是由于与此类系统使用的小型Web索引相关的较差结果，这是由于缺乏参与分布式Web爬行的经济激励，或者由于对协作Web索引感兴趣的用户社区的小规模性质造成的。这个项目旨在表明，通过激励机制来创建一个最新的、自然生成的、广泛提供给“大众”的网络索引，是实现搜索引擎空间民主化的途径。为了对用户参与产生可持续的激励，该项目利用了比特币的原理，但迫使矿工抓取和索引网络，而不是像比特币那样进行无目的的计算。该项目以Webcoin为特色，Webcoin是一种新型分布式数字货币，它将网络任务众包，并奖励进行网络爬行、抓取和索引的用户，同时提供与比特币相同的安全保证。Webcoin的新颖性、重点和独特贡献在于，它是第一个激励网络任务（而不是计算任务或个人内容）在几天内生成谷歌规模的网络索引的公司。这个项目的主要目标是设计、实现和评估Webcoin。这个项目有可能产生重大影响，因为它消除了阻碍新型搜索引擎扩散的关键障碍，使人们能够在没有重大初始成本的情况下创建这样的系统。如果成功，该系统将实现无偏见的搜索，从而增强人类自由，促进自由开放的社会。PI计划将Webcoin作为开源软件设计和传播。此外，Webcoin的设计原则可以在网络以外的领域被证明是有用的，在激励用户参与和大规模验证用户贡献方面。教育是该项目不可分割的一部分，项目负责人计划利用现有的机构和特殊项目，从代表性不足的群体中招收学生。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Four major search engines hold a unique position in enabling the use of the Internet, as they alone direct over 98% of Internet users to the content they seek, using proprietary indices. While the contribution of these companies is undeniable, their design is necessarily affected by their economic interests, which may or may not align with those of the users, raising concerns regarding their effect on the availability of information around the globe. One of the key barriers that hinders the proliferation of a larger number of search engines is the daunting, multi-billion-dollar-worth task of crawling and indexing the exabyte-scale Web space. Hence, this task is carried out by a small number of players capable of committing substantial resources to conduct 24/7 Web crawling and indexing. Despite prior efforts to democratize this space, i.e., promote the use of a distributed, objective and unbiased index instead, the use of such distributed systems has not become common. This is mostly due to inferior results associated with small Web indices used by such systems, induced by the lack of economic incentives for participation in distributed Web crawling, or because of the small-scale nature of the communities of users interested in collaborative Web indexing. This project seeks to show that generating incentives to create an up-to-date, organically-generated Web index, widely available to the "masses," is the way to democratize the search engine space. To generate sustainable incentives for user participation, this project utilizes Bitcoin's principles, yet forces miners to crawl and index the Web instead of conducting purposeless computations, as in Bitcoin. This project features Webcoin, a novel distributed digital currency which crowdsources networking tasks and rewards users for conducting Web crawling, scraping, and indexing, while providing the same security guarantees as Bitcoin. Webcoin's novelty, focus, and unique contribution lies in being the first to incentivize networking tasks, rather than computational tasks or personal content, to produce Google-scale Web indices within days. The key goal of this project is to design, implement, and evaluate Webcoin. This project has the potential to make a significant impact by removing the key barrier that hinders the proliferation of novel search engines, empowering the people to create such systems without significant initial costs. If successful, the proposed system will enable bias-free search, thus enhancing human liberties and promoting a free and open society. The PI plans to design and disseminate Webcoin as open-source software. Moreover, Webcoin's design principles could prove useful in domains beyond networking, in incentivizing user participation and in verifying user contributions, at scale. Education is an integral part of the project, and the PI plans to leverage existing institutional and special programs to recruit students from underrepresented groups into the project.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.