NeTS: Small: Incentivizing Internet-Scale Web Mining with Webcoin

四大搜索引擎在互联网的使用方面占据着独特的地位，因为仅它们就使用专有索引将98%以上的互联网用户引导到他们所寻求的内容。虽然这些公司的贡献是不可否认的，但它们的设计必然受到其经济利益的影响，这些利益可能与用户的利益一致，也可能不一致，从而引起人们对它们对地球仪的信息提供的影响的关切。阻碍大量搜索引擎扩散的关键障碍之一是令人生畏的、价值数十亿美元的抓取和索引EB级Web空间的任务。因此，这项任务是由少数能够投入大量资源进行24/7网络爬行和索引的玩家执行的。尽管先前努力使这一空间民主化，即，尽管我们没有推广使用分布式、客观和无偏见的指数，但这种分布式系统的使用并不普遍。这主要是由于劣质的结果与小的Web索引使用这样的系统，引起的参与分布式Web爬行的经济激励措施的缺乏，或因为小规模的性质，有兴趣在协作Web索引的用户社区。这个项目旨在表明，产生激励，以创建一个最新的，有机生成的Web索引，广泛提供给“群众”，是民主化的搜索引擎空间的方式。为了产生可持续的用户参与激励，该项目利用了比特币的原则，但迫使矿工抓取和索引Web，而不是像比特币那样进行无代价的计算。这个项目的特点是Webcoin，一种新型的分布式数字货币，它众包网络任务，并奖励用户进行网络抓取，抓取和索引，同时提供与比特币相同的安全保证。Webcoin的新奇，重点和独特的贡献在于它是第一个激励网络任务，而不是计算任务或个人内容，在几天内产生Google规模的Web索引。该项目的主要目标是设计、实现和评估Webcoin。这个项目有可能通过消除阻碍新型搜索引擎扩散的关键障碍，使人们能够在没有重大初始成本的情况下创建此类系统，从而产生重大影响。如果成功，拟议的系统将实现无偏见的搜索，从而加强人类自由，促进自由和开放的社会。PI计划将Webcoin作为开源软件设计和传播。此外，Webcoin的设计原则可以在网络之外的领域证明是有用的，可以激励用户参与并验证用户的贡献。教育是该项目不可或缺的一部分，PI计划利用现有的机构和特殊计划，从代表性不足的群体中招募学生加入该项目。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

Four major search engines hold a unique position in enabling the use of the Internet, as they alone direct over 98% of Internet users to the content they seek, using proprietary indices. While the contribution of these companies is undeniable, their design is necessarily affected by their economic interests, which may or may not align with those of the users, raising concerns regarding their effect on the availability of information around the globe. One of the key barriers that hinders the proliferation of a larger number of search engines is the daunting, multi-billion-dollar-worth task of crawling and indexing the exabyte-scale Web space. Hence, this task is carried out by a small number of players capable of committing substantial resources to conduct 24/7 Web crawling and indexing. Despite prior efforts to democratize this space, i.e., promote the use of a distributed, objective and unbiased index instead, the use of such distributed systems has not become common. This is mostly due to inferior results associated with small Web indices used by such systems, induced by the lack of economic incentives for participation in distributed Web crawling, or because of the small-scale nature of the communities of users interested in collaborative Web indexing. This project seeks to show that generating incentives to create an up-to-date, organically-generated Web index, widely available to the "masses," is the way to democratize the search engine space. To generate sustainable incentives for user participation, this project utilizes Bitcoin's principles, yet forces miners to crawl and index the Web instead of conducting purposeless computations, as in Bitcoin. This project features Webcoin, a novel distributed digital currency which crowdsources networking tasks and rewards users for conducting Web crawling, scraping, and indexing, while providing the same security guarantees as Bitcoin. Webcoin's novelty, focus, and unique contribution lies in being the first to incentivize networking tasks, rather than computational tasks or personal content, to produce Google-scale Web indices within days. The key goal of this project is to design, implement, and evaluate Webcoin. This project has the potential to make a significant impact by removing the key barrier that hinders the proliferation of novel search engines, empowering the people to create such systems without significant initial costs. If successful, the proposed system will enable bias-free search, thus enhancing human liberties and promoting a free and open society. The PI plans to design and disseminate Webcoin as open-source software. Moreover, Webcoin's design principles could prove useful in domains beyond networking, in incentivizing user participation and in verifying user contributions, at scale. Education is an integral part of the project, and the PI plans to leverage existing institutional and special programs to recruit students from underrepresented groups into the project.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.