NeTS: Small: Incentivizing Internet-Scale Web Mining with Webcoin

点击翻译按钮获取中文摘要

Four major search engines hold a unique position in enabling the use of the Internet, as they alone direct over 98% of Internet users to the content they seek, using proprietary indices. While the contribution of these companies is undeniable, their design is necessarily affected by their economic interests, which may or may not align with those of the users, raising concerns regarding their effect on the availability of information around the globe. One of the key barriers that hinders the proliferation of a larger number of search engines is the daunting, multi-billion-dollar-worth task of crawling and indexing the exabyte-scale Web space. Hence, this task is carried out by a small number of players capable of committing substantial resources to conduct 24/7 Web crawling and indexing. Despite prior efforts to democratize this space, i.e., promote the use of a distributed, objective and unbiased index instead, the use of such distributed systems has not become common. This is mostly due to inferior results associated with small Web indices used by such systems, induced by the lack of economic incentives for participation in distributed Web crawling, or because of the small-scale nature of the communities of users interested in collaborative Web indexing. This project seeks to show that generating incentives to create an up-to-date, organically-generated Web index, widely available to the "masses," is the way to democratize the search engine space. To generate sustainable incentives for user participation, this project utilizes Bitcoin's principles, yet forces miners to crawl and index the Web instead of conducting purposeless computations, as in Bitcoin. This project features Webcoin, a novel distributed digital currency which crowdsources networking tasks and rewards users for conducting Web crawling, scraping, and indexing, while providing the same security guarantees as Bitcoin. Webcoin's novelty, focus, and unique contribution lies in being the first to incentivize networking tasks, rather than computational tasks or personal content, to produce Google-scale Web indices within days. The key goal of this project is to design, implement, and evaluate Webcoin. This project has the potential to make a significant impact by removing the key barrier that hinders the proliferation of novel search engines, empowering the people to create such systems without significant initial costs. If successful, the proposed system will enable bias-free search, thus enhancing human liberties and promoting a free and open society. The PI plans to design and disseminate Webcoin as open-source software. Moreover, Webcoin's design principles could prove useful in domains beyond networking, in incentivizing user participation and in verifying user contributions, at scale. Education is an integral part of the project, and the PI plans to leverage existing institutional and special programs to recruit students from underrepresented groups into the project.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.