CRII: CSR: Towards Understanding and Mitigating the Impact of Web Robot Traffic on Web Systems

这个CSR-CRII项目是对网络机器人(又名：网络爬虫)在世界各地的Web系统上的流量-从十年前所有请求的大约20%上升到今天的60%以上。由于目前Web系统的优化假设服务的交通呈现出机器人没有的类似人类的模式，因此在Web上呈现机器人活动可能会悄悄地降低Web系统的性能、能源效率和可扩展性。随着网络继续向社交平台发展，在这个平台上，个人上传的即席想法和观察只能给组织带来即时价值，物联网概念预计将引入数百万台设备，从网络收集数据并自动向在线服务提交请求，机器人流量只会在数量和强度上迅速增加。因此，我们必须了解Web机器人流量对现代Web系统的影响，并设计出能够减轻其对系统性能、能源效率和可扩展性的影响的技术。这一努力将综合我们目前对机器人流量的理解，使用机器学习工具、统计分析和数据科学方法，这些方法以前没有在Web流量分析和用户行为建模的背景下考虑。它将通过以下方式提高我们理解机器人流量对Web系统影响的能力：(I)设计自动方法，根据机器人的功能和它们施加的需求对机器人进行分类；以及(Ii)开发新的机器人流量生成器，该生成器针对特定的机器人类型而量身定做，可以测试系统对不同强度和功能类型混合的机器人流量的反应。该项目还将探索一种具有机器人弹性的原型缓存系统，该系统可能会立即为现有的Web系统带来性能回报。该项目将产生初步的分析模型、经验结果和原型分析软件，从而导致更长期的研究努力。来自Web系统的最新数据可以立即用于该项目。该项目的结果可能会改变Web系统从单一服务器到大型云的设计和优化方式，从而降低性能、能源效率和服务机器人的财务成本。将有策略地招募参与该项目的学生，以扩大参与范围。教育活动将为学生提供有用但很少被教授的流量分析和网络系统安全，促进知识工程和网络安全之间更紧密的联系，学生和研究社区。

This CSR-CRII project responds to the sudden rise of Web robot (a.k.a. Web crawler) traffic on Web systems around the world - from approximately 20% of all requests a decade ago to over 60% today. Because present Web systems' optimizations assume that the traffic serviced exhibit human-like patterns that robots do not, present robot activity on the Web may silently degrade performance, energy efficiency, and scalability of Web systems. As the Web continues to evolve towards a social platform where individuals upload extemporaneous thoughts and observations that only carry instantaneous value to organizations, and where the Internet of Things concept is expected to introduce millions of devices that collect data from the Web and submit requests to online services automatically, robot traffic will only rapidly increase in volume and intensity. For this reason, it is essential that we understand the impact of Web robot traffic on modern Web systems and devise technologies capable of mitigating their impact on system performance, energy efficiency, and scalability.This effort will synthesize our present understanding of robot traffic with machine learning tools, statistical analysis, and data science methods not previously considered in the context of Web traffic analysis and user behavioral modeling. It will improve our ability to understand the impact of robot traffic on Web systems by: (i) devising automatic methods to classify robots by their functionality and by the demands they impose; and (ii) develop novel robot traffic generators, tailored to a specific profile of robot types that can test how a system reacts to robot traffic of varying intensity and functional type mixtures. The project will also explore a prototype robot-resilient caching system that could lead to immediate performance payoffs for existing Web systems. The project will result in preliminary analytical models, empirical results, and prototype analysis software leading to longer-term research endeavors. Recent data from Web systems that provide services across many Web domains are immediately available for the project.The results of the project potentially may transform the way Web systems from single servers to large clouds are designed and optimized mitigating performance, energy efficiency, and the financial cost of servicing robots. Students to work on this project will be strategically recruited to broaden participation. Educational activities will provide students useful yet infrequently taught traffic analysis and Web systems security fostering stronger ties between knowledge engineering and cybersecurity student and research communities.