CRII: CSR: Towards Understanding and Mitigating the Impact of Web Robot Traffic on Web Systems

这个CSR-CRII项目对全球Web系统上网络机器人（又名网络爬虫）流量的突然增长做出了回应——从十年前的大约20%到今天的60%以上。由于当前Web系统的优化假设所服务的流量呈现类似人类的模式，而机器人却没有，因此在Web上呈现机器人活动可能会无声地降低Web系统的性能、能源效率和可伸缩性。随着网络不断发展成为一个社交平台，在这个平台上，个人可以上传临时的想法和观察结果，而这些想法和观察结果只会对组织产生即时的价值，物联网概念预计将引入数百万台从网络收集数据并自动向在线服务提交请求的设备，机器人流量只会在数量和强度上迅速增加。由于这个原因，我们必须了解Web机器人流量对现代Web系统的影响，并设计出能够减轻它们对系统性能、能源效率和可伸缩性的影响的技术。这项工作将把我们目前对机器人流量的理解与机器学习工具、统计分析和数据科学方法结合起来，这些方法以前没有在网络流量分析和用户行为建模的背景下考虑过。它将通过以下方式提高我们理解机器人流量对Web系统影响的能力：(i)设计自动方法，根据机器人的功能和它们所施加的需求对机器人进行分类；（ii）开发新型机器人交通发生器，针对机器人类型的特定概况进行定制，可以测试系统如何对不同强度和功能类型混合的机器人交通做出反应。该项目还将探索一个原型机器人弹性缓存系统，该系统可以为现有的Web系统带来即时的性能回报。该项目将产生初步的分析模型、实证结果，以及导致长期研究努力的原型分析软件。来自跨许多Web域提供服务的Web系统的最新数据可立即用于项目。该项目的结果可能会改变Web系统从单个服务器到大型云的设计和优化方式，从而降低性能、能源效率和服务机器人的财务成本。我们将策略性地招募参与本项目的学生，以扩大参与范围。教育活动将为学生提供有用但不常教授的流量分析和网络系统安全，促进知识工程和网络安全学生与研究社区之间更牢固的联系。

This CSR-CRII project responds to the sudden rise of Web robot (a.k.a. Web crawler) traffic on Web systems around the world - from approximately 20% of all requests a decade ago to over 60% today. Because present Web systems' optimizations assume that the traffic serviced exhibit human-like patterns that robots do not, present robot activity on the Web may silently degrade performance, energy efficiency, and scalability of Web systems. As the Web continues to evolve towards a social platform where individuals upload extemporaneous thoughts and observations that only carry instantaneous value to organizations, and where the Internet of Things concept is expected to introduce millions of devices that collect data from the Web and submit requests to online services automatically, robot traffic will only rapidly increase in volume and intensity. For this reason, it is essential that we understand the impact of Web robot traffic on modern Web systems and devise technologies capable of mitigating their impact on system performance, energy efficiency, and scalability.This effort will synthesize our present understanding of robot traffic with machine learning tools, statistical analysis, and data science methods not previously considered in the context of Web traffic analysis and user behavioral modeling. It will improve our ability to understand the impact of robot traffic on Web systems by: (i) devising automatic methods to classify robots by their functionality and by the demands they impose; and (ii) develop novel robot traffic generators, tailored to a specific profile of robot types that can test how a system reacts to robot traffic of varying intensity and functional type mixtures. The project will also explore a prototype robot-resilient caching system that could lead to immediate performance payoffs for existing Web systems. The project will result in preliminary analytical models, empirical results, and prototype analysis software leading to longer-term research endeavors. Recent data from Web systems that provide services across many Web domains are immediately available for the project.The results of the project potentially may transform the way Web systems from single servers to large clouds are designed and optimized mitigating performance, energy efficiency, and the financial cost of servicing robots. Students to work on this project will be strategically recruited to broaden participation. Educational activities will provide students useful yet infrequently taught traffic analysis and Web systems security fostering stronger ties between knowledge engineering and cybersecurity student and research communities.