CRII: CSR: Towards Understanding and Mitigating the Impact of Web Robot Traffic on Web Systems

这个CSR-CRII项目是为了应对网络机器人（又称网络机器人）的突然兴起。网络爬虫）流量在世界各地的Web系统-从大约20%的所有请求十年前到今天的60%以上。由于目前的Web系统的优化假设的交通服务表现出类似人类的模式，机器人不，目前的机器人在Web上的活动可能会默默地降低性能，能源效率和Web系统的可扩展性。随着网络继续向社交平台发展，个人上传即兴的想法和观察，这些想法和观察只会对组织产生即时价值，物联网概念预计将引入数百万从网络收集数据并自动向在线服务提交请求的设备，机器人流量只会在数量和强度上迅速增加。因此，我们必须了解Web机器人流量对现代Web系统的影响，并设计能够减轻其对系统性能，能源效率和可扩展性的影响的技术。这项工作将综合我们目前对机器人流量的理解与机器学习工具，统计分析，以及以前在Web流量分析和用户行为建模中未考虑的数据科学方法。它将提高我们的能力，了解机器人流量对Web系统的影响：（一）设计自动方法，分类机器人的功能和他们施加的要求;和（二）开发新的机器人流量发生器，量身定制的机器人类型的特定配置文件，可以测试系统如何反应不同强度和功能类型的混合物的机器人流量。该项目还将探索一个原型机器人弹性缓存系统，可以为现有的Web系统带来即时的性能回报。该项目将产生初步的分析模型，实证结果和原型分析软件，从而导致长期的研究工作。该项目可以立即获得跨多个Web域提供服务的Web系统的最新数据。该项目的结果可能会改变Web系统的设计和优化方式，从单服务器到大型云，从而降低性能、能源效率和服务机器人的财务成本。将战略性地招募参与该项目的学生，以扩大参与。教育活动将为学生提供有用但不常教授的流量分析和网络系统安全，促进知识工程和网络安全学生和研究社区之间更紧密的联系。

This CSR-CRII project responds to the sudden rise of Web robot (a.k.a. Web crawler) traffic on Web systems around the world - from approximately 20% of all requests a decade ago to over 60% today. Because present Web systems' optimizations assume that the traffic serviced exhibit human-like patterns that robots do not, present robot activity on the Web may silently degrade performance, energy efficiency, and scalability of Web systems. As the Web continues to evolve towards a social platform where individuals upload extemporaneous thoughts and observations that only carry instantaneous value to organizations, and where the Internet of Things concept is expected to introduce millions of devices that collect data from the Web and submit requests to online services automatically, robot traffic will only rapidly increase in volume and intensity. For this reason, it is essential that we understand the impact of Web robot traffic on modern Web systems and devise technologies capable of mitigating their impact on system performance, energy efficiency, and scalability.This effort will synthesize our present understanding of robot traffic with machine learning tools, statistical analysis, and data science methods not previously considered in the context of Web traffic analysis and user behavioral modeling. It will improve our ability to understand the impact of robot traffic on Web systems by: (i) devising automatic methods to classify robots by their functionality and by the demands they impose; and (ii) develop novel robot traffic generators, tailored to a specific profile of robot types that can test how a system reacts to robot traffic of varying intensity and functional type mixtures. The project will also explore a prototype robot-resilient caching system that could lead to immediate performance payoffs for existing Web systems. The project will result in preliminary analytical models, empirical results, and prototype analysis software leading to longer-term research endeavors. Recent data from Web systems that provide services across many Web domains are immediately available for the project.The results of the project potentially may transform the way Web systems from single servers to large clouds are designed and optimized mitigating performance, energy efficiency, and the financial cost of servicing robots. Students to work on this project will be strategically recruited to broaden participation. Educational activities will provide students useful yet infrequently taught traffic analysis and Web systems security fostering stronger ties between knowledge engineering and cybersecurity student and research communities.