Improving Big Data Infrastructure to Effectively Utilize Resources for Data Processing and Services

大数据分析被誉为下一波创新浪潮的新兴驱动力。它已经形成了许多新应用的基础，涉及从商业工作流到工程活动和科学计算的广泛领域。一种典型的应用是面向企业和个人移动或车辆用户的基于云的网络服务，其中通过每日日志文件从固定或移动实体(例如配备了各种内置传感器(例如GPS、加速计和摄像头)的智能手机)收集数据，然后由大数据基础设施进行处理，以便将数据作为服务提供。例如，车辆轨迹可以从视频数据实时生成，以便于为交通管理分析交通流参数。显然，在这类应用中，用于数据处理的底层基础设施的响应时间对服务质量(Qos)有很大影响。*我正在进行的总体研究目标是通过更好地利用大数据基础设施中的资源来提高服务质量。本文的研究目标主要有两个方面：一是利用网络拓扑结构、作业特征和执行过程，更好地均衡网络流量，更高效地为应用共享云资源。这些拟议的改进对于在云中部署大数据基础设施的服务提供商尤为重要，因为调配的资源通常基于现收现付的计费策略，而有效利用资源意味着最大化云服务提供商的货币利润，并最大限度地降低大数据分析用户的成本。其次，我寻求一种有效的方式，将数据作为一种服务向外部世界传播。最终，我希望提高服务质量，同时提高大数据基础设施对各种服务中断故障的容忍度。*根据国际数据公司(International Data Corporation)撰写、领先分析软件开发商SAS赞助的大数据市场白皮书，“加拿大企业在大数据时代面临落后于国际竞争对手的风险”。特别是，这项调查和分析发现，与更愿意投资并在不久的将来制定更明确的大数据采用计划的国际公司相比，加拿大公司在采用能够处理大数据的技术方面较晚和缓慢。*我提议的研究将提供更具成本效益和更可靠的大数据基础设施，以提供更快的大数据处理速度。这将吸引更多的加拿大公司采用这项新技术，以提高创新能力和生产率。我的成果将为加拿大的电信和汽车行业带来显著的经济效益，并显著提升加拿大在这一领域的形象。

Big data analytics is hailed as an emerging driving force of the next innovation wave. It has formed the basis for many novel applications across a wide range of fields from business workflows to engineering activities and scientific computing. One such typical application is cloud-based network services for business and individual mobile or vehicular users, where data is collected through daily log files, from stationary or mobile entities such as smartphones equipped with a variety of built-in sensors (e.g., GPS, accelerometers and cameras) and then processed by a big-data infrastructure in order to provision data as a service. For example, vehicle trajectories can be generated in real time from video data to facilitate the analysis of traffic flow parameters for transportation management. Clearly, in such applications, the response time of the underlying infrastructure for data processing has great impact on Quality of Service (QoS). *******My ongoing overall research goal is the amelioration of QoS through better resource utilization in big data infrastructure. I approach this research goal from two directions: First, I aim at better balancing network traffic and more efficiently sharing cloud resources for applications by exploiting network topology, job characteristics and execution processes. These proposed improvements are particularly important for service providers deploying big data infrastructure in clouds, as the provisioned resources are typically based on pay-as-you-go billing strategies, and effectively utilizing the resources means maximizing the monetary profits for cloud service providers and minimizing cost for users of big data analytics. Second, I search for an effective way to disseminate data to the outside world as a service. Ultimately, I want to increase QoS while also improving the tolerance of big data infrastructure for a variety of service-disrupting faults. ******According to a recent whitepaper on the Big Data market written by International Data Corporation and sponsored by SAS, a leading developer of analytics software, "Canadian businesses are at risk of falling behind international competitors in the era of Big Data". In particular, this survey and analysis found that Canadian companies are late and slow in adoption of technology capable of processing Big Data in comparison to international companies that are more willing to invest in and have more defined plans for adoption of these technologies in the near future.******My proposed research will provide a more cost effective and more reliable Big Data infrastructure that offers faster processing of Big Data. This will attract more Canadian companies to adopt this new technology in order to become more innovative and productive. My results will bring significant economic benefits to Canada's telecommunication and automobile industries, in particular, and significantly enhance Canada's profile in this field.